Behind the Collapse of Xi'an Covid-19 App

Sun_Huixia

Sun_Huix...

· 1月11日

After an in-depth investigation by TMTPost, suppliers to the app were surfaced. And it looks that the architecture, not the firewall, is the culprit for the breakdowns.

January 4 was the deadline for the municipal government in Xi’an, the ancient capital of China with a population of 12 million, to reach its goal of zero new daily Covid-19 case. On January 1, there were 122 new infections; On January 2, the daily count fell to 90; On January 3, the figure was 95.

The new cases in 24 hours were not many compared with thousands in many cosmopolitan regions in North America during the ongoing fifth wave of the coronavirus pandemic since early 2000. But with the zero-case policy in place across China, the doulbe-digit new coronavirus infections have made the outbreak in Xi’an the second catastrophic heathcare event in two years, only after that in Wuhan, the once epicenter of the pandemic.

While many residents in the city rushed to the Covid-19 testing centers to find out if they were infected, their app that stores personal travel history and Covid-19 test results stopped working.

In China, everyone must show their travel records in the past 14 days in order to get on a bus, shop in a store or depart a city. If they visit an area with a single Covid-19 infection, the status of their app is showed in the color of yellow or red, which means they should quarantine for two weeks even if they have no symptoms at all.

Many residents affected by the app crash complained online that they could not even leave home to get tested. Let alone the government’s zero new Covid-19 case goal by January 4.

Every resident in high-risk neighbourhoods of Xi'an is required to get tested for Covid-19 every two days thanks to China’s enormous manufacturing capacity and the centralized administrative system. Many doctors and nurses from other cities were dispatched to the city to help.

Suppliers to App Project

Amid the fury of the public, it took about two hours for engineers to frantically solve the problem. Actually it was not the first app crash. On December 20, a similar breakdown occurred during the current round of outbreaks in Xi’an that started on December 9, when the first local case, as opposed to infections of overseas travellers to China, was reported. Last time, it took about 24 hours for the app to resume working following hectic repairs.

Rumors quickly circulated online that the municipal government spent only 270,000 yuan (about US$38,600) to build such a vital administrative system. A snapshot of a document showing the government’s invitation for offers from suppliers surfaced.

After a probe in the snapshot by TMTPost, the snapshot turned out to be the document for a different government procurement project.

The app system was developed in three days, according to a news story published in May 2020. However, amid the outcry over the crash, Xi'an Telecom said three days were just the time for developing a prototype, not the final version of the product. 

Based on the information accessed by TMTPost, the procurement project of the app named “One Barcode to Anywhere”, which was designed to curb the spread of the coronavius, was won by China Telecom’s subsidiary Xi’an Telecom. On February 18, 2020, about two months after the outbreak in Wuhan, the app was launched for its tral operations.

Xi’an Telecom divided the project into various parts, which were subcontracted by nearly 10 information technology companies. The project consists of development, operation and maintenance, Internet security-related products and services, server products, text message services and big data among many others.          

The app is built on another project named “Xi’an Digital Administrative Platform”, known as the “Administrative Cloud”. According to the information available to TMTPost,the storage of data, internet and computation services used by the app are supplied by the platform. The services bought by Xi’an Telecom are generally called “application services.”  

According to public information released by government agencies, Alibaba Cloud is the supplier of Administrative Cloud and text message services; Meilin is responsible for back-office algorithm; Xi’an Dongruan System Integration Co. Ltd. supplies software and maintenance services.

Meilin Data was blamed by some internet users for the crash on December 20. The company responded by saying that it provided algorithm, not operational services.

Dongruan were involved in platform software and customized development services. The platform was originally not designed for displaying the results of mass Covid-19 antigen tests on the app. After the first breakdown, the capacity was expanded to 40,000 QPS (Queries Per Second). However, it was still not enough when all 12 million residents were tested within two days.    

Official Suspended from Office 

Liu Jun, the head of Xi’an Big Data Resources Administration, apologized for the December 20 breakdown and cautioned residents not to use the app unnecessarily.

His advisory provoked a wave of online mocks. After all, who would use the app when not needed? One netizen said sarcastically: “Don’t show your barcode when you don’t have to; But when you have to, the barcode is dead.”

Liu also gave the reason for the systematic glitch. Too many users were using it at the same time, he said.

As to the cause of the January 4 breakdown, the authorities attributed to “a sudden surge in the number of users”, which was ten times as high as that in the pre-December 9 period. The unusually high volume of visitors led to “online overcrowding,” which triggered the firewall to block information packages. The core issue is that the architecture is not suited for a large number of simultaneous visits.

On January 5, Liu was publicly reproached by Xi’an Municipal Committee of the Communist Party of China in the wake of two breakdowns in two weeks. He was suspended from his position and ordered to hand in a self-criticism report.

According to public information, the “One Barcode to Anywhere” app amassed downloads of over 30 million by over 200,000 stores and companies as well as 50,000 operators and users of the subway system, buses and taxies. The app has been used for a total of 3 billion times in 43,000 neighborhoods. The app, along with big data, has been applied to conduct contact tracing of nearly 400,000 persons and for over 400,000 Covid-19 tests or vaccinations per day.  

Alibaba Not to Blame?

Some netizens pointed fingers at Alibaba Cloud after the two crashes. Alibaba Cloud appeared not only on the list of suppliers for the “One Barcode to Everywhere” app, but also on the list of suppliers for “Administrative Cloud.”  

A source close to the procurement project told TMTPost that Alibaba Cloud made a mistake in the analysis of DNS, or Domain Name Service. There are two VIP routes reserved for the app but one error occurred on one of the two routes. RDS, or Relational Database Service, contains many slow SQLs, which caused traffic congestion. The two problems were quickly solved in the wake of the breakdown.  

However, a spokesman of Alibaba Cloud denied the existence of two issues. “Alibaba Cloud provided underlying structure to the “One Barcode to Everywhere” app, but was not involved in the creation of architecture. During the Covid-19 outbreak in Xi’an, Alibaba Cloud platform has been running smoothly. There were no glitches in either DNS analysis or RDS. Our technical team has been on the spot. Alibaba Cloud is willing to make more contributions to the fight against the novel coronavirus,” he said.

An expert in cybersecurity and information technology told TMTPost on the condition of anonymity that traffic congestion triggered a blockage on the firewall and consequently failures of requests for the data. Therefore the personal information of users could be matched with the information on test results or travel history in the cloud, resulting in the failure of information display on a user’s app. Ultimately it was a problem of archteicture, not the firewall, he added.

On January 4, the city government failed to reach its goal. As of the midnight of January 10, Xi’an reported 15 new daily infections. On December 23, 2021, the ancient capital of China was locked down following an outbreak of the Delta variant-related infections in early December.

本文系作者Sun_Huixia授权钛媒体发表,并经钛媒体编辑,转载请注明出处、作者和本文链接
本文观点仅代表作者本人,钛媒体平台仅对用户提供信息及决策参考,本文不构成投资建议。
想和千万钛媒体用户分享你的新奇观点和发现,点击这里投稿 。创业或融资寻求报道,点击这里

敬原创,有钛度,得赞赏

”支持原创,赞赏一下“
钛粉30326 钛ae5u70 满级钛宝16422180875 马洋928 钛小超16395556011 钛粉25713
514人已赞赏 >
514换成打赏总人数514人赞赏钛媒体文章
关闭弹窗

挺钛度,加点码!

  • ¥ 5
  • ¥ 10
  • ¥ 20
  • ¥ 50
  • ¥ 100

支付方式

确认支付
关闭弹窗

支付

支付金额:¥6

关闭弹窗
sussess

赞赏金额:¥ 6

赞赏时间:2020.02.11 17:32

关闭弹窗 关闭弹窗

注册邮箱未验证

我们已向下方邮箱发送了验证邮件,请查收并按提示验证您的邮箱。

如果您没有收到邮件,请留意垃圾邮件箱。

更换邮箱

您当前使用的邮箱可能无法接收验证邮件,建议您更换邮箱

账号合并

经检测,你是“钛媒体”和“商业价值”的注册用户。现在,我们对两个产品因进行整合,需要您选择一个账号用来登录。无论您选择哪个账号,两个账号的原有信息都会合并在一起。对于给您造成的不便,我们深感歉意。