AWS Tokyo outage caused the downtime of Chinese exchanges. Why was Huobi so seriously affected? Why did Binance crash when there was a lot of traffic?

The old-fashioned "butterfly effect" story appeared in the Chinese crypto industry on the evening of the 19th. Due to the power outage of the cooling system of the AWS Tokyo node, Huobi first experienced a serious network outage, followed by Binance, Matcha, Coinex, KuCoin, etc., which also experienced small-scale failures. It can be said that almost all Chinese exchanges were affected.

At 10:00 pm Beijing time on the 19th, Huobi suddenly experienced a network failure, and users were kicked out of the login state, indicating that the network was not good. Huobi said that the login was affected due to a large-scale network failure in Amazon Cloud Service Area C. The other party has been contacted and is being repaired. The safety of customer assets is not affected.

At 11:00, Binance, Kucoin, Matcha APP, Coinex, etc. also experienced lag, but the impact was relatively small. The figure shows the fault diagram and cause of the AWS node, which is currently being repaired. AWS discovered the problem at 11:00 pm Japan time on the 19th. The core reason was that the cooling system was powered off, causing the temperature to rise. Most of it had been restored by 3:30 am Japan time on the 20th, and it was basically fully restored by 5:30 am.

Binance announced that due to the AWS outage affecting our platform as well as many other exchanges, Binance Network temporarily suspended deposits, withdrawals and fund transfers from 1:13 AM to 4:52 AM Hong Kong time on the 29th.

Although almost all Chinese exchanges have experienced outages, the reason why Huobi’s outage was so serious was explained by a Weibo user as follows, and this statement was also recognized by relevant people.

"The new CTO sacrificed high availability in pursuit of speed and forced all services to be migrated to a single availability zone, which is Zone C where the failure occurred. At the same time, there was no quick and effective emergency response plan. Once a single zone failure occurred, all services needed to be migrated out and data needed to be rebuilt, which was very time-consuming." (Reference Exclusive: Former Alibaba executive Zhu Ye becomes Huobi's new CTO, all Huobi executives exposed for the first time)

Industry insiders pointed out that AWS Japan is a relatively expensive region in the Asia-Pacific region and is also an earthquake zone. It is not a suitable choice to choose AWS Japan as the only available region. Objectively speaking, system availability, extreme transaction performance, and cost investment are an "impossible triangle". Pursuing speed at the expense of availability may lead to such a result.

As Bitcoin and some mainstream currencies continued to soar on the evening of the 19th, investors who shorted contracts may suffer losses because they cannot withdraw orders. At present, except for Huobi, the scope of failures in other exchanges is relatively small, so there is no problem of compensation. Huobi said that it will definitely introduce a corresponding compensation plan. What is more special is that OKEx may have basically no failures in this incident because it mainly uses Alibaba Cloud servers in Hong Kong.

Before the failure, Binance suddenly announced that it had suspended the withdrawal of Ethereum and related tokens, which triggered a lot of speculation and conspiracy theories, but it quickly resumed withdrawals. The reason is currently unknown. However, Binance's relatively high frequency of downtime and failures has also attracted attention. What is the reason? Binance itself attributed it to excessive traffic, and CZ said that the number of new users per day reached 30,000. However, some people questioned that Binance's profit last quarter was close to 1 billion US dollars, and it should not be reluctant to invest in technology.

Professionals pointed out that the reasons may be as follows:

First, the investment is still insufficient. Although Binance has a lot of profits, the system, software and hardware require a huge investment cost. The annual maintenance cost of the leading companies may reach hundreds of millions of dollars. Second, the cryptocurrency circle has a strong special feature, 7x24 hours trading, global users, various unpredictable emergencies and hacker attacks, which makes maintenance very difficult. Third, the internal emergency response capacity is insufficient and there is no perfect testing team. Traditional Internet companies will have a lot of pre-drills on failures and perfect mechanisms and systems. Because Binance is developing too fast, many related systems and mechanisms have not kept up.