Why did OKEx avoid the network outages on the 19th and 24th?

Wu said author | Colin Wu

Editor of this issue | Colin Wu

As we all know, on the 19th, due to the failure of the Tokyo AWS node, almost all Chinese exchanges experienced failures.

It is rumored that Huobi "sacrificed high availability in pursuit of speed and forced all services to be migrated to a single availability zone, namely Zone C where the failure occurred. At the same time, there was no quick and effective emergency response plan. Once a single-zone failure occurred, all services needed to be migrated out and data needed to be rebuilt, which was very time-consuming." This led to a very serious and long-lasting network outage.

Embarrassingly, due to the drastic market fluctuations on the 22nd, Huobi was the first to experience a full network failure, followed by Binance. In the past period of time, whenever there were drastic market fluctuations, Binance was often the first to have problems. Therefore, He Yi also expressed in the community that he was looking for technical personnel who participated in King of Glory and Double Eleven to solve the high concurrency problem.

Although cryptocurrency exchanges do not have as many users as traditional financial software, they also have many unique features, such as 24/7 non-stop trading, users from all over the world, and frequent sudden and drastic market fluctuations. Especially global exchanges like Binance are more prone to failures.

Many people are discussing that among the three major exchanges, OKEx, which has been relatively low-key recently, has indeed escaped unscathed from these two almost network-wide failures, which is somewhat different. Some users also ridiculed that OKEx's traffic has decreased. This seems illogical because second-tier exchanges such as Matcha, KuCoin, and Gate have also experienced failures many times.

It is rumored that OKEx attaches more importance to technology. In any case, what is the reason why OKEx was relatively unscathed during these two network outages? Let's hear what OKEx thinks. Wu Blockchain interviewed the technical director of OKEx. The content of the interview is relatively plain. The following interview content represents the views of OKEx.

1. AWS node failed on the 19th, why did OKEx not encounter any problems?

Reply: OKEx actually has some businesses running on AWS, but more services are on Alibaba Cloud servers in Hong Kong, so there was no failure this time. We chose Alibaba Cloud for historical reasons, because it is more familiar and has better services. But this is not necessarily an advantage, because Alibaba Cloud has also had similar (failure) problems in the past. In any case, if all services are really migrated to a single availability zone, this is definitely not a rational behavior.

2. Due to the surge in traffic on the 22nd, there were no problems with OKEx. What was the reason?

Reply: We rarely have downtime due to sudden traffic, and our team still believes that OKEx's system stability will be better. For the system to achieve a high level of service quality, it is a process that requires top-level design and systematic execution. Generally speaking, system service stability is usually measured by SLA indicators. For example, when we say that the system stability reaches 4 9s, that is, 99.99%, it means that the system service interruption time is within 50 minutes each year. We can be relatively close to this number.

3. There is also a view that the reason there was no downtime is because OKEx traffic is relatively small?

Reply: From the data, OKEx's spot trading is slightly lower, but its contract derivatives are still acceptable compared to its competitors, and it has some advantages. We believe that the reason for no downtime is that our own technical capabilities and trading system performance are better, and we can bear a larger concurrency.

4. OKEx actually had some problems in 2017-2018, but it has been relatively good in the past two years. What is the reason?

Reply: We rarely experience downtime due to sudden traffic, but there were more outages in the last month of last year, mainly because we were preparing to use a unified account and upgrades and maintenance were more frequent.

Technological development and upgrading is a process. It is not convenient to talk about the specific progress in detail, but there have indeed been many adjustments in technology and risk control.

For example, the mark price mechanism is adopted. The moving average mechanism is used in the calculation, which can smooth out the contract price fluctuations in a short period of time and effectively reduce unnecessary forced liquidation caused by abnormal fluctuations in the market. The latest version 3.0 of the underlying trading system has a lower latency than the previous version, and the instantaneous transaction peak exceeds 1 million times/minute. Compared with the old system, the stability and concurrency capabilities have been greatly improved.

5. The promotion of unified accounts has received a lot of discussion, but some people believe that there are difficulties in technology and risk control. What do you think?

Reply: The unified account does make the business more complicated, and the risk control logic is also more complicated. The unified account also has different modes. If the user chooses a more aggressive account mode, it may cause greater losses, but there are also relatively independent modes. At present, the unified account has been launched, and old users will gradually switch to it. We believe that the technical difficulties have been solved.