The long road to SegWit: How Bitcoin’s biggest protocol upgrade became a reality

Segregated Witness (SegWit) has been activated on Bitcoin. As of today, all SegWit-ready nodes on the Bitcoin network are already enforcing the new rules, marking the biggest protocol upgrade to date for Bitcoin.

However, activation is not easy to come by and cannot be achieved overnight.

This post is a review of the long road to SegWit.

question

A Bitcoin transaction consists of two main parts. One part is the "base transaction data," which includes what bitcoins are being moved and where they are being moved to (the address), as well as some other data. The second part is called the "witness," which contains some code that cryptographically signs the data to prove that the owner of the bitcoin actually wants to use it.

It is this signature data that introduces some complexity to Bitcoin transactions. In the so-called "malleability vulnerability," anyone can slightly change Bitcoin signatures after they are created without invalidating the signature. This means that the entire transaction, more specifically the transaction ID (txid), can be changed by the relayer or miner of the transaction.

Data from malleability attacks on the Bitcoin network in 2015. The red line roughly represents the number of transactions on the network that were malleable.

This in itself isn’t necessarily a big problem. Transactions still work, bitcoins still get transferred from one place to another, and all under exactly the same conditions. However, it does make it more cumbersome to create new transactions because of the presence of unconfirmed transactions: new transactions need to know the transaction IDs they depend on. This in turn makes it more difficult to build certain second-layer protocols on top of Bitcoin, such as bidirectional payment channels.

idea

The idea of ​​addressing the malleability vulnerability by “separating” signature data from other transaction data dates back several years.

Back in 2012, Bitcoin Core contributors Russell O’Connor, Matt Corallo, Luke Dashjr, and Gregory Maxwell, along with Bitcointalk moderator “Theymos,” discussed the issue on the IRC Bitcoin development channel, but at the time they did not find a successful way to separate signatures from the Bitcoin network.

Russell O'Connor, Gregory Maxwell, Luke Dashjr, and Theymos discussing the malleability vulnerability on IRC in 2012

A year later, in August 2013, the issue resurfaced as Bitcoin Core contributors Peter Todd and Gregory Maxwell had a similar discussion on IRC. But by then, both had made progress in combating malleability with their own ideas. “I was saying to [split the entire scriptsig],” Maxwell wrote. “I even suggested using transactions without scriptsigs as [transaction IDs].”

A month later, Maxwell discussed the malleability issue again on IRC with renowned cryptographer Dr. Adam Back. This time, Back suggested omitting signatures when calculating transaction IDs. Although, Maxwell commented: "Separating signatures from txids might help, but it's a deep hard fork change... and it's hard to actually guarantee security."

Sidechain

Blockstream originally proposed a sidechain expansion solution for the Bitcoin blockchain

In August 2014, Adam Back, Gregory Maxwell, and entrepreneur and investor Austin Hill teamed up with several Bitcoin Core developers (including Dr. Pieter Wuille) to found Bitcoin technology company Blockstream. The company focuses on sidechain technology, alternative blockchains that can effectively be pegged to Bitcoin.

By early 2015, Blockstream engineers decided to add a new feature to the company’s sidechain prototype, Elements, and announced it in June of the same year. This feature would ultimately solve the scalability problem on the sidechain by separating the underlying transaction data from the witness data into different data structures.

The name of this new feature is, of course, Segregated Witness.

Block size controversy

The debate over the size of the blockchain has been going on for some time: technically since October 2010, more specifically since October 2013, and finally came to light in the spring of 2015:

Former Bitcoin Core lead developer Gavin Andresen and Bitcoinj lead developer Mike Hearn believe that Bitcoin’s 1 megabyte size limit should be increased via a hard fork. A hard fork is a protocol change that is incompatible with the original Bitcoin system and requires almost the entire Bitcoin ecosystem to upgrade. This is not an easy task – because there is no consensus in the Bitcoin community for this change.

Regardless, in the summer of 2015, Andresen and Hearn announced that they would use the Bitcoin XT software client to advance their plans, a controversial move that put the Bitcoin development community and industry in a state of emergency.

To resolve this disagreement and help find a possible solution to the block size dispute, the Bitcoin community quickly organized two conferences (or workshops) in the second half of 2015: the Montreal Scaling Bitcoin Conference and the Hong Kong Scaling Bitcoin Conference.

One of the most promising scaling proposals to come out of the Montreal conference was the Lightning Network, a complex second-layer scaling solution that was detailed only a few months ago in a white paper by Joseph Poon and Thaddeus Dryja. The only problem: This solution requires a scalability fix.

Soft Fork

Lombrozo (CodeShark), Wladimir van der Laan (wumpus), Luke Dashjr (luke-jr), and Dr. Pieter Wuille (sipa) discuss SegWit as a possible soft fork on IRC

At that time, most people still believed that Segregated Witness could not be implemented on the Bitcoin main chain without a hard fork.

With the exception of Bitcoin Core contributor (and Bitcoin Konts maintainer) Luke Dashjr.

In October 2015, between two Bitcoin Scaling conferences, Bitcoin Core contributors Eric Lombrozo, Pieter Wuille, Wladimir van der Laan, and Luke Dashjr discussed a possible new model for soft forks on IRC. During this discussion, Dashjr pointed out that the mechanism proposed at the conference would not work for all potential soft forks, such as the SegWit soft fork.

Interestingly, the thing Dashjr thought was obvious — namely, deploying SegWit as a soft fork — had never occurred to anyone else. Even Dashjr himself didn’t seem to realize the implications of this possibility at first.

In order to deploy SegWit as a soft fork, the witness data had to be placed in a new, separate structure in the Bitcoin block. And the “anchor” for all of this witness data (the “Merkle tree root”) had to be moved to a somewhat unconventional part of the Bitcoin block: the coinbase transaction that rewards miners with new coins.

Although this approach was somewhat unusual, in the days and weeks that followed, Bitcoin Core contributors realized that this approach brought an additional benefit. By placing the witness data in a separate structure on the Bitcoin block, the size of the Bitcoin block would increase, but these changes would not be noticed by non-upgraded nodes. This could actually increase the size of Bitcoin blocks without increasing the existing block size of Bitcoin.

In the weeks leading up to the second Bitcoin Scaling Workshop, some Bitcoin Core contributors believe they may have finally found a solution to the block size limit dispute that could at least temporarily end. Segregated Witness would effectively raise the block limit in a backwards-compatible way while fixing a long-standing malleability vulnerability, enabling more advanced scaling solutions like the Lightning Network.

A win-win-win solution – or so they thought.

exhibit

Segregated Witness – as a soft fork – was first proposed by Pieter Wuille at the Scaling Bitcoin Workshop in Hong Kong in December 2015. This was the first time many people had heard of the proposal, and initially the proposal seemed to be received with enthusiasm.

Shortly after the second "Bitcoin Scaling" meeting, Gregory Maxwell proposed a plan called the "scaling roadmap" with SegWit at its core. This roadmap was quickly supported by the Bitcoin Core development team and other developers and users in the Bitcoin ecosystem.

criticism

Despite initial excitement about the proposal, SegWit has also received a lot of criticism.

Concerns about the proposed protocol upgrade vary. Former Bitcoin Core contributor Jeff Garzik — who founded his own development company, Bloq — doesn’t think SegWit is a sufficient short-term scaling solution. Meanwhile, Bitcoin XT lead developer Mike Hearn wasn’t entirely convinced by the proposal: He viewed the solution as an “accounting trick” and quit Bitcoin development altogether shortly after.

Jonathan Toomim, a developer of alternative software client Bitcoin Classic, called the proposal “ugly and clumsy” and suggested it would be better to try it as a hard fork. Even Bitcoin Core contributor Peter Todd has his own concerns, particularly related to mining.

However, most of these issues were deemed solvable, implausible, or worth the trade-offs by the Bitcoin Core development team. Soft fork upgrades began to develop.

develop

Although a version of Segregated Witness has been implemented on Elements, much of the code for the main version of Bitcoin has yet to be written, not just because it would need to be implemented as a soft fork, but also because SegWit on Bitcoin would include a host of new features not present in Elements: for example, the “witness discount” required to increase block size, new compatibility for peer-to-peer networks, etc.

The specific Bitcoin Improvement Plan BIP141 was authored by Pieter Wuille, Ciphrex CEO Eric Lombrozo, and independent Bitcoin Core contributor Johnson Lau. By early January 2016, as the debate over scaling continued to heat up, these people and other Bitcoin Core contributors launched an initial dedicated test network for the protocol upgrade, called SegNet. Two weeks later, this test network was made public for testing by the wider Bitcoin development community. By March, SegNet had been upgraded to a test version that could support the Lightning Network.

Development has been ongoing over the coming months, incorporating feedback from the Bitcoin development community, fixing bugs, improving the codebase, and launching new versions of SegNet.

The Segregated Witness GitHub page, where development and other issues are publicly visible and anyone can follow and contribute

At the same time, Bitcoin contributors have also communicated with more Bitcoin industry practitioners, and the number of companies and projects pledging to support Segregated Witness has been growing over time.

As of June of the same year, Segregated Witness added 4,743 new lines of code (including test code) and proposed to delete or modify 554 existing Bitcoin Core lines of code. After several contributors conducted a deeper review of the code, Bitcoin Core's main maintainer Wladimir van der Laan merged the code into Core's "main branch" by the end of June.

Meeting

While SegWit was being developed, the debate over Bitcoin’s block size heated up again in the Bitcoin community. This time, it was led by Bitcoin Classic, and some Bitcoin companies and miners seemed determined to use a hard fork to increase the block size limit to 2 megabytes.

In Hong Kong, several Bitcoin Core contributors, mining pool operators, and other members of the bitcoin industry held an emergency meeting to discuss the issue of scaling.

The meeting resulted in an agreement known as the Bitcoin Roundtable Consensus (or the Hong Kong Consensus). The Bitcoin Core contributors in attendance and the entire Bitcoin development community agreed to develop a safe hard fork based on the SegWit improvements. In turn, the miners agreed to run SegWit in production after Bitcoin Core released a version containing the hard fork code. The crisis seemed to have been averted - although it would soon become clear that not everyone was happy with the outcome of the agreement.

A few months later, more Bitcoin Core contributors and mining pool operators met in California. After the meeting, the Core contributors in attendance were convinced that Segregated Witness would be activated by miners.

release

About six months after the initial schedule — originally slated for release in April — Segregated Witness was officially rolled out in Bitcoin Core version 0.13.1 in October 2016. The protocol upgrade was also implemented in several other Bitcoin clients, such as Bitcoin Knots and Bcoin.

Using an activation method called "VersionBits" (BIP9) (designed to minimize network disruption), 95% of miners (by hash rate) must express support for SegWit to be activated on the Bitcoin network. Starting November 15, Bitcoin miners can express support for this protocol upgrade. At the same time, users are also encouraged to upgrade their clients. At present, it seems that many users have indeed upgraded.

As of August 2017, the majority of the Bitcoin network consists of SegWit-ready nodes (Source: luke.dashjr.org)

Based on meetings with mining pool operators, and the general belief that SegWit will be a boon for Bitcoin, many expected the soft fork to be activated soon.

politics

But that was not the case. It turns out that several participants in the Hong Kong roundtable disagreed on what they actually signed.

Bitmain’s co-CEO Jihan Wu specifically said he would only be willing to activate SegWit if the Bitcoin Core development team also implemented a hard fork to increase the block size limit. Other mining pools, including F2Pool, HaoBTC, and bitcoin.com, have also not signaled support for a soft fork.

Bitmain (and subsidiary AntPool) requested a hard fork to increase the block size limit, in return they will support SegWit activation

Additionally, a new mining pool has emerged in China: ViaBTC. Due to its close relationship with Bitmain, ViaBTC has enough hashrate to block SegWit activation single-handedly. Its operator, Yang Haibo, has positioned himself as a staunch critic of the proposed protocol upgrade.

The activation of SegWit seems far away.

UASF

Avatar of anonymous Bitcoin and Litecoin developer Shaolinfry

In February 2017, three months after SegWit was officially released, a new opportunity emerged.

An anonymous developer who used to contribute to Litecoin, going by the handle “Shaolinfry,” has proposed a new proposal on the Bitcoin development mailing list and the popular bitcointalk.org forum: a “user activated soft fork,” or “UASF.”

Shaolinfry said in his email that the hashrate activation mechanism that has become the standard for soft forks was never a “vote.” He wrote, “The signaling method is widely misunderstood to mean that the hashrate is voting on a proposal, and it is difficult to correct this misunderstanding in the larger community.

Shaolinfry proposed an alternative: a user-activated soft fork (UASF). Unlike hashrate activation, a user-activated soft fork would use a "designated day activation," where nodes would begin enforcing the new rules at a predetermined time in the future. "As long as the UASF is executed by a majority of economic participants, this should force a majority of miners to follow (or activate) the soft fork."

The idea immediately gained steam on Bitcoin forums and social media, and when former BTCC COO and outspoken SegWit advocate Michael Miu established a bounty fund for the development of UASF software, it seemed like the proposal might become a reality.

Patented Technology

In the first week of April 2017, Gregory Maxwell made a bombshell announcement via the Bitcoin mailing list.

Maxwell claims to have reverse engineered a manufacturer’s ASIC mining chips and found that they include the patented AsicBoost technology. More importantly, Maxwell also revealed that the secret use of this technology is incompatible with the deployment of SegWit via a soft fork. He said: “This incompatibility largely explains some of the puzzling behavior of some people in the mining ecosystem.”

While Maxwell’s email did not name a specific ASIC manufacturer, Bitmain acknowledged that it added patented technologies to its mining chips — though it denied using them on Bitcoin’s mainnet.

Regardless, for some users, the discovery reinforces the desire of some to enable a SegWit soft fork on the Bitcoin network. And, as hashrate activation becomes increasingly unlikely, a user-activated soft fork looks more and more like a possible solution to the problem.

BIP148 Proposal

After coming up with the general idea of ​​UASF, Shaolinfry and a dozen other members of the Bitcoin community opened a UASF channel on the Bitcoin Core community’s Slack.

The channel became a hub for discussion and organization about the initiative. The activation date was initially chosen as October 1, then moved to August 1 to account for a possible low hash rate. Shaolinfry wrote the specific Bitcoin Improvement Proposal: BIP148. Open Dim founder Rodolfo Novak also set up an information website to promote the idea.

The initial plan is to get exchanges and other businesses to support the UASF. If these companies support the proposal and enforce the soft fork, then the desired economic majority will be achieved in no time.

However, UASF has not gained the support that its backers had hoped. Although many companies and some developers appear to support BIP148, no major exchange or other enterprise has announced support, and some companies have even publicly opposed the initiative.

By mid-April, Gregory Maxwell stated on the Bitcoin development mailing list that he thought BIP148 was not a good idea either. As one of the most respected and influential Bitcoin Core contributors, Maxwell's rejection of the initiative had a significant impact: this version of UASF seemed to have lost all momentum.

Instead, some have begun working on an alternative to UASF: BIP149.

Altcoins

Many altcoins are based on Bitcoin’s codebase. This means that the SegWit code developed for Bitcoin is largely compatible with these alternative cryptocurrencies. Unsurprisingly, several altcoins decided to implement SegWit. Back in January 2017, Groestlcoin was the first to activate SegWit.

But other currencies are also struggling. Litecoin, Vertcoin and Viacoin all seem to be drawn into Bitcoin’s political games. These currencies rely heavily on the same miners as Bitcoin, and most of them have not signaled support for the upgrade.

Allegedly, this was due to technical issues or other reasons, but, as Viacoin lead developer Romano put it: “It seems more likely that the reason is that they are reluctant to activate SegWit on altcoins, as this would give them less reason to delay SegWit activation on Bitcoin.”

By April 2017, this attitude led Litecoin founder Charlie Lee to advocate for a user-activated soft fork on Litecoin. His initiative received an enthusiastic response from Litecoin users; soon, Litecoin miners, Lee and other members of the Litecoin ecosystem arranged an online meeting and reached the Litecoin Global Roundtable Resolution. In exchange for some commitments from Lee, miners agreed to activate SegWit. ShaolinFry and others considered this UASF effort a success.

A week after Segregated Witness was activated on Litecoin, an unknown person made a bold move. He (or she) sent $1 million worth of Litecoin to an address protected by Segregated Witness, challenging anyone to steal the funds. To date, the bounty remains unchanged, further boosting confidence in the technology.

New York Agreement

Meanwhile, the block size debate is still raging. As another client software that has increased the Bitcoin block size through a hard fork, Bitcoin Unlimited is very popular in the Bitcoin mining community. Especially with the support of Bitmain's Mr. Wu, the project seems to be moving towards a potential (contentious) hard fork.

This potential threat, and the possibility of a “split” in the Bitcoin blockchain, was the reason DCG founder and CEO Barry Silbert called a meeting ahead of the New York Consensus in 2017. The meeting was initially announced on a private email list for Bitcoin entrepreneurs and other prominent industry members, and brought together a large portion of the Bitcoin industry, including miners — though notably not Bitcoin Core contributors.

The outcome of that meeting is often referred to as the “New York Consensus.” Participants agreed on what they believed to be a compromise between those who wanted a hard fork and those who preferred to increase bitcoin’s block size with SegWit. According to the idea originally proposed by RSK founder Sergio Demian Lerner, SegWit would be activated under certain conditions, along with a hard fork that would double bitcoin’s “base block size limit.”

The “New York Consensus” and its two specific action points

But not everyone in the Bitcoin ecosystem supports the protocol, and one specific problem stands out. The conditions for SegWit activation under the New York Consensus are fundamentally incompatible with those proposed by the Bitcoin Core development team, whose code has been widely adopted by Bitcoin users.

The Intolerable Minority

Image of BIP148 UASF support released by Yongquan Miao

While many have switched from supporting the BIP148 UASF to BIP149, not everyone has completely abandoned the first UASF proposal.

When Shaolinfry proposed the concept, his premise was that it would be supported by the economic majority, otherwise, it should be terminated before the designated date. However, some users on the UASF Slack channel have different ideas. Some of them - including Bitcoin Core and Bitcoin Knots developer Luke Dashjr - are considering activating the soft fork regardless of how the rest of the Bitcoin ecosystem reacts. Even if they are in the minority, even if they will actually become an altcoin, they will go ahead with the UASF upgrade.

Around mid-May, Alphonse Pace linked this decision to a game theory concept described by statistician and author Nassim Nicholas Taleb: the "intolerable minority". In short, the idea is that even an economic minority should be able to force miners to activate the SeparateWit soft fork. Otherwise, they will lose their "customer base" (Bitcoin users).

Support for BIP148 is growing on social media and message boards again, this time with the backing of game theory, thanks to rumors about AsicBoost, the activation of SegWit on Litecoin, and dissatisfaction with the New York Agreement.

In addition, several articles discussed the potential of UASF development, and a lot of debate on social media, YouTube channels, and other discussion platforms followed. At the same time, Eric Lombrozo also fully supported the proposal, and the UASF hats distributed by Miao Yongquan became popular. Inspired by the code name of the upcoming Electrum wallet, August 1st was called "Bitcoin Independence Day."

The only problem is that the activation method of BIP148 and the New York Consensus is incompatible with the activation method proposed by the New York Consensus and the Bitcoin Core team.

Kludge

James Hillard, an engineer at Bitmain Warranty, came to the rescue. Hilliard proposed a slightly complicated but ingenious solution to make everything compatible with each other: the activation of Segregated Witness proposed by the Bitcoin Core development team, the BIP148 UASF, and the New York Consensus activation mechanism. His BIP91 could keep Bitcoin intact - at least when Segregated Witness is activated.

As long as a majority of miners activate BIP91 by August 1, all Bitcoin nodes should still be part of the same network. This is a relatively small window of time, as the solution was only proposed in late May, but Jeff Garzik, the lead developer of the New York Protocol, accepted the proposal and plans to release the resulting software client within a few weeks before August 1. It is feasible.

activation

Information website XBT.eu status at the time of BIP91 lock

By mid-July, Bitcoin miners had missed the opportunity to activate Segregated Witness in time using the method proposed by the Bitcoin Core development team to be compatible with BIP148. As a result, the market seemed nervous about the potential "split" between the BIP148 chain and the non-BIP148 chain. In a week, the exchange rate of Bitcoin fell from around $2,500 to $1,900: the lowest level in more than a month.

Perhaps spooked by these market moves, the Bitcoin mining community quickly began signaling support for BIP91, even earlier than the timeline set by the New York Agreement. On July 20, 10 days before the designated activation date of BIP148 (originally scheduled for August 1), BIP91 was locked in. Two days later, it was activated.

With BIP91 locked in, it was only a matter of time before Segregated Witness was officially locked in. Segregated Witness was finally locked in on August 9th — and as early as August 8th, things had reached the point of no return.

Bitcoin will “officially” begin implementing Segregated Witness after a two-week grace period.

use

SegWit logo designed by Albert Dros

Of course, the final step for SegWit is actual user adoption. Since SegWit has only just launched at the time of this article’s publication, it’s impossible to know how many users will actually use the upgrade, and at what speed. Some critics (perhaps the most notable of these is Garzik) predict that widespread adoption could take a year or even longer. Others, including some wallet and library developers, think they could have the feature in a few weeks, or that they’re ready. And other technologies that rely on the upgrade, like the Lightning Network, but also Merkelized Abstract Syntax Trees (MAST), atomic swaps, faster transaction signing for hardware wallets, and TumbleBit in payment processor mode, are all in various stages of development.

It’s been a long road, but starting today, anyone who wants to use SegWit should be able to do so.