What Caused The Solana Network To Halt In June 2022?

What Caused Solana To Go Down?

On June 1, 2022 at approximately 4:30 PM UTC, the Solana network stopped functioning. Solana validators ceased block production as a result of stalled consensus, which led the network to shut down. Solana is programmed to shutdown when validators are unable to reach consensus of >66%.

Solana users were unable to send and receive crypto transactions for approximately four and a half hours while the team at Solana Labs worked closely with validators to identify and resolve the issue. Validators found the issue quickly by comparing the final block and looking for the difference that led to a split in consensus - they identified a single durable nonce transaction that was processed differently. A single misconfigured Solana transaction shutdown the whole network.

Block production resumed at around 9:00 PM UTC after the team fixed the issue that caused the hiccup. The team temporarily disabled durable nonce transactions so the entire network could restart.

tldr on the crash
1. nonce account had an old blockhash
2. user incorrectly signed a nonce tx using recent blockhash
3. status cache checked that RBH != old blockhash and assumed TX was a normal tx
4. AdvanceNonce moved the old blockhash to RBH

- @aeyakovenko, CEO of Solana Labs. June 2, 2022

The transaction was erroneously signed using a recent blockhash, which it is not meant to do. The core Solana software running the network did not handle the error correctly, and the validators were not able to reach consensus as a result. High level, validators split consensus where >33% of the validators processed the transaction one way, and a separate group representing >33% of validators processed the transaction another way.

Solana blockchain is meant for a super majority of validators (>66%) to agree on a transaction. The split made it impossible for >66% of the validators to agree. The Solana network is programmed to shutdown when the validators cannot reach finalized consensus, so it proceeded to do so.

Recent Shutdown Is Not Related To Network Congestion

Most observers immediately assumed that the Solana network shutdown was related to congestion, which previously caused deteriorated performance at various points in 2021 and 2022. The team at Solana Labs stated specifically that the downtime experienced on June 1 was not due to network congestion.

Over 99.99% of Solana network transactions do not use the durable nonce format. As more institutions invest in Solana for the long-term, many choose to use cold wallet storage which can involve durable nonce accounts. The recent increase in durable nonce usage can explain why this bug remained hidden for so long.

Once the team identified the issue, they updated the way durable nonce transactions reference blockhashes so this issue would not recur. The issue was a strange hidden bug that has been fixed. The Solana Labs team did a great job addressing the issue, pushing an update, and publicly disclosing the details regarding the issue.

During an open community call hosted on Twitter Spaces, Anatoly Yakovenko, the Co-Founder of Solana and current CEO of Solana Labs, stated that the team is still actively engaged in updating the core software to address network congestion issues in coming weeks.

Resources

Solana Press Release
What is Solana?