Solana stakeholders raced to fix the network Saturday night, after an “insane amount” of data flooded the proof of-stake chain, killing validators and grinding down production.
Bots had attacked the Candy Machine, a popular non-fungible token (NFT), minting tool. They created an unprecedented tsunami of traffic with four million transactions requests every second and 100 gigabits per second. This was a new record for the network according to one Solana Foundation representative.
This swarm drove validators from consensus for reasons that are not yet known. At 4:32 p.m., block production was impossible and the network went dark. ET. At 11:59 p.m. ET: The cluster was restarted at slot 131973970 by validators (coordinating through Solana’s Discord channels, and a Google Doc created in part by one of the validators).
Anatoly Yakovenko was the co-founder of validator. He claimed he was travelling during the fracas and credited them with leading the mainnet recovery. On Saturday, he was criticized on Twitter for being “MIA” in a network crisis.
Saturday’s hard reboot did not result in new-and-improved codes populating across the validators, unlike last September’s 17 hour outage. It simply picked up the network’s seven-hour-old failure.
Validators debated whether to temporarily block Candy Machine transactions while they prepared for the restart. Discord members debated whether such a move was censorship. It would not be effective if only two-thirds validators agreed to it. On Saturday night, few were able to opt in.
Others quickly moved to strengthen their defenses. At 11:36 p.m. ET: Metaplex, the key steward for Solana NFT infrastructure, and one closely connected with Candy Machine, tweeted that it would soon implement a 0.01 SOL “botting penal” to help NFT projects stop excessive traffic.
Solana ecosystem services such as Phantom wallet and Mango Markets, a decentralized exchange, struggled to hold up after the RPC node providers returned online.
SOL markets suffered a brief but severe drawdown due to the outage. CoinGecko reports that Solana’s native currency fell to $83.13 at the end of the outage. It then recovered to $89 around three hours later.
CoinDesk was told by a member that Solana core developers are still trying to figure out what happened Saturday and how the apparent botting attack overcame existing safeguards.
S