Ethereum's Final Challenge: Is the Beacon Chain Still Alive?

Original author: Yicheng

Compilation of the original text: Deep Tide TechFlow

introduce

"The Beacon Chain has life." On May 11 and 12, 2023, Ethereum faced two temporary and final loss events, testing its resilience. Despite these challenges, the network remained alive and recovered from both events autonomously. We're about to dig into these noteworthy incidents, reviewing their impact and subsequent enhancements implemented to prevent similar incidents from happening in the future.

Event overview

May 11th and 12th, 2023 will be significant dates in the history of Ethereum because on these two days, Ethereum's resilience was severely tested. On May 11, at approximately 20:;19 UTC, the Ethereum mainnet network experienced a significant slowdown in the rate at which blocks were produced, causing finalization to be delayed by four epochs — a first for Ethereum. The next day, a similar event occurred, this time extending the delay to nine epochs and resulting in an inactivity penalty.

During these events, a significant dip in network engagement was observed. The first slip occurred at epoch 200,;551, causing finalization to temporarily stall until epoch 200,;555. The second drop in participation occurred at epoch 200,;750, causing finalization to be suspended again until epoch 200,;759.

Despite initial concerns, the ethereum network has demonstrated its inherent resilience by recovering on its own. These events not only confirmed the resilience of the Ethereum Beacon Chain, but also highlighted potential areas for improvement.

Inactivity Leak

During the non-final state, the Ethereum network deploys a key mechanism called "inactivity leak". This feature is rooted in Ethereum 2.0's PoS protocol, and is designed to maintain network functionality during major disruptions, such as events such as World War III or large-scale natural disasters, which can cause a large number of validators to go offline, thereby preventing block finalization.

Inactivity leak mode is triggered if the network cannot finalize a block for four consecutive epochs (approximately 16 minutes). In this mode, validators who do not attest to blocks will begin to lose some of their staked Ether (ETH). This penalty grows quadratically over time until the block is finalized and restored.

This model has a double deterrent effect. First, it removes the rewards for validator proofs. Second, it imposes incremental penalties on non-participating validators proportional to their inactivity time. This mechanism incentivizes validators to maintain active participation and accelerates network recovery. This is a cornerstone feature for maintaining network integrity during major disturbances.

Influence

For network participants (validators):

According to an estimate provided by Ben Edgington, assuming 65%; of validators were offline during the 8 epoch leak, the inactivity leak resulted in approximately 28 ETH being destroyed. This equates to a loss of ~0.0006 ETH per offline validator.

Additionally, during the outage, proof rewards were reduced to zero, resulting in an additional loss of ~50 ETH that could have been issued through other means. In total, the estimated total loss to validators, including inactivity penalties and lost proof rewards, is approximately 78 ETH.

For users:

In contrast, end users were minimally affected. Although the reduction in available block space has led to a reduction in transaction processing capacity, Gas prices have not seen a sharp increase and are still below their intraday peaks. What's more, the network remains active throughout these events.

This means that Ethereum continues to process transactions without any major disruption, demonstrating its resilience. As a result, users can maintain operations on the Ethereum network largely undisturbed, even in the face of challenges, underscoring the system's robust resilience.

reason

At the heart of Prysm's problem is the lack of a caching mechanism for block replay. This absence exacerbates system load, spawns too many go routines, and increases CPU pressure. In some cases, a new replay started before the previous one had finished, further stressing the system.

Another factor that exacerbated the problem was Prysm's mishandling of proofs from previous epochs - data that should have been ignored was not. This inefficiency, combined with suboptimal use of head state, puts pressure on the system, especially as deposits surge and validator registrations grow.

These events also revealed key differences between the strategies employed by different Ethereum clients. When faced with the problem of executing clients, Lighthouse chooses to discard proofs to keep the network alive, while Prysm and Teku etc. default to using old proofs to generate blocks.

Despite the challenges, these events are critical in providing insight into software inefficiencies, design choices, and network conditions, making the Ethereum network stronger. This sequence of events did not result in any permanent damage, but instead enhanced the resilience and diversity of the Ethereum network design.

recovery

During these events, the resilience of the Ethereum Beacon Chain was truly tested, and it performed extremely well. The Ethereum Beacon Chain appears to be alive and repairing itself.

A key factor for successful recovery is the diversity of clients on the Ethereum network. The presence of multiple clients, each with a unique way of handling the network, proved to be a boon. For example, while Prysm and Teku clients struggled under the load of old proofs, Lighthouse's policy of discarding proofs ensured that part of the network remained active and up and running.

Essentially, Ethereum’s resilience comes from the diversity of its clients, a factor that plays a key role in helping the network heal itself, eliminating any need for human intervention.

Lessons Learned

• Testnet vs Mainnet: These events highlight the differences between the testnet environment and mainnet. With over 600,000 validators on mainnet and a large number of withdrawal operations, it is clear that the complexity and unpredictability of live networks often exceeds that of test environments. This points to the need for more rigorous stress testing to better deal with real-world network conditions.
• Inactivity leak penalty: The effectiveness of the mainnet inactivity leak penalty has been strengthened during these events. These penalties play a vital role in promoting active validator participation, maintaining network liveness, and enabling network recovery.
• The importance of liveness: These events underscore the important role of liveness in blockchain networks. Under the design of the LMD Ghost protocol, Ethereum remained active throughout the process, ensuring that users were minimally affected. Unlike some blockchains that can face downtime during network issues, Ethereum prioritizes liveness over throughput. This approach protects users and the normal operation of the network, emphasizing that without liveness, regardless of throughput, network functionality and user security are compromised.
• Importance of client diversity: The recovery process emphasizes the value of having diverse clients. Different Ethereum clients have unique responses to network events, contributing to the overall resilience and robustness of the network.
• Network Resilience: These events are a strong testament to the resilience of the Ethereum network. Despite significant challenges, the network self-heals and becomes stronger, embodying the concept of anti-fragility in complex systems. This resilience sets a strong precedent for the broader crypto ecosystem and demonstrates the robustness of Ethereum's underlying architecture and design principles.

The events of May 11th and 12th, 2023 are pivotal moments in the evolution of Ethereum. They provide tangible evidence of the viability of the Beacon Chain, even in challenging circumstances. As Ethereum continues to evolve, it builds on these experiences to become not only more robust, but also more brittle-resistant — ready to continue its journey toward decentralization and beyond.