The guide to reliability in distributed systems

What is reliability?

Reliability is the ability for a system to remain available over a period of time. Reliable systems are those that can continuously perform their core functions without service disruptions, errors, or significant reductions in performance. However, there are many different ways a system can fail, especially as a system becomes larger, more dynamic, and more complex. Our systems—and the people operating those systems—must be able to recover from these failures. This recoverability is called resilience. In order to maximize availability, systems must be both reliable and resilient.

Why is reliability challenging?

Reliability isn’t just a matter of enabling replication on a database or having redundant systems running in a separate region. Reliability is an alignment of technology, engineering practices, and organization culture. Engineering teams must not only understand the importance of improving reliability, but have the knowledge, tools, practices, and organizational support to make reliability a part of their everyday workflow. For site reliability engineering (SRE) teams, this means prioritizing redundancy, fault-tolerance, automated healing, and similar mechanisms. And when failures do occur, incident management teams must have the knowledge and practice to respond to and fix them quickly.

Additionally, reliability isn’t a short-term, one-and-done project. It requires us to build a reliability practice in our organization, train our teams to develop and test for reliability, and address failure modes in our systems. Our systems also change over time as new employees get onboarded, new code gets deployed, and unanticipated behaviors emerge. Just as QA testing is an ongoing process that reveals defects and regressions in code, reliability testing is an ongoing process that reveals defects in systems.

{{block-content}}

Throughout this process, teams must also navigate obstacles and risks to their reliability efforts, including:

• Continuing to build and maintain applications.
• Responding to incidents and outages.
• Completing major technical initiatives, such as migrating from on-premises data centers to the cloud (16% of organizations cite performance and reliability as a top challenge to overall cloud adoption).
• Adopting new technologies, such as containers and Kubernetes.

For many organizations, these obstacles can become deterrents to reliability. Nonetheless, a reliability initiative has significant benefits that offset these challenges.

Why should you make reliability a priority?

The most obvious risk of not prioritizing reliability is an increase in outages, which are interruptions in a system’s normal operations. Outages have significant real-world consequences. According to Gartner, companies lose an average of $336,000 per hour of downtime, with top ecommerce sites risking up to $13 million in lost sales per hour of downtime. This doesn’t include other harmful long-term effects, such as losses in customer trust, decreased company valuation, and the indirect costs of fixing outages (engineer’s salaries, restoring from backups, spinning up additional infrastructure, etc). For companies that provide critical services, outages can have a direct impact on customer lives.

There are other benefits too. From an operational perspective, improving reliability lets teams spend less time mitigating the damage caused by outages, and more time focusing on their core competencies. According to the 2021 State of Chaos Engineering report, organizations that prioritize reliability have:

• Fewer high-severity incidents per month.
• Lower incident resolution times (measured by the mean time to resolution, or MTTR).
• Fewer bugs shipping into production.

Improving reliability also improves the customer experience, builds trust in your brand, and gives you a competitive advantage. Ultimately, reliability isn’t just a technology investment, but an investment in the growth and success of the organization.

For every dollar spent in failure, learn a dollar’s worth of lessons

{{name-component}}

How do you start improving reliability?

As mentioned earlier, reliability is a journey that involves multiple facets of the business. We can distill this journey into three main elements: culture, observability, and technology. To see how each element contributes to reliability, click on the links below.