A HOW-TO GUIDE FOR SREs

Reducing MTTD For High-Severity Incidents

Download Your Copy

Thanks for requesting Reducing MTTD for High-Severity Incidents: A How-To Guide for SREs! View the guide here. (A copy has also been sent to your email.)


About the Authors

Jordan Pritchard

Director of Infrastructure & Site Reliability Engineering

Michael Kehoe

Architect of reliable, scalable infrastructure

Rodney Lester

Technical Lead, Reliability Pillar of Well Architected Program

Tammy Butow

Principal SRE

Jay Holler

Manager, Site Reliability Engineering

Ramin Keene

Founder

Get an introduction to the fundamentals of incident management classification and explore methods for reducing the time it takes SRE teams to discover their downstream effects.

This ebook dives into:

  • Incident classification: SEV descriptions and levels, and SEV and time-to-detection (TTD) timelines
  • Organization-wide critical service monitoring, including key dashboards and KPI metrics emails
  • Service ownership and metrics for organizations maintaining a microservices architecture
  • Effective on-call principles for site reliability engineers, including rotation structure, alert threshold maintenance, and escalation practices
  • Chaos Engineering practices to identify random and unpredictable behavior in your system
  • Monitoring and metrics to detect incidents caused by self-healing systems
  • Creating a high-reliability culture by listening to people in your organization

  • Incident classification: SEV descriptions and levels, and SEV and time-to-detection (TTD) timelines

  • Organization-wide critical service monitoring, including key dashboards and KPI metrics emails

  • Service ownership and metrics for organizations maintaining a microservices architecture

  • Effective on-call principles for site reliability engineers, including rotation structure, alert threshold maintenance, and escalation practices

  • Chaos Engineering practices to identify random and unpredictable behavior in your system

  • Monitoring and metrics to detect incidents caused by self-healing systems

  • Creating a high-reliability culture by listening to people in your organization

For companies such as Amazon, Dropbox, and Gremlin, the term high severity incident. (SEV) signifies drops in network availability, product feature issues, data loss, revenue loss, and security risks. These high-impact bugs occur when coding, automation, testing, and other engineering practices create issues that eventually reach the customer—issues that can exist without detection for hours, days, weeks, and even years.

With this in-depth ebook, SREs, SRE managers, VPs of engineering, and CTOs will learn powerful methods for reducing MTTD through incident classification and leveling, tooling, monitoring, KPI metrics, alerting, observability, and Chaos Engineering. The authors share real-life experiences to explain how they achieved MTTD reduction results for companies including Gremlin, LinkedIn, Twitter, Amazon Web Services, Fuzzbox, and Samba TV.

Over a decade of collective experience unleashing chaos at companies like

Download your copy

Avoid downtime. Use Gremlin to turn failure into resilience.

Gremlin empowers you to proactively root out failure before it causes downtime. See how you can harness chaos to build resilient systems by requesting a demo of Gremlin.

Product Hero ImageShape