The Reliability Survival Guide
A Summer Read for Building Systems That Do Not Break
Your system is down. It costs you $50,000 per minute. The incident commander is asking why no one is waking up. Your on-call engineer has no runbook. Your backup database is 18 hours out of sync. Your CEO is asking when it will be back.
This has happened to you. Or it will.
This book is not about preventing that moment. It is about understanding why it happens, what choices led to it, and what you can actually do about it when you have limited money, limited staff, and limited visibility into what your infrastructure is really doing.
What This Book Is
This is a survival guide for operators, architects, and leaders running real systems under real constraints.
Primary audience: senior engineers, principal engineers, and architects who make design and operational decisions.
Secondary audience: CTOs and business leaders who need a translation layer between reliability decisions and financial consequences.
Not “what reliability looks like in ideal conditions.” Not another essay about nine nines.
The Google SRE Book is the source of truth for building and running an SRE function. Google coined the term. The principles in that book are foundational, and this book references them throughout. If you are standing up or scaling a dedicated SRE team, that book should be on your desk.
This book is different. It covers the work that has to happen before an SRE team is viable, and the ongoing work that SRE teams depend on but cannot build alone. The economic framing, the tiering decisions, the incentive structures, the provider failure constraints, and the governance artifacts are not in the Google SRE Book. They are the foundation it assumes you have already built. Most organizations have not built it.
If you have an SRE team, this book is the briefing they need on how your organization actually works. If you do not have one yet, this book is the groundwork that makes one viable.
Note: This book uses technical terms. See the Glossary for definitions.
This is what you need to know when:
- Cost pressure fights reliability ambitions
- Your team is 4 people covering 47 services
- Your cloud provider’s SLA is not your system’s SLA
- Your system has failures you cannot see
- Your on-call rotation is burning people out
- You discover reliability only after the outage
Each chapter does four things: names an uncomfortable truth, explains a model you can use, shows what fails in practice, and gives you something to do this quarter.
This principle runs through every page:
Reliability is not purchased at deployment. Your team continuously makes tradeoffs between reliability features, business incentives, and time pressure.
How to Read Claims in This Guide
This guide uses three types of claims, each labeled clearly:
Proven statistic Data published by other companies or benchmarks you can trust.
Observed pattern Behavior we have seen repeat across multiple organizations in production.
Model assumption An assumption we made to reason through the tradeoffs.
These are not the same. A pattern can be useful without being universal. Look for the label to understand how strong the claim is.
The Book Map
The stack diagram is your unifying reference model. Each chapter deepens one layer. Read them in order, or jump to the layer you need to understand.
Cross-Chapter Failure Taxonomy
Use this list to classify incidents and design decisions consistently:
- Organizational domain: who decides what, who owns what, and what is incentivized
- Control plane domain: login, deployment, configuration, policy, and scaling control
- Data plane domain: serving requests, correctness of results, and uptime
- Dependency domain: other companies’ systems (AWS, payment processors, etc.)
- Economic domain: budget limits and number of servers you can afford
Most production incidents involve multiple domains. One problem rarely explains the whole outage.
Part 1: The Truth
Why Reliability Fails Before Infrastructure Fails
Chapter 1: Reliability Is an Economic Decision Money buys reliability. But only if you understand what reliability actually costs. Most organizations spend on the wrong things.
Chapter 2: Systems Fail According to Incentives Your system does not fail because it is poorly engineered. It fails because someone is incentivized to make a choice that leads to failure. Find whose incentive is creating the problem.
Chapter 3: The Things That Actually Break The specific failure modes that wake your on-call engineer at 3 AM. No abstractions. What actually breaks, not what you think will break.
Chapter 3b: Shared Responsibility, Accountability Vacuum When reliability becomes “everyone’s job”, it becomes “no one’s job”. This chapter explains why, and what structure works instead.
Part 2: The Model
How to Think About Reliability as Economics, Recovery, and Risk
Control plane reliability does not equal customer reliability. The database is healthy but customers cannot log in. This distinction shows up in every chapter.
Chapter 4: The Reliability Equation—A Financial Model How to think about reliability using recovery time, data loss, and business impact. This model runs through the rest of the book.
Chapter 5a: Provider Failures as System Constraints Your cloud provider’s 99.9% uptime is your starting point, not your destination. Understand what this SLA does NOT promise.
Chapter 5b: Identity—The System Kill Switch In modern systems, identity is often the most critical part. When login breaks, everything breaks. This chapter shows why and what to do.
Part 3: The Reality
What Actually Breaks in Production
Chapter 6a: Partial Failure and Control Plane Failures Systems do not fail completely. They fail partially, unpredictably, and in ways that surprise everyone.
Chapter 6b: Silent Outages—When Data Corruption Looks Like Success Your system returns OK. Error rate is near zero. Your data is corrupted. The most dangerous failures look like success.
Chapter 7a: How You See (and Miss) Reality You have not failed in 18 months. Your uptime is 99.9%. You are exactly when you are most vulnerable. This chapter breaks your confidence before teaching you to rebuild it.
Chapter 7b: Change—The Failure You Deploy Yourself Deployments, config changes, migrations. Change is the most common cause of outages. This chapter explains how to make change safer.
Part 4: The Trade-offs
Where Cost, Burnout, and Reliability Start Fighting
Chapter 7c: The Hidden Cost of Reliability Tooling Faster detection costs more money. Better monitoring needs more storage. Every reliability improvement costs something.
Chapter 8: Reliability Trade-offs—On-Call, FinOps, and the Negotiation You cannot optimize for reliability, cost, AND human burnout at the same time. This chapter shows the tradeoffs your organization is making.
Part 5: The System
What to Implement If You Want Reliability to Be Governable
Chapter 9: Reliability Governance—ADRs, Ledgers, and Indicators You cannot govern what you do not measure. You cannot measure what you do not define. This chapter builds the governance system that makes reliability decisions repeatable.
Chapter 12: Reliability Pricing and the SaaS Margin Trap The moment you commoditize reliability in your pricing is the moment you lock yourself into a reliability cost. This chapter explains why, and what SaaS companies get wrong about margin.
Chapter 13: Reliability Maturity and Organizational Adoption The hardest part: getting an organization to actually adopt a reliability system. This chapter maps the 4 phases of adoption, why organizations reject systems, and how adoption actually happens.
Part 6: The Execution
What to Do Starting This Quarter
Chapter 10: Reliability Execution—The Quarterly Plan Theory is useless. This chapter is your operational checklist: what to do in the next 90 days to move the needle on reliability in your organization.
Appendix: Operating Artifacts and Policy Templates Drop-in templates, policy language, and worked examples you can adopt Monday morning: SLO policies, on-call policies, incident post-mortems, tiering frameworks, ADR templates.
How to Use This Guide
If you have 2 hours: Read Chapters 1, 2, 4, and 7b. You will understand the principle and the failure mode that causes most outages.
If you have a weekend: Read Chapters 1–10 in order. You will have the complete mental model and a quarterly plan.
If you are on-call tonight: Jump to Chapter 0 and Chapter 6b (Silent Outages). They explain what to do first and what most monitors miss.
If you are building a reliability program: Start with Chapters 1–4 (establish the model), then Chapter 9–10 (build the governance), then Appendix (copy the templates).
If you are a CTO or platform leader: Chapters 1, 2, 8, 12, 13 are your read. They explain the negotiation that reliability actually is.
Field Examples (Azure, Explicitly Labeled)
These examples are included because this is the environment I know best. The underlying pattern is provider-neutral.
-
Example 1: Control plane impairment (Azure). If Azure Resource Manager operations fail while compute remains healthy, customer impact still occurs because scaling, redeployment, policy changes, and secret rotation are blocked. Equivalent failure classes exist in AWS and GCP control plane APIs.
-
Example 2: Identity dependency concentration (Azure). If Managed Identity token acquisition degrades, authentication and authorization paths can fail even when application code is healthy. Equivalent identity dependency risks exist with IAM and token services in every major cloud.
-
Example 3: Region strategy confusion (Azure). Paired-region and zone-redundant options are implementation choices, not business strategy. The strategy is still your declared RTO, RPO, and tested failover and failback path. The same discipline applies in AWS and GCP multi-region designs.
The Book Promise
Most reliability writing explains how systems should work in ideal conditions.
This book is about how reliability survives real companies: with cost pressure, limited staff, partial visibility, provider failures, dependency concentration, and leadership teams that often discover reliability only after the outage.
Every chapter is written from the field, not the lab.
Want to Publish This?
If you are an editor, publisher, or agent who wants to turn this into a real book, reach out. I am open to it.
I work at Microsoft. The views expressed here are my own and based solely on publicly available information. This content is for educational purposes and does not represent official Microsoft guidance or commitments.