Embracing Risk

The Paradox of Perfect Reliability

Here’s a thought experiment:

Imagine you run a web service. A customer asks: “Can you guarantee 100% uptime?”

You could do it. Here’s how:

1. Replicate everything 10 times. If one copy fails, nine others handle the load.
2. Store every piece of data in 20 places. The odds of losing all 20 copies simultaneously are near-zero.
3. Have engineers on-call 24/7, in shifts. Someone is always ready to fix issues in seconds.
4. Run chaos tests constantly. Find every possible failure mode.
5. Never deploy code changes. Code changes cause outages; frozen code is stable code.
6. Use only the most expensive, reliable hardware. Avoid cheaper components that fail more often.

Result: 100% uptime achieved!

Cost: About $X million per year for a modest service.

Also: You haven’t shipped a feature in 18 months. Competitors have moved into your market. Engineers are burned out. The business is bankrupt.

This is why perfect reliability is a trap.

The Fundamental Tension: Reliability vs. Velocity

SRE exists because of a core tension in software development:

Reliability <----> Velocity
(Safe, stable)   (Fast, features)

In a traditional organization:

• Dev teams want to ship fast. “We’ll deploy this feature daily. If it breaks, we’ll fix it.”
• Ops teams want stability. “We’ll freeze deployments. No changes on Friday. No changes during peak hours.”

Both are responding rationally to incentives:

• Devs are measured on features shipped
• Ops is measured on uptime maintained

Result: Conflict, slow deployments, frustration, on-call engineers waking up at 3am.

SRE’s insight: This doesn’t have to be a tradeoff. You can have both — if you’re intentional about risk.

The Error Budget: Quantifying Risk

Here’s the core SRE concept: the error budget.

Definition: An error budget is the amount of downtime (or errors) a service can tolerate while still meeting its reliability target.

Example calculation:

You decide your service should be 99.9% available (three 9s).

Total time in a month: 
  30 days × 24 hours × 60 minutes = 43,200 minutes

Error budget:
  43,200 × (1 - 0.999) = 43,200 × 0.001 = 43.2 minutes

Interpretation:
  Your service can be down for 43.2 minutes per month 
  and still meet the 99.9% target.

Why this matters:

1. It’s a budget, not a ceiling. You can use it, and you should use it.
2. It’s shared between outages and deployments. If you have a major outage that burns 30 minutes, you have 13 minutes left. That 13 minutes is your “safe deployment window.”
3. It’s a business decision, not a technical one. Your CTO and product lead decide the target, not your SRE team.

Using Your Error Budget: Three Scenarios

Scenario 1: You Have Error Budget Remaining

Your service is 99.95% available for the month (better than the 99.9% target).

You have 26 minutes of error budget left.

Decision: Deploy that risky feature.

Why? Because:

• The feature is valuable to the business
• You have buffer
• If it causes a 10-minute outage, you’re still within SLO
• The learning from a small failure is worth more than waiting

Without error budget thinking: “We’ve been stable. Let’s never change anything.” (Stagnation)

With error budget thinking: “We have room for a small failure. Let’s ship this.”

Scenario 2: You’re Out of Error Budget

It’s mid-month. A bug caused a 25-minute outage yesterday. You’ve now exceeded your monthly error budget.

What do you do?

1. Freeze risky deployments. Only deploy critical bugfixes.
2. Reduce blast radius. For necessary deployments, roll out to 5% of traffic, watch it for 1 hour, then 50%, then 100%.
3. Increase monitoring. Add extra alerts so you catch issues faster.
4. Investigate root cause. Why did the bug get through? Improve tests or code review.
5. Plan for next month. If you’re regularly out of budget, your SLO is too aggressive.

The key: Out-of-budget is a signal, not a punishment. It tells you to slow down, not to panic.

Scenario 3: You Never Use Your Error Budget

Your service is 99.95% available, but your SLO is 99.9%.

This is actually a problem, not a success.

Why?

1. Your SLO is not honest. If you never risk it, it’s not your real target; it’s a guess.
2. You’re leaving value on the table. You could ship faster, test riskier features, or invest in other services.
3. Ops will tighten the SLO next year. “You were 99.95%, so let’s target 99.95% now.” This continues until you’re chasing perfection.

Healthy error budget usage:

• You have a 99.9% SLO
• Most months you’re at 99.93%
• A couple times a year, you deploy something that slightly risks it
• Maybe once a year you hit an outage that brings you down to 99.85%
• Your error budget is working as intended

Making Risk Decisions

Not all risks are equal. Here’s a framework for deciding whether to take a risk:

1. What’s the potential impact?

• Small: Affects <1% of users, recoverable in <5 minutes
• Medium: Affects 1–10% of users, recoverable in 5–30 minutes
• Large: Affects >10% of users, recoverable in >30 minutes

2. How likely is this impact?

• Low (< 5% chance): “Probably won’t happen”
• Medium (5–20% chance): “Could happen, but not likely”
• High (> 20% chance): “Probably will happen”

3. What’s the value of taking this risk?

• High value: Critical feature, or major reliability improvement
• Medium value: Nice-to-have feature, moderate improvement
• Low value: Cosmetic change, minor optimization

4. Can you mitigate?

• Yes (use canary deployments, gradual rollout): Reduce the risk substantially
• Partial (add monitoring, prepare rollback): Reduce impact if it goes wrong
• No (it’s either broken or working): All-or-nothing

Decision rule:

Error Budgets in Practice: Real Example

The Scenario:

Your team runs a payment processing service. SLO: 99.95% uptime.

End of month: You’re at 99.94%, so you’ve got about 4 minutes of budget left.

Your team wants to deploy a new feature: Batch payment processing (processes 1000 payments in one transaction instead of one at a time).

The analysis:

1. Impact: If it breaks, all payments in that batch fail. High risk. Potential to affect hundreds or thousands of users.
2. Likelihood: Your tests pass, but it’s complex code. You’d estimate 15% chance of a production issue.
3. Value: Customers have been asking for this for months. Competitive advantage. High value.
4. Mitigation: You can deploy to 1% of traffic first, monitor for 2 hours, then 10%, then 100%.

Decision:

With 4 minutes of budget and 15% failure risk, you don’t deploy now. But here’s what you do:

1. Prepare for next month. Plan to deploy on Day 1 of next month with a fresh budget.
2. Implement canary strategy. Start with 1% traffic, let it run for 4 hours (covers business-hour variations).
3. Add observability. Instrument the feature heavily; log every batch start and end.
4. Prepare rollback. Have a manual rollback plan in case it goes wrong.
5. On-call prep. Brief the on-call engineer on what this feature does and how to debug it.

Result: The business gets the feature. The service stayed within SLO. No 3am pages. Everyone wins.

SLO Targets: Choosing the Right Number

How do you decide on an SLO? Some guidance:

Rules of thumb:

1. Never promise more than you deliver consistently. If you say 99.9% but average 99.95%, that’s fine. If you say 99.95% but average 99.9%, you’re in trouble.
2. Align SLO with business impact. If downtime costs $50k/minute, you need higher availability than if it costs $5/minute.
3. Be realistic about dependencies. If you depend on AWS, you can’t promise better uptime than AWS provides. (AWS typically guarantees 99.99% per-AZ, but that’s still an external risk.)
4. Review annually. As your system matures and usage grows, your reliability should improve. Tighten your SLO.

SRE Interview Tip

99.9%
= 43.2 min/month downtime

99.99%
= 4.32 min/month downtime

43.2 ÷ 4.32 = 10x