Error Budget
1 - Concept
1.1 Définition
Error Budget = Quantité d'erreurs/indisponibilité autorisée par le SLO.
1.2 Formule
error_budget:
formula: "1 - SLO"
example:
slo: 99.9%
error_budget: 0.1%
monthly_budget:
total_minutes: 43200 # 30 days * 24h * 60min
error_budget: 43.2 minutes # 0.1%
1.3 Philosophie
error_budget_philosophy:
key_insight: |
"100% reliability is wrong target.
Beyond a certain point, more reliability
provides diminishing returns."
balance:
- Too reliable → Slow innovation
- Not reliable enough → Poor UX
- Error budget → Right balance
2 - Calcul de l'Error Budget
2.1 Request-based
# 30 jours, 1M requêtes/jour
request_based:
total_requests: 30_000_000
slo: 99.9%
error_budget:
percentage: 0.1%
requests: 30_000 # Requêtes qui peuvent échouer
current_period:
failed_requests: 15_000
budget_consumed: 50%
budget_remaining: 50%
2.2 Time-based
# 30 jours
time_based:
total_time: 43200 # minutes
slo: 99.9%
error_budget:
percentage: 0.1%
minutes: 43.2
current_period:
downtime_minutes: 20
budget_consumed: 46.3%
budget_remaining: 53.7%
2.3 Prometheus Implementation
# Error budget remaining (ratio)
(
1 - (
(1 - (sum(rate(http_requests_total{status=~"2.."}[30d])) / sum(rate(http_requests_total[30d]))))
/
(1 - 0.999) # SLO target
)
)
# Error budget consumed (percentage)
100 * (
(1 - (sum(rate(http_requests_total{status=~"2.."}[30d])) / sum(rate(http_requests_total[30d]))))
/
(1 - 0.999)
)
3 - Error Budget Policy
3.1 Document Policy
# error-budget-policy.yaml
service: payment-api
effective_date: 2024-01-01
stakeholders:
- Product Manager
- Engineering Lead
- SRE Lead
slo:
metric: availability
target: 99.9%
window: 30 days rolling
thresholds:
healthy:
condition: "> 50% remaining"
actions:
- Normal development velocity
- Standard release cadence
warning:
condition: "25-50% remaining"
actions:
- Notify stakeholders
- Review recent incidents
- Reduce risk of releases
critical:
condition: "< 25% remaining"
actions:
- Freeze feature releases
- Focus on reliability improvements
- Daily standup on reliability
exhausted:
condition: "0% remaining"
actions:
- Emergency mode
- Only critical bug fixes
- Post-mortem required
- Executive escalation
reset_conditions:
- New budget window starts
- Major architecture improvement approved
3.2 Decision Framework
4 - Burn Rate
4.1 Concept
Burn Rate = Vitesse de consommation du budget.
burn_rate:
definition: "Actual consumption rate / Expected consumption rate"
examples:
- burn_rate: 1.0 # On track
- burn_rate: 2.0 # 2x faster, budget gone in 15 days
- burn_rate: 0.5 # Half speed, budget lasts 60 days
4.2 Calcul
# 1h burn rate
(
1 - (sum(rate(http_requests_total{status=~"2.."}[1h])) / sum(rate(http_requests_total[1h])))
)
/
(
(1 - 0.999) / 720 # SLO budget per hour (30d = 720h)
)
4.3 Multi-window Alerting
# Google SRE recommandation
burn_rate_alerts:
- name: "Fast burn"
short_window: 5m
long_window: 1h
burn_rate: 14.4 # 2% budget en 1h
severity: page
- name: "Slow burn"
short_window: 30m
long_window: 6h
burn_rate: 6 # 5% budget en 6h
severity: page
- name: "Gradual burn"
short_window: 2h
long_window: 1d
burn_rate: 3 # 10% budget en 1 jour
severity: ticket
- name: "Low burn"
short_window: 6h
long_window: 3d
burn_rate: 1 # Budget sur la bonne voie
severity: ticket
4.4 Alerting Rules
groups:
- name: slo-burn-rate
rules:
# Fast burn - 1h window
- alert: ErrorBudgetFastBurn
expr: |
(
sum(rate(http_requests_total{status!~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))
) > (14.4 * (1 - 0.999))
and
(
sum(rate(http_requests_total{status!~"2.."}[1h]))
/
sum(rate(http_requests_total[1h]))
) > (14.4 * (1 - 0.999))
for: 2m
labels:
severity: critical
annotations:
summary: "Fast error budget burn detected"
# Slow burn - 6h window
- alert: ErrorBudgetSlowBurn
expr: |
(
sum(rate(http_requests_total{status!~"2.."}[30m]))
/
sum(rate(http_requests_total[30m]))
) > (6 * (1 - 0.999))
and
(
sum(rate(http_requests_total{status!~"2.."}[6h]))
/
sum(rate(http_requests_total[6h]))
) > (6 * (1 - 0.999))
for: 15m
labels:
severity: warning
5 - Utilisation stratégique
5.1 Budget pour l'innovation
innovation_spending:
scenario: "Budget healthy (70% remaining)"
opportunities:
- Risky deployments
- New features
- Infrastructure changes
- Experiments
strategy:
- Use canary deployments
- Plan for potential rollbacks
- Monitor closely during changes
5.2 Conversations Product vs Reliability
product_conversation:
when: "Product wants to ship fast"
with_error_budget:
product: "We need to ship feature X now"
sre: "Budget is at 30%, we can ship with extra monitoring"
outcome: "Data-driven decision"
without_error_budget:
product: "We need to ship feature X now"
sre: "It's too risky"
outcome: "Subjective argument"
5.3 Blameless attribution
budget_consumption_tracking:
sources:
- planned_maintenance: 10%
- deployments: 25%
- infrastructure: 30%
- external_dependencies: 20%
- unknown: 15%
action:
- Focus on top consumers
- Invest in reducing each category
- Track improvements over time
6 - Dashboard Error Budget
dashboard_panels:
- row: "Current Status"
panels:
- title: "Error Budget Remaining"
type: gauge
query: slo:error_budget:remaining * 100
thresholds: [25, 50, 75]
- title: "Burn Rate (1h)"
type: stat
query: slo:burn_rate:1h
thresholds: [1, 6, 14.4]
- title: "Time to Exhaustion"
type: stat
query: slo:time_to_exhaustion_hours
- row: "Trends"
panels:
- title: "Error Budget Over Time"
type: timeseries
query: slo:error_budget:remaining * 100
- title: "Budget Consumption by Source"
type: piechart
queries:
- deployments
- infrastructure
- external
Résumé
Dans ce chapitre, nous avons appris :
- Le concept d'Error Budget
- Les méthodes de calcul
- Les Error Budget Policies
- Le Burn Rate et multi-window alerting
- L'utilisation stratégique du budget
- Les dashboards Error Budget
Prochaine étape
Dans le prochain chapitre, nous verrons Élimination du Toil.
→ Chapitre suivant : Élimination du Toil