Exercices et Projets

---

1 - Exercices pratiques

Exercice 1 : Définir des SLOs

Objectif : Définir des SLIs et SLOs pour un service web.

Contexte : API e-commerce avec endpoints checkout, search, listing.

Solution

# slo-specification.yaml
service: ecommerce-api

slis:
  availability:
    description: "Proportion of successful HTTP requests"
    good_events: "HTTP 2xx, 3xx responses"
    total_events: "All HTTP responses (excluding 4xx)"
    
  latency_checkout:
    description: "Checkout completion within 3 seconds"
    good_events: "Checkout requests < 3s"
    total_events: "All checkout requests"
    
  latency_search:
    description: "Search results within 500ms"
    good_events: "Search requests < 500ms"
    total_events: "All search requests"

slos:
  availability:
    target: 99.9%
    window: 30 days rolling
    error_budget: 43.2 minutes
    
  latency_checkout:
    target: 99%
    threshold: 3 seconds
    window: 30 days rolling
    
  latency_search:
    target: 95%
    threshold: 500ms
    window: 30 days rolling

# Prometheus queries

# Availability SLI
sum(rate(http_requests_total{status=~"2..|3.."}[5m])) 
/ 
sum(rate(http_requests_total{status!~"4.."}[5m]))

# Latency SLI (checkout)
sum(rate(http_request_duration_seconds_bucket{endpoint="/checkout",le="3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{endpoint="/checkout"}[5m]))

---

Exercice 2 : Error Budget Policy

Objectif : Créer une Error Budget Policy.

Solution

# error-budget-policy.yaml
service: ecommerce-api
version: 1.0
effective_date: 2024-01-01

stakeholders:
  - Product Manager: [email protected]
  - Engineering Lead: eng-[email protected]
  - SRE Lead: sre-[email protected]

slo:
  metric: availability
  target: 99.9%
  window: 30 days rolling
  error_budget: 0.1%

thresholds:
  healthy:
    condition: "Budget > 50%"
    actions:
      - Normal velocity
      - Standard deployments
      - Innovation encouraged
      
  warning:
    condition: "Budget 25-50%"
    actions:
      - Alert stakeholders weekly
      - Review recent incidents
      - Prioritize reliability fixes
      - Reduce risky deployments
      
  critical:
    condition: "Budget < 25%"
    actions:
      - Daily status meetings
      - Feature freeze
      - Focus exclusively on reliability
      - Require SRE approval for changes
      
  exhausted:
    condition: "Budget = 0%"
    actions:
      - Emergency mode
      - Only critical bug fixes
      - Executive escalation
      - Mandatory post-mortem

reset_conditions:
  - New 30-day window starts
  - Major architecture improvement approved by SRE

review_cadence:
  - Weekly: Budget status review
  - Monthly: Policy effectiveness review
  - Quarterly: SLO targets review

---

Exercice 3 : Post-mortem

Objectif : Rédiger un post-mortem pour un incident fictif.

Scénario : Le service de paiement a été down pendant 45 minutes à cause d'une mise à jour de configuration mal testée.

Solution

# Post-mortem: Payment Service Outage

## Summary
**Date:** 2024-01-15
**Duration:** 45 minutes (14:00 - 14:45 UTC)
**Severity:** SEV1
**Author:** Jane Doe
**Status:** Final

### Impact
- 100% of payment transactions failed
- ~2,500 users affected
- Estimated revenue impact: $15,000
- Customer support tickets: 150

## Timeline (UTC)

| Time | Event |
|------|-------|
| 13:45 | Config change deployed to production |
| 14:00 | First alerts for payment failures |
| 14:05 | On-call engineer acknowledged |
| 14:10 | Incident Commander assigned |
| 14:15 | Identified recent config change |
| 14:25 | Decision to rollback |
| 14:35 | Rollback completed |
| 14:45 | Service fully recovered |

## Root Cause
A configuration change to the payment gateway timeout settings 
was deployed without proper testing. The new 5-second timeout 
was too aggressive for the payment processor's actual response 
times, causing all requests to time out.

## Contributing Factors
1. **Missing staging test**: Config change bypassed staging
2. **No canary deployment**: 100% traffic hit at once
3. **Unclear rollback procedure**: Delay in rollback decision
4. **Alert gap**: No alert for timeout errors specifically

## Lessons Learned

### What went well
- Fast detection (15 minutes)
- Good incident communication
- Clean rollback

### What didn't go well
- Config change not tested
- Rollback decision delayed
- Missing specific monitoring

### Where we got lucky
- Happened during low traffic period
- No data corruption

## Action Items

| Action | Priority | Owner | Due | Status |
|--------|----------|-------|-----|--------|
| Add staging gate for config changes | P1 | @alice | 01/22 | Open |
| Implement config canary deployment | P1 | @bob | 01/29 | Open |
| Add timeout error alerting | P1 | @carol | 01/20 | Done |
| Update rollback runbook | P2 | @dave | 01/25 | Open |
| Load test payment timeouts | P2 | @eve | 02/01 | Open |

---

Exercice 4 : Toil Identification

Objectif : Identifier et prioriser le toil dans une équipe.

Solution

# toil-analysis.yaml
team: Platform SRE
period: Q4 2024
total_work_hours: 4160  # 4 people * 40h * 26 weeks

toil_inventory:
  - task: "Manual database failover"
    frequency: "Monthly"
    time_per_occurrence: "2 hours"
    monthly_hours: 2
    automation_effort: "40 hours"
    automation_savings: "24 hours/year"
    roi_months: 20
    priority: Low
    
  - task: "Certificate renewal"
    frequency: "10 per month"
    time_per_occurrence: "30 min"
    monthly_hours: 5
    automation_effort: "20 hours"
    automation_savings: "60 hours/year"
    roi_months: 4
    priority: High
    
  - task: "User access requests"
    frequency: "50 per month"
    time_per_occurrence: "15 min"
    monthly_hours: 12.5
    automation_effort: "80 hours"
    automation_savings: "150 hours/year"
    roi_months: 6
    priority: High
    
  - task: "Manual deployments"
    frequency: "20 per week"
    time_per_occurrence: "30 min"
    monthly_hours: 40
    automation_effort: "120 hours"
    automation_savings: "480 hours/year"
    roi_months: 3
    priority: Critical

summary:
  total_toil_hours: 59.5/month
  toil_percentage: 14.3%
  target_percentage: 10%
  
action_plan:
  q1:
    - Automate deployments (Critical)
    - Automate certificate renewal (High)
  q2:
    - Self-service access requests (High)
  q3:
    - Automate DB failover (Low)

---

2 - Projet complet : SRE Implementation

Architecture

SLO Recording Rules

# prometheus-rules.yaml
groups:
  - name: slo_rules
    rules:
      # Availability SLI
      - record: sli:api_availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="api",status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))
          
      # Latency SLI (P99 < 500ms)
      - record: sli:api_latency:ratio_rate5m
        expr: |
          sum(rate(http_request_duration_seconds_bucket{job="api",le="0.5"}[5m]))
          /
          sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
          
      # Error budget (30 day window)
      - record: slo:api_availability:error_budget_remaining
        expr: |
          1 - (
            (1 - avg_over_time(sli:api_availability:ratio_rate5m[30d]))
            /
            (1 - 0.999)
          )
          
      # Burn rate (1h)
      - record: slo:api_availability:burn_rate_1h
        expr: |
          (1 - avg_over_time(sli:api_availability:ratio_rate5m[1h]))
          /
          (1 - 0.999)

Alerting Rules

groups:
  - name: slo_alerts
    rules:
      - alert: ErrorBudgetFastBurn
        expr: slo:api_availability:burn_rate_1h > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn"
          description: "Burn rate is {{ $value | printf \"%.1f\" }}x"
          runbook: "https://wiki/runbooks/error-budget"
          
      - alert: ErrorBudgetLow
        expr: slo:api_availability:error_budget_remaining < 0.25
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error budget below 25%"
          
      - alert: SLOBreached
        expr: slo:api_availability:error_budget_remaining < 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "SLO has been breached"

Grafana Dashboard

{
  "title": "SRE Overview",
  "panels": [
    {
      "title": "Error Budget Remaining",
      "type": "gauge",
      "targets": [{
        "expr": "slo:api_availability:error_budget_remaining * 100"
      }],
      "fieldConfig": {
        "defaults": {
          "min": 0,
          "max": 100,
          "unit": "percent",
          "thresholds": {
            "steps": [
              { "color": "red", "value": 0 },
              { "color": "yellow", "value": 25 },
              { "color": "green", "value": 50 }
            ]
          }
        }
      }
    },
    {
      "title": "Burn Rate",
      "type": "stat",
      "targets": [{
        "expr": "slo:api_availability:burn_rate_1h"
      }],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "color": "green", "value": 0 },
              { "color": "yellow", "value": 1 },
              { "color": "red", "value": 6 }
            ]
          }
        }
      }
    }
  ]
}

---

3 - Quiz de révision

1. Quelle est la formule de l'Error Budget ?

2. Quels sont les 3 types de métriques SLI les plus courants ?

3. Qu'est-ce que le Toil et comment le réduire ?

4. Quels sont les rôles clés dans un incident ?

5. Pourquoi la culture "blameless" est-elle importante ?

Réponses

1. Error Budget = 1 - SLO (ex: SLO 99.9% → Error Budget 0.1%)

2. Availability (uptime), Latency (temps de réponse), Error Rate (taux d'erreurs)

3. Toil = travail manuel, répétitif, automatisable. Réduction : automatisation, élimination, self-service.

4. Incident Commander (coordination), Communications Lead (communication), Technical Lead (investigation/fix)

5. Permet de partager ouvertement les erreurs, apprendre des incidents, et améliorer les systèmes plutôt que blâmer les individus.

---

4 - Certifications et Ressources

Certifications recommandées

Certification	Fournisseur	Focus
Google Cloud Professional Cloud DevOps	Google	SRE on GCP
AWS Certified DevOps Engineer	AWS	AWS operations
CKA + CKS	CNCF	Kubernetes

Livres essentiels

• Site Reliability Engineering (Google)
• The Site Reliability Workbook (Google)
• Implementing Service Level Objectives (Alex Hidalgo)
• Incident Management for Operations (Rob Schnepp)

Ressources

• SRE.google
• SRE Weekly Newsletter
• Learning from Incidents

---

Résumé du cours

Félicitations ! Vous avez complété le cours Site Reliability Engineering.

Vous maîtrisez maintenant :

• Les principes SRE de Google
• Les SLIs, SLOs, SLAs
• La gestion de l'Error Budget
• L'élimination du Toil
• La gestion des incidents
• Les Post-mortems blameless
• Le Capacity Planning
• Le Release Engineering

Prochaines étapes

• Implémenter les SLOs dans votre organisation
• Créer une Error Budget Policy
• Établir une culture blameless
• Mesurer et réduire le toil

---

← Retour à la table des matières