Bonnes pratiques SRE
1 - Organisation SRE
1.1 Modèles d'équipe
team_models:
centralized:
description: "Une équipe SRE pour toute l'organisation"
pros:
- Expertise centralisée
- Standards cohérents
- Partage de connaissances
cons:
- Peut devenir bottleneck
- Moins de contexte produit
best_for: "PME, début de transformation"
embedded:
description: "SRE intégrés dans les équipes produit"
pros:
- Contexte produit fort
- Réponse rapide
- Ownership clair
cons:
- Standards inconsistants
- Silos de connaissance
best_for: "Grandes organisations matures"
hybrid:
description: "Core SRE + SREs embeddés"
pros:
- Balance des avantages
- Flexibilité
cons:
- Complexité organisationnelle
best_for: "Organisations en croissance"
1.2 Staffing
sre_staffing:
ratios:
critical_services: "6-8 devs : 1 SRE"
standard_services: "10-12 devs : 1 SRE"
mature_services: "15-20 devs : 1 SRE"
team_size:
minimum: 4 # Pour rotation on-call viable
optimal: 6-8
maximum: 12 # Au-delà, splitter
2 - On-Call Excellence
2.1 Setup optimal
oncall_setup:
rotation:
duration: "1 week"
overlap: "2 hours handoff"
backup: "Secondary on-call"
expectations:
response_time:
sev1: "5 minutes"
sev2: "15 minutes"
sev3: "1 hour"
availability: "Phone + laptop accessible"
handoff:
- Review active incidents
- Check error budget status
- Review recent deploys
- Update contact info
2.2 Sustainable On-Call
sustainable_oncall:
interruptions:
target: "< 2 per shift"
max_acceptable: "5 per shift"
review_threshold: "If > 5, investigate"
quality_of_life:
- Compensatory time off
- On-call bonus
- No consecutive weeks
- Max 1 in 4 weeks
after_incident:
- Time off following major incidents
- Debrief support
- Mental health resources
2.3 On-Call Anti-patterns
oncall_antipatterns:
alert_fatigue:
problem: "Too many alerts"
solution: "Review and tune thresholds"
hero_culture:
problem: "Same person always responds"
solution: "Enforce rotation, document knowledge"
no_handoff:
problem: "Issues dropped between shifts"
solution: "Mandatory handoff meeting"
no_followup:
problem: "Same issues recur"
solution: "Track and fix root causes"
3 - Documentation
3.1 Types essentiels
documentation_types:
runbooks:
purpose: "Step-by-step incident response"
format: "Actionable, clear steps"
review: "After each incident"
architecture:
purpose: "System understanding"
format: "Diagrams + text"
review: "Quarterly"
playbooks:
purpose: "Common procedures"
format: "Process documentation"
review: "After process changes"
postmortems:
purpose: "Learning from incidents"
format: "Standard template"
review: "Within 7 days of incident"
3.2 Runbook Template
# Runbook: [Service] - [Issue Type]
## Summary
Brief description of the issue and when to use this runbook.
## Prerequisites
- Access to [system]
- Permissions for [action]
## Detection
How to identify this issue:
- Alert: `[alert name]`
- Symptoms: [symptoms]
## Steps
### 1. Assess Severity
```bash
# Check service status
kubectl get pods -n production
2. Immediate Mitigation
[Steps to mitigate]
3. Root Cause Investigation
[Investigation steps]
4. Resolution
[Resolution steps]
Escalation
- If step X fails: Contact [team]
- If severity increases: Page [person]
Related
- [Link to dashboard]
- [Link to related runbook]
---
## 4 - Automation First
### 4.1 Automation Pyramid
```mermaid
graph TB
subgraph "Level 4: Self-healing"
HEAL[Auto-remediation]
end
subgraph "Level 3: Triggered"
TRIG[Runbook automation]
end
subgraph "Level 2: Scripted"
SCRIPT[Scripts disponibles]
end
subgraph "Level 1: Documented"
DOC[Runbooks manuels]
end
subgraph "Level 0: Ad-hoc"
ADHOC[Intervention manuelle]
end
ADHOC --> DOC --> SCRIPT --> TRIG --> HEAL
4.2 What to Automate
automation_priorities:
high:
- Deployments
- Scaling
- Certificate renewal
- Backup/restore testing
- Common incident responses
medium:
- Environment provisioning
- Configuration updates
- Log rotation
- Report generation
low:
- One-time tasks
- Rare procedures
- Complex judgment calls
5 - Métriques SRE
5.1 Key Metrics
sre_metrics:
reliability:
- SLO compliance rate
- Error budget remaining
- MTTR (Mean Time To Recovery)
- MTTD (Mean Time To Detect)
operational:
- Incident frequency
- Pages per on-call shift
- Toil percentage
- Automation coverage
team:
- On-call satisfaction score
- Action item completion rate
- Knowledge sharing frequency
5.2 Dashboard
sre_dashboard:
overview:
- Current SLO status (all services)
- Error budget burn rate
- Active incidents
trends:
- Incident frequency over time
- MTTR trend
- Toil percentage
team_health:
- On-call load distribution
- Action items backlog
- Training completion
6 - Production Readiness
6.1 PRR Checklist
prr_checklist:
architecture:
- [ ] Design doc approved
- [ ] Dependencies documented
- [ ] Failure modes analyzed
- [ ] Data flow documented
observability:
- [ ] SLIs defined
- [ ] SLOs agreed
- [ ] Dashboards created
- [ ] Alerts configured
- [ ] Logging implemented
reliability:
- [ ] Load tested
- [ ] Failure scenarios tested
- [ ] Rollback tested
- [ ] Disaster recovery plan
security:
- [ ] Security review done
- [ ] Secrets management
- [ ] Access controls
operations:
- [ ] Runbooks written
- [ ] On-call training
- [ ] Escalation path defined
6.2 Graduation Criteria
graduation_criteria:
from_alpha:
- Basic SLOs defined
- Monitoring in place
- Primary on-call assigned
from_beta:
- SLOs met for 1 month
- Load testing complete
- Runbooks available
to_ga:
- SLOs met for 3 months
- No critical incidents
- Team trained
- Documentation complete
7 - Culture
7.1 Blameless Culture
blameless_practices:
postmortems:
- Focus on systems, not people
- Ask "what" not "who"
- Public and shared
psychological_safety:
- Safe to ask questions
- Safe to admit mistakes
- Safe to challenge decisions
learning:
- Regular incident reviews
- Cross-team sharing
- External learning (conferences, papers)
7.2 Collaboration Dev-SRE
collaboration:
shared_ownership:
- "You build it, you run it" (with support)
- Shared on-call for critical services
- Joint post-mortems
communication:
- Regular sync meetings
- Shared Slack channels
- Joint planning sessions
knowledge_transfer:
- Embedded SRE rotations
- Training sessions
- Pair programming
8 - Continuous Improvement
8.1 Quarterly Review
quarterly_review:
topics:
- SLO performance
- Incident trends
- Toil reduction progress
- Team health metrics
outputs:
- Updated priorities
- New automation targets
- Training needs
- Process improvements
8.2 SRE Maturity Model
sre_maturity:
level_1_reactive:
- Manual operations
- Ad-hoc monitoring
- Reactive incident response
level_2_defined:
- SLOs defined
- Basic automation
- Runbooks exist
level_3_managed:
- SLOs met consistently
- Automation significant
- Toil tracked
level_4_optimized:
- Error budget driven
- Self-healing systems
- Proactive improvement
level_5_leading:
- Industry-leading reliability
- Innovation in practices
- Contributing to community
Résumé
Dans ce chapitre, nous avons couvert :
- L'organisation des équipes SRE
- L'excellence On-Call
- La documentation essentielle
- L'automation first
- Les métriques SRE
- La Production Readiness
- La culture SRE
- L'amélioration continue
Prochaine étape
Dans le prochain chapitre, nous mettrons en pratique avec des Exercices et Projets.
→ Chapitre suivant : Exercices et Projets