Post-mortems
1 - Philosophie Blameless
1.1 Principes
blameless_culture:
core_principles:
- "Focus on systems, not individuals"
- "Assume good intentions"
- "Seek to understand, not to blame"
- "Learn and improve"
key_questions:
- "What happened?" (not "Who did it?")
- "What can we improve?"
- "How can we prevent this?"
1.2 Pourquoi Blameless ?
2 - Quand faire un post-mortem
2.1 Critères
postmortem_triggers:
always:
- SEV1 incidents
- Data loss or breach
- Customer-facing outage > 30 min
- SLO breach
- Repeat incidents
consider:
- SEV2 incidents
- Near misses
- Interesting learnings
- Process failures
2.2 Timeline
postmortem_timeline:
day_0: "Incident resolved"
day_1-2: "Draft postmortem written"
day_3-5: "Review meeting scheduled"
day_5-7: "Postmortem finalized"
day_7-14: "Action items assigned"
ongoing: "Track action items to completion"
3 - Template de Post-mortem
# Post-mortem: [Title]
## Summary
**Date:** YYYY-MM-DD
**Duration:** X hours Y minutes
**Severity:** SEV-X
**Author:** [Name]
**Status:** Draft / In Review / Final
### Impact
- [Number] users affected
- [Duration] of impact
- [$X] estimated revenue impact
## Timeline
All times in UTC
| Time | Event |
|------|-------|
| 14:00 | Alert triggered for high error rate |
| 14:05 | On-call engineer acknowledged |
| 14:10 | Incident Commander assigned |
| 14:30 | Root cause identified |
| 15:00 | Fix deployed |
| 15:15 | Metrics normalized |
| 15:30 | Incident resolved |
## Root Cause
[Detailed technical explanation of what caused the incident]
## Contributing Factors
1. [Factor 1 - e.g., missing monitoring]
2. [Factor 2 - e.g., configuration issue]
3. [Factor 3 - e.g., unclear runbook]
## Detection
- How was the incident detected?
- Could we have detected it sooner?
- What monitoring was missing?
## Response
- What went well?
- What could have been better?
- Were runbooks helpful?
## Resolution
- How was the incident resolved?
- What was the fix?
## Lessons Learned
### What went well
- Fast detection (5 minutes)
- Clear communication
- Effective collaboration
### What didn't go well
- Runbook was outdated
- Escalation was delayed
- Missing dashboard
### Where we got lucky
- Low traffic period
- Expert was available
## Action Items
| Action | Priority | Owner | Due Date | Status |
|--------|----------|-------|----------|--------|
| Add monitoring for X | P1 | @alice | 2024-01-20 | Done |
| Update runbook Y | P1 | @bob | 2024-01-22 | In Progress |
| Fix configuration Z | P2 | @carol | 2024-01-30 | Open |
## Supporting Information
- [Link to incident Slack channel]
- [Link to dashboards]
- [Link to logs]
4 - Root Cause Analysis
4.1 5 Whys
five_whys_example:
incident: "Website down for 30 minutes"
analysis:
why_1: "Why was the website down?"
answer_1: "The database was unreachable"
why_2: "Why was the database unreachable?"
answer_2: "The database server ran out of disk space"
why_3: "Why did it run out of disk space?"
answer_3: "Log files grew without rotation"
why_4: "Why weren't logs rotated?"
answer_4: "Log rotation wasn't configured"
why_5: "Why wasn't log rotation configured?"
answer_5: "No checklist for new servers"
root_cause: "Missing server setup checklist"
action: "Create and enforce server setup checklist"
4.2 Fishbone Diagram
4.3 Contributing Factors
contributing_factors:
technical:
- Missing monitoring
- No auto-scaling
- Single point of failure
process:
- Outdated runbook
- Unclear escalation
- Missing testing
human:
- Fatigue (overnight)
- Knowledge gap
- Miscommunication
environmental:
- Unexpected traffic spike
- Third-party outage
- Network issues
5 - Review Meeting
5.1 Agenda
review_meeting:
duration: "45-60 minutes"
agenda:
1_intro: "5 min"
- Welcome
- Reminder of blameless culture
2_timeline: "10 min"
- Walk through events
- Clarify any gaps
3_root_cause: "15 min"
- Review analysis
- Discuss contributing factors
4_lessons: "10 min"
- What went well
- What to improve
5_actions: "15 min"
- Review proposed actions
- Assign owners
- Set deadlines
6_wrap: "5 min"
- Next steps
- Questions
5.2 Facilitation Tips
facilitation:
do:
- Keep focus on systems, not people
- Encourage all voices
- Ask open questions
- Document everything
- Follow up on action items
dont:
- Allow finger pointing
- Skip the meeting if busy
- Let discussions go off-topic
- Create too many action items
- Forget to follow up
6 - Action Items
6.1 Types d'actions
action_types:
prevent_recurrence:
- Fix the bug
- Add validation
- Improve architecture
improve_detection:
- Add monitoring
- Create alerts
- Improve dashboards
improve_response:
- Update runbooks
- Add automation
- Training
improve_process:
- Change review process
- Update checklists
- Improve testing
6.2 SMART Actions
smart_actions:
bad_example:
action: "Improve monitoring"
problem: "Vague, unmeasurable"
good_example:
action: "Add alert for database disk usage > 80%"
specific: "Database disk usage alert"
measurable: "Alert exists and fires at 80%"
achievable: "Simple Prometheus alert"
relevant: "Prevents disk space incidents"
time_bound: "Due in 1 week"
6.3 Tracking
action_tracking:
tools:
- JIRA with "postmortem" label
- Dedicated tracker
- OKR alignment
metrics:
- Actions created per postmortem
- Time to close actions
- Percentage completed on time
- Recurring incidents reduced
7 - Sharing Learnings
7.1 Communication
sharing_channels:
internal:
- Team standup
- Engineering all-hands
- Postmortem newsletter
- Wiki/documentation
external:
- Tech blog (sanitized)
- Conference talks
- Industry sharing
7.2 Postmortem Review Rotation
review_rotation:
frequency: "Weekly"
format: "30 min meeting"
agenda:
- Review 1-2 recent postmortems
- Discuss patterns
- Share cross-team learnings
benefits:
- Knowledge sharing
- Pattern identification
- Culture reinforcement
8 - Métriques
postmortem_metrics:
process:
- Time to draft (target: < 2 days)
- Time to finalize (target: < 7 days)
- Action items per postmortem
effectiveness:
- Repeat incident rate
- MTTR improvement
- Action completion rate
culture:
- Postmortem participation
- Blameless sentiment
- Knowledge sharing frequency
Résumé
Dans ce chapitre, nous avons appris :
- La philosophie Blameless
- Quand faire un post-mortem
- Le template complet
- L'analyse Root Cause (5 Whys)
- La Review Meeting
- Les Action Items SMART
- Le partage des learnings
Prochaine étape
Dans le prochain chapitre, nous verrons Capacity Planning.
→ Chapitre suivant : Capacity Planning