Aller au contenu principal

Gestion des Incidents

1 - Cycle de vie d'un incident​

2 - Sévérité des incidents​

2.1 Classification​

SévéritéDescriptionExempleResponse Time
SEV1Outage completSite down5 min
SEV2Dégradation majeureFeature critique down15 min
SEV3Dégradation mineureLatence élevée1h
SEV4Impact minimalBug non-critical4h

2.2 Critères de classification​

severity_criteria:
  sev1_critical:
    user_impact: "> 50% users affected"
    revenue_impact: "Direct revenue loss"
    data_impact: "Data loss or breach"
    
  sev2_major:
    user_impact: "10-50% users affected"
    revenue_impact: "Potential revenue impact"
    feature_impact: "Critical feature unavailable"
    
  sev3_minor:
    user_impact: "< 10% users affected"
    performance: "Noticeable degradation"
    
  sev4_low:
    user_impact: "Minimal"
    workaround: "Available"

3 - Rôles pendant un incident​

3.1 Incident Command System​

3.2 Responsabilités​

roles:
  incident_commander:
    responsibilities:
      - Overall coordination
      - Decision making
      - Resource allocation
      - Escalation decisions
    skills:
      - Leadership
      - Communication
      - Technical understanding
      
  communications_lead:
    responsibilities:
      - Status updates
      - Stakeholder communication
      - Customer communication
      - Documentation
      
  operations_lead:
    responsibilities:
      - Coordinate technical work
      - Track actions
      - Manage responders
      
  technical_lead:
    responsibilities:
      - Technical investigation
      - Solution implementation
      - Validate fixes

4 - Process de réponse​

4.1 Workflow​

incident_workflow:
  1_detect:
    triggers:
      - Alerting system
      - Customer report
      - Internal discovery
    actions:
      - Acknowledge alert
      - Create incident ticket
      
  2_triage:
    actions:
      - Assess severity
      - Assign Incident Commander
      - Start incident channel
      - Notify stakeholders
      
  3_investigate:
    actions:
      - Gather information
      - Check recent changes
      - Review dashboards/logs
      - Form hypothesis
      
  4_mitigate:
    actions:
      - Implement fix/workaround
      - Validate improvement
      - Monitor closely
      
  5_resolve:
    actions:
      - Confirm resolution
      - Update status page
      - Close incident
      
  6_follow_up:
    actions:
      - Schedule post-mortem
      - Track action items
      - Update runbooks

4.2 Incident Channel Template​

## Incident: [Brief Description]

**Status:** 🔴 Active / 🟡 Investigating / 🟢 Resolved
**Severity:** SEV-X
**Start Time:** YYYY-MM-DD HH:MM UTC
**Incident Commander:** @name

### Impact
- [User impact description]
- [Business impact if known]

### Timeline
- HH:MM - Alert triggered
- HH:MM - IC assigned
- HH:MM - [Update]

### Current Actions
- [ ] Action 1 - @owner
- [ ] Action 2 - @owner

### Related Links
- [Dashboard]()
- [Logs]()
- [Runbook]()

5 - Communication​

5.1 Status Page Updates​

status_page_template:
  investigating:
    title: "Investigating issues with [Service]"
    body: |
      We are currently investigating issues affecting [service].
      Some users may experience [symptoms].
      We will provide updates as we learn more.
      
  identified:
    title: "Issue identified - [Service]"
    body: |
      We have identified the issue affecting [service].
      [Brief explanation without exposing vulnerabilities]
      We are working on a fix and will update shortly.
      
  monitoring:
    title: "Fix deployed - Monitoring [Service]"
    body: |
      We have deployed a fix for the issue affecting [service].
      We are monitoring to ensure stability.
      
  resolved:
    title: "Resolved - [Service]"
    body: |
      The issue affecting [service] has been resolved.
      [Duration] - From HH:MM to HH:MM UTC
      We apologize for any inconvenience.
      A post-mortem will be conducted.

5.2 Stakeholder Communication​

communication_cadence:
  sev1:
    internal: "Every 15 minutes"
    customer: "Every 30 minutes"
    executive: "Immediate + every 30 min"
    
  sev2:
    internal: "Every 30 minutes"
    customer: "Every hour"
    executive: "Daily summary"
    
  sev3:
    internal: "Every hour"
    customer: "If requested"
    executive: "Weekly summary"

6 - Escalation​

6.1 Matrice d'escalation​

escalation_matrix:
  level_1:
    who: "On-call engineer"
    when: "Alert triggered"
    authority: "Standard mitigation"
    
  level_2:
    who: "Tech Lead / Senior Engineer"
    when: "No progress in 15 min"
    authority: "Major changes"
    
  level_3:
    who: "Engineering Manager"
    when: "No progress in 30 min"
    authority: "Resource allocation"
    
  level_4:
    who: "Director / VP"
    when: "Customer/business impact"
    authority: "Cross-team coordination"
    
  level_5:
    who: "C-level"
    when: "Major outage > 1h"
    authority: "Full company resources"

6.2 Quand escalader​

escalation_triggers:
  always_escalate:
    - Data breach or security incident
    - Revenue impact > $X
    - SLA breach imminent
    - Media attention
    
  consider_escalate:
    - No progress after X minutes
    - Need expertise not available
    - Customer escalation
    - Multiple teams needed

7 - On-Call Best Practices​

7.1 Setup​

on_call_setup:
  rotation:
    type: "Weekly"
    overlap: "1 hour handoff"
    timezone: "Follow the sun (optional)"
    
  tools:
    paging: "PagerDuty / Opsgenie"
    chat: "Slack #incidents"
    video: "Zoom bridge"
    
  expectations:
    response_time: "< 5 min (SEV1)"
    availability: "Laptop + phone accessible"
    handoff: "Written summary required"

7.2 Sustainable On-Call​

sustainable_oncall:
  frequency:
    max: "1 week in 4"
    
  interruptions:
    target: "< 2 pages per shift"
    review: "If > 5, investigate"
    
  compensation:
    - Time off in lieu
    - On-call bonus
    - Meals during incidents
    
  wellbeing:
    - Post-incident debrief
    - Mental health support
    - Mandatory time off after major incidents

8 - Automation​

8.1 Auto-remediation​

auto_remediation_examples:
  pod_restart:
    trigger: "Pod health check failed"
    action: "Kubernetes auto-restart"
    
  scale_up:
    trigger: "CPU > 80% for 5 min"
    action: "HPA scales pods"
    
  failover:
    trigger: "Primary database down"
    action: "Auto-failover to replica"
    
  rollback:
    trigger: "Error rate > 5% after deploy"
    action: "Auto-rollback to previous version"

8.2 Incident Bot​

# Slack incident bot commands
/incident create "API latency high"
/incident severity SEV2
/incident assign @username
/incident status investigating
/incident timeline "Identified root cause: DB overload"
/incident resolve "Scaled database, latency normalized"

Résumé​

Dans ce chapitre, nous avons appris :

  • Le cycle de vie d'un incident
  • La classification par sévérité
  • Les rôles (Incident Commander, etc.)
  • Le process de réponse
  • La communication (status page, stakeholders)
  • L'escalation matrix
  • Les best practices on-call

Prochaine étape​

Dans le prochain chapitre, nous verrons les Post-mortems.

→ Chapitre suivant : Post-mortems

← Retour à la table des matières