Alerting

---

1 - Principes d'alerting

1.1 Caractéristiques d'une bonne alerte

1.2 Anti-patterns

Anti-pattern	Problème	Solution
Alert fatigue	Trop d'alertes	Réduire, consolider
Flapping alerts	On/off fréquent	Ajouter hysteresis
No runbook	Pas d'action claire	Documenter
Missing context	Pas assez d'info	Enrichir annotations

---

2 - Alertmanager

2.1 Architecture

2.2 Configuration

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'secret'
  slack_api_url: 'https://hooks.slack.com/services/xxx'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - receiver: 'critical-pagerduty'
      match:
        severity: critical
      continue: true
      
    - receiver: 'warning-slack'
      match:
        severity: warning
        
    - receiver: 'team-backend'
      match_re:
        service: 'api|database'
        
    - receiver: 'team-frontend'
      match:
        service: 'web'

receivers:
  - name: 'default-receiver'
    slack_configs:
      - channel: '#alerts'
        
  - name: 'critical-pagerduty'
    pagerduty_configs:
      - service_key: 'xxx'
        severity: critical
        
  - name: 'warning-slack'
    slack_configs:
      - channel: '#alerts-warning'
        
  - name: 'team-backend'
    slack_configs:
      - channel: '#backend-alerts'
    email_configs:
      - to: '[email protected]'
        
  - name: 'team-frontend'
    slack_configs:
      - channel: '#frontend-alerts'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

---

3 - Alert Rules (Prometheus)

3.1 Structure

# alerts/rules.yml
groups:
  - name: http-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }}"
          runbook_url: "https://runbooks.company.com/high-error-rate"
          dashboard: "https://grafana.company.com/d/xxx"

3.2 Alertes essentielles

groups:
  - name: infrastructure
    rules:
      # CPU
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          
      # Memory
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
        for: 5m
        labels:
          severity: warning
          
      # Disk
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          
      # Disk prediction
      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes[6h], 24*3600) < 0
        for: 1h
        labels:
          severity: warning

  - name: application
    rules:
      # Latency
      - alert: HighLatency
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)) > 1
        for: 5m
        labels:
          severity: warning
          
      # Availability
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
          
      # Error budget burn
      - alert: ErrorBudgetBurn
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
          / 
          sum(rate(http_requests_total[1h])) by (service)
          > 0.01
        for: 5m
        labels:
          severity: warning

---

4 - Intégrations

4.1 Slack

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        username: 'AlertManager'
        icon_emoji: ':alert:'
        send_resolved: true
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'
        actions:
          - type: button
            text: 'Runbook'
            url: '{{ (index .Alerts 0).Annotations.runbook_url }}'
          - type: button
            text: 'Dashboard'
            url: '{{ (index .Alerts 0).Annotations.dashboard }}'

4.2 PagerDuty

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'xxx'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
          num_firing: '{{ .Alerts.Firing | len }}'
          resolved: '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'

4.3 Webhook (Custom)

receivers:
  - name: 'custom-webhook'
    webhook_configs:
      - url: 'https://api.company.com/alerts'
        send_resolved: true
        http_config:
          bearer_token: 'xxx'

---

5 - Silencing

5.1 Via API

# Créer un silence
curl -X POST http://alertmanager:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighCPUUsage", "isRegex": false},
      {"name": "instance", "value": "server-01", "isRegex": false}
    ],
    "startsAt": "2024-01-15T10:00:00Z",
    "endsAt": "2024-01-15T12:00:00Z",
    "createdBy": "admin",
    "comment": "Maintenance planifiée"
  }'

5.2 Via UI

Alertmanager UI → Silences → New Silence

# Matchers
alertname = HighCPUUsage
cluster = production
service =~ api.*

# Duration
Start: 2024-01-15 10:00
End: 2024-01-15 12:00

# Comment
Maintenance planifiée - Ticket JIRA-123

---

6 - Templates

# alertmanager.yml
templates:
  - '/etc/alertmanager/templates/*.tmpl'

{{/* templates/slack.tmpl */}}
{{ define "slack.default.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.default.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Description:* {{ .Annotations.description }}
*Details:*
{{ range .Labels.SortedPairs }}  • {{ .Name }}: {{ .Value }}
{{ end }}
{{ end }}
{{ end }}

---

7 - Best Practices

7.1 Niveaux de sévérité

Sévérité	Critère	Notification
Critical	Impact client, perte revenue	PagerDuty immédiat
Warning	Dégradation, risque	Slack, email
Info	Informatif	Dashboard uniquement

7.2 SLA-based Alerting

# Alert basée sur SLO
- alert: SLOBreach
  expr: |
    (
      sum(rate(http_requests_total{status!~"5.."}[30d]))
      /
      sum(rate(http_requests_total[30d]))
    ) < 0.999
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "SLO breach: availability below 99.9%"

7.3 Checklist

Pour chaque alerte:
- [ ] Actionnable? Que faire quand elle se déclenche?
- [ ] Runbook documenté?
- [ ] Testée en staging?
- [ ] Seuils basés sur des données?
- [ ] For duration appropriée?
- [ ] Labels corrects pour le routing?
- [ ] Annotations informatives?

---

Résumé

Dans ce chapitre, nous avons appris :

• Les principes d'un bon alerting
• La configuration d'Alertmanager
• Les Alert Rules Prometheus
• Les intégrations (Slack, PagerDuty)
• Le silencing des alertes
• Les templates personnalisés
• Les bonnes pratiques

---

Prochaine étape

Dans le prochain chapitre, nous verrons Distributed Tracing.

→ Chapitre suivant : Distributed Tracing

---

← Retour à la table des matières