Skip to content

SRE Incident Command & Blameless Post-Mortem Standard

Skill metadata

Name incident-management-and-postmortem · Level staff · Tags sre incident-response postmortem ics runbooks

"SRE incident command: ICS role assignment (IC, Ops, Comms, Scribe), severity matrix, live mitigation and comms protocol, and blameless post-mortem with tracked action items. Use when running or improving incident response, declaring severity, coordinating an active outage, or writing a post-mortem."

Source: skills/sre-slo-sla-observability/incident-management-and-postmortem/SKILL.md

When to Use This Skill

Triggers — load this skill when:

  • An active outage needs roles, comms cadence, and mitigation discipline
  • A severity matrix or escalation policy must be defined or applied
  • A blameless post-mortem and its action items need to be written and tracked

Route elsewhere when:

  • Security compromise rather than availability failure -> secops-incident-triage-forensics
  • Error-budget policy that triggers freezes -> sli-slo-error-budget-design
  • DR failover execution -> backup-and-disaster-recovery

1. Incident Command System (ICS) Roles

                   [Incident Commander (IC)]
                               |
            +------------------+------------------+
            |                                     |
    [Operations Lead]                      [Communications Lead]
  Executes diagnostic checks             Updates statuspage, executives,
  and deployment rollbacks               and customer support channels

2. Severity Classification Matrix

Severity Customer Impact MTTD Target MTTR Target Action Required
SEV-1 Critical system down (>25% users affected / financial loss) < 5 mins < 30 mins Immediate all-hands bridge, executive notification every 30m
SEV-2 Significant degradation / core feature failing < 15 mins < 2 hours Dedicated Slack channel + SRE bridge
SEV-3 Minor non-blocking issue or single customer affected < 1 hour < 1 business day Standard ticket queue triage

3. Blameless Post-Mortem Template

# Incident Post-Mortem: [YYYY-MM-DD] [Incident Title]

## Executive Summary
- **Impact Duration**: 42 minutes (14:12 UTC to 14:54 UTC)
- **User Impact**: ~14,200 checkout attempts failed with HTTP 504.
- **Root Cause**: Database connection pool exhaustion caused by missing query index during peak traffic burst.

## Timeline (UTC)
- **14:12** - Latency alert fires for `checkout-service`.
- **14:15** - Incident declared SEV-1; Incident Commander appointed.
- **14:28** - Operations lead discovers 100% DB thread pool saturation.
- **14:38** - Hotfix index deployed to RDS reader/writer.
- **14:54** - Latency and error rates return to nominal SLO levels.

## Lessons Learned & Action Items
| Action Item | Type | Owner | Priority | Target Date |
|---|---|---|---|---|
| Implement query timeout limit on checkout repo | Preventative | @alice | P1 | Next Sprint |
| Add DB connection pool saturation SLO alert | Detection | @bob | P1 | 3 Days |

4. Scribe Discipline & Role Handoff

The Scribe is not optional. Without one, the timeline is reconstructed afterwards from memory and Slack scroll-back, which is exactly when detail is lost and blame appears.

The Scribe records, in the incident channel, with timestamps in UTC:

  • every state change ("error rate back under 1%"), not commentary;
  • every action taken and by whom, including the ones that did not work;
  • every hypothesis raised and how it was ruled in or out;
  • every decision the Incident Commander makes, and the time it was made.

That log is the post-mortem's timeline section, written for free during the incident.

Explicit handoff. Fatigue causes second incidents. Every role hands over out loud and in writing after ~2 hours or at a shift boundary:

HANDOFF 14:05Z — Incident Commander: @alex -> @sam
Current state: SEV2, error rate 4% (was 18%), degraded checkout only.
Working theory: connection-pool exhaustion after the 12:40Z deploy.
In flight: @dana rolling back to build 4471 (ETA 14:15Z).
Ruled out: DNS, upstream provider, node pressure.
Next decision point: 14:20Z — if rollback does not clear it, escalate to SEV1 and page DBRE.

The outgoing IC states it, the incoming IC confirms in channel, and the Scribe logs both. No silent handoffs, and never more than one Incident Commander at a time.