Sample After-Action Report

After-Action Report: CD-⁠01 The 02:14 Certificate (annual incident response test)

1. Participants

Roles played by the facilitator: CEO (Dana), CTO (Marcus), Security Lead (Ines), Head of Support (Priya), secondary on-call SRE (Leo), Head of Platform (Tom, auto-reply only), Enterprise customer (Brightwater Dental).

2. Objectives

1. Recognise and declare a partial, growing outage quickly.
2. Separate a tempting red herring (a recent deploy) from the evidence.
3. Find a certificate chain failure in an estate that "auto-renews everything".
4. Handle a break-glass access problem when the one person who knows is unavailable.
5. Keep customers and Enterprise accounts informed while impact grows.

3. Scenario summary

As presented to players: At 02:19 UTC on a Tuesday, the on-call engineer is paged because the auth-svc login success rate has fallen to 71%, having been 99.6% all evening. Most of the company is asleep. Impact grows over the following hour, and an Enterprise customer, support, the CTO and the CEO all need attention.

What was really happening (revealed at debrief): The internal intermediate certificate authority, created by hand three years earlier with a 3-year validity, expired at 02:14:07 UTC. cert-manager renews only leaf certificates and nobody monitored the intermediate, so every certificate showed READY. Existing connections kept working and only new connections failed, so new logins broke first and the damage spread. A 01:52 web-app deploy was an unrelated red herring. The fix was a new intermediate issued from the root key, held behind two-of-three break-glass approval, followed by re-issuing leaf certificates and rolling restarts.

4. Timeline

Key moments: detection T+00:00 → declared T+00:12 · first external comms T+00:17 · mitigation started T+00:50 · resolution not reached (logins 74.6% at T+01:15; queue drain and reconciliation still pending).

5. Results against the rubric

Hints used: 1 · Nudges given: 0

6. What went well

• Evidence-led diagnosis after the hint. Ana inspected the full chain at T+00:30 and had the expired intermediate identified by T+00:34.
• Fix prepared in parallel. The ClusterIssuer update and cmctl re-issue were ready before access arrived, and each restart wave was verified.
• Command and delegation. Sam phoned the approvers directly (T+00:41 to T+00:43), gave Leo the notify-svc backlog (T+00:25) and kept Marcus off the cluster while giving him CEO and Enterprise work (T+00:50).
• Honest, proactive communication. The T+00:22 update told customers logged-in users could keep working, Jo emailed Enterprise accounts at T+00:44 before being asked, and Jo gave the CTO a clear SLA and credits position.
• Candid reflection. The team named its own restart mistake and the "reacted before we'd looked" lesson.

7. What to improve

• Rollback before evidence (T+00:02 to T+00:10). The deploy was CSS-only, and nobody checked whether the errors were TLS-related. Eight minutes were spent, and nobody else was woken until the second page had fired.
• Restart approved without evidence (T+00:12). Login success fell from 41.5% to about 5%. No one asked what evidence supported it, and the effect of dropping pooled connections was not considered.
• First status update was vague (T+00:17). It gave no impact or workaround. The T+00:22 update corrected this.
• Status page cadence. At T+01:15 the page still said "next update 03:20" and was past that time.
• Break-glass single point of failure. Tom, the third approver, was unreachable, and there was no fallback approver or phone list.
• Emergency options not weighed. Disabling mTLS verification was never considered, so no owner or exception process was tested.

8. Improvement actions

9. Plan and documentation updates

• Add the "evidence before rollback/restart" checklist to the incident template (action 3).
• Publish a break-glass runbook with a fourth named approver, phone numbers and a 2 a.m. call order (action 2).
• Add status page templates and a 30-minute update reminder to the communications runbook (action 4).
• Add a service-credit decision guide for Enterprise SLAs. Owner: Marcus. Due: 23 Oct 2026.
• Book the post-incident review within five working days, as the incident process requires (by 7 Oct 2026). Owner: Sam.
• Record the emergency mTLS-off exception procedure in the security policy (action 7).

10. Evidence summary

This exercise tested availability incident response and recovery procedures, including a scenario in which a key person was unavailable. It is intended to support evidence for SOC 2 Trust Services Criteria CC7.4 and CC7.5 and, where Availability is in scope, A1.3 (testing of recovery plan procedures). It also tests the documented incident process: declaration, roles, status-page cadence and post-incident review. Whether it satisfies a specific requirement is a matter for the organisation and its auditor.

Evidence attached / retained:

• This report (signed)
• Attendance record
• Exercise log / transcript
• Improvement actions entered in the tracker
• Updated plan or runbooks (if any)

11. Sign-off

Report prepared with the Calmdrill Game Master (AI-facilitated). Reviewed and approved by: ________________ Role: ________________ Date: ________