Incident response runbooks that actually run

It's 3 a.m. The pager goes off. You open the runbook — a wiki page with eleven manual steps, two of which reference a dashboard that was renamed last quarter. You start copying commands into a terminal, one at a time, while the error rate climbs. Half the steps are "check X" and you're checking them sequentially because the doc is a list, not a program.

The diagnostics in a runbook don't depend on each other. There's no reason to check the API, then the database, then the queue one after another — you check them to build a picture, and you want the whole picture now. A runbook should fan those out at once, decide what severity you're actually in, and route accordingly — and it should leave a record without you typing one.

The runbook as a workflow

Classify the severity, run every diagnostic in parallel, branch on severity, then write the incident record from what the diagnostics found.

name: incident-runbooks
defaults:
  signal: "error_rate"
  sev: "2"

tasks:
  # `orchstep run respond --var sev=1`
  respond:
    steps:
      - name: classify
        func: shell
        do: echo "incident from {{ vars.signal }} classified as SEV{{ vars.sev }}"
        outputs:
          level: "{{ vars.sev }}"

      - name: diagnostics
        desc: "Pull signals from every subsystem at once"
        parallel:
          - name: check_api
            func: shell
            do: echo "API_ERROR_RATE=high"
            outputs:
              status: '{{ result.output | regexFind "API_ERROR_RATE=(.+)" }}'
          - name: check_db
            func: shell
            do: echo "DB_CONNECTIONS=saturated"
            outputs:
              status: '{{ result.output | regexFind "DB_CONNECTIONS=(.+)" }}'
          - name: check_queue
            func: shell
            do: echo "QUEUE_DEPTH=growing"
            outputs:
              status: '{{ result.output | regexFind "QUEUE_DEPTH=(.+)" }}'

      - name: route
        if: '{{ le (atoi vars.sev) 1 }}'
        then:
          - name: page
            func: shell
            do: echo "SEV1 — paging on-call + incident commander, opening a bridge"
        else:
          - name: ticket
            func: shell
            do: echo "SEV{{ vars.sev }} — filed a ticket for business-hours follow-up"

      - name: record
        func: shell
        do: |
          echo "incident record: sev={{ steps.classify.level }} api={{ steps.check_api.status }} db={{ steps.check_db.status }} queue={{ steps.check_queue.status }}"

The parallel: block runs all three checks simultaneously, and each one captures a named output via regexFind. The record step then reads each steps.check_api.status output and friends — so the incident record is built from what the diagnostics actually saw, not from what you remembered to type.

Parallel diagnostics, not a sequential checklist

This is the part a wiki can't do. Three checks, one wall-clock cost:

- name: diagnostics
  parallel:
    - name: check_api
      func: shell
      do: echo "API_ERROR_RATE=high"
    - name: check_db
      func: shell
      do: echo "DB_CONNECTIONS=saturated"
    - name: check_queue
      func: shell
      do: echo "QUEUE_DEPTH=growing"

When you're triaging, the difference between "check three things sequentially" and "check three things at once" is the difference between a clear picture in two seconds and a partial picture you're still assembling while the incident grows. More: Parallel & Composition.

Severity routing, not a guess

A SEV1 and a SEV3 are not the same call. The route step makes the branch explicit — page a human and open a bridge for the severe ones, file a ticket for the rest:

- name: route
  if: '{{ le (atoi vars.sev) 1 }}'
  then:
    - name: page
      func: shell
      do: echo "SEV1 — paging on-call + incident commander, opening a bridge"
  else:
    - name: ticket
      func: shell
      do: echo "SEV2+ — filed a ticket for business-hours follow-up"

The on-call person doesn't have to decide whether this one is bad enough to wake someone up — the runbook encodes the policy your team already agreed on, calmly, in advance. Branching details: Conditionals.

Rehearse it on a Tuesday afternoon

The reason this matters: you can run the whole thing when nothing is on fire.

orchstep run respond --var sev=1 --dry-run

A dry run shows the SEV1 branch taking the paging path before you ever rely on it at 3 a.m. That's how a runbook earns trust — you read its resolved plan in daylight, not while panicking. See Previewing with Dry Run.

What you actually gained

A workflow won't replace human judgment in a genuinely novel outage — and it shouldn't. But the 90% of incidents that look like incidents you've seen before deserve a runbook that runs, not one you read.

Where to go next

• Parallel & Composition — run independent work concurrently
• Conditionals — if / elif / else routing
• Previewing with Dry Run — rehearse the runbook before you need it

Turn your gnarliest wiki runbook into a workflow and run orchstep run respond --dry-run — read the plan when you're calm, not when you're paged.