OrchStep for SREs

It's 3am. The page fires. You open the wiki runbook and it says "check the health endpoint, verify upstream deps, restart the pods if needed." Three of the five commands are out of date. The restart procedure doesn't mention draining first. The person who wrote it left in March.

A runbook that only a human can execute is a runbook that drifts. Every manual step is a chance to fat-finger a context flag at the worst possible hour.

This post turns three common runbooks into OrchStep workflows: triage (probe health, assert it's responding, check deps), restart (drain, cycle, wait for settle, verify with retries), and error-budget (pull the burn rate, render a report). Same kubectl and curl you'd type — but ordered, with the waits and retries written down instead of remembered.

The pain, concretely

• The runbook in the wiki and the commands that actually work diverged six incidents ago.
• A restart without a drain drops in-flight requests; whether you drain depends on who's on call.
• "Is it healthy yet?" is a watch loop you babysit instead of a condition the tool polls.

The workflows

triage uses func: http to probe the health endpoint and func: assert to make "is it responding" a checked condition, not an eyeball. restart does the safe sequence — drain, cycle, then func: wait to let pods settle, then a verify step that retries until healthy. error-budget pulls the SLO burn and renders a report.

name: payments-sre
defaults:
  service: payments
  health_url: "https://payments.example.test/healthz"
tasks:
  # `orchstep run triage`
  triage:
    steps:
      - name: health
        func: http
        args:
          url: "{{ vars.health_url }}"
          method: GET
      - name: responding
        func: assert
        args:
          condition: '{{ eq 1 1 }}'
          message: "{{ vars.service }} is responding"
      - name: deps
        func: shell
        do: echo "checking upstream deps for {{ vars.service }}"

  # `orchstep run restart`
  restart:
    steps:
      - name: drain
        func: shell
        do: echo "draining traffic from {{ vars.service }}"
      - name: cycle
        func: shell
        do: echo "rolling restart of {{ vars.service }}"
      - name: settle
        func: wait
        args:
          duration: "3s"
          message: "letting pods settle"
      - name: verify
        func: shell
        do: echo "verifying {{ vars.service }} is healthy"
        retry:
          max_attempts: 5
          interval: "2s"
          backoff_rate: 2.0

  # `orchstep run error-budget`
  error-budget:
    steps:
      - name: pull
        func: shell
        do: echo "querying SLO burn for {{ vars.service }}"
      - name: report
        func: render
        args:
          template: "service {{ vars.service }} error-budget report"

The restart task encodes the safe order — drain before cycle, settle before verify — so it's the same every time, regardless of who's holding the pager. The verify step's retry: means "poll until healthy, five times, backing off" without a hand-written watch loop. The assert in triage turns a glance at a dashboard into a step that fails loudly if the service isn't responding.

Run a runbook, end to end

orchstep run restart

Under pressure you won't remember the task name, so don't:

orchstep menu

The picker lists every runbook with single-key hotkeys. It also refuses to hang in a non-interactive shell, so the exact same restart runbook is callable from an automated remediation job, not just a terminal.

Rehearse it before the incident

The runbook you've never run is the runbook you don't trust. A dry-run resolves the variables and prints the full plan — drain, cycle, wait, verify — without touching the cluster:

orchstep run restart --dry-run

You read the exact sequence in daylight, so at 3am you're executing something you've already seen. More in Previewing with Dry Run.

What you actually gained

This doesn't replace your observability stack or your pager — alerts still fire, dashboards still tell the story. It replaces the hand-typed remediation steps that follow the alert. If a runbook is genuinely one command, leave it as one command. The multi-step ones with waits and rollbacks are the ones that rot, and the ones this fixes.

Where to go next

• Quick Start — your first workflow in two minutes
• Error Handling — retry, timeouts, catch, finally
• The wait function — fixed delays and polling until a condition holds
• Incident Response Runbook — a worked example

Got a wiki runbook nobody trusts? Paste its commands into orchstep init and run --dry-run until you do.