Agent runbooks for on-call

It's 3am. The pager goes off: 5xx rate on checkout is climbing. You ask your coding agent to dig in, and it does a decent job — pulls the error rate, checks pod saturation, notices the deploy from 14 minutes ago, and walks you to a rolling restart that clears it. You go back to bed.

Three weeks later the same alert fires. The fix is somewhere in a chat transcript you can't search, half-remembered by whoever was on-call last time. So the agent re-derives it from scratch, slower, while the error budget burns.

The triage was good work. The problem is it evaporated. A runbook is the durable form of that work — and OrchStep gives you a runbook you can actually run, not a wiki page that rots.

From transcript to runbook

The steps the agent took are already a workflow: check signals, decide, remediate, verify. Write them down as one, and the next incident starts from "run the runbook" instead of "re-investigate."

name: oncall
defaults:
  service: checkout
  target: production

tasks:
  # `orchstep run triage` — the signals the agent pulled, in order
  triage:
    steps:
      - name: error-rate
        func: shell
        do: echo "checking 5xx rate for {{ vars.service }} in {{ vars.target }}"
        outputs:
          rate: '{{ result.output | regexFind "[0-9]+" }}'
      - name: saturation
        func: shell
        do: echo "checking pod saturation for {{ vars.service }}"
      - name: recent-deploy
        func: shell
        do: 'echo "last deploy of {{ vars.service }} was 14m ago"'

  # `orchstep run remediate --var service=checkout`
  remediate:
    steps:
      - name: triage
        task: triage              # reuse the triage steps, don't re-type them
      - name: restart
        func: shell
        do: echo "rolling restart of {{ vars.service }} in {{ vars.target }}"
        retry:
          max_attempts: 3
          interval: "2s"
          backoff_rate: 2.0
      - name: verify
        func: assert
        args:
          condition: '{{ eq vars.target "production" }}'
          message: "remediation only runs against a real target"

The service is a variable, so the same runbook covers checkout today and api next week with --var service=api. The triage task is called as a step from remediate — gather signals first, every time, with no copy-paste. The retry handles the registry or control-plane flaking that always seems to happen mid-incident.

Let the agent capture it

You don't have to transcribe the session by hand. After the agent resolves the incident, the orchstep-capture skill reads the session history — the shell commands, the HTTP checks, the git operations — drops the noise (ls, cd, the failed retries), and writes a clean workflow with the hardcoded values pulled out into vars.* placeholders. The 3am investigation becomes oncall.yml before you've closed the incident channel.

Replay beats remember

Next time the alert fires, the on-call engineer — or the agent acting on their behalf — runs one command:

orchstep run remediate --var service=checkout

And before anything executes, they can see exactly what's about to happen:

orchstep run remediate --var service=checkout --dry-run

The dry-run resolves the variables and prints the plan without touching production — so a half-awake human (or an agent you're not 100% sure about yet) can sanity-check the fix before it runs. Full tour: Previewing with Dry Run.

What you actually gained

This isn't about replacing judgment during an incident — the agent's triage is still where the thinking happens. It's about making sure the second time you see a failure mode, you're not paying full price for it again.

Where to go next

• LLM Agent Integration — how agents author, run, and capture workflows
• Skills — the capture skill that turns a session into a runbook
• Previewing with Dry Run — see the plan before it runs
• Error Handling — retry, catch, finally, timeouts

Got an incident your agent just cleared? Capture it now, while the steps are still on screen — orchstep init gives you the shape to drop them into.