Rollback runbooks you can rehearse

Here is the uncomfortable truth about most rollback procedures: the first time anyone runs them for real is during the incident they were supposed to prevent from getting worse. The runbook is a wiki page. The commands are copy-pasted from memory. The on-call engineer is doing destructive operations on production, at 3am, under pressure, having never executed these exact steps before.

A rollback you've never run isn't a plan. It's a hope with a confidence problem.

The fix isn't a better wiki page. It's making the rollback a thing you can rehearse — run end to end, safely, on a calm Tuesday afternoon, and watch it resolve without touching production. This post writes the rollback as an OrchStep workflow you dry-run to rehearse and run for real when you need it, with the easy-to-forget parts handled by catch: and finally:.

The rollback as a workflow

Locate the last-known-good version, drain traffic, revert, verify. The revert step — the one that can fail — gets a catch: to escalate and a finally: to always notify:

name: rollback
# Rehearse first:  orchstep run rollback --var to=2.3.0 --dry-run
# Then for real:   orchstep run rollback --var to=2.3.0
defaults:
  service: api
  target: production
  to: "2.3.0"

tasks:
  rollback:
    steps:
      - name: locate
        func: shell
        do: echo "selecting last-known-good {{ vars.service }} {{ vars.to }} on {{ vars.target }}"

      - name: drain
        func: shell
        do: echo "draining traffic from {{ vars.service }} on {{ vars.target }}"

      - name: revert
        func: shell
        do: echo "reverting {{ vars.service }} to {{ vars.to }}"
        catch:
          - name: escalate
            func: shell
            do: echo "rollback failed — paging on-call for {{ vars.service }}"
        finally:
          - name: notify
            func: shell
            do: echo "rollback of {{ vars.service }} to {{ vars.to }} on {{ vars.target }} complete"

      - name: verify
        func: shell
        do: echo "verifying {{ vars.service }} health after rollback"

The two blocks that matter most are the ones nobody remembers to write at 3am:

• catch: is the "it went wrong" path. If the revert itself fails, you don't want a half-rolled-back service and silence. The catch: escalates and pages on-call automatically — the failure is loud, not lost.
• finally: is the "tell everyone" path. Whether the revert succeeds or the catch: fires, the finally: notifies. The classic incident sin — rolling back and forgetting to update the status channel — is structurally impossible here.

Swap the echos for your real kubectl rollout undo, traffic-shift, and PagerDuty calls and the shape is unchanged.

Rehearse it before you need it

This is the part that turns a hope into a plan. A dry-run walks the entire rollback — resolves the target version, the service, every step, the catch: and finally: paths — and executes nothing:

orchstep run rollback --var to=2.3.0 --dry-run

Run that in a team review. Everyone sees the exact order of operations, confirms the target version is right, and checks that the escalation and notification wording is what they'd want during an actual incident. You've now "run" the rollback as a group without any risk — which is the entire difference between a procedure people trust and one they pray works. Full tour: Previewing with Dry Run.

Run it for real

When the bad deploy lands, it's the same workflow minus the flag:

orchstep run rollback --var to=2.3.0

The on-call engineer isn't improvising. They're running the exact steps the team rehearsed last week, with escalation and notification wired in. The --var to= lets them target whichever known-good version applies without editing anything under pressure.

What you actually gained

Where OrchStep stops

OrchStep orders the rollback and guarantees the catch:/finally: paths; it does not decide when to roll back or replace your deployment tool. The echos are placeholders for your real revert and paging commands. If your platform offers one-click rollback you trust and rehearse, lean on it. This is for the rollbacks that are still a sequence of manual commands on a wiki page — the ones you'd really rather rehearse on a Tuesday than improvise at 3am.

Where to go next

• Error Handling — catch, finally, retries, and timeouts in depth
• Previewing with Dry Run — rehearse the whole runbook, execute nothing
• Variables & Outputs — targeting a version with --var

Runnable as-is — every step only echos. orchstep run rollback --var to=2.3.0 --dry-run is your rehearsal; drop the flag when it's real.