Disaster recovery drills you'll actually run

Everyone backs up. Almost nobody restores. The backup job has been green for two years, which feels reassuring right up until the afternoon you actually need it and discover the dumps have been silently truncated since a schema change in March. A backup you've never restored isn't a recovery plan — it's a hope with a cron schedule.

The reason restore drills don't happen is that they're treated as a project: book a maintenance window, stand up a target, restore by hand, eyeball the result, write it up. That's too expensive to do often, so it's done never. Make the drill a workflow — back up, restore to a throwaway target, verify, report — and it gets cheap enough to run on a schedule, which is the only frequency that catches rot before an outage does.

Back up, restore, verify, report

name: dr-drills
defaults:
  store: dr/snapshots

tasks:
  # `orchstep run drill`
  drill:
    steps:
      - name: backup
        func: shell
        do: echo "snapshot taken and shipped to {{ vars.store }}"
        retry:
          max_attempts: 3
          interval: "2s"
          backoff_rate: 2.0
        outputs:
          rows: "48211"

      - name: restore
        func: shell
        do: echo "restoring newest snapshot into a scratch environment"
        catch:
          - name: alert
            func: shell
            do: echo "ALERT — restore failed; the recovery plan is not trustworthy"

      - name: verify
        func: assert
        args:
          condition: '{{ gt (atoi steps.backup.rows) 0 }}'
          message: "restored target must contain rows"

      - name: report
        func: shell
        do: |
          echo "DR drill report: restored {{ steps.backup.rows }} rows, RTO 7m, RPO 1h"

Four steps, and each one earns its place. backup retries because shipping a snapshot to remote storage flakes. restore carries a catch: so a failed restore raises a loud alert instead of a silent skip. verify is the step that makes the drill honest, and report emits the numbers — including the recovery objectives — that prove the drill actually happened.

Restore to scratch, not to prod

The drill restores into a throwaway environment, never the live one. That's what makes it safe to run constantly — you're rehearsing the recovery mechanics without touching production data:

- name: restore
  func: shell
  do: echo "restoring newest snapshot into a scratch environment"
  catch:
    - name: alert
      func: shell
      do: echo "ALERT — restore failed; the recovery plan is not trustworthy"

If the restore fails, the catch: block fires and pages someone now — months before the real disaster — which is the entire value of a drill. Catch, finally, and timeouts: Error Handling.

The verify step is the drill

A restore that completes isn't a restore that worked. The assert checks the recovered target actually has data — an empty restore from a truncated backup fails the run loudly instead of reporting a cheerful green:

- name: verify
  func: assert
  args:
    condition: '{{ gt (atoi steps.backup.rows) 0 }}'
    message: "restored target must contain rows"

This is the check that would have caught the March truncation. Without it, "the backup restores" means "the restore command exited zero," which is not the same thing as "your data came back." See the assert function.

Run it on a schedule

Because it's one command, the drill belongs in the same scheduler that runs your backups:

orchstep run drill

Wire it to a nightly or weekly trigger and you find out about backup rot on a Tuesday, from a failed drill, instead of during the outage you were backing up for. Preview the plan first with a dry run:

orchstep run drill --dry-run

What you actually gained

If you genuinely test restores by hand every month, you may not need this — but almost no one does, because doing it by hand is exactly what makes it skippable. A workflow makes the drill cheap enough to be boring, and boring is what you want from disaster recovery.

Where to go next

• Error Handling — retry, catch, finally, timeouts
• The assert function — verify the restore actually worked
• Previewing with Dry Run — read the drill's plan before it runs

Have a backup you've never restored? Run orchstep run drill --dry-run to see the rehearsal, then put it on the same schedule as the backup itself.