Backups and restore drills you can trust

Everyone has backups. Far fewer have ever restored one. The backup cron job has been "succeeding" for two years, which means it has been writing something to a bucket for two years — whether that something is a restorable database is a question nobody asks until the night the primary dies.

The gap is that backing up and restoring are treated as different projects. The backup runs nightly; the restore is a runbook someone wrote once and hopes never to use. So the backup never proves itself, and the restore path quietly bit-rots.

This post puts both in one OrchStep workflow: a backup that verifies its own integrity and prunes old copies, and a restore drill you run on a schedule to prove the backups actually come back — with a catch: alert on either path so failure is loud, not silent.

Backup that checks itself

The backup task dumps, uploads with a retry, verifies the archive integrity, and prunes to a retention window. The upload step has a catch: so a failed push pages you instead of vanishing into a log:

name: backups
defaults:
  store: backups/db
  keep: "14"

tasks:
  # `orchstep run backup`
  backup:
    steps:
      - name: dump
        func: shell
        do: echo "pg_dump | gzip > db.sql.gz"
      - name: upload
        func: shell
        do: echo "upload db.sql.gz to {{ vars.store }}"
        retry:
          max_attempts: 3
          interval: "2s"
          backoff_rate: 2.0
        catch:
          - name: alert
            func: shell
            do: echo "ALERT - backup upload to {{ vars.store }} failed"
      - name: verify
        func: shell
        do: echo "gunzip -t db.sql.gz (integrity check)"
      - name: prune
        func: shell
        do: echo "keep the {{ vars.keep }} most recent backups"

  # `orchstep run restore-drill` — prove the backups actually restore.
  restore-drill:
    steps:
      - name: fetch
        func: shell
        do: echo "fetch newest backup from {{ vars.store }}"
        outputs:
          rows: "4211"
      - name: restore
        func: shell
        do: echo "restore into a scratch database"
        catch:
          - name: alert
            func: shell
            do: echo "ALERT - restore drill failed, backups are not trustworthy"
      - name: check
        func: assert
        args:
          condition: '{{ gt (atoi steps.fetch.rows) 0 }}'
          message: "restored database should not be empty"
      - name: done
        func: shell
        do: echo "restore drill passed with {{ steps.fetch.rows }} rows"

The verify step is what makes the backup more than a guess — gunzip -t proves the archive isn't truncated garbage before you ever depend on it. prune enforces retention in the same task that wrote the new copy, so the bucket doesn't grow forever.

The restore drill is the real test

A backup is a hypothesis. The restore-drill task is the experiment: fetch the newest backup, restore it into a throwaway database, and assert it isn't empty. The assert step turns "looks fine" into a pass/fail the run enforces:

      - name: check
        func: assert
        args:
          condition: '{{ gt (atoi steps.fetch.rows) 0 }}'
          message: "restored database should not be empty"

If the restore produces an empty database, the assert fails the run and the catch: on the restore step fires an alert. You find out the backups are worthless during a drill, on a Tuesday afternoon — not during an outage at 2am.

Run it

orchstep run backup
orchstep run restore-drill

The demo steps only echo, so you can watch the whole flow safely; swap them for pg_dump, your object-store CLI, and a scratch-database restore to make it real. Point the same restore-drill task at a systemd timer and the rehearsal happens weekly without anyone remembering to do it.

What you actually gained

OrchStep doesn't replace pg_dump or your storage — it makes the guarantee testable. A backup that verifies itself and a restore you rehearse on a schedule is the difference between "we have backups" and "we can recover." If you already run regular restore drills and trust them, keep doing exactly that. If your last successful restore was never, start here.

Where to go next

• Error Handling — retry, catch, finally, timeouts
• The assert function — turn "looks fine" into a pass/fail gate
• Quick Start — your first workflow in two minutes

When did you last restore a backup? Make the drill a task and let the schedule answer that question for you.