OrchStep for data engineers

The extract job times out against a flaky source, so the transform runs on nothing, and the load writes an empty partition over yesterday's good data. You find out from a dashboard the next morning. Now you're hand-writing a backfill: a for loop over date partitions in a shell script you'll throw away after, hoping you got the boundaries right.

A lot of data work is this — ordering steps that must not run out of order, retrying the ones that flake, and quarantining the batches that fail a quality check instead of loading them. Cron gives you a trigger and nothing else. The dependency logic ends up smeared across a shell script, a Makefile, and a half-remembered runbook.

This post writes that logic down as an OrchStep workflow: an etl task with a retrying extract, a quality gate, and a catch: that quarantines bad batches, plus a backfill task that loops over partitions. Same aws s3, dbt, and SQL you already run — with the ordering and recovery as syntax.

The pain, concretely

• A flaky extract poisons the whole run because transform and load don't know it failed.
• A bad batch loads anyway because the quality check is a step you forgot to add.
• Backfills are throwaway shell loops you rewrite every time, with off-by-one date bugs.

The workflows

etl runs extract (with retries), transform, a func: assert quality gate, then load — and a catch: on load quarantines instead of corrupting the table. backfill loops over partition keys with loop.item, so reprocessing a date range is one task, not a bespoke script.

name: events-pipeline
defaults:
  source: "s3://acme-raw/events"
  warehouse: "analytics.events"
tasks:
  # `orchstep run etl`
  etl:
    steps:
      - name: extract
        func: shell
        do: echo "extracting from {{ vars.source }}"
        retry:
          max_attempts: 3
          interval: "5s"
          backoff_rate: 2.0
        outputs:
          rows: "{{ result.output }}"
      - name: transform
        func: shell
        do: echo "transforming batch from {{ vars.source }}"
      - name: quality
        func: assert
        args:
          condition: '{{ gt 1 0 }}'
          message: "batch is non-empty"
      - name: load
        func: shell
        do: echo "loading into {{ vars.warehouse }}"
        catch:
          - name: quarantine
            func: shell
            do: echo "moving bad batch to quarantine"

  # `orchstep run backfill`
  backfill:
    steps:
      - name: partition
        func: shell
        loop:
          items: '{{ list "2026-01-01" "2026-01-02" "2026-01-03" }}'
        do: echo "reprocessing partition {{ loop.item }}"

The retry: on extract means a flaky source gets three attempts with backoff before the run fails — instead of a single timeout poisoning everything downstream. The quality assert stops a bad batch from reaching load; if load itself fails, the catch: routes the batch to quarantine rather than leaving the table half-written. The backfill loop replaces the throwaway date-range script with one named task you keep.

Run a backfill without writing a script

orchstep run backfill

Need a different range? Swap the partition list with --var, or drive it from a variable. The task is the same; only the input changes. And to see every pipeline task at a glance:

orchstep menu

Because the picker is non-interactive-safe, the same etl task is callable from your scheduler — cron still pulls the trigger, OrchStep owns the steps. More on loops in Loops.

Preview before you touch the warehouse

Loading the wrong thing is expensive to undo. A dry-run resolves variables, expands the loop, and prints the plan — extract, transform, quality gate, load, quarantine path — without running a query:

orchstep run etl --dry-run

You confirm the quality gate sits before the load, and the quarantine path exists, before any data moves. More in Previewing with Dry Run.

What you actually gained

This isn't an orchestrator competing with Airflow or Dagster for DAG scheduling — keep those for cross-job scheduling, lineage, and the calendar. OrchStep is the body of a single job: the ordered, retrying, recoverable steps that today live in a shell script the scheduler shells out to. If your transform is one idempotent command, leave it. The multi-stage extract-transform-load with recovery is where this earns its place.

Where to go next

• Quick Start — your first workflow in two minutes
• Loops — iterate over partitions, files, or shards
• Error Handling — retry, catch, finally, timeouts
• Modular Data Pipeline — a worked pipeline example

Got a backfill script you rewrite every quarter? Make it a backfill task and stop rewriting it.