Data pipeline orchestration at scale

A nightly ETL has a few jobs that a hand-rolled script does badly. It should extract independent sources at the same time, not one after another. It should refuse to load a dataset that failed a quality check, not load garbage and alert later. It should page someone when the load breaks. And no matter how it ends, it should drop the staging tables — because the one thing worse than a failed pipeline is a failed pipeline that leaves half a terabyte of scratch data behind.

Scripts get maybe two of those four right. This post builds all four as one OrchStep workflow: a parallel: extract, a quality gate per dataset pulled from a reusable module, a catch: that alerts, and a finally: that guarantees cleanup.

The pipeline

The shape is fan-out, gate, load. Extraction fans out across three sources; each dataset is gated by a shared quality module; the load alerts on failure and always cleans up.

name: warehouse
# Extract three sources in parallel, gate each on quality via a shared module,
# then load — with a cleanup that always runs, even when a source fails.
defaults:
  run_date: "2026-04-23"

modules:
  - name: quality
    source: "./modules/quality"

tasks:
  # `orchstep run etl --var run_date=2026-04-23`
  etl:
    steps:
      # Fan-out: every source extracts at the same time.
      - name: extract
        parallel:
          - name: orders
            func: shell
            do: echo "extracting orders for {{ vars.run_date }}"
          - name: events
            func: shell
            do: echo "extracting events for {{ vars.run_date }}"
          - name: billing
            func: shell
            do: echo "extracting billing for {{ vars.run_date }}"

      # Quality gates, reused from the module — one call per dataset.
      - name: gate_orders
        module: quality
        task: check
        with:
          dataset: orders
          min_rows: "100"

      - name: gate_events
        module: quality
        task: check
        with:
          dataset: events
          min_rows: "500"

      - name: load
        func: shell
        do: echo "loading vetted datasets into the {{ vars.run_date }} partition"
        catch:
          - name: alert
            func: shell
            do: echo "load failed for {{ vars.run_date }} — paging data on-call"
        finally:
          - name: cleanup
            func: shell
            do: echo "dropping staging tables for {{ vars.run_date }}"

Every step is echo-only, so the whole pipeline — fan-out, module gates, and cleanup — runs anywhere. In production, the extract branches shell out to your warehouse's load tools and the module's rows step runs a real SELECT count(*).

The four things scripts get wrong

Parallel extract. The three sources don't depend on each other, so they shouldn't wait in line. The parallel: block runs orders, events, and billing concurrently — each is a full step that can carry its own retry: or timeout:. Wall-clock time drops to the slowest source instead of the sum.

A quality gate you don't rewrite per dataset. The quality module is imported once and called per dataset with different with: values — min_rows: "100" for orders, "500" for events. The check logic lives in one place; when you add a freshness assertion, every gate gets it. That's the difference between a module and a copy-pasted function.

Alerting that's part of the run. The load step's catch: fires only on failure and pages on-call. It's not a separate monitoring job that notices five minutes late — it's the failure path of the step itself.

Cleanup that always runs. The finally: drops the staging tables whether the load succeeded, failed, or a gate rejected a dataset upstream. Guaranteed cleanup is the whole reason finally: exists: the resource leak that a crashed script leaves behind simply can't happen here.

Preview the whole pipeline first

A dry run resolves the run date, expands the parallel: block, and descends into each module call — printing the full plan without extracting a row:

orchstep run etl --var run_date=2026-04-23 --dry-run

You see the fan-out, both gates with their thresholds, and the load's catch/finally in one plan. More: Previewing with Dry Run.

What you gained

If your pipeline is one extract and one load, a script is honest. Once it's many sources with per-dataset rules, the parallel/module/finally structure is what keeps it from rotting.

Where to go next

• Modules — packaging reusable tasks like the quality gate
• Error Handling — catch:, finally:, retry:, timeouts
• Variables & Outputs — passing with: args into a module

Got a nightly ETL that runs sequentially and leaks staging tables when it dies? Fan it out, gate it with a module, and let finally: guarantee the cleanup.