Shard your test suite for speed

The test job that used to take four minutes now takes twenty-two. Nobody added twenty-two minutes of tests on purpose — it crept. And a twenty-two-minute job is a twenty-two-minute feedback loop, which means people stop waiting for it, start merging on faith, and the suite quietly stops mattering.

The fix is old and boring: split the suite into shards and run them at the same time. The annoying part has always been the bookkeeping — generating the shard list, fanning out the runs, collecting every result back, and refusing to call it green unless all of them reported. That's the part that ends up as a brittle matrix block in YAML you can't run locally.

Here's the same idea as an OrchStep workflow: one loop, one aggregate, one gate.

The workflow

name: ci
# Split the suite into N shards, run each, then gate on a full set of results.
defaults:
  shards: [1, 2, 3, 4]
  total: 4

tasks:
  # `orchstep run test`
  test:
    steps:
      - name: shard
        loop:
          items: "{{ vars.shards }}"
          as: n
        func: shell
        do: echo "running shard {{ loop.n }}/{{ vars.total }}"
        outputs:
          result: "shard {{ loop.n }} ok"

      - name: aggregate
        func: shell
        do: echo "collected {{ len steps.shard.outputs }} shard results"

      - name: gate
        func: assert
        args:
          condition: '{{ eq (len steps.shard.outputs) 4 }}'
          message: "every shard must report before the suite passes"

The shard step loops over the shard list. In a real suite the do: line is your runner with a shard flag — go test -shard, pytest --splits, or jest --shard — pointed at the current shard number and the total. Each iteration records an entry in outputs, so after the loop steps.shard.outputs is an array with one result per shard.

The part that matters: the gate

A sharded suite has a failure mode plain parallelism doesn't: a shard that never ran looks exactly like a shard that passed. If shard 3 silently dies and you only check the shards that came back, you ship with a quarter of your tests skipped.

That's what the assert step is for. It refuses to pass unless the number of collected results equals the number of shards you asked for:

- name: gate
  func: assert
  args:
    condition: '{{ eq (len steps.shard.outputs) 4 }}'
    message: "every shard must report before the suite passes"

steps.shard.outputs collects one entry per loop iteration, so len is your "did everyone show up?" count. No full set, no green.

Tune it without editing the file

The shard count is a variable, so dialing parallelism up or down is a CLI flag, not a diff:

orchstep run test --var total=8 --var shards='[1,2,3,4,5,6,7,8]'

And before you wire it into CI, see the fan-out without running a thing:

orchstep run test --dry-run

The dry-run resolves the loop and prints one planned shard step per item, so you can confirm you're getting eight shards and not one before you spend a runner on it. Full tour: Previewing with Dry Run.

What you gained

This won't make your tests faster — it makes them finish faster by running them at once, and it makes "all shards reported" a thing the workflow enforces instead of a thing you hope for.

Where to go next

• Loops — items, range, and per-iteration outputs
• Variables & Outputs — the four-level scoping model
• Functions: assert — gate a step on a condition

Install is one line — curl -fsSL https://orchstep.dev/install.sh | sh — then orchstep run test --dry-run on the demo above to watch the shards fan out.