Quarantine flaky tests automatically

Everyone knows which tests are flaky. There's a test that touches a webhook and fails one run in twenty. There's a search test that breaks whenever the index rebuilds mid-suite. They're not bugs in the product — they're bugs in the tests — but they fail the build, and the build is what gates the merge.

So the team learns the worst habit in CI: hit re-run on red. Once "red sometimes means nothing" is the culture, red always means nothing, and a real regression sails through because the third re-run happened to pass.

The honest move is to treat flaky tests as a known set: retry them a couple of times, and if they still fail, route them to a quarantine list and let the pipeline go green — loudly, with a record, not silently. Here's that as a workflow.

The workflow

name: ci
# Retry known-flaky tests; on exhaustion, record them to a quarantine list
# instead of failing the whole pipeline.
defaults:
  quarantine:
    - "TestCheckout_Webhook"
    - "TestSearch_Reindex"

tasks:
  # `orchstep run test`
  test:
    steps:
      - name: stable
        func: shell
        do: echo "running the stable suite"

      - name: flaky
        loop:
          items: "{{ vars.quarantine }}"
          as: test
        func: shell
        do: echo "retrying known-flaky {{ loop.test }}"
        retry:
          max_attempts: 3
          interval: "1s"
          backoff_rate: 2.0
        catch:
          - name: record
            func: shell
            do: echo "QUARANTINED {{ loop.test }} -> reports/quarantine.txt"

      - name: report
        func: shell
        do: echo "quarantine list holds {{ len vars.quarantine }} test(s)"

Two things are doing the work here, and they're both syntax, not scripting.

Retry is a field, not a loop

The flaky step runs each known-flaky test (the do: line would be your runner pointed at a single test by name, like go test -run) and retries it up to three times with backoff:

retry:
  max_attempts: 3
  interval: "1s"
  backoff_rate: 2.0

That's the entire "give it another shot" logic. No counter variable, no until loop, no sleep $((2**n)). A test that's flaky one run in twenty almost never fails three attempts in a row, so most flakes clear here and you never hear about them.

Catch turns a hard failure into a record

The interesting case is the test that fails all three attempts. Without a plan, that fails the build and you're back to re-run culture. The catch: block changes the outcome: when retries are exhausted, it records the test instead of letting the step blow up.

catch:
  - name: record
    func: shell
    do: echo "QUARANTINED {{ loop.test }} -> reports/quarantine.txt"

In a real pipeline that line appends to a file, opens an issue, or posts to a channel — the point is the test that couldn't pass three times in a row becomes a tracked item, not a blocked merge. The pipeline stays green, and you have a list of tests that owe someone an explanation.

A word of honesty: quarantine is a holding cell, not a graveyard. If a test lives on that list for a month, it's not flaky — it's broken, or it's testing something that no longer matters. The list exists so flakes are visible and assigned, not so they're ignored with extra steps.

What you gained

Where to go next

• Error Handling — retry, catch, finally, and timeouts
• Loops — iterate a known set with per-iteration outputs
• Functions: shell — run your existing test commands

Install with curl -fsSL https://orchstep.dev/install.sh | sh, then orchstep run test on the demo to watch a flake get retried and quarantined.