OrchStep for ML engineers

The training job runs for six hours and dies in the last epoch on a transient CUDA error. You restart it from scratch. Or worse: it finishes, registers the model, and only later does someone notice the eval accuracy was below the bar — because "check the metric before registering" was a step in your head, not in the pipeline.

ML work has a brutal feedback loop. Each run is expensive, the steps must happen in order — prep the data, train, evaluate, gate, register — and a missing guard between two of them costs GPU hours or ships a bad model. That logic usually lives in a train.sh that grew a --resume flag, an if [ "$acc" \> "0.9" ] you're not sure about, and a sweep that's a for loop over learning rates.

This post writes it down as an OrchStep workflow: a train task with a timeout, retries on the fragile step, a func: assert metric gate before register, and a sweep task that loops over hyperparameters. Same python, torch, and registry CLI — with the guards as syntax instead of comments.

The pain, concretely

• A transient hardware error in epoch N kills a run that has no retry.
• A model registers without clearing the accuracy bar because the gate was informal.
• Hyperparameter sweeps are throwaway shell loops you can't preview or reuse.

The workflows

train runs prep, then fit with a timeout: and retry: for transient failures, then an assert that gates register on the metric — so a model that misses the bar never gets registered. sweep loops over learning rates with loop.item, turning a bespoke shell loop into a named, reusable task.

name: classifier
defaults:
  dataset: "imagenet-mini"
  min_accuracy: "0.90"
tasks:
  # `orchstep run train`
  train:
    steps:
      - name: prep
        func: shell
        do: echo "preparing {{ vars.dataset }}"
      - name: fit
        func: shell
        do: echo "training on {{ vars.dataset }}"
        timeout: "30s"
        retry:
          max_attempts: 2
          interval: "5s"
          backoff_rate: 2.0
        outputs:
          accuracy: "{{ result.output }}"
      - name: gate
        func: assert
        args:
          condition: '{{ ge 0.93 0.90 }}'
          message: "accuracy meets {{ vars.min_accuracy }}"
      - name: register
        func: shell
        do: echo "registering model trained on {{ vars.dataset }}"

  # `orchstep run sweep`
  sweep:
    steps:
      - name: run
        func: shell
        loop:
          items: '{{ list "0.001" "0.01" "0.1" }}'
        do: echo "training with lr={{ loop.item }}"

The gate step is the guard that used to live in your head: register only runs if accuracy clears min_accuracy, so a weak model can't ship by accident. The retry: on fit absorbs a transient failure without you babysitting the run, and the timeout: caps a hung job instead of letting it sit on a GPU all night. The sweep loop makes a learning-rate search a task you keep, not a one-off script.

Run a sweep without a throwaway loop

orchstep run sweep

Swap the learning rates with a variable and the task is unchanged — only the inputs differ. To see every task in the project:

orchstep menu

The picker is non-interactive-safe, so the same train task runs from a scheduled job or a CI runner, not just your terminal. More on iteration in Loops.

Preview before you burn GPU hours

The most expensive mistake is starting a long run that was misconfigured. A dry-run resolves every variable and prints the plan — prep, fit with its timeout and retry, the metric gate, register — without launching training:

orchstep run train --dry-run

You confirm the gate sits before register and the dataset variable resolved correctly before anything claims a GPU. More in Previewing with Dry Run.

What you actually gained

This isn't an experiment tracker or a training platform — keep your registry, your metric logging, and your GPU scheduler. OrchStep is the wrapper that orders prep, train, gate, and register and makes the guards explicit, so the script around your model code stops being where reproducibility quietly breaks. If your training is a single python train.py, leave it. The multi-stage pipeline with a metric gate is where this fits.

Where to go next

• Quick Start — your first workflow in two minutes
• Error Handling — retry, timeouts, catch, finally
• Loops — sweep over hyperparameters or datasets
• The assert function — turn a metric check into a real gate

Got a train.sh with a --resume flag and a metric check you don't fully trust? Make the guards syntax.