Testing agent-generated workflows

An agent will write you a workflow in seconds, and it'll look right. That's exactly the problem. "Looks right" is how a reference to a variable nothing supplies, or a step that reads an output an earlier step never produced, sails straight into a real environment.

You already know not to merge agent-written code without review. Agent-written workflows deserve the same treatment — and OrchStep gives you a test harness for them that runs in under a second and touches nothing: lint, validate, then --dry-run. Three stages, each catching a different class of mistake, all before execution.

The workflow an agent handed you

Here's a plausible nightly ETL the agent generated. It parses, it looks reasonable — but "reasonable" isn't "verified":

name: nightly-etl
defaults:
  source: events
  target: warehouse
  batch: "500"

tasks:
  etl:
    steps:
      - name: extract
        func: shell
        do: echo "extracting {{ vars.batch }} rows from {{ vars.source }}"
        outputs:
          count: '{{ result.output | regexFind "[0-9]+" }}'
      - name: transform
        func: shell
        do: echo "transforming {{ steps.extract.count }} rows"
      - name: load
        func: shell
        do: echo "loading into {{ vars.target }}"
        retry:
          max_attempts: 3
          interval: "5s"
          backoff_rate: 2.0
      - name: verify
        func: assert
        args:
          condition: '{{ ne steps.extract.count "" }}'
          message: "extract must return a row count"

Stage 1: lint — is it structurally sane?

First question: is this even a well-formed workflow, or did the agent invent a field that doesn't exist?

orchstep lint

Lint parses the file and flags anti-patterns — a duplicated variable, a malformed retry block, a step that's shaped wrong. It's the fastest gate and it catches the agent's structural hallucinations. Until it prints a clean result, nothing else is worth running. (Agents can call this themselves through the MCP server's lint tool before they ever hand you the file.)

Stage 2: validate — do the variables resolve?

The most common agent mistake isn't a syntax error — it's a vars.* reference that no layer supplies. It parses fine and renders to empty at runtime. validate catches it statically:

orchstep validate etl --strict

It walks every variable reference in the task and reports which ones are supplied and which resolve to nothing — across defaults, env files, and --var. With --strict, an unresolved reference fails the check instead of warning. This is the stage that catches "the agent referenced vars.region but never declared it." See Variables & Outputs.

Stage 3: dry-run — does the plan match intent?

Structure is sound and variables resolve. Last question: does the plan actually do what you asked the agent for? Resolve everything and execute nothing:

orchstep run etl --dry-run

The dry-run prints the fully resolved sequence — every rendered command, every branch the conditionals take. You read it the way you'd read a diff: does extract really feed transform, does the verify assert guard the right thing, is target pointed where you meant? This is where you catch logic the agent got plausibly-but-wrongly. Full tour: Previewing with Dry Run.

Wire it as the harness

The three stages compose into one gate you run on any agent output before it's allowed near a real run:

orchstep lint                      # structure
orchstep validate etl --strict     # variable resolution
orchstep run etl --dry-run         # resolved plan

Each stage is fast and side-effect-free, so there's no reason to skip them — and together they turn "the agent says it works" into "I checked, and here's what it'll do." That's the difference between trusting output and verifying it.

Where OrchStep stops

This harness verifies the workflow — its structure, its variables, its plan. It doesn't verify that the commands inside your shell steps are the right commands, the same way a linter won't tell you your business logic is wrong. For that, you still review what the steps actually do. But you'll be reviewing a workflow that's already proven to parse, resolve, and plan correctly — which is most of the noise gone.

Where to go next

• LLM Agent Integration — how agents author workflows
• Variables & Outputs — what validate --strict checks
• Previewing with Dry Run — reading the resolved plan
• MCP Server — the lint tool agents call on themselves

Got a workflow an agent just wrote? Run the three-stage harness on it before the first real run — it takes less than a second.