Agents in CI: deterministic ops

An agent in a terminal is great at deploying a service. An agent in your CI pipeline is a liability. CI needs to be deterministic, headless, and auditable — three things a probabilistic model in a hot loop is not. You don't want your release pipeline calling a model provider, you don't want it to hang on an interactive prompt, and you don't want "it deployed differently this time" to ever be a sentence anyone says.

The fix isn't to put the agent in CI. It's to put the agent's result in CI. Let the agent solve the deploy interactively once, capture it as a workflow, and have CI replay that workflow — no model in the loop, the same steps every run.

The workflow CI replays

This is what gets committed and runs on every push. It's a normal OrchStep workflow with a retry on the flaky step and an assertion that the target was actually set:

name: ci-deploy
defaults:
  target: staging
tasks:
  deploy:
    steps:
      - name: build
        func: shell
        do: echo "building artifact"
      - name: deploy
        func: shell
        do: echo "deploying to {{ vars.target }}"
        retry:
          max_attempts: 3
          interval: "5s"
          backoff_rate: 2.0
      - name: verify
        func: assert
        args:
          condition: '{{ ne vars.target "" }}'
          message: "deploy target must be set"

The retry the agent did by hand when the registry flaked is now retry: syntax. The check it did by eye is an assert that fails the build loudly. No interpretation, no variation — the same three steps, every time.

Make it headless and parseable

Two things make this CI-safe.

Refuse to hang. In a pipeline, an interactive prompt is a deadlock waiting to happen. Set ORCHSTEP_NON_INTERACTIVE so the run never waits on a TTY:

export ORCHSTEP_NON_INTERACTIVE=1
orchstep run deploy --var target=production --output json

Emit structured output. --output json gives you a machine-readable result instead of text your pipeline has to scrape. Your CI step can read the step statuses and gate on them directly — fail the build if any step failed, surface the failing step's name in the log, attach the JSON as an artifact for the audit trail.

Because replay needs no model, there's no API key in your CI secrets for this, no rate limit to hit, and no provider outage that blocks a release. It's a binary running a YAML file.

Lint it like any other config

A captured workflow is a file in your repo, so treat it like one. Add a lint step so a malformed workflow fails fast, before it tries to deploy anything:

orchstep lint

That catches the broken-YAML class of failure in CI review instead of mid-deploy. The same orchstep.lint check is available over the MCP server if an agent is the one preparing the workflow, but in the pipeline itself you just call the binary.

The division of labor

The pattern is a clean split:

The agent does the open-ended work once, where a human is watching. CI does the repeatable work forever, headless and deterministic. The captured workflow is the contract between them.

When the agent should stay in the loop

Be honest about the boundary. If a task genuinely needs judgment on every run — deciding whether to deploy based on fuzzy signals, triaging a novel failure — that's reasoning, and reasoning belongs with the agent, not in a replayed recipe. Replay is for the deterministic execution that follows the decision. Don't try to bake a judgment call into a YAML file; capture the steps, keep the judgment human (or agent).

Where to go next

• LLM Agent Integration — --output json and agent execution
• Capture skill — turning the interactive deploy into this workflow
• Error Handling — the retry and assertions CI relies on

Got an agent that nails the deploy by hand? Capture it, commit it, and let CI replay it headless — same steps, zero tokens.