A coverage matrix that catches missing vars

Once your config is spread across per-environment files, a new failure mode appears: you set target in prod-production.yml, ship it, and three weeks later someone runs --env staging — where you forgot to set target — and the deploy goes out with an empty cluster name. Nothing errored at parse time. The value just rendered as <no value> and a tool downstream did something quietly wrong.

A template engine won't catch this, because the variable is valid syntax — it is simply unset for that one environment. You need something that knows about all your environments at once and tells you which one has a hole. That is orchstep validate.

The matrix

validate scans every vars.X a task references, then checks each environment to see whether it supplies that var. The output is a coverage matrix — every referenced variable down the side, every environment across the top:

name: coverage
# `orchstep validate deploy` scans every vars.X this task references and reports
# which environments would leave one unresolved — as a coverage matrix.
env_config:
  env_dir: environments
tasks:
  deploy:
    steps:
      - name: ship
        func: shell
        do: 'echo "deploy {{ vars.version }} to {{ vars.target }} in {{ vars.region }}"'

The deploy step references three vars: version, target, and region. Here is what validate reports against those env files:

$ orchstep validate deploy

referenced variables (3): region  target  version

coverage matrix (✓ supplied · – absent):

  variable  (none)  nonprod  nonprod-dev  prod  prod-production
  region    –       –        –            –     –
  target    –       –        ✓            –     ✓
  version   ✓       ✓        ✓            ✓     ✓

findings:
  ⚠ nonprod         : 2 unresolved (region, target)
  ⚠ nonprod-dev     : 1 unresolved (region)
  ⚠ region          : referenced but supplied by NO environment (typo or forgotten declaration?)

Read across version: supplied everywhere, because it lives in defaults.yml. Read across target: present in the concrete nonprod-dev and prod-production files, absent in the group layers — which is fine, a nonprod group isn't meant to be a deploy target. Read across region: absent everywhere. That last row is the real bug — a var the task references that no environment provides. A typo, or a declaration someone deleted.

Two kinds of gap, and only one fails the build

This distinction is the whole reason the tool is useful instead of noisy:

• A group-layer gap (target missing from nonprod) is a warning. Group files are shared layers, not always complete targets, so it's expected that you'd only ever select a concrete environment.
• A supplied-by-no-environment gap (region) is the one that bites. It means the variable can never resolve, no matter which environment you pick.

--strict keys on exactly that second case — it fails the build when a referenced var is supplied by no environment at all:

$ orchstep validate deploy --strict; echo "exit: $?"
exit: 1

Drop the region reference (or add it to an env file) and --strict goes green. So a CI gate stays quiet about the expected group-layer gaps and fires only on the genuinely unresolvable var — no alert fatigue.

Wire it into CI

validate is static — no deploy, no credentials, no side effects — so it belongs in the same lane as your linters:

# in your pipeline
- run: orchstep lint
- run: orchstep validate deploy --strict

For dashboards or custom gates, --json gives the same matrix machine-readable:

orchstep validate deploy --json

{
  "task": "deploy",
  "referenced": ["region", "target", "version"],
  "matrix": {
    "region": { "nonprod-dev": "absent", "prod-production": "absent" },
    "target": { "nonprod-dev": "supplied", "prod-production": "supplied" }
  }
}

Want a per-step breakdown for one target instead of the whole matrix? Pass --env <name>.

What you gained

Where to go next

• orchstep validate — the full command and flags
• Externalize env config into files — the env files this validates
• Environments & Variable Files — groups vs concrete environments

The matrix above is reproducible in orchstep-demos — run orchstep validate deploy --strict and watch it catch region.