Assertions & Schemas

Stepbook gives you two layers of correctness checking on a unit's output: a Zod schema (structural validation) and assertions (semantic checks). Both live on the unit (defineStep) and run automatically after its runner returns — on every call. They surface as the checks on each trajectory node, so a unit an agent calls five times is checked five times, once per turn (decide#0, decide#1, …).

Schemas: structural validation

A step's optional schema is a Zod type. The step's output is validated against it before assertions run. A schema violation becomes a first-class schema valid: check with the offending Zod issues attached.

import { z } from 'zod'
import { defineStep } from '@stepbook/runner'

export default defineStep({
  type: 'llm',
  schema: z.object({
    summary: z.string(),
    entities: z.array(
      z.object({
        name: z.string(),
        type: z.enum(['person', 'organization', 'location', 'concept']),
      }),
    ),
  }),
  runner: async (input, ctx) => callModel(input),
})

Schemas are especially valuable for llm units: the model can return anything, and the schema is your guardrail. If the output doesn't match, you get a failed check pointing at exactly which field was wrong — instead of the next unit the runner calls crashing on a missing property.

Schema + structured outputs

When your LLM step uses structured outputs (e.g. the Anthropic SDK's output_config: { format: zodOutputFormat(Schema) }), use the same Zod schema for both the model call and the step's schema. The model is asked to conform, and stepbook verifies it did.

Assertions: semantic checks

assertions is a function from the step's output to an array of Checks. It runs after schema validation. Each check is a named boolean with optional context:

type Check = {
  name: string
  pass: boolean
  reason?: string // human-readable explanation, shown inline
  offending?: unknown[] // the items that caused a failure
}

A real example from the demo's parse step:

assertions: (o) => [
  { name: 'wordCount > 0', pass: o.wordCount > 0, reason: `count=${o.wordCount}` },
  {
    name: 'wordCount matches words.length',
    pass: o.wordCount === o.words.length,
    reason: `wordCount=${o.wordCount}, words.length=${o.words.length}`,
  },
  {
    name: 'no words longer than 12 chars',
    pass: longWords.length === 0,
    offending: longWords, // surfaced inline in the dashboard + lab
  },
]

reason vs offending

• reason is a short string explaining the result — shown next to the check in both passing and failing states.
• offending is the list of items that caused a failure. The dashboard and the CLI surface these inline so you can see exactly which array elements or fields broke the check. Use it whenever a check is "all items satisfy X" — pass the ones that didn't.

Async assertions

Assertions can be async, which lets you call out to an LLM judge or another service:

assertions: async (o) => {
  const verdict = await judge(o.summary)
  return [{ name: 'summary is faithful', pass: verdict.faithful, reason: verdict.note }]
}

How results surface

After a run, every check is recorded in the run blob and indexed in state/runs.jsonl with per-node totals. You can see them:

• In the CLI — the run/check output shows ✓ N pass · M fail per node.
• In the dashboard — the Checks panel shows every assertion, grouped by node, with offending items expanded inline.
• As an exit code — stepbook check exits 1 if any check fails, which is what makes it CI-friendly.

stepbook check                        # exits 1 if any assertion fails
stepbook check --json | jq '.data.totals.fail'   # 0 when everything passed

check vs run

• run executes the pipeline and persists a snapshot. Assertions run as part of it.
• check is the CI verb: it runs the pipeline and exits non-zero if any check fails. After editing one unit, --from <unit> re-runs just that unit while the rest replay from cache.

The two share the same assertion machinery; check just turns the result into an exit code your CI can gate on.

Why per-unit

The reason assertions live on the unit — rather than as a separate test suite — is that it tells you where a regression is. When an eval fails, you don't get "the pipeline is worse"; you get "the classify unit's labels are known assertion failed on these three items, on turn 2." Because each check hangs off its own trajectory node, the failure points at the exact call — that locality is the whole point.