Introduction

You're building a multi-step pipeline — maybe it parses some input, calls a model or two, derives a result, and scores it. You compose those steps in plain TypeScript: an entrypoint calls them in a line, or a loop calls them turn after turn. And the inner loop of working on it hurts:

• You change step 3 and have to re-run steps 1 and 2 to see the effect — even though they didn't change, and even though step 2 costs real money every time.
• You want to know what actually changed between this run and the last, but you're squinting at two blobs of JSON.
• Your model step gives a slightly different answer every time and you have no idea how much it varies or whether your last prompt edit made it worse.
• A regression shows up and your eval suite tells you the pipeline got worse, but not which step broke.

Stepbook is built for that inner loop. It runs your pipeline locally, records a trajectory — a tree in which every step call is a first-class node with its own input, output, checks, and cache entry — and lets you fix one step without re-running the rest, diff runs structurally, and measure how much a step drifts over time. Nothing leaves your machine.

Status: alpha

The runner, CLI, and dashboard all ship test suites (500+ tests across the workspace). APIs may still shift before v1.0 as the public surface settles — feedback is welcome.

How it earns its keep

Each pain point above maps to something stepbook does:

• Re-running the whole pipeline to debug one step is wasteful. Stepbook caches each step call's output, keyed by the step's input and the hash of its source file. Edit a prompt or a parser and only that step — plus any step whose input changed as a result — re-runs. Everything else is served instantly from cache. See The Cache.
• Diffing outputs by eye doesn't scale. Snapshot-diff two runs of the same step, field by field. See Drift & Stability.
• You don't know if a model step is stable. Run it ten times; stepbook clusters the outputs by similarity and flags outliers, so you can see at a glance whether the step is consistent.
• Eval failures don't tell you where. Assertions live at the step level, so a failure points at the exact step and the exact items that broke it. See Assertions & Schemas.
• You don't want to ship your data to a SaaS. Everything is local: a state/ directory in your repo, a JSONL run store, a file-based snapshot cache.

The first two work as a pair, and together they're the heart of stepbook: the cache makes re-running one step cheap, and the diff shows you what that re-run changed. Edit → cheap re-run → diff is the loop everything else is built around — the rest is what makes each turn of it trustworthy.

General, with an LLM focus

Stepbook works for any pipeline of typed input → parse → derive → eval style steps — there's nothing model-specific about the core. But the features that set it apart (stability clustering, drift detection, cost and token telemetry) are the ones you reach for most when steps are non-deterministic, which in practice means LLM-heavy work. That's where this kind of dev tooling is most missing, so that's where stepbook leans.

Two ways in: a UI and a CLI

Stepbook reads the same local data through two front doors:

• The dashboard — a visual inspector for humans. Browse run history, compare runs, inspect each step's input/output/code, and run stability and query analyses interactively. See The Dashboard.
• The CLI — a headless command surface that both humans and AI coding agents use. Every command emits a versioned --json envelope with stable exit codes, so an agent (Claude Code, Cursor, Aider, …) can drive stepbook as a tool. See Driving Stepbook from an Agent.

The dashboard is also extensible — its own panels are built on a public plugin contract you can add to. See Extending the Dashboard.

What stepbook is not

• Production observability of running LLM apps. That lane belongs to LangSmith / Braintrust / Langfuse. Stepbook is for the dev loop — the inner cycle of building and refining a pipeline before it ships.

Next steps

• Getting Started — install, scaffold, run.
• The Dashboard — the visual inspector and its panels.
• Steps & Step Types — the core building block.