How it works
Pipeline overview
The stages a job goes through, checkpointed in one DuckDB file.
A job is a folder with one DuckDB file. Each stage is checkpointed: a killed job resumes from its last completed stage, and an interrupted stage is reset and re-run, giving the same result as an uninterrupted run.
stateDiagram-v2 [*] --> INGEST INGEST --> PROFILE PROFILE --> METRICS_DRAFT METRICS_DRAFT --> AWAIT_CONFIRM: interactive METRICS_DRAFT --> MEASURE: --yes (assumptions flagged) AWAIT_CONFIRM --> MEASURE: user confirms/edits MEASURE --> DETECT DETECT --> DECOMPOSE DECOMPOSE --> INDEX_DOCS INDEX_DOCS --> WEB_CONTEXT: web enabled INDEX_DOCS --> EXPLAIN: web disabled WEB_CONTEXT --> EXPLAIN EXPLAIN --> DECIDE DECIDE --> VERIFY VERIFY --> RENDER RENDER --> [*]
| Stage | Does | Model |
|---|---|---|
| INGEST | Detects tables in spreadsheets and CSV; parses PDF, DOCX, EML, MD, TXT into sections | — |
| PROFILE | Types, units, dates, duplicates, outliers, entity matching; screens table text for injections | — |
| METRICS_DRAFT | Drafts metric definitions, validated by code | optional |
| AWAIT_CONFIRM | Pauses for review unless --yes / auto_confirm | — |
| MEASURE · DETECT · DECOMPOSE | Insight templates and decompositions, all SQL | — |
| INDEX_DOCS | Screens and indexes documents: BM25 + vectors | — |
| WEB_CONTEXT | Prepares the sanitized web lane (--web only) | — |
| EXPLAIN | Investigates each top finding with fixed tools | yes |
| DECIDE | Candidate actions, impact ranges computed in code | yes |
| VERIFY | V1–V8; nothing unverified ships | judge |
| RENDER | report.html, self-contained | — |
Every stage emits events (the API streams them over SSE). One web-enabled run of the example (scripted planner, offline web fixture, no LLM), stage by stage: the rail and each panel’s engine stages follow the run’s event log (which stages ran, in which order), facts are shown with their ids from the verified report, and the V1–V8 counts are the verifier’s own check table. The data is docs/assets/readme/pipeline.json (regenerate with tsx apps/docs/scripts/pipeline-data.ts --write); a test reruns the engine and fails if it drifts.
One real run · examples/coffee-roaster · scripted planner, no LLM
01 Files INGEST
| File | Kind | Read as |
|---|---|---|
| email-0412.eml | headers and body | |
| ops-notes.docx | document | text, by section and page |
| price-list.pdf | document | text, by section and page |
| sales-2026.xlsx | table | sheets, split into table regions |
02 Profile PROFILE
- Tables
- 3
- Document sections
- 9
- Data-quality issues
- 3
Every column gets a role and a type before anything is measured; the issues go into the report.
03 Metric METRICS_DRAFT
revenue = sum of amount, per month · USD · assumed
Drafted in plain words and YAML, confirmed or assumed, then compiled to SQL:
SELECT strftime("date", '%Y-%m') AS period, sum("amount") AS value
FROM "t_t1"04 SQL MEASURE · DETECT · DECOMPOSE
| Revenue, 2026-03 | $44,100 N12 |
|---|---|
| Revenue, 2026-04 | $36,200 N11 |
| Change | −$7,900 N29 |
| Change, percent | −17.9% N30 |
| By channel: wholesale | −$8,500 N144 |
| Wholesale's share of the change | 108% N147 |
05 Evidence INDEX_DOCS · WEB_CONTEXT · EXPLAIN · DECIDE
-
After the new price list we are pausing our standing order while we compare suppliers.
D1 email-0412.eml · Purchasing, Hotel Group · Standing order · document supported -
Wholesale prices increase 12% from 15 March 2026.
D7 price-list.pdf · p.2 · correlated -
Specialty coffee roasters typically see refunds of 1–2% of sales.
W1 external · coffee-benchmarks.example · a benchmark page from this run's offline web fixture, compared with the run's own numbers, never in place of them
06 Verify VERIFY
- V1 placeholders ran 19 times · pass
- V2 recompute ran 101 times · pass
- V3 re-derive ran 3 times · pass
- V4 definitions ran 18 times · pass
- V5 citations ran 73 times · pass
- V6 causal ran 18 times · 2 caught
- V7 grounding ran 2 times · pass
- V8 external ran 18 times · pass
101 numbers recomputed, 0 mismatched · 6 citations checked · 2 claims dropped, 2 downgraded
07 Report RENDER
Revenue fell after March, driven by wholesale (a change of −$7,900).
#1 action of 2 · medium confidence Offer wholesale accounts a volume-tier discount to win back paused standing orders: +$3,995 N350 to +$7,191 N351 , next quarter.
Checked later against: revenue >= $44,100 by 2026-09-30