Learn
The verifier (V1–V8)
Fluent text is persuasive whether or not it is true, so every report is re-checked by code before you read it.
Eight checks every report passes before you see it, what each one caught on the example, and why a wrong number stops the job.
Stage 7 of 10. VERIFY takes the draft report and checks it. Nothing reaches report.json without passing.
The problem it solves
Section titled “The problem it solves”Language models are fluent, and fluent text is persuasive whether or not it is true. A model may cite a number that doesn’t exist, say “caused by” when it only knows two things happened together, or recommend an action no finding supports. A careful human analyst catches those by re-checking. The verifier is that re-check, done by code, every time.
What happens on the example
Section titled “What happens on the example”VERIFY · the coffee-roaster run
In
- draft claims18
- facts to recompute101
VERIFY
- 1 · V1 placeholders19 checks, 0 failed
- 2 · V2 recompute101 checks, 0 failed
- 3 · V3 re-derive3 checks, 0 failed
- 4 · V4 definitions18 checks, 0 failed
- 5 · V5 citations73 checks, 0 failed
- 6 · V6 causal language18 checks, 2 failed
- 7 · V7 grounding2 checks, 0 failed
- 8 · V8 external18 checks, 0 failed
Out
- V119/19all passed
- V2101/101all passed
- V33/3all passed
- V418/18all passed
- V573/73all passed
- V616/182 failed
- V72/2all passed
- V818/18all passed
- dropped2
- rewritten or downgraded2
- mismatched numbers0
From the verifier's check table (2 failed checks in all).
On this run the verifier recomputed 101 numbers, and 0 differed. It checked 6 citations. Only V6 (causal language) found problems. What it removed or changed:
| Claim in the draft | What happened | Why |
|---|---|---|
| dropped | unknown fact N99999 | |
| dropped | no finding supports it | |
| The drop was caused by the price increase. | document_supported → correlated | no source document cited |
| The drop was caused by the price increase. | reworded | V6: causal wording without a supporting document; reworded to “The drop coincided with the price increase.” |
16 claims shipped. The same draft against the verified report, side by side (from the build’s own run of the example):
Draft → verified · what the verifier changed
- Wholesale also lost {{N99999}} in May.dropped · unknown fact N99999
- Open a second shopdropped · no finding supports it
- The drop was caused by the price increase.document_supported → correlated · no source document cited
- The drop was caused by the price increase.correlated → correlated · V6: causal wording without a supporting document; reworded to “The drop coincided with the price increase.”
15 claims shipped; 2 dropped and 2 reworded or downgraded on the way, each listed in the report's verification section.
The eight checks
Section titled “The eight checks”| Check | Asks | If it fails |
|---|---|---|
| V1 placeholders | Is every number in model text a {{N…}} placeholder for a fact that exists? | drop the claim |
| V2 recompute | Run every fact’s SQL or function again: same value (within 1e-9)? | stop the job |
| V3 re-derive | Do headline totals match when computed another way (summing the members)? | stop the job |
| V4 definitions | Does each fact match its metric’s definition, period and filters, and the wording its sign? | drop |
| V5 citations | Does each quote exist at its source, and do cited rows feed the fact? A judge model checks support. | drop |
| V6 causal language | Do “caused”, “because”, “led to” have a document behind them? | reword or downgrade |
| V7 grounding | Does each recommendation cite a surviving finding, with impact facts and a future check date? | drop the action |
| V8 external | Does each web number cite its exact web source and stay labelled External? | drop |
Why do V2 and V3 stop the job instead of dropping a claim? Because a fact that doesn’t recompute is not a model mistake, it is a bug in the engine. Shipping a report around it would hide the bug. The job fails with a diagnostics file instead.
The ideas behind it
Section titled “The ideas behind it”Verify, don’t trust
Section titled “Verify, don’t trust”The model’s text is treated as a draft from a junior colleague: useful, and checked line by line. The checks are cheap because every number is a fact with its computation attached; re-running it is one query.
Causal grades
Section titled “Causal grades”Saying why something happened is the riskiest thing an analyst does. Every claim carries a causal grade. observed: the numbers themselves. correlated: two things happened together. document_supported: a document says so, with a quote. V6 enforces the vocabulary: causal words need a document.
Hallucination
Section titled “Hallucination”When a model states something not supported by its input, that is called a hallucination. Ji et al. survey the kinds of hallucination and how they are measured; see Further reading. You can’t make a model stop; you can make sure nothing it states unsupported is shown to the reader as fact.
Try it
Section titled “Try it”The verifier’s own tests, and the injection suite that runs every report through it:
bunx vitest run packages/core/test/verify.test.ts packages/core/test/injection.test.tsEvery report carries a verification section with these counts; see Report & fact IDs.
In the code
Section titled “In the code”packages/verify/src/verify.ts: V1–V8 and the drop/rewrite/stop policy.packages/verify/src/recompute.ts: recomputing SQL and function facts.- The spec:
docs/specs/agent-and-verifier.md.
Go deeper: The verifier.
Further reading
Section titled “Further reading”- Ji et al., Survey of Hallucination in Natural Language Generation, 2022. arXiv:2202.03629
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, 2023. arXiv:2305.14251