Learn
Retrieval
The numbers can say what fell, but only your documents can say why.
How the engine finds the passage that explains a number, with BM25, embeddings, reciprocal rank fusion and a reranker, explained from scratch.
Stage 4 of 10. INDEX_DOCS indexes the documents; the planner’s search_documents tool searches them. This is the engine’s RAG: retrieval-augmented generation.
The problem it solves
Section titled “The problem it solves”The numbers say revenue fell in April and wholesale did it. They can’t say why. The why is usually written somewhere: a price list, an email, a note. But you can’t hand every document to a language model on every question (too long, too slow, and anything in them could steer it), and a plain keyword search misses passages that say the same thing in other words.
So the documents are cut into small passages called chunks, and a search picks the few that matter for one question. Only those are read, and only by the quarantined extractor.
What happens on the example
Section titled “What happens on the example”The planner, investigating the revenue drop, searched for “wholesale price increase standing order paused”. The 9 chunks of the four documents were searched two ways and the lists combined:
INDEX_DOCS · search_documents · the coffee-roaster run
In
- querywholesale price increase standing order paused
- chunks indexed9hash-ngram-1024 · english stemmer
INDEX_DOCS · search_documents
- 1 · BM25Keyword search: chunks that share the query's words, weighted by how rare the words are.
- 2 · VectorsMeaning search: chunks whose embedding points the same way as the query's.
- 3 · Fuse (RRF)Each list votes 1/(60 + rank); a chunk near the top of both wins.
- 4 · RerankA cross-encoder would reorder the top chunks. Not run offline: the result says so.
Out
- email BM25 rank3email-0412.eml
- email vector rank3
- email RRF score0.0317
- rerankskipped (degraded: rerank)
From searchDocuments() on the run's index, with the planner's own query.
Every chunk the search returned, with its rank in each list and the fused score:
| Chunk | From | BM25 rank | Vector rank | RRF score | Starts with |
|---|---|---|---|---|---|
| C7 | price-list.pdf | 1 | 1 | 0.0328 | Wholesale terms Wholesale prices increase 12% from 15 March 2026. Espresso blend 1kg ... 2… |
| C8 | price-list.pdf | 2 | 2 | 0.0323 | Payment terms Invoices are due within 14 days. Late payments may pause standing orders.… |
| C1 | email-0412.eml › Standing order | 3 | 3 | 0.0317 | Hello, After the new price list we are pausing our standing order while we compare supplie… |
| C9 | price-list.pdf | 4 | 5 | 0.0310 | Contact orders@roaster.example Northside Coffee Roasters, Tashkent… |
| C6 | price-list.pdf | 5 | 4 | 0.0310 | Northside Coffee Roasters Price list 2026 Retail (per bag) Espresso blend 250g ... 9.50 Fi… |
| C4 | ops-notes.docx › 3. Packaging | – | 6 | 0.0152 | 3. Packaging Switched to the new packaging supplier on 28 April. The new valve bags are ch… |
| C3 | ops-notes.docx › 2. Staff | – | 7 | 0.0149 | 2. Staff One new barista started at the shop counter.… |
| C2 | ops-notes.docx › 1. Roasting | – | 8 | 0.0147 | 1. Roasting Roast schedule moved to Monday and Thursday mornings.… |
The hotel’s email (C1) came third in both lists, behind two pages of the price list. That’s fine: all three are relevant, and the planner reads extracts from the top few. The Word notes about roasting and staff share no words with the query, so BM25 doesn’t rank them at all (”–”), and the vector lane puts them last.
Rerank was skipped: this offline run has no reranker model, and the result says so (degraded: rerank) instead of hiding it. The fused order stands.
Extract. The top chunk went to the quarantined extractor, a model call with no tools that returns typed records with exact quotes, checked to be substrings of the chunk:
{
"id": "E1",
"kind": "event",
"file": "price-list.pdf",
"date": "15 March 2026",
"text": "wholesale prices went up",
"quote": "Wholesale prices increase 12% from 15 March 2026.",
"citation": {
"source_id": "D7",
"quote": "Wholesale prices increase 12% from 15 March 2026.",
"char_start": 16,
"char_end": 65,
"page": 2
}
}
That quote, with its page and character span, is what a citation in the report points at.
Step through another query on the same files (from the build’s own run):
Query: “why did the hotel stop its wholesale order”
DuckDB full-text search, stemmed. Chunks without the words don't rank.
- 1
email-0412.eml› Standing order
Hello, After the new price list we are pausing our standing order while we compare suppliers. We will let you know about…BM25 #1 - 2
price-list.pdf
Wholesale terms Wholesale prices increase 12% from 15 March 2026. Espresso blend 1kg ... 24.60 Filter blend 1kg ... 23.5…BM25 #2 - 3
price-list.pdf
Contact orders@roaster.example Northside Coffee Roasters, Tashkent…BM25 #3 - 4
price-list.pdf
Payment terms Invoices are due within 14 days. Late payments may pause standing orders.…BM25 #4
The ideas behind it
Section titled “The ideas behind it”Retrieval-augmented generation means: before a model writes, find the few passages that are relevant and give it those, so it writes from evidence instead of memory. Here the model never sees raw passages at all; it sees the extractor’s typed records.
BM25: keyword search
Section titled “BM25: keyword search”BM25 scores a passage by the query words it contains, giving rare words more weight than common ones and not rewarding a long passage just for being long. It is exact and explainable, and it misses synonyms. The engine runs it inside DuckDB’s full-text index, with stemming, so “increase” also matches “increases”.
Embeddings: meaning search
Section titled “Embeddings: meaning search”An embedding is a list of numbers that places a text in a “meaning space”, where texts about the same thing sit close together. The query is embedded too, and the nearest passages win, even if they share no words. The default model, bge-m3, runs in-process with no server; this page’s run used hash-ngram-1024, an offline stand-in that hashes words, so tests and docs need no download.
Reciprocal rank fusion
Section titled “Reciprocal rank fusion”The two lists disagree. RRF combines them without comparing their scores (which are on different scales): each list gives a passage 1/(60 + rank) points, and points add up. A passage near the top of both lists beats one that tops only one. The email: 1/(60+3) + 1/(60+3) = 0.0317.
Rerankers
Section titled “Rerankers”A reranker (a cross-encoder) reads the query and one passage together and scores how well the passage answers it. It is slower than both lanes, so it only reorders the top few. When it can’t run, the engine says the result is degraded.
Try it
Section titled “Try it”The retrieval benchmark runs both labelled corpora offline:
bun run --filter @agentic-analytics/retrieval eval:retrievalAnd the documents tutorial shows what the documents change in a report.
In the code
Section titled “In the code”packages/retrieval/src/chunk.ts: chunking with page, heading and span.packages/retrieval/src/search.ts: the query flow.packages/retrieval/src/retrieval.sql.ts: BM25 and vector search in DuckDB.packages/retrieval/src/rrf.tsandrerank.ts: fusion and reranking.packages/retrieval/src/embed.ts: in-process embeddings and the offline hash embedder.packages/agent/src/extractor.ts: the quarantined extractor.- The spec:
docs/specs/retrieval.md.
Go deeper: Documents lane.
Further reading
Section titled “Further reading”- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009. doi:10.1561/1500000019
- Reimers and Gurevych, Sentence-BERT, 2019. arXiv:1908.10084
- Chen et al., BGE M3-Embedding, 2024. arXiv:2402.03216
- Cormack, Clarke and Büttcher, Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009. doi:10.1145/1571941.1572114
- Nogueira and Cho, Passage Re-ranking with BERT, 2019. arXiv:1901.04085
- DuckDB, Full-text search extension.