Skip to content

How it works

Documents lane

Chunking, BM25, vectors, reciprocal-rank fusion, rerank and quarantined extraction — step by step.

Documents answer the why: a price list, an email, an ops note. The lane is built so that text from your files can inform the analysis without ever steering it.

  1. Chunk each section into overlapping passages with their page, heading path and character span.
  2. Search two ways in the same DuckDB file: BM25 full-text (stemmed for the document’s language) and exact vector search over in-process embeddings (bge-m3 by default; an offline hash embedder for tests).
  3. Fuse the two rankings with reciprocal-rank fusion: each list votes 1/(60 + rank).
  4. Rerank the top passages with a cross-encoder; if it is unavailable the result is marked degraded, not hidden.
  5. Extract. Only the quarantined extractor — a model call with no tools — reads the passage text. It returns typed events and statements with exact quotes, checked to be substrings of the passage. The planner sees only those extracts, inside a delimited data block.

Step through a real query on the example:

Query: “why did the hotel stop its wholesale order”

DuckDB full-text search, stemmed. Chunks without the words don't rank.

  1. 1email-0412.eml › Standing order
    Hello, After the new price list we are pausing our standing order while we compare suppliers. We will let you know about…    BM25 #1
  2. 2price-list.pdf
    Wholesale terms Wholesale prices increase 12% from 15 March 2026. Espresso blend 1kg ... 24.60 Filter blend 1kg ... 23.5…    BM25 #2
  3. 3price-list.pdf
    Contact orders@roaster.example Northside Coffee Roasters, Tashkent…    BM25 #3
  4. 4price-list.pdf
    Payment terms Invoices are due within 14 days. Late payments may pause standing orders.…    BM25 #4

Retrieval recall@8 is measured on two labelled corpora (bun run --filter @agentic-analytics/retrieval eval:retrieval): 1.0 on both with the offline embedder.