Quality
Eval results
The latest results per model, from evals/RESULTS.md.
Generated by aa eval from evals/results/*.json (docs/EVALS.md §4). Each table is the latest run of one model on one suite. Weak results are shown, not hidden.
Datasets: 30 generated businesses (evals/generator, 10 archetypes × 3 variants) with planted events, distractors and data-quality issues. Retrieval recall@8 and the injection suite run separately (bun run --filter @agentic-analytics/retrieval eval:retrieval, bunx vitest run packages/core/test/injection.test.ts).
The engine’s detection and ranking rules (M11, M13a) were tuned on these same 30 datasets, so every full and smoke score here is optimistic for unseen data. The held-out generated set (10 reseeded businesses and two archetypes no rule was written against, docs/specs/accuracy.md §9) is scored only by release runs (aa eval --suite heldout); the smoke suite includes one of its datasets as a CI check (smoke runs before 2026-10-22 had three datasets, now four). Of the Groq tables, the three single-model smoke runs (gpt-oss-120b, gpt-oss-20b, qwen3.8-27b alone) predate the M13a engine changes (seasonal guard, ranking, driver split, grouping dimensions). The “planner, extractor, judge” smoke run used the M13a engine, before the merge’s representative-rule fix (a template with no measured change keeps its priority), which changes nothing per dataset on the offline baseline. Its full run (2026-10-20/21, over two nights) used the engine just before the M14a fixes (#44–#49) and the scoring before code-quality fix 9: it is the before of the M14a before/after, and the after run follows. A run with unfinished jobs is partial: its rates count those jobs as misses and its pass column reads n/a. A nightly run is carried over two passes (aa eval --passes 2, scripts/eval-daily.sh): a job the first night’s quota stopped runs again the next night, and the run is partial only if jobs are still unfinished after that. The ten v3 datasets changed on 2026-10-22 (code-quality fix 11b: their noise blip moved from 2025-12 to 2025-11, off the planted December peak): v3 rows of runs before that date used the old data.
baseline · full suite · 2026-10-21
Section titled “baseline · full suite · 2026-10-21”Baseline: a scripted heuristic (keyword extractor, rule-based planner and word-overlap judge), not a language model. Its rules were written knowing the generator’s document templates (e.g. it treats a sentence with “after” and “pausing” as a stated cause), so its cause-grade and action scores are optimistic. It checks the harness end to end; it is not evidence of model quality.
30 datasets · 120 s · 23100 tokens
Usage: no provider calls (scripted model)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 100.0% | 100% | yes |
| Number mismatches | 0 | 0 | yes |
| Planted recall@5 | 96.0% | ≥ 80% | yes |
| Driver accuracy | 100.0% | ≥ 90% | yes |
| Cause grade accuracy | 100.0% | ≥ 80% | yes |
| Distractor FP rate | 0.0% | ≤ 10% | yes |
| Quality-issue recall | 100.0% | ≥ 90% | yes |
| Citation precision | 100.0% | ≥ 95% | yes |
| Definition adherence | 100.0% | 100% | yes |
| Recommendation grounding | 100.0% | 100% | yes |
| Recommendation relevance | 100.0% | ≥ 70% | yes |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 5 | 4.1 s |
| coffee_roaster-v2 | yes | 2/2 | none | 1/1 | 5 | 5.0 s |
| coffee_roaster-v3 | yes | 2/2 | none | 1/1 | 5 | 6.0 s |
| printing_shop-v1 | yes | 1/1 | none | 2/2 | 3 | 2.2 s |
| printing_shop-v2 | yes | 2/2 | none | 1/1 | 5 | 3.7 s |
| printing_shop-v3 | yes | 2/2 | none | 1/1 | 5 | 4.3 s |
| clinic-v1 | yes | 1/1 | none | 2/2 | 5 | 2.9 s |
| clinic-v2 | yes | 2/2 | none | 1/1 | 5 | 4.1 s |
| clinic-v3 | yes | 2/2 | none | 1/1 | 5 | 4.7 s |
| online_store-v1 | yes | 1/1 | none | 2/2 | 4 | 3.1 s |
| online_store-v2 | yes | 2/2 | none | 1/1 | 5 | 4.5 s |
| online_store-v3 | yes | 2/2 | none | 1/1 | 5 | 5.4 s |
| distributor-v1 | yes | 1/1 | none | 2/2 | 5 | 2.3 s |
| distributor-v2 | yes | 1/2 | none | 1/1 | 5 | 3.3 s |
| distributor-v3 | yes | 2/2 | none | 1/1 | 5 | 3.4 s |
| restaurant-v1 | yes | 1/1 | none | 2/2 | 5 | 3.8 s |
| restaurant-v2 | yes | 2/2 | none | 1/1 | 5 | 5.0 s |
| restaurant-v3 | yes | 2/2 | none | 1/1 | 5 | 6.7 s |
| saas-v1 | yes | 1/1 | none | 2/2 | 5 | 3.7 s |
| saas-v2 | yes | 2/2 | none | 1/1 | 5 | 4.4 s |
| saas-v3 | yes | 2/2 | none | 1/1 | 5 | 5.2 s |
| logistics-v1 | yes | 1/1 | none | 2/2 | 5 | 3.2 s |
| logistics-v2 | yes | 2/2 | none | 1/1 | 5 | 4.6 s |
| logistics-v3 | yes | 2/2 | none | 1/1 | 5 | 5.6 s |
| construction_supplier-v1 | yes | 1/1 | none | 2/2 | 5 | 3.0 s |
| construction_supplier-v2 | yes | 2/2 | none | 1/1 | 5 | 3.6 s |
| construction_supplier-v3 | yes | 2/2 | none | 1/1 | 5 | 4.3 s |
| agency-v1 | yes | 1/1 | none | 2/2 | 5 | 1.9 s |
| agency-v2 | yes | 1/2 | none | 1/1 | 5 | 2.6 s |
| agency-v3 | yes | 2/2 | none | 1/1 | 5 | 3.7 s |
configured (small-model mode) · full suite · 2026-10-19
Section titled “configured (small-model mode) · full suite · 2026-10-19”History, not a headline: the first scheduled nightly pass (cron, 2026-10-19 22:00 UTC), under the old one-night rule. 21 of 30 jobs finished; then Groq’s free tier (200k tokens a day for the qwen/qwen3.8-27b planner) stopped the rest with RATE_LIMITED, so the rates count those jobs as misses. Same Groq roles as the smoke run (planner qwen/qwen3.8-27b, extractor openai/gpt-oss-120b, judge openai/gpt-oss-20b), small-model mode. Since #37 a nightly run spans two nights and is labelled partial only if jobs are still unfinished after the second; this pass is kept as the record that led to that change.
30 datasets · 2012 s · 801284 tokens
Usage: n/a (tokens per call not recorded for this run)
Served by: extractor → openai-compatible:groq:openai/gpt-oss-120b (448 calls, 335 cached); judge → openai-compatible:groq:openai/gpt-oss-20b (92 calls, 79 cached); planner → openai-compatible:groq:qwen/qwen3.8-27b (131 calls, 47 cached, 31 failed)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 70.0% | 100% | no |
| Number mismatches | 0 | 0 | n/a (partial run) |
| Planted recall@5 | 48.0% | ≥ 80% | n/a (partial run) |
| Driver accuracy | 100.0% | ≥ 90% | n/a (partial run) |
| Cause grade accuracy | 70.8% | ≥ 80% | n/a (partial run) |
| Distractor FP rate | 0.0% | ≤ 10% | n/a (partial run) |
| Quality-issue recall | 70.0% | ≥ 90% | n/a (partial run) |
| Citation precision | 100.0% | ≥ 95% | n/a (partial run) |
| Definition adherence | 100.0% | 100% | n/a (partial run) |
| Recommendation grounding | 100.0% | 100% | n/a (partial run) |
| Recommendation relevance | 38.1% | ≥ 70% | n/a (partial run) |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 3 | 40.0 s |
| coffee_roaster-v2 | yes | 2/2 | none | 1/1 | 3 | 7.0 s |
| coffee_roaster-v3 | yes | 1/2 | none | 1/1 | 3 | 5.9 s |
| printing_shop-v1 | yes | 1/1 | none | 2/2 | 3 | 4.1 s |
| printing_shop-v2 | yes | 1/2 | none | 1/1 | 3 | 6.1 s |
| printing_shop-v3 | yes | 1/2 | none | 1/1 | 3 | 5.4 s |
| clinic-v1 | yes | 1/1 | none | 2/2 | 3 | 4.4 s |
| clinic-v2 | yes | 1/2 | none | 1/1 | 3 | 8.0 s |
| clinic-v3 | yes | 1/2 | none | 1/1 | 3 | 29.6 s |
| online_store-v1 | yes | 1/1 | none | 2/2 | 3 | 133.2 s |
| online_store-v2 | yes | 1/2 | none | 1/1 | 3 | 186.6 s |
| online_store-v3 | yes | 1/2 | none | 1/1 | 3 | 90.1 s |
| distributor-v1 | yes | 1/1 | none | 2/2 | 3 | 100.4 s |
| distributor-v2 | yes | 1/2 | none | 1/1 | 3 | 164.8 s |
| distributor-v3 | yes | 1/2 | none | 1/1 | 3 | 93.8 s |
| restaurant-v1 | yes | 1/1 | none | 2/2 | 3 | 148.7 s |
| restaurant-v2 | yes | 2/2 | none | 1/1 | 3 | 155.5 s |
| restaurant-v3 | yes | 1/2 | none | 1/1 | 3 | 61.4 s |
| saas-v1 | yes | 1/1 | none | 2/2 | 3 | 184.4 s |
| saas-v2 | yes | 2/2 | none | 1/1 | 3 | 4.9 s |
| saas-v3 | yes | 1/2 | none | 1/1 | 3 | 123.6 s |
| logistics-v1 | no (RATE_LIMITED: DECIDE failed: every route is rate-limited: op) | 0/1 | none | 0/2 | 0 | 160.8 s |
| logistics-v2 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 6.1 s |
| logistics-v3 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 10.6 s |
| construction_supplier-v1 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/1 | none | 0/2 | 0 | 2.7 s |
| construction_supplier-v2 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 10.6 s |
| construction_supplier-v3 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 118.6 s |
| agency-v1 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/1 | none | 0/2 | 0 | 12.9 s |
| agency-v2 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 69.0 s |
| agency-v3 | no (RATE_LIMITED: EXPLAIN failed: every route is rate-limited: o) | 0/2 | none | 0/1 | 0 | 62.2 s |
openai-compatible:groq:qwen/qwen3.8-27b planner, gpt-oss-120b extractor, gpt-oss-20b judge (small-model mode) · full suite · 2026-10-21
Section titled “openai-compatible:groq:qwen/qwen3.8-27b planner, gpt-oss-120b extractor, gpt-oss-20b judge (small-model mode) · full suite · 2026-10-21”The before of the M14a real-model before/after: run over two nights (2026-10-20 and 2026-10-21) on the engine before the M14a fixes (#44–#49), and scored with the scoring of that time (before code-quality fix 9). The next nightly runs measure the merged fixes.
Passes: 2026-10-20, 2026-10-21 (2 of up to 2; every job finished).
30 datasets · 1681 s · 1007951 tokens
Usage: n/a (tokens per call not recorded for this run)
Served by: extractor → openai-compatible:groq:openai/gpt-oss-120b (540 calls, 467 cached); judge → openai-compatible:groq:openai/gpt-oss-20b (120 calls, 111 cached); planner → openai-compatible:groq:qwen/qwen3.8-27b (177 calls, 96 cached, 35 failed)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 100.0% | 100% | yes |
| Number mismatches | 0 | 0 | yes |
| Planted recall@5 | 68.0% | ≥ 80% | no |
| Driver accuracy | 100.0% | ≥ 90% | yes |
| Cause grade accuracy | 67.6% | ≥ 80% | no |
| Distractor FP rate | 0.0% | ≤ 10% | yes |
| Quality-issue recall | 100.0% | ≥ 90% | yes |
| Citation precision | 100.0% | ≥ 95% | yes |
| Definition adherence | 96.7% | 100% | no |
| Recommendation grounding | 100.0% | 100% | yes |
| Recommendation relevance | 30.0% | ≥ 70% | no |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 3 | 4.5 s |
| coffee_roaster-v2 | yes | 2/2 | none | 1/1 | 3 | 7.0 s |
| coffee_roaster-v3 | yes | 1/2 | none | 1/1 | 3 | 5.7 s |
| printing_shop-v1 | yes | 1/1 | none | 2/2 | 3 | 4.0 s |
| printing_shop-v2 | yes | 1/2 | none | 1/1 | 3 | 5.9 s |
| printing_shop-v3 | yes | 1/2 | none | 1/1 | 3 | 5.3 s |
| clinic-v1 | yes | 1/1 | none | 2/2 | 3 | 4.4 s |
| clinic-v2 | yes | 1/2 | none | 1/1 | 3 | 7.8 s |
| clinic-v3 | yes | 1/2 | none | 1/1 | 3 | 5.2 s |
| online_store-v1 | yes | 1/1 | none | 2/2 | 3 | 4.5 s |
| online_store-v2 | yes | 1/2 | none | 1/1 | 3 | 11.1 s |
| online_store-v3 | yes | 1/2 | none | 1/1 | 3 | 8.4 s |
| distributor-v1 | yes | 1/1 | none | 2/2 | 3 | 2.8 s |
| distributor-v2 | yes | 1/2 | none | 1/1 | 3 | 5.0 s |
| distributor-v3 | yes | 1/2 | none | 1/1 | 3 | 70.2 s |
| restaurant-v1 | yes | 1/1 | none | 2/2 | 3 | 3.7 s |
| restaurant-v2 | yes | 2/2 | none | 1/1 | 3 | 36.7 s |
| restaurant-v3 | yes | 1/2 | none | 1/1 | 3 | 7.4 s |
| saas-v1 | yes | 1/1 | none | 2/2 | 3 | 137.7 s |
| saas-v2 | yes | 2/2 | none | 1/1 | 3 | 4.9 s |
| saas-v3 | yes | 1/2 | none | 1/1 | 3 | 6.5 s |
| logistics-v1 | yes | 1/1 | none | 2/2 | 3 | 76.8 s |
| logistics-v2 | yes | 2/2 | none | 1/1 | 3 | 119.5 s |
| logistics-v3 | yes | 1/2 | none | 1/1 | 3 | 190.8 s |
| construction_supplier-v1 | yes | 1/1 | none | 2/2 | 3 | 215.3 s |
| construction_supplier-v2 | yes | 1/2 | none | 1/1 | 3 | 248.0 s |
| construction_supplier-v3 | yes | 1/2 | none | 1/1 | 3 | 176.8 s |
| agency-v1 | yes | 1/1 | none | 2/2 | 3 | 206.1 s |
| agency-v2 | yes | 1/2 | none | 1/1 | 3 | 89.6 s |
| agency-v3 | yes | 1/2 | none | 1/1 | 3 | 8.9 s |
openai-compatible:groq:openai/gpt-oss-120b (small-model mode) · smoke suite · 2026-10-16
Section titled “openai-compatible:groq:openai/gpt-oss-120b (small-model mode) · smoke suite · 2026-10-16”Groq free tier. The daily token quota (200k/day) ran out after the first job, so two jobs failed with a rate-limit error; rerun when the quota resets. Its one completed job was scored before the metric-concept mapping and its report was not kept, so that row may undercount (the run predates —keep-reports).
3 datasets · 192 s · 30752 tokens
Usage: n/a (tokens per call not recorded for this run)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 33.3% | 100% | no |
| Number mismatches | 0 | 0 | n/a (partial run) |
| Planted recall@5 | 0.0% | ≥ 80% | n/a (partial run) |
| Driver accuracy | n/a | ≥ 90% | n/a (partial run) |
| Cause grade accuracy | n/a | ≥ 80% | n/a (partial run) |
| Distractor FP rate | 0.0% | ≤ 10% | n/a (partial run) |
| Quality-issue recall | 50.0% | ≥ 90% | n/a (partial run) |
| Citation precision | n/a | ≥ 95% | n/a (partial run) |
| Definition adherence | 100.0% | 100% | n/a (partial run) |
| Recommendation grounding | 100.0% | 100% | n/a (partial run) |
| Recommendation relevance | 0.0% | ≥ 70% | n/a (partial run) |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 0/1 | none | 2/2 | 3 | 151.0 s |
| restaurant-v3 | no (PROVIDER: EXPLAIN failed: Rate limit reached for model `open) | 0/2 | none | 0/1 | 0 | 37.0 s |
| saas-v2 | no (PROVIDER: EXPLAIN failed: Rate limit reached for model `open) | 0/2 | none | 0/1 | 0 | 3.9 s |
openai-compatible:groq:openai/gpt-oss-20b (small-model mode) · smoke suite · 2026-10-16
Section titled “openai-compatible:groq:openai/gpt-oss-20b (small-model mode) · smoke suite · 2026-10-16”Groq free tier. Run before DECIDE degraded gracefully on invalid output: two jobs failed at DECIDE on Groq’s own JSON validation (fixed since; rerun pending quota). Rescored from the kept report with the final scoring.
3 datasets · 703 s · 91087 tokens
Usage: n/a (tokens per call not recorded for this run)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 33.3% | 100% | no |
| Number mismatches | 0 | 0 | n/a (partial run) |
| Planted recall@5 | 0.0% | ≥ 80% | n/a (partial run) |
| Driver accuracy | 0.0% | ≥ 90% | n/a (partial run) |
| Cause grade accuracy | 100.0% | ≥ 80% | n/a (partial run) |
| Distractor FP rate | 0.0% | ≤ 10% | n/a (partial run) |
| Quality-issue recall | 50.0% | ≥ 90% | n/a (partial run) |
| Citation precision | 100.0% | ≥ 95% | n/a (partial run) |
| Definition adherence | 100.0% | 100% | n/a (partial run) |
| Recommendation grounding | n/a | 100% | n/a (partial run) |
| Recommendation relevance | 0.0% | ≥ 70% | n/a (partial run) |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 3 | 279.7 s |
| restaurant-v3 | no (PROVIDER: DECIDE failed: Failed to validate JSON. Please adj) | 0/2 | none | 0/1 | 0 | 216.9 s |
| saas-v2 | no (PROVIDER: DECIDE failed: Failed to validate JSON. Please adj) | 0/2 | none | 0/1 | 0 | 206.5 s |
openai-compatible:groq:qwen/qwen3.8-27b (small-model mode) · smoke suite · 2026-10-16
Section titled “openai-compatible:groq:qwen/qwen3.8-27b (small-model mode) · smoke suite · 2026-10-16”Groq free tier, small-model mode. The model drafted metrics on the Costs sheet, so the planted ×1000 cost typo surfaced as the top finding (an engine gap recorded in STATE). Rescored from the kept reports after the review fix that stopped Costs-sheet sums counting as revenue.
3 datasets · 1174 s · 100559 tokens
Usage: n/a (tokens per call not recorded for this run)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 100.0% | 100% | yes |
| Number mismatches | 0 | 0 | yes |
| Planted recall@5 | 0.0% | ≥ 80% | no |
| Driver accuracy | 0.0% | ≥ 90% | no |
| Cause grade accuracy | 50.0% | ≥ 80% | no |
| Distractor FP rate | 0.0% | ≤ 10% | yes |
| Quality-issue recall | 100.0% | ≥ 90% | yes |
| Citation precision | 100.0% | ≥ 95% | yes |
| Definition adherence | 100.0% | 100% | yes |
| Recommendation grounding | 100.0% | 100% | yes |
| Recommendation relevance | 0.0% | ≥ 70% | no |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 3 | 359.7 s |
| restaurant-v3 | yes | 1/2 | none | 1/1 | 3 | 359.7 s |
| saas-v2 | yes | 0/2 | none | 1/1 | 3 | 454.1 s |
openai-compatible:groq:qwen/qwen3.8-27b planner, gpt-oss-120b extractor, gpt-oss-20b judge (small-model mode) · smoke suite · 2026-10-16
Section titled “openai-compatible:groq:qwen/qwen3.8-27b planner, gpt-oss-120b extractor, gpt-oss-20b judge (small-model mode) · smoke suite · 2026-10-16”Groq free tier, every role on Groq (planner qwen/qwen3.8-27b, extractor openai/gpt-oss-120b, judge openai/gpt-oss-20b), small-model mode, run with —model configured. All 3 jobs finished. The free judges tried first (Gemma 4 on Google AI Studio, OpenRouter free, Mistral) were dropped: overloaded (503/429, 44–127 s per call) or rate-limited on the first request. Bars missed: cause grade 75% (< 80%) and recommendation relevance 66.7% (< 70%).
3 datasets · 37 s · 97619 tokens
Usage: n/a (tokens per call not recorded for this run)
Served by: extractor → openai-compatible:groq:openai/gpt-oss-120b (54 calls, 54 cached); judge → openai-compatible:groq:openai/gpt-oss-20b (15 calls, 13 cached); planner → openai-compatible:groq:qwen/qwen3.8-27b (15 calls, 14 cached)
| Metric | Result | v0.1 bar | Pass |
|---|---|---|---|
| Jobs completed | 100.0% | 100% | yes |
| Number mismatches | 0 | 0 | yes |
| Planted recall@5 | 80.0% | ≥ 80% | yes |
| Driver accuracy | 100.0% | ≥ 90% | yes |
| Cause grade accuracy | 75.0% | ≥ 80% | no |
| Distractor FP rate | 0.0% | ≤ 10% | yes |
| Quality-issue recall | 100.0% | ≥ 90% | yes |
| Citation precision | 100.0% | ≥ 95% | yes |
| Definition adherence | 100.0% | 100% | yes |
| Recommendation grounding | 100.0% | 100% | yes |
| Recommendation relevance | 66.7% | ≥ 70% | no |
Per dataset
| Dataset | Done | Planted found | Distractors reported | Quality found | Findings | Time |
|---|---|---|---|---|---|---|
| coffee_roaster-v1 | yes | 1/1 | none | 2/2 | 3 | 5.7 s |
| restaurant-v3 | yes | 1/2 | none | 1/1 | 3 | 10.5 s |
| saas-v2 | yes | 2/2 | none | 1/1 | 3 | 20.9 s |
Real public data
Section titled “Real public data”Public datasets run as they are published (evals/real/README.md): bun run eval:real:fetch downloads them and checks every SHA-256, bun run eval:real runs the engine and scores each report against the answer files in evals/real/answers/. Each check reads only report.json, and its expected numbers come from SQL written in the answer file. A failed check is an engine gap, listed in docs/progress/STATE.md; checks are not tuned to pass.
baseline · real public data · 2026-10-23
Section titled “baseline · real public data · 2026-10-23”Baseline: the scripted heuristic, no language model. On real files it drafts metrics with the deterministic fallbacks (or uses the answer file’s confirmed definitions) and finds changes with the analysis library; it writes no causes from documents (there are none). Each check reads report.json; its expected numbers come from the SQL in evals/real/answers/.
Hotel booking demand datasets (Data in Brief (Elsevier), CC BY 4.0)
Section titled “Hotel booking demand datasets (Data in Brief (Elsevier), CC BY 4.0)”Antonio, N., de Almeida, A., & Nunes, L. (2019). Hotel booking demand datasets. Data in Brief, 22, 41–49. https://doi.org/10.1016/j.dib.2018.11.126. File read: hotel-bookings.csv (SHA-256 3b22c26428ec…). Completed in 26.2 s · 4/6 checks pass.
| Check | Pass | What the report shows |
|---|---|---|
| C1: The job completes on 119,390 bookings with the published padding and “NULL” text left in | yes | report with 5 findings, 3 data-quality items |
| C2: Every number re-computes (no mismatch, re-derivation mismatch or definition violation) | yes | 107 numbers recomputed before the report shipped (a mismatch would have stopped the job), 0 claims dropped for breaking a definition |
| C3: The summer peak: August is the top month of room revenue in 2015, 2016 and 2017 (G2) | yes | 2015 peak 2015-08; 2016 peak 2016-08; 2017 peak 2017-08; 3/3 values equal SQL |
| C4: The autumn fall is not reported as news: bookings in November are below October in both years (G2) | no | 1 of 5 findings: F2 2015-10→2015-11 (contribution.v1) |
| C5: Missing values: Children is NA in 4 bookings (G1) | yes | Q3 missing_value: 4 rows (N624), SQL 4; cites rows 40602, 40669, 40681… |
| C6: The extreme ADR: 5,400 in one city-hotel booking against a median of 99.9 (G1), a likely decimal-point slip | no | not reported (kinds reported: unparseable_value, unit_error, missing_value) |
Metrics: room_revenue, room_nights, bookings, cancellation_rate. Data-quality kinds: unparseable_value, unit_error, missing_value. Findings: trend_break: revenue 2017-04 → 2017-05 (contribution.v1); anomaly: bookings 2015-10 → 2015-11 (contribution.v1); anomaly: room_nights 2015-10 → 2015-11 (contribution.v1); improvement: revenue[customertype=Group] 2016-08 → 2017-08 (period_change.v1); improvement: revenue 2017-08 (trend.v1).
Warehouse and Retail Sales (Montgomery County, Maryland), 2017–2020 (Montgomery County, MD (Department of Liquor Control, on data.montgomerycountymd.gov), Public domain)
Section titled “Warehouse and Retail Sales (Montgomery County, Maryland), 2017–2020 (Montgomery County, MD (Department of Liquor Control, on data.montgomerycountymd.gov), Public domain)”Source: Montgomery County, MD. Warehouse and Retail Sales, data.montgomerycountymd.gov/d/v76h-r7br (Public Domain). File read: montgomery-warehouse-sales.csv (SHA-256 b1f5450e132c…). Completed in 141.7 s · 2/2 checks pass.
| Check | Pass | What the report shows |
|---|---|---|
| C1: The job completes on a file with a split year and month and three money-like measures | yes | report with 0 findings, 3 data-quality items |
| C2: Every number re-computes (no mismatch, re-derivation mismatch or definition violation) | yes | 2 numbers recomputed before the report shipped (a mismatch would have stopped the job), 0 claims dropped for breaking a definition |
Metrics: none. Data-quality kinds: unparseable_value, missing_id, missing_value. Findings: none.
Electronic card transactions, September 2026 (Stats NZ (Tatauranga Aotearoa), CC BY 4.0)
Section titled “Electronic card transactions, September 2026 (Stats NZ (Tatauranga Aotearoa), CC BY 4.0)”This work is based on Stats NZ’s data (Electronic card transactions: September 2026) which are licensed by Stats NZ for reuse under the Creative Commons Attribution 4.0 International licence. File read: nz-card-spend-by-industry.csv (SHA-256 63ff45be4c78…). Completed in 6.7 s · 5/5 checks pass.
| Check | Pass | What the report shows |
|---|---|---|
| C1: The job completes on the official CSV layout (Period as year.month, e.g. 2002.10) | yes | report with 4 findings, 1 data-quality item |
| C2: Every number re-computes (no mismatch, re-derivation mismatch or definition violation) | yes | 380 numbers recomputed before the report shipped (a mismatch would have stopped the job), 0 claims dropped for breaking a definition |
| C3: The December peak: December is the top month of 2023, 2024 and 2025 (G1) | yes | 2023 peak 2023-12; 2024 peak 2024-12; 2025 peak 2025-12; 3/3 values equal SQL |
| C4: The regular January fall is not reported as news (G4: January is below December in 24 of 24 years) | yes | none |
| C5: Suppressed values: Data_value is blank for 2000-01 to 2002-09 (G3: 264 lines) | yes | Q1 missing_value: 264 rows (N403), SQL 264; cites rows 2, 3, 4… |
Metrics: card_spend. Data-quality kinds: missing_value. Findings: anomaly: card_spend 2020-03 → 2020-04 (contribution.v1); anomaly: card_spend 2020-04 → 2020-05 (contribution.v1); trend_break: card_spend 2016-09 → 2016-10 (contribution.v1); improvement: card_spend 2026-09 (trend.v1).
Online Retail II (UCI Machine Learning Repository, CC BY 4.0)
Section titled “Online Retail II (UCI Machine Learning Repository, CC BY 4.0)”Chen, D. (2012). Online Retail II [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C5CG6D. File read: online_retail_II.sample10.csv (SHA-256 6d8dff8c0bff…). Completed in 13.9 s · 7/7 checks pass.
| Check | Pass | What the report shows |
|---|---|---|
| C1: The job completes on a real 105,797-line file | yes | report with 2 findings, 3 data-quality items |
| C2: Every number re-computes (no mismatch, re-derivation mismatch or definition violation) | yes | 32 numbers recomputed before the report shipped (a mismatch would have stopped the job), 0 claims dropped for breaking a definition |
| C3: Cancellations: invoice numbers starting with C (G1: 2,089 lines, 816 invoices) | yes | Q1 prefixed_id: 2089 rows (N120), SQL 2089; cites rows 58, 117, 118… |
| C4: Negative quantities (G1: 2,431 lines) | yes | Q2 negative_quantity: 2431 rows (N121), SQL 2431; cites rows 58, 117, 118… |
| C5: Missing customer IDs (G1: 21,051 lines) | yes | Q3 missing_id: 21051 rows (N122), SQL 21051; cites rows 156, 157, 158… |
| C6: The Nov–Dec peak: November is the top month of 2010 and 2011 (G2) | yes | 2010 peak 2010-11; 2011 peak 2011-11; 3/3 values equal SQL |
| C7: No finding about December 2011, a partial month (G3: the data ends 2011-12-09) | yes | no finding at 2011-12 |
Metrics: revenue, orders. Data-quality kinds: prefixed_id, negative_quantity, missing_id. Findings: improvement: revenue 2011-11 (trend.v1); anomaly: orders 2010-11 → 2010-12 (anomaly.v1).
Baseline report (M14)
Section titled “Baseline report (M14)”The runs
Section titled “The runs”Today’s engine on the M14 measuring tools (docs/plans/m14.md PR 8), published as it is, gaps included. Every run here is offline and scripted (2026-10-22, no language model), each run through scripts/heavy.sh: the scripted baseline for the engine, and canned FakeModel answers built from the answer keys for ablation arms A and B. So these are harness checks: they show the tools measure what they should, not what a model does, and nothing here says the engine adds accuracy. The real-model runs are DEFERRED to the nightly runs (below).
Golden v1 is the dev split of three real datasets’ golden sets (Online Retail II, NZ card spend, hotel bookings; 23 items, 3 traps). EIA-861M is DEFERRED (its host doesn’t answer from this machine), so golden v1 has three datasets until it runs. Held-out golden items are scored only in release runs (M21), and the held-out generated set is not part of this report. Golden items are agent-reviewed, not maintainer-reviewed.
| Run | Command | Result |
|---|---|---|
| Golden v1, scripted (dev split) | bun run eval -- --suite golden --model baseline (bun run eval:golden is the same run: the model defaults to baseline) | 7/23 items pass: hotel bookings 2/8, NZ card spend 3/7, Online Retail II 2/8. Counted toward the bars: 7/20 numeric and categorical, 0/3 traps. 7 of the 8 items read from the report (facts and drivers) pass; the baseline answers no follow-up question, so all 15 ask items (traps included) are misses |
| Consistency on smoke, scripted | bun run eval -- --suite smoke --repeat 5 --model baseline | 4/4 jobs agree in all 5 runs on every measure (numbers, findings, driver, cause grade, action), 0 disagreements; the cache is n/a (no provider calls). It checks the repeat harness: the scripted engine is deterministic |
| Ablation on smoke, scripted | bun run eval:ablation -- --arms A,B,C --suite smoke | Planted recall@5: A 40%, B 60%, C 100% (5 tasks over 3 datasets). C − A 0.600 [0.200, 1.000]; C − B 0.400 [0.000, 0.800]; B − A 0.200 [−0.600, 1.000] |
| Ablation on golden v1, scripted | bun run eval:ablation -- --arms A,B,C --suite golden | Numeric accuracy: A 8/16, B 0/16, C 6/16; traps: A 0/3, B 3/3, C 0/3 (23 tasks). C − A −0.217 [−0.478, 0.043]; C − B 0.174 [−0.087, 0.435]; B − A −0.391 [−0.652, −0.087] |
Each difference is paired by task, with its 95% interval from the paired bootstrap (10,000 resamples, fixed seed 20261022, docs/specs/accuracy.md §5). The canned answers decide these numbers: arm A answers every other numeric item right by design, and arm B replays each item’s answer SQL, which reads the raw file with read_csv, so the locked sandbox refuses every one (SQL_REFUSED: the containment working) and arm B ends in a refusal. Arm A saw a sample of each file (the cut is stated below with its numbers).
Not run here (DEFERRED to the nightly runs, docs/progress/STATE.md “Deferred checks”): golden v1 on the configured Groq routes (bun run eval:real:fetch && bun run eval -- --suite golden --model configured); consistency over 5 runs on smoke with the configured model, cache off (bun run eval -- --suite smoke --repeat 5 --model configured); the ablation on a free model (bun run eval:ablation -- --arms A,B,C --suite smoke --model openai-compatible:groq:openai/gpt-oss-120b and the same with --suite golden); EIA-861M’s pinning and golden set.
Gaps (docs/progress/STATE.md “Known gaps carried forward”): hotel bookings C4 and C6 in the real tier; retail golden item G7 (no driver on anomaly findings); the held-out generated set’s drop (recall@5 75.0% against 96.0% on the development 30); the post-ship citation judge, deferred to M21.
Golden v1: baseline · dev split · 2026-10-22
Section titled “Golden v1: baseline · dev split · 2026-10-22”Items are agent-reviewed, not maintainer-reviewed. An item counts toward the bars only when its independent re-derivation agreed. The baseline is the scripted heuristic with no language model: it answers no follow-up question, so every ask item (traps included) is a miss, never n/a.
hotel-bookings
Section titled “hotel-bookings”Job completed in 25.6 s · 2/8 items pass · counted toward the bars: 2/7 numeric and categorical, 0/1 traps; 0 not counted.
| Item | Kind | Source | Pass | Expected | Got | Question |
|---|---|---|---|---|---|---|
| G1 | numeric | fact | yes | 1809324.75 | 1809324.7500000042 | What was room revenue (ADR × nights, bookings not cancelled) for arrivals in August 2016, both hotels together (EUR)? |
| G2 | numeric | fact | yes | 1970181.6 | 1970181.6000000036 | What was room revenue (ADR × nights, bookings not cancelled) for arrivals in August 2017, the last month in the data, both hotels together (EUR)? |
| G3 | numeric | ask | no | 41.73 | no answer (FakeModel has no scripted response for role=planner promptHash=24a1546f: Answer the question.) | What share of the city hotel’s bookings, in percent, were cancelled (all arrival dates)? |
| G4 | categorical | ask | no | Non Refund | no answer (FakeModel has no scripted response for role=planner promptHash=0307b5ac: Answer the question.) | Which deposit type has the highest cancellation rate, both hotels and all arrival dates together? |
| G5 | numeric | ask | no | 118669.91 | no answer (FakeModel has no scripted response for role=planner promptHash=4df2b94d: Answer the question.) | By how much did room revenue (ADR × nights, bookings not cancelled), both hotels together, change from April 2017 to May 2017 arrivals (May minus April, EUR)? |
| G6 | numeric | ask | no | 81.98 | no answer (FakeModel has no scripted response for role=planner promptHash=d5bd2436: Answer the question.) | What share of all bookings, in percent, came through the travel agent and tour operator channel (TA/TO), cancelled bookings included? |
| G7 | categorical | ask | no | Resort Hotel | no answer (FakeModel has no scripted response for role=planner promptHash=39ada550: Answer the question.) | Which hotel had more room revenue (ADR × nights, bookings not cancelled) for arrivals in August 2016, the resort hotel or the city hotel? |
| G8 | trap | ask | no | refusal | no answer (FakeModel has no scripted response for role=planner promptHash=b1d0731f: Answer the question.) | What was the gross operating profit margin of the two hotels in 2016? |
nz-card-spend
Section titled “nz-card-spend”Job completed in 5.3 s · 3/7 items pass · counted toward the bars: 3/6 numeric and categorical, 0/1 traps; 0 not counted.
| Item | Kind | Source | Pass | Expected | Got | Question |
|---|---|---|---|---|---|---|
| G1 | numeric | fact | yes | 11321.6 | 11321.6 | What was total card spending, all industry groups together, in December 2025 (NZD million)? |
| G2 | numeric | fact | yes | 9190.7 | 9190.7 | What was total card spending, all industry groups together, in September 2026, the latest month (NZD million)? |
| G3 | categorical | driver | yes | Durables | Durables | Which industry group contributed the largest decrease in NZD to the fall in card spending from March to April 2020? |
| G4 | numeric | ask | no | -3803.7 | no answer (FakeModel has no scripted response for role=planner promptHash=86b3b549: Answer the question.) | By how much did total card spending, all industry groups together, change from March 2020 to April 2020 (NZD million)? |
| G5 | numeric | ask | no | 29.015 | no answer (FakeModel has no scripted response for role=planner promptHash=acc46eaf: Answer the question.) | What share of 2025 card spending, in percent, was in the Consumables industry group? |
| G6 | numeric | ask | no | 1.25 | no answer (FakeModel has no scripted response for role=planner promptHash=e17aab3b: Answer the question.) | By what percentage was total card spending, all industry groups together, in September 2026 above September 2025, the same month a year earlier? |
| G7 | trap | ask | no | refusal | no answer (FakeModel has no scripted response for role=planner promptHash=0020d93a: Answer the question.) | What was the average value of a card transaction in 2025? |
online-retail-ii
Section titled “online-retail-ii”Job completed in 11.8 s · 2/8 items pass · counted toward the bars: 2/7 numeric and categorical, 0/1 traps; 0 not counted.
| Item | Kind | Source | Pass | Expected | Got | Question |
|---|---|---|---|---|---|---|
| G1 | numeric | fact | yes | 140447.11 | 140447.10999999937 | What were net sales (quantity × price over all lines, cancellations, postage, fees and manual adjustments included) in November 2010? |
| G2 | numeric | fact | yes | 201 | 201 | How many distinct invoices (cancellation invoices included) were there in December 2010? |
| G3 | numeric | ask | no | 145480.93 | no answer (FakeModel has no scripted response for role=planner promptHash=8881ba09: Answer the question.) | What were net sales (quantity × price over all lines, cancellations, postage, fees and manual adjustments included) in the best-selling month of 2011? |
| G4 | numeric | ask | no | 37570.84 | no answer (FakeModel has no scripted response for role=planner promptHash=c3216eac: Answer the question.) | By how much did net sales (quantity × price over all lines, cancellations, postage, fees and manual adjustments included) change from October 2011 to November 2011? |
| G5 | numeric | ask | no | 13.7 | no answer (FakeModel has no scripted response for role=planner promptHash=3d621d90: Answer the question.) | What share of 2010 net sales, in percent, came from the ten customers with the highest 2010 net sales? Lines with no customer ID stay in the total. |
| G6 | numeric | ask | no | 3.58 | no answer (FakeModel has no scripted response for role=planner promptHash=024601a7: Answer the question.) | By what percentage were November 2011 net sales (quantity × price over all lines, cancellations, postage, fees and manual adjustments included) above November 2010, the same month a year earlier? |
| G7 | categorical | driver | no | United Kingdom | no driver | Which country accounted for most of the fall in invoices from November to December 2010? |
| G8 | trap | ask | no | refusal | no answer (FakeModel has no scripted response for role=planner promptHash=9a70101b: Answer the question.) | What was the gross margin on 2011 sales? |
Consistency: baseline · smoke suite · 5 runs · 2026-10-22
Section titled “Consistency: baseline · smoke suite · 5 runs · 2026-10-22”Response cache: n/a (scripted model, no provider calls). Datasets: coffee_roaster-v1, restaurant-v3, saas-v2, tool_hire-v1.
- Run 1: 4/4 jobs finished, 0 provider calls (0 cached)
- Run 2: 4/4 jobs finished, 0 provider calls (0 cached)
- Run 3: 4/4 jobs finished, 0 provider calls (0 cached)
- Run 4: 4/4 jobs finished, 0 provider calls (0 cached)
- Run 5: 4/4 jobs finished, 0 provider calls (0 cached)
| Measure | Jobs where every run agrees | Bar (§4.7, over 5 runs) | Pass |
|---|---|---|---|
| (a) numbers: every fact value identical | 4/4 (100.0%) | 100% | yes |
| (b) findings: the same top-5 findings | 4/4 (100.0%) | 95% | yes |
| (c) driver: the same top driver per finding | 4/4 (100.0%) | 95% | yes |
| (d) cause grade: the same grade per finding | 4/4 (100.0%) | 90% | yes |
| (e) action: the same top recommendation scenario | 4/4 (100.0%) | 90% | yes |
Disagreements
Section titled “Disagreements”None: every run agreed on every measure.
Ablation: smoke, baseline (2026-10-22)
Section titled “Ablation: smoke, baseline (2026-10-22)”Harness check, not an engine-vs-model comparison. Arms A and B gave canned FakeModel answers built from the answer keys, and arm C ran the scripted planner; the numbers show the harness works, not what a model does.
Jobs: coffee_roaster-v1, restaurant-v3, saas-v2. Context window for arm A: 32768 tokens.
| Measure | A (model alone) | B (model + SQL) | C (the engine) |
|---|---|---|---|
| Planted recall@5 (with driver) | 40% (2/5) | 60% (3/5) | 100% (5/5) |
| Driver accuracy | 0% (0/3) | 100% (3/3) | 100% (3/3) |
| Distractor FP rate | 60% (3/5) | 0% (0/5) | 0% (0/5) |
| Tokens in / out | 30 / 15 | 60 / 30 | 1570 / 785 |
| Time (s) | 0.1 | 0.4 | 14.5 |
Arm A saw a sample. coffee_roaster-v1: The input is cut to fit the model’s context window: t_t1 (sales.xlsx#Orders) shows rows 1–843 of 11564. Answers about the whole data rest on this sample. restaurant-v3: The input is cut to fit the model’s context window: t_t1 (sales.xlsx#Orders) shows rows 1–907 of 31389. Answers about the whole data rest on this sample. saas-v2: The input is cut to fit the model’s context window: t_t1 (orders.csv) shows rows 1–896 of 14628. Answers about the whole data rest on this sample.
| Paired difference | Tasks | Mean | 95% interval | Adds accuracy? |
|---|---|---|---|---|
| C − A | 5 | 0.600 | [0.200, 1.000] | interval above 0, but canned answers: no claim |
| C − B | 5 | 0.400 | [0.000, 0.800] | not shown (interval includes 0 or is below) |
| B − A | 5 | 0.200 | [-0.600, 1.000] | not shown (interval includes 0 or is below) |
Bootstrap: 10,000 paired resamples, fixed seed (accuracy.md §5).
Per task
Section titled “Per task”| Arm | Job | Task | Score | Got |
|---|---|---|---|---|
| C | coffee_roaster-v1 | P1 | 1 | found at rank 1 |
| A | coffee_roaster-v1 | P1 | 0 | found at rank 1, wrong driver |
| B | coffee_roaster-v1 | P1 | 1 | found at rank 1 |
| C | restaurant-v3 | P1 | 1 | found at rank 1 |
| C | restaurant-v3 | P3 | 1 | found at rank 2 |
| A | restaurant-v3 | P1 | 0 | found at rank 1, wrong driver |
| A | restaurant-v3 | P3 | 1 | found at rank 2 |
| B | restaurant-v3 | P1 | 1 | found at rank 1 |
| B | restaurant-v3 | P3 | 0 | not found |
| C | saas-v2 | P1 | 1 | found at rank 1 |
| C | saas-v2 | P2 | 1 | found at rank 2 |
| A | saas-v2 | P1 | 0 | found at rank 1, wrong driver |
| A | saas-v2 | P2 | 1 | found at rank 2 |
| B | saas-v2 | P1 | 1 | found at rank 1 |
| B | saas-v2 | P2 | 0 | not found |
Ablation: golden, baseline (2026-10-22)
Section titled “Ablation: golden, baseline (2026-10-22)”Harness check, not an engine-vs-model comparison. Arms A and B gave canned FakeModel answers built from the answer keys, and arm C ran the scripted planner; the numbers show the harness works, not what a model does.
Jobs: hotel-bookings, nz-card-spend, online-retail-ii. Context window for arm A: 32768 tokens.
| Measure | A (model alone) | B (model + SQL) | C (the engine) |
|---|---|---|---|
| Numeric accuracy | 50% (8/16) | 0% (0/16) | 38% (6/16) |
| Categorical accuracy | 100% (4/4) | 0% (0/4) | 25% (1/4) |
| Trap pass rate | 0% (0/3) | 100% (3/3) | 0% (0/3) |
| Unsupported-number rate | 47% (9/19) | n/a (0 of 0) | — |
| … without source cells | 53% (10/19) | n/a (0 of 0) | — |
| Citation validity (SQL re-run) | — | n/a (0 of 0) | — |
| Tokens in / out | 230 / 115 | 460 / 230 | 380 / 190 |
| Time (s) | 7.7 | 1.6 | 42.6 |
Arm A saw a sample. hotel-bookings: The input is cut to fit the model’s context window: t_t1 (hotel-bookings.csv) shows rows 1–394 of 119390. Answers about the whole data rest on this sample. nz-card-spend: The input is cut to fit the model’s context window: t_t1 (nz-card-spend-by-industry.csv) shows rows 1–476 of 2568. Answers about the whole data rest on this sample. online-retail-ii: The input is cut to fit the model’s context window: t_t1 (online_retail_II.sample10.csv) shows rows 1–603 of 105797. Answers about the whole data rest on this sample.
The unsupported-number support set is the golden answers, arm C’s facts for the same job and the source cells of the columns each item names; the second row leaves the cells out. It is wider than “the data or the golden set”, which favours arms A and B.
| Paired difference | Tasks | Mean | 95% interval | Adds accuracy? |
|---|---|---|---|---|
| C − A | 23 | -0.217 | [-0.478, 0.043] | not shown (interval includes 0 or is below) |
| C − B | 23 | 0.174 | [-0.087, 0.435] | not shown (interval includes 0 or is below) |
| B − A | 23 | -0.391 | [-0.652, -0.087] | not shown (interval includes 0 or is below) |
Bootstrap: 10,000 paired resamples, fixed seed (accuracy.md §5).
Per task
Section titled “Per task”| Arm | Job | Task | Score | Got |
|---|---|---|---|---|
| C | hotel-bookings | G1 | 1 | 1809324.7500000042 |
| C | hotel-bookings | G2 | 1 | 1970181.6000000036 |
| C | hotel-bookings | G3 | 0 | no answer |
| C | hotel-bookings | G4 | 0 | no answer |
| C | hotel-bookings | G5 | 0 | no answer |
| C | hotel-bookings | G6 | 0 | no answer |
| C | hotel-bookings | G7 | 0 | no answer |
| C | hotel-bookings | G8 | 0 | no answer |
| A | hotel-bookings | G1 | 1 | 1,809,324.75 |
| A | hotel-bookings | G2 | 0 | 2,167,200 |
| A | hotel-bookings | G3 | 1 | 41.73 |
| A | hotel-bookings | G4 | 1 | Non Refund |
| A | hotel-bookings | G5 | 1 | 118,669.91 |
| A | hotel-bookings | G6 | 0 | 90 |
| A | hotel-bookings | G7 | 1 | Resort Hotel |
| A | hotel-bookings | G8 | 0 | answered: It will be about 512,000. |
| B | hotel-bookings | G1 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G2 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G3 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G4 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G5 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G6 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G7 | 0 | refused: SQL_REFUSED |
| B | hotel-bookings | G8 | 1 | refused: no column answers it (SQL_REFUSED) |
| C | nz-card-spend | G1 | 1 | 11321.6 |
| C | nz-card-spend | G2 | 1 | 9190.7 |
| C | nz-card-spend | G3 | 1 | Durables |
| C | nz-card-spend | G4 | 0 | no answer |
| C | nz-card-spend | G5 | 0 | no answer |
| C | nz-card-spend | G6 | 0 | no answer |
| C | nz-card-spend | G7 | 0 | no answer |
| A | nz-card-spend | G1 | 1 | 11,321.6 |
| A | nz-card-spend | G2 | 0 | 10,110 |
| A | nz-card-spend | G3 | 1 | Durables |
| A | nz-card-spend | G4 | 0 | -4,184 |
| A | nz-card-spend | G5 | 1 | 29.02 |
| A | nz-card-spend | G6 | 0 | 1 |
| A | nz-card-spend | G7 | 0 | answered: It will be about 512,000. |
| B | nz-card-spend | G1 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G2 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G3 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G4 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G5 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G6 | 0 | refused: SQL_REFUSED |
| B | nz-card-spend | G7 | 1 | refused: no column answers it (SQL_REFUSED) |
| C | online-retail-ii | G1 | 1 | 140447.10999999937 |
| C | online-retail-ii | G2 | 1 | 201 |
| C | online-retail-ii | G3 | 0 | no answer |
| C | online-retail-ii | G4 | 0 | no answer |
| C | online-retail-ii | G5 | 0 | no answer |
| C | online-retail-ii | G6 | 0 | no answer |
| C | online-retail-ii | G7 | 0 | no driver |
| C | online-retail-ii | G8 | 0 | no answer |
| A | online-retail-ii | G1 | 1 | 140,447.11 |
| A | online-retail-ii | G2 | 0 | 221 |
| A | online-retail-ii | G3 | 1 | 145,480.93 |
| A | online-retail-ii | G4 | 0 | 41,328 |
| A | online-retail-ii | G5 | 1 | 13.7 |
| A | online-retail-ii | G6 | 0 | 4 |
| A | online-retail-ii | G7 | 1 | United Kingdom |
| A | online-retail-ii | G8 | 0 | answered: It will be about 512,000. |
| B | online-retail-ii | G1 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G2 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G3 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G4 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G5 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G6 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G7 | 0 | refused: SQL_REFUSED |
| B | online-retail-ii | G8 | 1 | refused: no column answers it (SQL_REFUSED) |