Fall Through on Doubt: Pre-Registered Two-Domain Evidence for a Deterministic Serve Layer in Front of a Local LLM

Abstract

A language model asked about a user's private record has two failure modes: it invents an answer, or — if well calibrated — it declines almost everything. Retrieval-augmented generation (RAG) narrows the gap but keeps the defining defect: when it fails, it still answers. We present and evaluate a deterministic serve layer that, placed in front of a local LLM, answers bounded-domain record questions directly from a curated knowledge graph — and falls through to the model on any doubt, at every stage, carrying its near-miss facts as grounding. The layer is stated as a seven-point substrate contract that any system can satisfy; we describe one instantiation.

The evaluation is pre-registered throughout: verdict bands were frozen before the exam fixtures existed, workloads and knowledge graphs are hash-pinned in git, per-cell outcomes were predicted in writing from the shipped code before every run, and both baseline arms ran under criteria frozen before the arms were built. Across two synthetic domains — a domotic household and a medication-and-care record chosen for grammatical distance — the layer served 376 answers with zero wrong (a count, never a pooled rate: the domains are never pooled, and the count sums the three measured legs, domain 1 plus domain 2 zero-shot and post-growth, which put the same questions to two configurations of the layer), over-routed 0 controls on every leg (0/40 in domain 1, Wilson upper bound 8.8%; 0/40 on each of domain 2's two legs), and separated 84 of 84 pre-authored near-neighbor question pairs — where retrieval over the same facts in the same words asserted between 39 and 44 wrong answers in domain 1 and between 55 and 63 in domain 2, each a range spanning the frozen judge and a lenient bound (§4.6), and conflated 14 pairs. Zero-shot on the second domain, the layer served 36 questions — all correct — and refused the rest, with the fall-throughs sitting exactly where its vocabulary tables end; after a human wrote 159 vocabulary rows it reached exactly its domain-1 score, cell for cell, at 100% served-precision. The adaptation-cost criterion contained a designed falsifier, and it fired: two non-table code changes were required, and we quote the pre-named FAIL sentence verbatim. Against dense retrieval the correctness comparison is a statistical tie in domain 1 (p = 0.694 under the frozen judge; p = 1.000 under the lenient bound, whose point estimate straddles the layer) and an established win in domain 2 (p = 0.0052) — a lead robust in direction but, for the dense retriever, sitting at the edge of significance under that bound (p = 0.0817), a sensitivity we quantify with one symmetric audit applied identically to every arm of both exams, which moves every baseline and leaves the layer unmoved. Supplying the declined facts as grounding flips fall-through answers from wrong to right in both domains (exact McNemar p < 0.001 and p = 0.003). A served answer returns in about a millisecond at roughly three orders of magnitude (~10³×) less energy than the model call it displaces, metered with hardware counters under a pinned power profile.

The full text — methods, tables, figures, appendices — lives in the PDF.

The journalistic first read: The Assistant That Refuses to Guess · Questions, replications, disagreements: contact@ame-labs.eu