Draft. Idea proposition, no paper yet, 2026-09-04. Working name: stale-policy-retrieval
A site can keep an old policy sentence on one page while a reworded replacement lives on another, with no date a crawler can trust. We do not yet know how often that happens, or whether question-answering systems then serve the old one. This poster states the question and how we will test it.
The failure, by example
Illustrative. The page layout is lib.usm.edu's; the day counts are invented. Neither page carries a date a crawler can trust, and the stale page is the newer one by page date.
The question
The question is about the corpus, so it applies to any kind of system, and the diagnostic is reported per system. "Retrieval miss versus resolution miss" is the retrieval-only case of stages 2 and 3.
Nearby work assumes one of three things: both versions are among the retrieved passages (ConflictRAG, DRAGged), candidates carry reliable timestamps (FRESCO, recency rerankers), or facts are already aligned as triples (MemStrata, Graphiti). Help centers and institutional sites break all three at once.
Why every kind of system serves the stale one
| Hybrid retrieval | FAQ matches the question word for word; the policy sentence ranks low and is never shown. |
|---|---|
| Recency reranker | Prefers the FAQ: a typo fix last month gave it the newer page date. |
| Conflict detector | Sees no conflict, because the policy sentence was not retrieved. |
| Search agent | Opens the FAQ, reads 120 days, stops. Nothing it saw suggests a contradiction exists. |
| Long context | Has both statements, undated, and must decide from content alone. Whether it can is an experiment. |
| Graph memory | Extracts "borrow period" and "loan period" as different edges; the old one is never invalidated. |
Walk-through of the ①-example against each system kind (PROPOSITION §1). These are the four kinds the paper will evaluate, plus two variants.
What the literature already shows, in nearby settings
Numbers as reported by the cited papers; none reproduced here. The difference from our setting in every row: their misleading documents are wrong or adversarial and both sides were retrieved. Ours is a once-true statement, undated, with the replacement not retrieved.
Two things the first crawl already shows
Data: library-chatbot crawl of 2026-09-04. 1,782 HTML pages, 66 PDFs, 1,429 unique bodies. 55 pages extracted empty: 4 LibCal widgets, 14 forms or embeds, the rest image-only exhibit pages. In-degree counts one vote per linking page. This is the "not ingested" stage in the wild; no stale value has been caught yet, because there is one snapshot.
Hypothesis
1. Nothing prompts a search against the first fit. Once a system finds a statement that answers the question, it keeps searching within the frame that statement set, and rarely against it. Say "prefers confirming evidence", not "stops early": the same literature reports agents that over-search. More rounds deepen the same line.
2. The replacement is worded like a policy. Even a deliberate search for a contradiction has to bridge the gap between "when are my books due" and "loan periods". If so, better search alone will not close it.
Mechanism to test: pages written in the user's language (FAQ, how-to) match questions well and go stale because nobody owns them. Policy pages match poorly and are the ones that get updated. The stale copy is found first because it was written to be found.
A good timestamp fixes stage 3 only. A page date is not a fact date, and when the current value was removed the right answer is "unknown", which no date rule produces.
What gets measured, and what kills it
Corpus, from real edit histories
GitHub Docs and Mozilla Support (git and revision history), a Wayback-archived help center, and lib.usm.edu from daily snapshots. The evaluated system gets a flat crawl. The timeline stays sealed. Wikipedia is a control only.
Three conditions, plus oracle dates
Easy: the fact changed in place. Hard: it changed on one page and a reworded copy went stale on another. Stale-only: the current value was removed, so the right answer is "unknown". Oracle statement-level dates measure how much timestamps alone close.
Four kinds of system
Hybrid retrieval (plus recency, FRESCO-style, ConflictRAG-style variants), long context (isolates stage 3), agentic search (plus falsification prompting and evidence-aware termination), one graph or memory system. Then a simple baseline: an offline index that aligns reworded statements about one fact and marks the older one replaced.
Kill criteria, set before any data
Gate 1, first week: across two edit-history corpora, fewer than 25% of changed mutable facts had a reworded copy on another page. Then the hard case is rare and there is no paper. Also: an existing reranker or detector resolves over 85% of the hard condition; most failures land at stage 3; or falsification prompting alone closes it.
TL;DR
What we ask
When a stale statement and its reworded replacement sit on different pages with no trustworthy dates, does the system serve the current value, and if not, did the current statement fail to be ingested, fail to reach the model, or lose the final choice?
What we have
A charter with dated kill criteria, a prior-art table, and one crawl of lib.usm.edu in which the policy text and the hours are the two things that do not ingest while their question-shaped restatements do. No result yet.
What happens next
Gate 1: clone two documentation repos, find commits that change a mutable fact, and grep the old value across the repo at that commit. If reworded cross-page copies are under 25% of changes, the product ships and the paper does not.