Findings of EMNLP 2026, arXiv:2605.24060, cs.IR

Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

Sugam Panthi and Rabab Abdelfattah

TIAP: a fixed-output audit of scoring-target choice in conversational-memory evaluation

Scan for paper

The problem

Rescoring the same ranked output changes the winning system

Memory systems store one piece of evidence in several forms: the raw turn, a source-linked observation, and a canonical fact. Benchmarks silently pick which form earns retrieval credit, so conclusions flip while the ranked output is held completely fixed. We call it target non-invariance.

Worked example: the rank-1 derived memory answers the query, but Raw scores a miss because the raw-turn ID is absent, while Source and Canonical score hits. — The retrieved item never moves. Raw scores it a miss (no raw-turn ID); Source and Canonical score the same item a hit.

Method

A deterministic script rescores saved output against three targets

TIAP pipeline: construct Raw, Source, and Canonical scoring targets; rescore saved ranked outputs on shared subsets without rerunning retrieval; measure score shifts, winner flips, and design flips, plus a 1,902-case semantic audit.

We construct three targets for each query, then a deterministic script rescores the same saved outputs: Raw (exact turn), Source (same source family), and Canonical (serving form). The script never reruns retrieval. LoCoMo & LongMemEval-S, 4 retrievers, transfer: Mem0, MemoryOS.

83.4–94.0% of shared queries change per-query nDCG when only the credited target changes.

Core result: nDCG@60, fixed ranked outputs (n = 899–963 / 299–332)

Changing only the credited target moves every score

The margin swings six-fold, +0.068 to +0.420. Source beats Canonical on every LoCoMo subset; the order reverses on LongMemEval-S. Data: paper Table 2.

Engineering consequence: fixed subset, n = 453

Pick a target, pick a different memory-store design

Raw

Source

Canonical

Lexical, F1 vs F5

Lexical, F1 vs F8

Lexical, F5 vs F8

all-MiniLM, F1 vs F5

all-MiniLM, F1 vs F8

all-MiniLM, F5 vs F8

Data: paper Table 5.

Transfer: full-run nDCG@60

It follows real architectures, in no clean direction

Rescoring published Mem0 and MemoryOS runs preserves non-invariance, but the ordering flips between datasets: two monotonic, two non-monotonic. On Mem0 with LongMemEval-S, raw-vs-canonical switching changes MRR on 466 of 470 queries.

Relaxed credit is fully justified only 29.2% of the time

Cases where Raw scored a miss but Source/Canonical scored a hit, audited for whether the credited memory actually answers the query. Human validation: Fleiss κ = 0.83 on a 115-case subset. Permissive targets are not free, but the failures do not rescue the raw-only default either. Data: paper Table 6.

Takeaway

Report the target, or the benchmark reports noise

Canonical is the primary target when the deployed system serves transformed memories.
Always report Raw alongside it as a strict floor.
Source family is a stress test, not a standalone leaderboard target.
Rescore on shared subsets whenever ontology coverage differs across systems.
Treat winner flips across targets as a result, not noise to smooth away.

TL;DR

What we did

TIAP rescored saved ranked outputs (retrieval never rerun) under three scoring targets (Raw, Source · Canonical) on LoCoMo and LongMemEval-S, plus Mem0 and MemoryOS transfer runs.

What we found

The target choice alone changes per-query nDCG on 83.4–94.0% of shared queries, flips which system wins, and reverses store-design recommendations; relaxed credit is fully justified only 29.2% of the time.

What you should do

A memory benchmark score is meaningless without its target. Declare it, report Canonical with a Raw floor, and treat cross-target disagreement as a finding; otherwise the benchmark measures your ontology choice, not your retriever.

This poster online