Poster: Same Ranking, Different Winner

Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

Sugam Panthi, Rabab Abdelfattah

The problem

Rescoring the same ranked output changes the winning system

Memory systems store one piece of evidence in several forms: the raw turn, a source-linked observation, and a canonical fact. Benchmarks silently pick which form earns retrieval credit, so conclusions flip while the ranked output is held completely fixed. We call it target non-invariance.

Worked example: the rank-1 derived memory answers the query, but Raw scores a miss because the raw-turn ID is absent, while Source and Canonical score hits. — The retrieved item never moves. Raw scores it a miss (no raw-turn ID); Source and Canonical score the same item a hit.

Method

A deterministic script rescores saved output against three targets

TIAP pipeline: construct Raw, Source, and Canonical scoring targets; rescore saved ranked outputs on shared subsets without rerunning retrieval; measure score shifts, winner flips, and design flips, plus a 1,902-case semantic audit.

We construct three targets for each query, then a deterministic script rescores the same saved outputs: Raw (exact turn), Source (same source family), and Canonical (serving form). The script never reruns retrieval. LoCoMo & LongMemEval-S, 4 retrievers, transfer: Mem0, MemoryOS.

Changing only the credited target moves every score

The margin swings six-fold, +0.068 to +0.420. Source beats Canonical on every LoCoMo subset; the order reverses on LongMemEval-S. Data: paper Table 2.

Pick a target, pick a different memory-store design

Raw

Source

Canonical

Lexical, F1 vs F5

Lexical, F1 vs F8

Lexical, F5 vs F8

all-MiniLM, F1 vs F5

all-MiniLM, F1 vs F8

all-MiniLM, F5 vs F8

Data: paper Table 5.

Transfer: full-run nDCG@60

It follows real architectures, in no clean direction

Rescoring published Mem0 and MemoryOS runs preserves non-invariance, but the ordering flips between datasets: two monotonic, two non-monotonic. On Mem0 with LongMemEval-S, raw-vs-canonical switching changes MRR on 466 of 470 queries.

Relaxed credit is fully justified only 29.2% of the time

Cases where Raw scored a miss but Source/Canonical scored a hit, audited for whether the credited memory actually answers the query. Human validation: Fleiss κ = 0.83 on a 115-case subset. Permissive targets are not free, but the failures do not rescue the raw-only default either. Data: paper Table 6.

Takeaway

Report the target, or the benchmark reports noise

Canonical is the primary target when the deployed system serves transformed memories.
Always report Raw alongside it as a strict floor.
Source family is a stress test, not a standalone leaderboard target.
Rescore on shared subsets whenever ontology coverage differs across systems.
Treat winner flips across targets as a result, not noise to smooth away.

TL;DR

What we did

TIAP rescored saved ranked outputs (retrieval never rerun) under three scoring targets (Raw, Source · Canonical) on LoCoMo and LongMemEval-S, plus Mem0 and MemoryOS transfer runs.

What we found

The target choice alone changes per-query nDCG on 83.4–94.0% of shared queries, flips which system wins, and reverses store-design recommendations; relaxed credit is fully justified only 29.2% of the time.

What you should do

A memory benchmark score is meaningless without its target. Declare it, report Canonical with a Raw floor, and treat cross-target disagreement as a finding; otherwise the benchmark measures your ontology choice, not your retriever.

This poster online

Accepted at EMNLP Findings 2026