Memory Benchmark Reporting Checklist

A benchmark number means nothing without its setup.

Eight rules, three verdicts, one machine-readable result file. MBRC does not tell you what to build or which benchmark to run. It specifies what has to be written down before a number can be compared to anyone else's.

Same systemsame benchmark · one quarter

Vendor figure, top model as reader and judge 90.2

Vendor's own preprint 71.2

Unified re-measurement, 12 systems 48.0

Three accuracy figures for one memory system on one benchmark within one quarter. None of them is a lie — they ran under different readers, judges and question subsets, and nobody wrote down which. The third carries its own caveat, see R7.

Version1.0 Statusopen for comment Licencereuse and adapt with attribution MaintainerPatric Zeller

01 — Why

A checklist, not a benchmark

The field does not lack benchmarks. It lacks a shared way of writing results down.

LongMemEval, BEAM, Memora, MemoryAgentBench, LoCoMo and a dozen others were published inside two years. The consequence is visible every quarter: vendors publish figures that contradict each other, re-measurements contradict the vendors, and every comparison collapses into an argument about setup.

Medicine solved a version of this with reporting standards rather than method mandates. CONSORT and PRISMA never told anyone how to run a trial — they specified what a published report must contain. MBRC borrows that shape deliberately: neutral on architecture, neutral on which benchmark you pick, opinionated only about disclosure.

Adoption cost is the design constraint. The rules below are the reasoning; the form in section 04 is the standard in practice. Fill it in, publish the file next to your number, and the number becomes comparable.

02 — The eight rules

Each rule exists because of a documented failure

Not because of a principle. Sources are named so you can check whether the failure was real.

R1Fix the reader, vary the memory

A memory benchmark measures memory only if the answering model is identical across all arms: same model, same version, same temperature, same context budget, same answer prompt. Repeating results with a different reader is a separate run, reported separately — not the same number.

R2Pin the judge, calibrate it, publish the calibration

LLM-as-a-judge is the de-facto standard, and an uncalibrated judge produces figures with no information in them. Before any run: pin judge model, version and prompt; run an adversarial set of at least 100 deliberately wrong but plausible answers, of which the judge must reject at least 95 %; double-score at least 100 samples against human annotation and report both the agreement rate and Cohen's κ.

If the adversarial threshold fails, the judge is unfit. That is not a finding about the system under test.

Why the threshold is set where it is

A judge configuration in common use for LoCoMo scoring accepted 62.81 % of deliberately wrong but topically plausible answers (Penfield Labs, 08.04.2026). Penfield operates a competing system and is not an independent auditor — the analysis is citable because its underlying data is published and checksum-verifiable, which is the standard this checklist applies to itself.

Memora (arXiv:2604.20006) is the only memory benchmark publishing judge–human agreement at all, at 88.3 %. The κ values in that paper span more than one comparison axis, which is why the form below asks which axis a κ belongs to.

R3Score nuggets, not right or wrong

Decompose each answer into atomic criteria, score each at 0 / 0.5 / 1, average. Materially less noisy than a binary verdict, and it gives partial credit where partial credit is what actually happened.

R4At least ten runs, report the dispersion

A single run is not a result. Report median, standard deviation and number of runs. For two-arm comparisons use a paired test across tasks (Wilcoxon), not a t-test across means.

In a re-analysis of LoCoMo category comparisons, 56 % were statistically indistinguishable from noise when measured once.

R5Accuracy without cost is half a result

Every accuracy figure ships with three others: tokens per query, split into input, cached input and output; latency p50 and p95, split into retrieval time and end-to-end; and context reduction against what full text would have cost.

A system reaching 95 % by pushing 100,000 tokens into context has moved the problem, not solved it. Where no language model sits in the evaluated path, R5 has no object — say so rather than leaving the fields blank.

R6Measure modules separately, not just end to end

A weak aggregate cannot be attributed. Report retrieval, extraction, routing and maintenance as separate figures, then the full system with a fixed reader. The module figures are the diagnostic ones; the end-to-end figure is the comparable one.

R7Never compare against a foreign blog figure

A comparison holds only if reader model, judge, question subset and setup are shared and published. Everything else is marked not comparable in the report — explicitly devalued, not quietly omitted.

This rule cuts both ways. A re-measurement landing far below a vendor figure is a comparability problem, not a truth claim, and deserves the same disclosure scrutiny — including whether an author of the re-measurement has a system in the test field.

R8Not measurable means not measurable

Three outcomes, never two. A criterion that could not be measured is never reported as passed, and never as breached with a zero. Where a metric has no object for a given unit at all, that is a fourth state and also not a failure.

03 — Verdicts

Three outcomes, per criterion, never aggregated

A composite score lets a clean result in one dimension pay for a broken one in another. Report each criterion on its own.

The permitted outcomes of a single criterion under MBRC 1.0
Outcome	Meaning
Passed	Measured under the disclosed setup, met its stated threshold.
Breached	Measured under the disclosed setup, missed its threshold. A finding for that criterion, not for the system as a whole.
Not measurable	The criterion applies and would be informative, but the measurement failed or could not be carried out. Never collapsed into either of the other two.
No object	The criterion has no subject matter for this unit — the metric is undefined here rather than unmeasured. Reported, not judged.

Why undefined values are never counted as zero

Excluding undefined values from an aggregate without counting them separately rewards the broken system: the worse the retrieval, the more queries drop out of scoring. Count them, and declare the metric not measurable for that unit once they exceed a third of the queries.

04 — The reporting form

This is the standard in practice

Fill it in, generate the file, publish it alongside your number.

Fields marked * are required — including limitations and competing interest, which is why the form will not complete without them. Everything happens in your browser; nothing is transmitted anywhere.

MBRC result recordschema mbrc/1.0

System under test

System name *

Version or build *

Commit or artifact id

Run date *

Configuration affecting results *

Benchmark

Benchmark *

Subset or split *

Questions *

Exclusions and why

R1 — Reader

Reader model *

Reader version *

Temperature *

Context budget, tokens *

Answer prompt identical across arms? *

R2 — Judge and calibration

Judge in the scoring path? *

Judge model and version

Judge prompt version

Adversarial rejection rate, %

Adversarial set size

Judge–human agreement, %

Cohen's κ

κ comparison axis

If no judge sits in the scoring path, say so. Computed metrics are stronger evidence than judged ones and should be labelled as such.

R3 and R4 — Scoring and dispersion

Scoring scheme *

Runs per cell *

Aggregation *

Dispersion reported *

Test for arm comparison

Statistical unit

R5 — Cost alongside accuracy

Does R5 apply? *

Tokens per query, input

Tokens, cached input

Tokens, output

Retrieval latency p50 / p95

End-to-end latency p50 / p95

Context reduction vs full text

R6 — Module breakdown

Retrieval

Extraction

Routing

Maintenance and forgetting

Full system, fixed reader *

R8 and determinism

Criteria passed *

Criteria breached *

Not measurable *

Embedding model and version *

Reranker and version *

Instance state before the run *

Warm-up runs

Raw data availability *

Disclosure

Competing interest *

Foreign figures cited, and their status

Limitations * — a report without stated limits is not a report

Nothing generated yet.

mbrc-result.json

05 — Who runs this

The obvious objection, answered first

Competing interest

MBRC is maintained by Patric Zeller, who also builds NEXUS — a long-term memory system sitting in exactly the category this checklist is used to evaluate. That is a competing interest, and hiding it would be the same failure the checklist exists to prevent.

The reason to publish anyway: independence is not what makes a measurement trustworthy — reproducibility is. The most-cited critical re-analysis in this field came from a competitor and is quoted precisely because its data is published and checksum-verifiable. The same standard applies here.

Every NEXUS figure is published as a self-measurement, on a separate page, with a completed MBRC record attached. None of it appears in this document.

The rules above were written to be architecture-neutral. Dimensions where NEXUS is structurally unusual — bi-temporal correctness, formal reasoning, deletion and tenant isolation compliance — are deliberately not part of this checklist. They are a separate proposal, to be argued separately, once the reporting baseline holds.

Corrections

If a rule, a cited figure or its stated conditions is wrong here, send it. It gets fixed, the change is recorded, and the person who found it is named.

info@patric-zeller.de