MBRC · reporting checklist

Memory Benchmark Reporting Checklist

A benchmark number means nothing without its setup.

Ten rules, one verdict per criterion, one machine-readable result file. MBRC does not tell you what to build or which benchmark to run. It specifies what has to be written down before a number can be compared to anyone else's.

An independent proposal by its author. Not issued, endorsed, accredited or reviewed by any standards body, journal or conference.

One product nameone benchmark · three protocols
Vendor research page, no visible date reader and judge both gpt-5.4, 500 questions, retrieval depths stated1 90.2
The same vendor's preprint, 20 Jan 2025 reader gpt-4o, judge GPT-4o, subset LongMemEval_S2 71.2
Unified re-measurement of twelve systems, 23 Jun 2026 system version, subset and question count not stated3 48.0
The two dated figures are seventeen months apart, and each is correctly reported for the protocol it ran under. They still cannot be read against each other.

What this card shows

Three defensible measurements of one product name. Two of the three document their protocol in some detail — more than this field usually manages, and more than the reader of a single headline number would guess. They are still not comparable: seventeen months separate the two that carry a date, the third shows none, the reader models belong to different generations, and no source states which version of the system was measured. That is a reporting-convention problem, not a vendor problem.

The same card could be drawn for our own system, and we expect it would look no better. The standard of scrutiny applied here to a vendor figure applies equally to a re-measurement that lands far below one — see R7. Every figure on this page is listed with its source, its date and the conditions it was measured under in section 07; a figure without those three things does not belong on a page that asks others to supply them.

Version1.0 Statusopen for comment LicenceCC BY 4.0 · schema CC0 1.0 DOI10.5281/zenodo.22878931 MaintainerPatric Zeller

01 — Why

A checklist, not a benchmark

The field does not lack benchmarks, and since early 2026 it no longer lacks reporting frameworks either. What is missing is one that knows what a memory system can fail at.

LongMemEval7, BEAM9, Memora5, MemoryAgentBench10, LoCoMo11 and a dozen others were published inside two years. The consequence is visible every quarter: vendors publish figures that contradict each other, re-measurements contradict the vendors, and every comparison collapses into an argument about setup. The card at the top of this page is one instance of it, and it is not the interesting one — the interesting part is that all three of those numbers were reported in good faith.

Independent work has reached the same conclusion in print: memory leaderboards are not interpretable without the full protocol — ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, judge choice.33 MBRC is an attempt to write that protocol down once, so that it stops being rediscovered one paper at a time.

What already exists

A new reporting format has to justify itself against the ones already in use, because a sixth format that duplicates five others raises the cost of compliance without creating comparability. Here is the field as of August 2026, and what each of them leaves open for a measurement of a memory system.

Existing reporting frameworks, and the gap each leaves for this class of system
Framework What it settles What it leaves open here
NIST AI 800-2, public draft21 Protocol design, judge quality, contamination, uncertainty quantification, run logging, qualified claims. Practice 3.2 recommends an interoperable reporting schema. Recommends a schema without specifying one, and is deliberately general: no control arm against a reader with no memory, no ingestion cost, no staleness.
Evaluation Cards22 An interpretive layer over published results, with reporting completeness and score comparability as explicit signals, backed by a live registry. Benchmark- and model-centred. It reads results that exist; it does not specify the conditions of a single run of a system under test.
Agentic Benchmark Checklist20 Task validity, outcome validity and thirteen reporting checks for agent benchmarks, with ten benchmarks audited against them. No normed machine-readable result record, and agent task completion rather than recall, staleness and forgetting over time.
Rollout Cards23 A reproducibility record for agent research: typed row schemas, versioned scoring scripts, a drops manifest. Episode evidence. Not the accuracy, cost and control-arm triple that decides whether a memory figure means anything.
BenchmarkCards24 Documentation of a benchmark: purpose, data, methodology, targeted risks, ethical and legal considerations. Documents the benchmark, not the run. Prompts, seeds, significance tests and judge configuration are out of scope by design.
Minimum Reproducibility Disclosure, 202532 An eight-dimension disclosure checklist for memory evaluation — temporal governance, freshness, leakage auditing, model and parameters, retrieval, memory settings, cost, limitations — with a YAML template, in the appendix of a survey. Disclosure without verdicts: no control arms, no outcome per criterion, no pinned checksums, no validator. To our knowledge no published evaluation has used it.
Agent Memory Leaderboard, August 202638 One evaluation contract for several dozen systems: fixed answer model, hashed datasets and pipeline, versioned submissions. Judge model and prompts undisclosed, no run counts or intervals published, and vendors on the board among the partners — the three things R2, R4 and R7 exist for.

The closest thing to a predecessor is not in the table. A 2025 survey of memory in language models carries, in an appendix, a "Minimum Reproducibility Disclosure" checklist — eight dimensions from temporal governance to memory settings, with a YAML template — and a call for a minimally sufficient evaluation card.32 It is a disclosure schema, and to our knowledge no published evaluation has used it. MBRC takes the same view of the problem and goes three steps further: a verdict per criterion instead of a disclosure, mandatory control arms, and a record a machine can reject. Two of its ten rules exist only because the subject is memory. R9 requires three control arms — no memory, full context, corrupted memory — because without them a memory figure cannot be distinguished from the reader's own knowledge or from a long context window. R10 requires the question set, the answer key and the judge prompt to be pinned by checksum, because in this corner of the field answer keys demonstrably change. Of the frameworks above, none asks for the control arms, and only the leaderboard pins its data by hash — for its own runs, not for anyone else reporting a figure.

Compatibility is a debt, not a claim: MBRC 1.0 states its own fields, and a field-by-field mapping onto NIST AI 800-2 §3.2 and onto the dataset metadata already mandated at one major venue is the first thing the next version owes.

Where the shape comes from, and where it does not

The shape is borrowed from medicine. CONSORT and PRISMA never told anyone how to run a trial; they specified what a published report must contain. That is what MBRC is: neutral on architecture, neutral on which benchmark you pick, opinionated only about disclosure.

The process is not borrowed, and claiming otherwise would be the first thing a reader should disbelieve. Those guidelines were built through Delphi rounds with hundreds of participants, multi-day consensus meetings, piloting, and explanation documents running to tens of thousands of words.26 This document has one author. The closer precedent outside medicine is REFORMS — thirty-two items agreed by nineteen researchers across five disciplines after a workshop, without a formal Delphi25 — and nineteen is still nineteen. So: this is a proposal, published so that it can be argued with, with the disclosure it asks of others applied to itself. Section 06 says what it does not have.

Adoption cost is the design constraint. The rules below are the reasoning; the form in section 04 is the standard in practice. Fill it in, publish the file next to your number, and the number becomes comparable.

02 — The ten rules

Each rule exists because of a documented failure

Not because of a principle. Sources are named so you can check both things: whether the failure was real, and whether the rule follows from it.

The ten rules in one screen. Each is set out with its reasoning below; if a rule is violated, the run is not evaluable and is not reported.
No.RuleBecause
R1Fix the reader, vary the memoryotherwise the comparison measures the language model, not the memory
R2Pin the judge, calibrate it in both directionsa rejection-only test is passed by a judge that rejects everything
R3Score nuggets, not right or wrongbinary verdicts hide partial correctness and add noise
R4Enough tasks first, then enough runsthe sampling unit is the task; repeating one run measures the serving stack
R5Accuracy without cost is half a resulttokens, latency and ingestion cost belong in the same table
R6Measure modules separatelyan aggregate does not say which part carries it
R7Never compare against a foreign blog figurediffering conditions; such figures are marked, not used
R8Not measurable means not measurablefour states, never two: passed, breached, not measurable, no object
R9Run the controls that can falsify the resultwithout a no-memory, full-context and corrupted-memory arm, the figure has no referent
R10Declare what the reader already knewpublic benchmarks age into training data, and answer keys get corrected

R1Fix the reader, vary the memory

A memory benchmark measures memory only if the answering model is identical across all arms: same model, same version, same temperature, same answer prompt, same random seed, same task order. Repeating results with a different reader is a separate run, reported separately — not the same number.

Measure every arm inside one window and say which. Hosted models change underneath a study; a comparison run two months apart is a comparison of two model versions wearing the names of two memory systems.

The context budget is a ceiling, identical across arms — not an identical fill. The distinction is not pedantry: if every arm had to place the same number of tokens in context, there would be no context reduction left to report under R5, and the full-context control arm of R9 could be trimmed at will to flatter the memory system.

R2Pin the judge, calibrate it in both directions

LLM-as-a-judge is the de-facto standard, and an uncalibrated judge produces figures with no information in them. Before any run, pin judge model, version and prompt. Then calibrate both directions, because a one-directional test is trivially passed:

  • Rejection — at least 150 deliberately wrong but plausible answers.
  • Acceptance — at least 150 independently verified correct answers.
  • Gate — the lower bound of the 95 % Wilson interval of both rates is at least 0.90.

Why both directions: a judge that rejects everything passes a rejection-only test at 100 %, and any judge can be pushed past a one-sided threshold by sharpening its prompt until it gets there. An over-strict judge does not fail loudly — it compresses the field and makes two unequal systems look alike.

Why an interval and not a bare count: at 95 correct rejections out of 100, the Wilson interval runs from 0.888 to 0.978, so the observation is compatible with a true rate of 89 %. Conversely a judge whose true rate is exactly 0.95 fails a "95 of 100" gate in 38 % of attempts. A threshold that decides by coin flip in its own neighbourhood is not a threshold.

Agreement with human annotation: at least 300 stratified samples, reported as the raw agreement rate, the full contingency table, and both Gwet's AC1 and Cohen's κ with bootstrap confidence intervals. κ on its own misleads here, because it corrects against a chance expectation computed from the observed marginals: a judge that accepts most answers can show κ near 0.04 while agreeing with the human 85 % of the time.13 Report the rate and the table, not only the coefficient.

Report as well, for the judge: swap consistency on pairwise judgements, the shift in verdict at ±50 % answer length, test-retest agreement over three repetitions of the same prompt, stability across two prompt rewordings, and whether judge and reader come from the same model family.12 A calibration holds only for the task categories it was measured on.

None of this is peculiar to memory. A 2026 analysis of judge-agreement reporting shows that protocol choices alone — the judgment scale, which cases are retained, how abstentions are handled, how verdicts are pooled — move a reported accuracy from 0.551 to 0.899 and carry κ across zero without changing a single verdict; it distils the remedy into a reporting checklist that makes agreement claims reconstructible and comparable.34 A second line of work estimates a judge's true-positive and false-positive rates on a small human-labelled calibration set and derives a corrected decision threshold from them — the same two directions, put to statistical use.35

If either gate fails, the judge is unfit. That is not a finding about the system under test.

What is at stake is measurable. In a published audit of LoCoMo scoring, a judge configuration in common use accepted 62.81 % of deliberately wrong but topically plausible answers. The same judge caught concrete factual errors — wrong name, wrong date — at around 89 %, so the failure is specific to vague-but-relevant text rather than general.4

We have not independently reproduced that result, and its publisher sells a memory product of its own rather than acting as an independent auditor. We cite it because the underlying data, the scripts and all 99 disputed cases are published and checksum-verifiable — which is the standard this checklist applies to itself.

On where the threshold comes from

The 0.90 lower bound is a convention, not a derivation. It sits below the highest judge–human agreement published for a memory benchmark that we are aware of — 88.3 %, reported by Memora5, with LongMemEval reporting agreement for its own evaluator as well7 — and far above the 62.81 % measured in the audit. Setting it as an interval bound rather than a point estimate is the part that is derived: it is what makes the gate insensitive to the sample size a reporter happens to choose.

A worked justification, with the operating characteristic of the gate at several sample sizes, belongs in the explanation document that accompanies the next version. Until it exists, treat the number as what it is: a defensible convention, published so that it can be argued with.

R3Score nuggets, not right or wrong

Decompose each answer into atomic criteria, score each at 0 / 0.5 / 1, average. Materially less noisy than a binary verdict, and it gives partial credit where partial credit is what actually happened.

Report precision over nuggets alongside recall. An average over the criteria a system hit, with no penalty for what else it said, rewards saying everything. Weight the criteria as well: a system that misses the central claim and catches five side details should not outscore the reverse.

State where the nugget set came from and pin it. If the model that generates the criteria is the model that matches them, the score is circular; if the set is pooled from the candidate answers, it inherits their blind spots. State also whether the average runs over nuggets or over tasks — the two give different numbers.

For ordering tasks, name the coefficient. Kendall tau-b and Spearman weight every position equally, which is the wrong shape for retrieval, where an error at rank 1 costs more than one at rank 20; rank-biased overlap is head-weighted and handles lists of different length. Declare the tie treatment either way.

R4Enough tasks first, then enough runs

A single run is not a result — but the run count is the smaller half of the problem. The sampling unit is the task, not the run.14

State the smallest difference you would act on, and the number of tasks that gives 80 % power to detect it. Paired across tasks, five percentage points needs roughly 280 tasks and three points roughly 790. A report that prescribes a run count and says nothing about the task count is regulating the wrong quantity; underpowered comparisons are the documented norm in this literature, not the exception.15

Then replicate: at least five complete replicates with documented, distinct random seeds and permuted ingestion order. Ten repetitions of an identical run at temperature zero mostly measure the batching non-determinism of the serving stack — in one published measurement, 1,000 identical requests at temperature zero produced 80 distinct outputs.16 Name what varies: seeds, ingestion order, index construction. Repeating what does not vary is not replication.

Report the mean with its standard error across tasks, clustered where questions are nested; the between-replicate dispersion separately from it; and every seed.

For two-arm comparisons use a paired test across tasks: a paired permutation test or a paired bootstrap, McNemar where outcomes are binary. Wilcoxon assumes a symmetric difference distribution and degrades badly on the ties that 0 / 0.5 / 1 scoring produces. Report the mean paired difference with its 95 % interval, not a p-value on its own.

Where several categories are compared, name the primary hypothesis in advance, control the false discovery rate across the rest, and report how many tests were run in total. At ten categories and α = 0.05, at least one false positive is more likely than none — 0.40.

None of these prescriptions is idiosyncratic. A 2025 survey of memory evaluation arrives at the same set — paired permutation or bootstrap tests, effect sizes, agreement coefficients with intervals — from the other side of the field.32

The reason this matters here: in a re-analysis of published LoCoMo results, 56 % of adjacent-pair per-category comparisons fell inside each other's 95 % Wilson intervals — indistinguishable from noise at a single measurement. Category sizes in that benchmark range from 96 to 841 questions, a ratio of 8.8, so in the smallest category two systems need a gap of more than 15 points before they can be told apart at all.6 Same provenance and same caveat as under R2: a market participant, not reproduced by us, published with its data.

R5Accuracy without cost is half a result

Every accuracy figure ships with the cost of producing it, on both sides of the system.

Query side. Tokens per query, split into input, cached input and output; latency p50, p95 and p99, split into retrieval time and end-to-end; and context reduction against what full text would have cost.

Ingestion side. Tokens and wall time to ingest, embed, compress, consolidate and index — once and recurring, per stored unit — plus storage and its growth over time. A system that spends a hundred times more on ingestion than on querying looks inexpensive in a query-only table, and the comparison against a full-context arm becomes meaningless.

The scale of the ingestion side is not hypothetical. A controlled harness study in August 2026 moved three widely used production systems out of its main comparison and into a separate cost appendix, because their pipelines carry substantially different auxiliary-model budgets from the rest of the field.36 A figure that hides that difference is not comparable to one that does not.

Conditions. A latency figure without its conditions is not a figure: hardware and region, concurrency and request rate, warm-up and cache-hit rate, the number of measurements per cell, and the measurement window. Below roughly 200 measurements per cell the upper confidence bound of a p95 is the observed maximum — report the count, or do not report the percentile.

A system reaching 95 % by pushing 100,000 tokens into context has moved the problem, not solved it. Where no language model sits in the evaluated path, R5 has no object — say so rather than leaving the fields blank.

R6Measure modules separately, not just end to end

A weak aggregate cannot be attributed. Report retrieval, extraction, routing and maintenance as separate figures, then the full system with a fixed reader. The module figures are the diagnostic ones; the end-to-end figure is the comparable one.

Where a system has no separable module of a given kind, that entry is no object under R8 — not a gap, and not a zero.

R7Never compare against a foreign blog figure

A comparison holds only if reader model, judge, question subset and setup are shared and published. Everything else is marked not comparable in the report — explicitly devalued, not quietly omitted.

This rule cuts both ways. A re-measurement landing far below a vendor figure is a comparability problem, not a truth claim, and deserves the same disclosure scrutiny — including whether an author of the re-measurement has a system in the test field.

R8Not measurable means not measurable

Four states, never two. A criterion that could not be measured is never reported as passed, and never as breached with a zero: not measurable is an outcome in its own right and carries no penalty. Where a criterion has no subject matter for the unit under test at all, that is the fourth state, no object, and it is likewise not a failure.

The two are kept apart because they are repaired differently. A not-measurable result is a gap in the run and can be closed by running it again properly. A no-object result is a property of what was tested and will not change. Collapsing either into a zero turns an absent measurement into a bad one, and penalises a system for a criterion the run could not establish. Quietly dropping it does the opposite: it rewards whoever measured least. Counting it as a state of its own is the only treatment that does neither.

R9Run the controls that can falsify the result

A setup that can only confirm confirms nothing. Three control arms, all mandatory, all reported whether or not they flatter the result:

  • No memory. The same reader, the same questions, no retrieved context at all. Whatever it scores is the floor the benchmark hands out for free. If the margin over this arm is small, the benchmark is measuring the reader's prior knowledge, not memory.
  • Full context. The same reader with the entire source text in context and no memory system. Wherever the corpus fits a current context window this arm is not optional: without it, every figure can be answered with "you measured context length". Peer-reviewed setups now run to ten million tokens, so the excuse that it cannot be done has a short remaining life.9
  • Corrupted memory. The same run with the store emptied, or filled with wrong entries. If accuracy does not fall, the memory is not being used and the number belongs to something else.

The informative quantity is the distance between the full system and these arms, not the absolute value. Report all three distances. Where a distance is smaller than the dispersion established under R4, say so plainly — that is a legitimate result, and under R8 it is reported as a finding rather than hidden as a rounding.

R10Declare what the reader already knew, and pin what you measured

Public benchmarks age into training data. State whether the question set is public and since when, how that relates to the reader model's training cutoff, and whether canary strings or a private holdout were used. Contamination in closed models is documented practice, not a hypothetical.18 R9's no-memory arm is the empirical half of this rule; this is the declared half.

Pin the material: semantic version and SHA-256 of the question set, of the gold answers, of the nugget set and of the judge prompt. Answer keys do get corrected — the audit cited under R2 found 99 score-corrupting errors in 1,540 questions, which puts the ceiling for a flawless system at 93.57 % rather than 100 %.4 A figure measured against a corrected key is not comparable to one measured against the previous key, and neither is wrong; they are simply different measurements, which is the whole subject of this document.

The case is already real. A revised LoCoMo released in April 2026 revised 337 of its question–answer pairs, set the count at 1,382 questions and changed the judge; figures on the revised set now circulate next to figures on the original, and the two are not the same measurement.37 In one survey of the field, 62 % of all benchmark–system pairs draw on LoCoMo and LongMemEval, so the question of which key was used is not a corner case.36

Keep the development set and the test set apart. Prompt engineering, threshold tuning and retrieval parameters belong to the development set. Report how many times the test set was evaluated, and report every run that was carried out, including the ones not used. Selective disclosure of runs and variants is a documented practice in public leaderboards.19

03 — Verdicts

Four states, per criterion, never aggregated

A composite score lets a clean result in one dimension pay for a broken one in another. Report each criterion on its own, in exactly one of four states: two of them are the outcome of a measurement, and two of them record that no measurement stands behind the entry.

The four permitted states of a single criterion under MBRC 1.0. A record may not use any other.
OutcomeMeaning
Passed Measured under the disclosed setup, met its stated threshold.
Breached Measured under the disclosed setup, missed its threshold. A finding for that criterion, not for the system as a whole.
Not measurable The criterion applies and would be informative, but the measurement failed or could not be carried out. Never collapsed into any of the others.
No object The criterion has no subject matter for this unit — the metric is undefined here rather than unmeasured. Reported, not judged.

How undefined values are handled

Never as zero, and never silently dropped. Excluding them from an aggregate without counting them rewards the broken system: the worse the retrieval, the more queries fall out of scoring. Counting them as zero does the opposite and penalises a system for a question the run could not put to it.

Report the metric in three forms instead — on the defined cases only, with every undefined case counted as a failure, and with every one counted as a success. The latter two are the identification bounds: the range the true value must lie in. Report the count and the share of undefined cases per arm and per category, and state whether the failures look independent of the result or correlated with it. In a memory system they are usually correlated, because a value goes undefined precisely where retrieval failed — which is the case in which ignoring them distorts most.

Above 10 % undefined, the point estimate is quotable only together with its bounds. There is deliberately no share above which a unit drops out of the report: a percentage cut-off has no methodological basis — the missing-data literature says so in its title17 — and a cut-off creates an incentive to get past it rather than to measure. What decides is the failure mechanism, not the share. For scale: at one third undefined and 80 % accuracy on the rest, the bounds run from 0.53 to 0.87, which is a third of the scale.

04 — The reporting form

This is the standard in practice

Fill it in, generate the file, publish it alongside your number.

Fields marked * are required. Some become required only once an earlier answer makes them apply — declare that no language model sits in the evaluated path and the whole cost block turns into no object rather than into empty fields. Most reports will leave at least one block that way, and that is the intended use.

This form runs entirely in your browser. What you type is never sent to this site or to anyone else — the JSON file is built locally on your device. You do not have to take that on trust: the page is served with a content policy that forbids it from opening any network connection at all and from submitting a form anywhere (connect-src 'none'; form-action 'none'), which your browser enforces whatever this page says. It stores nothing on your device either — no cookies, no local storage. Loading the page is an ordinary web request to the hosting provider, which records standard server log data; see the privacy notice linked at the foot of the page (German, legally binding, and an English translation).

MBRC result recordschema MBRC/1.0
System under test
Benchmark
R1 — Reader
R2 — Judge and calibration

If no judge sits in the scoring path, say so above and this block becomes no object. Computed metrics are stronger evidence than judged ones and should be labelled as such.

R3 and R4 — Scoring, power and dispersion
R5 — Cost alongside accuracy
R6 — Module breakdown
R9 — Control arms
R10 — Provenance and prior knowledge
R8 — Verdict states and undefined values
Determinism and availability
Disclosure
Nothing generated yet.
mbrc-result.json

      

    


    
A record is conformant with MBRC 1.0 when every field required for
    the branches it declares is present and non-empty. Conformance is self-declared. The schema at
    /MBRC/1.0/mbrc.schema.json and the checker at
    /MBRC/1.0/mbrc-validate.py — one file, no dependencies
    — let a machine reject an incomplete record. That is the whole of what they do: neither can
    tell whether the values in a record are true.

  


  
  

    
05 — Who runs this

    
The obvious objection, answered first


    

      
Competing interest

      
MBRC is maintained by Patric Zeller, who also builds NEXUS — a long-term memory system
      sitting in exactly the category this checklist is used to evaluate. That is a competing
      interest, and hiding it would be the same failure the checklist exists to prevent.

    


    
    
The reason to publish anyway: independence is not what makes a measurement
    trustworthy — reproducibility is. Two of the figures cited on this page come from a market
    participant rather than from an independent auditor, and they are cited regardless, because the
    data, the scripts and every disputed case are published and checksum-verifiable
    (sources 4 and 6). That is the standard this document asks
    of others, so it is the standard by which it asks to be judged itself.


    
No NEXUS figure appears in this document, and none will. When NEXUS figures
    are published they will be published as self-measurements, on their own page, each with a
    completed record attached, under these rules and with this disclosure. That page does not exist
    yet. When it does, it will be linked from here — and if it is ever linked without a record
    attached, this paragraph is the thing to hold up.


    
The rules above were written to be architecture-neutral. Dimensions where NEXUS
    is structurally unusual — bi-temporal correctness, formal reasoning, deletion and tenant
    isolation compliance — are deliberately not part of this checklist. They are a separate
    proposal, to be argued separately, once the reporting baseline holds.


    

      
Corrections

      
If a rule, a cited figure or its stated conditions is wrong here, send it. It gets fixed,
      the change is recorded, and the person who found it is named.

      
info@patric-zeller.de · or open an issue at github.com/patricznr1/mbrc

    

  



  
  

    
06 — Limitations

    
What this document cannot do

    
The form refuses a record without a limitations field. This is
    ours.


    
    
Everything in section 04 is self-declared, and self-declaration has a measured
    track record that is not good. In a study of twelve trials published in journals that require a
    reporting checklist, one selected the applicable checklist; of the six that got that far, one
    was fully consistent with its own published text.31
    At one large machine-learning conference, only 34 % of reviewers found checklist answers useful
    for their assessment.30 Model cards, which nobody is
    obliged to complete, are present on 44.2 % of models, with the evaluation section filled in
    15.4 % of the time and the limitations section 17.4 %.29
    And the evidence that reporting guidelines improve reporting is itself observational: the
    Cochrane review of CONSORT found no experimental evaluation among fifty, with fewer than 8 %
    adjusting for confounding.27


    
Two consequences are built into the form rather than argued away. Where a field
    can be checked by a machine it is a value, not a claim: commit hash, benchmark checksum, model
    identifier and version, seeds, run count, standard error, measurement counts. "We were careful
    about contamination" is not checkable; "the question set has been public since October 2024 and
    the reader's training cutoff is August 2025" is. And a record that cannot be validated is not a
    record, which is why the schema is published rather than described — a machine can reject an
    incomplete record without anyone's good faith being involved.


    

      
No enforcement, and no pretence of it

      
Every reporting standard that measurably changed behaviour did so through a gate: trial
      registration as a precondition for review, a camera-ready process, a desk-reject rule, a
      metadata check inside submission. Code submission at one conference rose from 40 % at
      submission to 74.4 % at camera-ready once the requirement sat inside the process rather than
      beside it.30

      
MBRC sits beside the process. It has no gate and will not acquire one by being written more
      firmly. The realistic path is to make the record a by-product of running an evaluation instead
      of a task that follows it. Until that exists, adoption rests on whether the rules are worth
      following — which is a weaker force than a gate, and should be recognised as one.

    


    
What version 1.0 does not have

    
Named plainly, because the alternative is that someone else names them:

    

    
Scope

    
MBRC governs how a measurement is reported. It does not say which benchmark is
    worth running, it certifies nothing, it produces no ranking, and a completed record is a
    self-declaration and not a seal. It is also written for a field that measures in English: no
    German-language memory benchmark exists, and every method named on this page evaluates English
    conversation.


    
The author's competing interest is stated in section 05 and
    is not repeated here — but it belongs in this list, and a reader who weighs the limitations
    above should weigh that one with them.

  


  
  

    
07 — Sources

    
Every figure on this page, with its conditions

    
A checklist that asks others for provenance owes its own.


    
    
Peer-reviewed — reviewed and published at a conference or in a
    journal. Preprint — published, not reviewed. Audit, open — not reviewed, but
    reproducible from published data and scripts. Primary source — a report by a laboratory
    with documented method, not independently reviewed. Standards draft — issued for public
    comment by a standards body. Vendor figure — a self-measurement by a party with an
    interest in the result; under R7 it is not comparable to anything measured differently.
    Where a source has an interest in the outcome, that is stated in the entry rather than left to
    the reader.


    

      
    
        
  1. Vendor research page. 90.2 % on LongMemEval (451 / 500). Its
        "Reproducibility notes" state reader and judge as gpt-5.4 with chain-of-thought grading, the
        retrieval depths, and the question counts for both benchmarks. The page shows no date; its
        embedded metadata gives a last-modified date of 27 August 2026, and archived copies show the
        figures and the notes unchanged since 14 June 2026. getzep.com/research, retrieved 29 August
        and 15 September 2026.Vendor figure
    2. 

        
  2. Rasmussen, Paliychuk, Beauvais, Ryan, Chalef: "Zep: A Temporal Knowledge
        Graph Architecture for Agent Memory", arXiv:2501.13956v1, 20 January 2025. 71.2 % with
        reader gpt-4o, judge GPT-4o using the question-type prompts from the LongMemEval paper,
        subset LongMemEval_S (Table 2).Preprint
    3. 

        
  3. Zhou et al. (Shanghai Jiao Tong University, Tsinghua University,
        MemTensor): "Are We Ready For An Agent-Native Memory System?", arXiv:2606.24775v1,
        23 June 2026. Twelve memory systems and two reference baselines under uniform conditions,
        decomposed into four modules. The 48.0 quoted here is that study's LLM-judge accuracy for
        the same system named in sources 1 and 2 (§4.1); the study does not state which version of
        that system was tested, which subset was used, or how many questions. One of the evaluated
        systems is operated by co-authors of the study. Its code repository pins the vendor's client
        library at version 2.22.1 and configures a truncated longmemeval_s subset at 150,000 tokens;
        the service version behind the client and the question count remain
        unstated.Preprint
    4. 

        
  4. Penfield Labs: LoCoMo audit, 8 April 2026. 99 score-corrupting errors in
        1,540 questions (6.4 %), theoretical scoring ceiling 93.57 %, and 62.81 % acceptance of
        vague-but-topical wrong answers by a commonly used judge configuration — against roughly
        89 % detection for concrete factual errors. Data, scripts and all 99 cases published under
        CC BY-NC 4.0 with SHA-256 provenance for the upstream dataset. Published under a name that
        also sells a memory product of its own (penfield.app); no individual authors are named on
        the publication or the repository, and no correction to the figures had been issued as of 15
        September 2026.Audit, open
    5. 

        
  5. Uddin, Shubham, Wang, Blanco, Baral (Arizona State University, Genies,
        University of Arizona): "From Recall to Forgetting: Benchmarking Long-Term Memory for
        Personalized Agents" (Memora), arXiv:2604.20006, 21 April 2026. A metric that penalises use
        of superseded information; 88.3 % average agreement between LLM and human judgment;
        CC BY 4.0.Preprint
    6. 

        
  6. Penfield Labs: benchmark proposal, 9 April 2026. Wilson 95 % intervals
        make 56 % of adjacent-pair per-category comparisons statistically indistinguishable;
        category sizes range from 96 to 841 questions. Same provenance and same caveat as source
        4.Audit, open
    7. 

        
  7. Wu, Wang, Yu, Zhang, Chang, Yu: "LongMemEval: Benchmarking Chat
        Assistants on Long-Term Interactive Memory", ICLR 2025, arXiv:2410.10813. 500 curated
        questions across five long-term memory abilities; the evaluator model and its prompts are
        specified in the paper and prescribed in the repository, and the paper reports the
        evaluator's agreement with human experts.Peer-reviewed
    8. 

        
  8. Wu et al. (UCLA): "LongMemEval-V2: Evaluating Long-Term Agent Memory
        Toward Experienced Colleagues", arXiv:2605.12493, 12 May 2026. Holds the downstream reader
        fixed across all arms — the precedent for R1.Preprint
    9. 

        
  9. Tavakoli, Salemi, Ye, Abdalla, Zamani, Mitchell (University of Alberta,
        UMass Amherst): "Beyond a Million Tokens" (BEAM), ICLR 2026, arXiv:2510.27246. 100
        conversations across 19 domains, scales to 10 M tokens, 0 / 0.5 / 1 nugget scoring, and
        Kendall tau-b for event ordering — the precedent for R3.Peer-reviewed
    10. 

        
  10. Hu, Wang, McAuley (UC San Diego): "Evaluating Memory in LLM Agents via
        Incremental Multi-Turn Interactions" (MemoryAgentBench), ICLR 2026, arXiv:2507.05257. Four
        competencies; no tested method masters all four, and selective forgetting is the hardest of
        them.Peer-reviewed
    11. 

        
  11. Maharana, Lee, Tulyakov, Bansal, Barbieri, Fang: "Evaluating Very
        Long-Term Conversational Memory of LLM Agents" (LoCoMo), ACL 2024, arXiv:2402.17753. The
        dataset audited in sources 4 and 6.Peer-reviewed
    12. 

        
  12. Zheng, Chiang et al.: "Judging LLM-as-a-Judge with MT-Bench and Chatbot
        Arena", NeurIPS 2023 Datasets and Benchmarks, arXiv:2306.05685. Documents position bias,
        verbosity bias and self-enhancement bias in LLM judges, and reports that strong judges reach
        agreement with human preference at about the level humans reach with each
        other.Peer-reviewed
    13. 

        
  13. Feinstein and Cicchetti: "High agreement but low kappa", Journal of
        Clinical Epidemiology 43 (1990), together with Gwet: "Computing inter-rater
        reliability and its variance in the presence of high agreement", British Journal of
        Mathematical and Statistical Psychology 61 (2008). The prevalence and bias paradoxes of
        Cohen's κ, and the AC1 coefficient proposed in
        response.Peer-reviewed
    14. 

        
  14. Miller: "Adding Error Bars to Evals: A Statistical Approach to Language
        Model Evaluations", arXiv:2411.00640. Standard errors, clustered standard errors and paired
        differences for model evaluations, with the question rather than the run as the sampling
        unit.Preprint
    15. 

        
  15. Card, Henderson, Khandelwal, Jia, Mahowald, Jurafsky: "With Little Power
        Comes Great Responsibility", EMNLP 2020. Typical comparisons in this literature are
        underpowered for the effect sizes they report.Peer-reviewed
    16. 

        
  16. Thinking Machines Lab: "Defeating Nondeterminism in LLM Inference",
        2025. Attributes run-to-run variation at temperature zero primarily to batch-size dependence
        in serving rather than to floating-point arithmetic; 1,000 identical requests produced 80
        distinct outputs.Primary source
    17. 

        
  17. Madley-Dowd, Hughes, Tilling, Heron: "The proportion of missing data
        should not be used to guide decisions on multiple imputation", Journal of Clinical
        Epidemiology 110 (2019).Peer-reviewed
    18. 

        
  18. Balloccu, Schmidtova, Lango, Dusek: "Leak, Cheat, Repeat: Data
        Contamination and Evaluation Malpractices in Closed-Source LLMs", EACL
        2024.Peer-reviewed
    19. 

        
  19. Singh, Hooker et al.: "The Leaderboard Illusion", arXiv:2504.20879,
        NeurIPS 2025. Documents selective disclosure of privately tested model variants and
        asymmetric data access on a public leaderboard.Peer-reviewed
    20. 

        
  20. Zhou et al.: "Establishing Best Practices for Building Rigorous Agentic
        Benchmarks" (Agentic Benchmark Checklist), arXiv:2507.02825. Nineteen authors across UIUC,
        Stanford, Berkeley, Yale, Princeton, MIT, MLCommons and the UK AI Safety Institute; ten
        checks on task validity, 23 on outcome validity and 13 on benchmark reporting. Ten widely
        used agent benchmarks were audited and all ten had reporting
        defects.Preprint
    21. 

        
  21. NIST: "Practices for Automated Benchmark Evaluations of Language
        Models", NIST AI 800-2, initial public draft, January 2026, Center for AI Standards and
        Innovation, DOI 10.6028/NIST.AI.800-2.ipd. Covers protocol design, LLM-as-a-judge,
        contamination, uncertainty quantification, run logging and qualified claims; Practice 3.2
        recommends an interoperable schema for evaluation
        reporting.Standards draft
    22. 

        
  22. EvalEval Coalition: "Evaluation Cards: An Interpretive Layer for AI
        Evaluation Reporting", arXiv:2606.09809, June 2026. Thirty-three institutions; four
        interpretive signals including reporting completeness and score comparability, with a live
        registry that listed 1,396 benchmarks and 501,917 results in its snapshot of 7 September
        2026.Preprint
    23. 

        
  23. Masters, Liu, Albrecht: "Rollout Cards: A Reproducibility Standard for
        Agent Research", arXiv:2605.12131, May 2026. Typed row schemas, versioned scoring scripts
        and a drops manifest for long-horizon agent runs.Preprint
    24. 

        
  24. BenchmarkCards: "Standardized Documentation for Large Language Model
        Benchmarks", arXiv:2410.12974, NeurIPS 2025 Datasets and Benchmarks, University of Notre
        Dame and IBM Research. Documents the benchmark itself; does not prescribe prompts, seeds,
        significance tests or judge configuration.Peer-reviewed
    25. 

        
  25. Kapoor, Narayanan et al.: "REFORMS: Consensus-based Recommendations for
        Machine-learning-based Science", Science Advances 10(18):eadk3452, 2024. Thirty-two items in
        eight sections, agreed by nineteen researchers across five disciplines after a workshop,
        without a formal Delphi process.Peer-reviewed
    26. 

        
  26. Moher, Schulz, Simera, Altman: "Guidance for Developers of Health
        Research Reporting Guidelines", PLoS Medicine 7(2):e1000217, 2010. The eighteen-step method
        behind the CONSORT family, including the Delphi round, the consensus meeting, piloting and
        the explanation-and-elaboration document.Peer-reviewed
    27. 

        
  27. Turner et al.: Cochrane methodology review MR000030, 2012. CONSORT
        endorsement against completeness of reporting across 16,604 randomised trials; no included
        evaluation used an experimental design and fewer than 8 % adjusted for
        confounding.Peer-reviewed
    28. 

        
  28. Biderman, Schoelkopf, Sutawika et al.: "Lessons from the Trenches on
        Reproducible Evaluation of Language Models", arXiv:2405.14782. Five reporting practices in
        Appendix D; section 3.2 calls for reporting standards
        explicitly.Preprint
    29. 

        
  29. Analysis of 32,111 model cards across 74,970 repositories,
        arXiv:2402.05160. 44.2 % of models carry a card at all; the evaluation section is filled in
        for 15.4 % of them and the limitations section for
        17.4 %.Preprint
    30. 

        
  30. Pineau et al.: "Improving Reproducibility in Machine Learning Research
        (A Report from the NeurIPS 2019 Reproducibility Program)", arXiv:2003.12206. Code submission
        rose from 40 % at submission to 74.4 % at camera-ready once the checklist sat inside the
        process; only 34 % of reviewers found the checklist answers useful for their
        assessment.Peer-reviewed
    31. 

        
  31. Study of twelve randomised trials published between 2016 and 2017 in
        journals that require a reporting checklist, Trials 19 (2018). One of the twelve selected
        the applicable CONSORT extension; of the six with a correct checklist, one was fully
        consistent with its own published text.Peer-reviewed
    32. 

        
  32. Zhang, Li, Song, Lu, Li, Yang, Li: "Memory in Large Language Models:
        Mechanisms, Evaluation and Evolution", arXiv:2509.18868, 23 September 2025. Survey. Section
        4.1.4 proposes a Minimum Reproducibility Disclosure checklist with eight dimensions —
        temporal governance, freshness, leakage auditing, model and parameters, retrieval and
        re-ranking, memory settings, resources and cost, limitations — and a YAML template; section
        4.1.5 asks for agreement coefficients with 95 % intervals, paired permutation or bootstrap
        tests and effect sizes. The closest predecessor to this checklist that we are aware
        of.Preprint
    33. 

        
  33. Sheverev, Finkelstein, Nikolenko: "Beyond Memory Leaderboards:
        Evaluating Scientific Memory as Budgeted Context Restoration", arXiv:2607.16848, 18 July
        2026. Eight systems and a no-retrieval baseline on two full-text benchmarks; concludes that
        memory leaderboards are not interpretable without the full protocol — ingestion
        granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit and
        judge choice.Preprint
    34. 

        
  34. Rao, Callison-Burch: "Agreement Metrics for LLM-as-Judge Evaluation:
        What to Report and Why", arXiv:2606.00093, v2 of 31 July 2026. Shows on a rubric benchmark
        with per-criterion human labels that protocol choice alone moves reported accuracy from
        0.551 to 0.899 and carries κ across zero without altering a verdict; distils a reporting
        checklist that makes agreement claims reconstructible and
        comparable.Preprint
    35. 

        
  35. Feng, Shen, Balashankar, Gerner-Beuerle, Rodrigues: "Noisy but Valid:
        Robust Statistical Evaluation of LLMs with Imperfect Judges", ICLR 2026, arXiv:2601.20913.
        Estimates a judge's true-positive and false-positive rates on a small human-labelled
        calibration set and derives a variance-corrected decision threshold with finite-sample
        error control.Peer-reviewed
    36. 

        
  36. Huang, Zhang, Wu, Chen, Jiang, Yang, Yang, Zou, Zhang, Wu, Wu, Chang, Yu,
        Liu, Caliskan: "Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory
        Agents", arXiv:2608.15008, 15 August 2026. Controlled harness over three backbone models and
        four benchmark suites; reports Zep, Mem0 and MemGPT in a separate auxiliary-cost analysis
        rather than the main comparison because their pipelines carry substantially different
        auxiliary-model budgets; its survey finds 62 % of all benchmark–system pairs drawing on
        LoCoMo and LongMemEval.Preprint
    37. 

        
  37. LoCoMo-Refined (mem-eval-suite), released 14 April 2026. 1,382 questions
        with 337 question–answer pairs revised through screening and human audit, judge Qwen3-14B,
        reported judge–human agreement rising from 43.67 % on the original to 86.33 %; CC BY-NC 4.0.
        The publisher is not identified in the repository.Primary source
    38. 

        
  38. Agent Memory Leaderboard, agentmemoryleaderboard.ai, first public list
        12 August 2026. Organised by a group of universities with memory vendors as partners,
        several of which appear on the board. Fixes the answer model, hashes datasets and pipeline
        code and versions submissions; judge model and prompts are not disclosed and no run counts
        or intervals are published. The system in sources 1 to 3 is not among the listed
        systems.Primary source
    39. 
      

    


    

      
How to cite, and how to correct

      
Memory Benchmark Reporting Checklist (MBRC) 1.0, Patric Zeller, 2026, CC BY 4.0, doi:10.5281/zenodo.22878931. All
      external figures were checked against their primary source on 29 August 2026 and again on 15
      September 2026; where a source
      could not be dated or its conditions could not be established, that is stated in the entry
      rather than omitted.


@misc{zeller2026mbrc,
  author       = {Zeller, Patric},
  title        = {Memory Benchmark Reporting Checklist ({MBRC}) 1.0},
  year         = {2026},
  month        = {9},
  howpublished = {\url{https://patric-zeller.de/MBRC}},
  doi          = {10.5281/zenodo.22878931},
  note         = {Version 1.0, published 15 September 2026. Licence CC BY 4.0.}
}

      
Changes

      
    
        
  • 21 September 2026 — DOI 10.5281/zenodo.22878931 registered for version 1.0 (Zenodo, release v1.0.0 of
        github.com/patricznr1/mbrc). An English convenience translation of the privacy notice is
        linked at /privacy, and corrections can also be filed as issues at github.com/patricznr1/mbrc.
        Changes by the maintainer; no external report.
    • 
        
  • 15 September 2026 — Version 1.0 published: ten rules, four verdict states, source
        list with evidence class per entry, result schema and validator under /MBRC/1.0/. External
        figures re-checked on 15 September 2026.
    • 
        
  • 5 August 2026 — an unannounced draft with eight rules was reachable at this
        address; it is superseded in full and no records were issued against it.
    • 
      

      
An earlier draft of this page was reachable from 5 August 2026. It differs from this
      version in its rules and in its record schema, it was never announced, and no records were
      issued against it — but it was public, so it is named here rather than quietly replaced.

      
If an entry here misstates a figure, its date or its conditions, send it to
      info@patric-zeller.de or open an issue at
      github.com/patricznr1/mbrc. It gets fixed, the change is
      recorded with its date, and the person who found it is named.