Verified results

These are real runs against a live LLM (DeepSeek), not simulated or hand-picked. They're not yet competitive with the numbers the bigger players in this space publish — we're saying that plainly rather than letting a nicer headline number do the talking.

LoCoMo

84 QA pairs from LoCoMo-10 (conv-26, first 150 turns).

Before the date-folding fix After the date-folding fix

View as a table

	Before the date-folding fix	After the date-folding fix
Recall@10	61.9%	52.4%
Accuracy (strict)	11.9%	19.0%
Accuracy (LLM-judged)	29.8%	44.0%

The trade-off we didn't hide

A prior version scored lower on accuracy but higher on recall (62.0%). A fix that folds dates into extracted predicates (so "when did I say X" is answerable) traded some retrieval recall for a real +14.3-point accuracy gain — and instead of stopping at "accuracy went up," we went and found out why recall dropped: the date-folding instruction sometimes misfires on content-free small talk ("Thanks!" gets folded into "thanked on 2023-07-03"), and those extra low-value triples crowd real ones out of the fixed top-10 retrieval window.

By question category (after the fix)

View as a table

	Accuracy (LLM-judged)
Single-hop fact	50.0%
Temporal reasoning	50.0%
Multi-hop association	50.0%
Open-domain	55.9%
Adversarial (unanswerable)	15.0%

LongMemEval

xiaowu0162/longmemeval-cleaned, oracle variant, first 10 of 500 questions.

100.0% Recall@10

30.0% Accuracy (strict match)

80.0% Accuracy (LLM-judged)

Why these numbers need context, not just a headline

Only 10 of 500 questions were run — each ingests ~27 turns on average (real extraction + generation + judge calls through a proxy with real latency); the full dataset would take tens of hours. This is a real run, not a mock, but it's a small sample and shouldn't be read as generalizing to the full dataset.
All 10 happen to be temporal-reasoning type — the dataset also has a multi-session type, and taking the first 10 in file order wasn't a stratified sample.
Recall@10 = 100% is largely an artifact of the oracle variant's design — it pre-filters each question's haystack down to only the relevant sessions (no distractor sessions), considerably easier than a real deployment's memory store. This isn't the same task as the full (non-oracle) benchmark and shouldn't be compared directly against numbers other products report on that harder variant.
Strict-match accuracy (30%) is far below LLM-judged accuracy (80%) — the same pattern as LoCoMo: substring matching undercounts answers that are correct but worded differently.

Raw data and run scripts on GitHub →
Full methodology (what's verified vs. still open) →