Verified results
These are real runs against a live LLM (DeepSeek), not simulated or hand-picked. They're not yet competitive with the numbers the bigger players in this space publish — we're saying that plainly rather than letting a nicer headline number do the talking.
LoCoMo
84 QA pairs from LoCoMo-10 (conv-26, first 150 turns).
| Recall@10 (did the right evidence turn make it into context) | 52.4% |
| Accuracy — strict substring match | 19.0% |
| Accuracy — LLM-judged (wording-tolerant) | 44.0% |
The trade-off we didn't hide
A prior version scored lower on accuracy but higher on recall (62.0%). A fix that folds dates into extracted predicates (so "when did I say X" is answerable) traded some retrieval recall for a real +14.3-point accuracy gain — and instead of stopping at "accuracy went up," we went and found out why recall dropped: the date-folding instruction sometimes misfires on content-free small talk ("Thanks!" gets folded into "thanked on 2023-07-03"), and those extra low-value triples crowd real ones out of the fixed top-10 retrieval window.
LongMemEval
xiaowu0162/longmemeval-cleaned, oracle variant, first 10 of 500 questions.
| Recall@10 | 100% (10/10) |
| Accuracy — strict substring match | 30% |
| Accuracy — LLM-judged | 80% |
Why these numbers need context, not just a headline
- Only 10 of 500 questions were run — each ingests ~27 turns on average (real extraction + generation + judge calls through a proxy with real latency); the full dataset would take tens of hours. This is a real run, not a mock, but it's a small sample and shouldn't be read as generalizing to the full dataset.
- All 10 happen to be temporal-reasoning type — the dataset also has a multi-session type, and taking the first 10 in file order wasn't a stratified sample.
- Recall@10 = 100% is largely an artifact of the oracle variant's design — it pre-filters each question's haystack down to only the relevant sessions (no distractor sessions), considerably easier than a real deployment's memory store. This isn't the same task as the full (non-oracle) benchmark and shouldn't be compared directly against numbers other products report on that harder variant.
- Strict-match accuracy (30%) is far below LLM-judged accuracy (80%) — the same pattern as LoCoMo: substring matching undercounts answers that are correct but worded differently.
Raw data and run scripts on GitHub →
Full methodology (what's verified vs. still open) →