We Ran mem0's Benchmark Against Ourselves. Here's Everything.
Out of the box versus out of the box, Yohanun scores 62.4% to mem0's 48.8% on LoCoMo under one strict judge. This is the full method, every number we measured, and the null results we're publishing alongside the wins.
Anton Mannering
Founder & Chief Architect
A while ago I asked our own system a blunt question: are we actually better than the alternatives, or do we just believe we are? This post is the answer, with the full method, every number, and two null results we could have quietly left out.
The headline, then the homework
On LoCoMo, the long-term conversational memory benchmark introduced by the mem0 team, judged by one strict model across 1,540 questions:
| Category (n) | Yohanun, as shipped | mem0, its defaults | Delta |
|---|---|---|---|
| Overall (1,540) | 62.4% | 48.8% | +13.6 |
| Single-hop (841) | 77.3% | 58.1% | +19.2 |
| Multi-hop (282) | 45.0% | 37.2% | +7.8 |
| Temporal (321) | 46.4% | 39.6% | +6.8 |
| Open-domain (96) | 36.5% | 32.3% | +4.2 |
"As shipped" means exactly that. We provisioned a fresh tenant, ingested the benchmark's ten conversations through our public API, and touched nothing. The platform's defaults did the rest, including automatically summarizing all 272 conversation sessions, because that is simply what it does for every customer now. No configuration on either side of the table. That is the sentence this post exists to earn, and the rest of it is the homework.
Method
Both systems answered the same 1,540 questions (LoCoMo categories 1 through 4; we excluded the adversarial category, as mem0's own paper does). Both were answered and judged by the same frozen model, Claude Haiku 4.5, with identical prompts. Yohanun ran through its full production retrieval path: the access gate, lifecycle filtering, temporal weighting, all of it live, the same code that serves customers. mem0 ran as its open-source release with its default configuration and its own extraction pipeline building its store.
One thing you cannot do with these numbers: compare them to anyone's published absolutes, including mem0's own paper, which reports mid-60s for their system. Different judges score very differently; the same mem0 that publishes mid-60s on its own harness measures 48.8 on ours, and our numbers would move on theirs. Deltas transfer across judges; absolutes do not. Every number in this post lives on one scale, and we never mix scales. Stricter judging, for what it's worth, widens the gap in our favor, for a reason we find meaningful: our retrieval returns raw conversational grain with exact wording and dates, which survives strict matching, where extraction-based answers tend to be paraphrase-shaped.
Neither system was tuned for this benchmark. What you see is our shipped default against theirs: the same retrieval depth, window expansion, and automatic session summaries that every new tenant gets, doing on the benchmark exactly what they do in production.
The fairness ledger
Comparisons like this usually hide their asymmetries. Here are ours, both directions.
In mem0's favor: their optional extras (a BM25 keyword leg and spaCy entity extraction) were not installed, so they ran on their vector core. In our favor, and symmetrically: our own entity graph was disconnected from retrieval in every headline number here. Both systems competed retrieval-core against retrieval-core, with assists off on both sides. We did not run Zep or other systems; one honest comparison seemed worth more than three rushed ones. And mem0's paper reports that their graph-augmented variant adds about 1.5 points in their own hands, which brings us to the part of this post we suspect you'll trust most.
The null results
We built graph-hop retrieval: an entity graph, extracted by an LLM at session consolidation, that nominates memories connected to entities named in a query, one hop out, with the nominations flowing through the same access gate and scoring as everything else. The hypothesis was that it would lift multi-hop questions, our weakest category relative to our own single-hop. We pre-registered the decision rule before measuring: it ships on by default only if multi-hop moves.
Three controlled runs on the identical tenant, judge, and questions, with only the mechanism varying, including one where we rebuilt the graph at much finer grain between runs. Every cell landed within noise of the baseline, and a tempting temporal bump in the second run was erased by the third. We flag single-run readings as exactly that, and the same discipline is the reason to trust the headline above.
Why doesn't the graph help here? Our working theory, which we find more interesting than a win: conversational text is self-contextualizing. People restate who and what they're talking about, so the relationships a graph encodes are already present in the text, and therefore already in the embeddings. Our session summaries then concentrate those relationships further; they are, in effect, joins materialized at write time. A read-time join-finder arriving after write-time join materialization has nothing left to find. Graphs earn their keep where relationships exist in no single retrievable text: citation networks, authority structures, org walls. We use ours for exactly those things elsewhere in the platform. For conversational recall, on this benchmark, its measured contribution is zero, and we ship it accordingly: enabled as infrastructure whose value we believe compounds with tenant history, instrumented so real usage will tell us, and claimed as worth nothing today.
Caveats that travel with every claim here
One benchmark, and benchmarks are narrow. Small categories carry real noise; the open-domain +4.2 on 96 questions is suggestive, not established. LoCoMo is mem0's benchmark, which we consider a point in favor of using it, but no benchmark measures what long-lived memory is really for. And the numbers say nothing about the things we actually consider our moat, which no public benchmark tests: access decided before generation, per-read audit trails, memory that revives when you return to old work.
Reproduce it, or argue with it
The harness is a phased, resumable script: provision, ingest, answer, judge, report, with the model seats and retrieval depth as environment variables. If you want to reproduce this, argue with the method, or run your own system through the same judge, email us and we'll share it along with the exact prompts. We'd genuinely like to see a third party run this, including one that proves us wrong somewhere; we've published our own nulls, so one more would hardly break the habit.
And if what you actually care about is the part benchmarks can't see, the governance underneath is documented at yohanun.com/governance, and the product this memory serves is Guilds.