LoCoMo self-run
notes

Headline

Overall accuracy

Multi-signal retrieval (lexical/entity + RRF fusion + MMR), session-level extraction, evidence-aware judging. Same question set and judge stack as v0.2 — not a different dataset.

Method

How we measured

Fixed stack, disclosed end to end — so numbers can be compared fairly across our own releases.

Dataset

LoCoMo-10 (non-adversarial)

Questions

1540 (adversarial category excluded)

Harness

mem0 open eval harness (memory-benchmarks-style)

Extract

gpt-4o-mini

Answer

gpt-4o-mini

Judge

gpt-4o-mini

Embeddings

text-embedding-3-small

Recall budget

6,500 tokens · k=150

v0.2 store

InMemoryBeliefStore (accuracy focus)

v0.3 store

Postgres + pgvector (production store shape)

Telemetry

v0.3 on Postgres

Recall p50 442ms / p95 913ms / mean 526ms (includes embedding API); packed context mean 1,252 tokens, max 3,213 (cap 6,500).

Packed context stays well under the 6,500 cap (mean ~1.3k tokens vs mem0 published mean ~7k) while accuracy moves up — useful for cost, not just score.

Categories

By question type

Grey = v0.2 · Red = v0.3.

Product suite

Our wedge

Staleness/conflict harness: mem01 5/5. Same harness, mem0 OSS scored 2/5.

Internal staleness/conflict product suite (location flip, preference flip, job change, multi-fact, scope isolation). Not LoCoMo — the failure mode we optimize for.

Honesty

What we are not
claiming

• mem0’s published ~92.5% LoCoMo figure uses a stronger gpt-4o-class answer/judge stack. Our self-run uses gpt-4o-mini throughout for cost and reproducibility.
• Single full run per version; LLM judge noise is roughly ±2 points on 1,540 questions.
• Two conversations dipped slightly vs v0.2 while eight improved — overall still +2.6 pts.
• v0.2 used in-memory store; v0.3 used Postgres. Accuracy is treated as store-independent; latency claims are from the Postgres path only.