I’ve cited the same headline numbers throughout this series: zero tasks for no-memory, eight for vector RAG, ten for Borg, with stale-fact rates of 0%, 11.5%, and 2.5%. Numbers like that are easy to publish and easy to game. This post is about how the benchmark was built and, just as important, what its own authors say it does not prove.
The reproducible harness and the ground-truth task definitions live in the repo, under bench/. You can rerun it. That’s the first property that matters: if you can’t reproduce it, don’t trust it.
The tasks are real work
The ten tasks come from real production episodes in cloud-infrastructure engineering, generalized for public use: incident response, schema migration, architecture review. They’re spread deliberately across task types. Four are debug tasks, like diagnosing API gateway 502 errors or tracing message-bus unauthorized errors. Three are architecture tasks, like designing event-bus remediation or MCP gateway authentication. Two are compliance tasks, like an event-sync scoping decision or a CI security-audit fix. One is a writing task.
The spread is the point. Each task type exercises a different kind of memory: debug leans on prior episodes and procedures, architecture on semantic facts and entities, compliance on approval decisions and change boundaries. A benchmark that was all “recall this one fact” would flatter any storage system. Rooting the tasks in multi-step engineering work means success depends on assembling the right context, which is the thing actually being tested.
Three conditions, one evaluator
The benchmark runs three conditions against the same ten tasks, the same model, the same grader, and the same ground-truth labels:
Condition A is no memory. The model answers each task with only the task prompt, no prior-session context. The floor.
Condition B is top-10 vector RAG. Conversation logs are chunked, embedded with the same model Borg uses, and the ten most similar chunks are prepended verbatim to every prompt. This is the real baseline, the approach most teams reach for, not a strawman.
Condition C is Borg, the full pipeline: classify intent, retrieve across facts, episodes, and graph, rank, and compile a token-budgeted package. The compiled output is injected, not raw matches.
The only variable across conditions is the memory system. Identical tasks, identical scoring, identical ground truth. That isolation is what lets you attribute the difference to the approach.
What gets measured
The benchmark tracks five metrics, not one. Task success is binary: does the final answer cover every element the grader labeled required. Retrieval precision is the share of injected context that actually got used in the answer. Stale-fact rate is the fraction of compiled facts that were superseded or deprecated at query time. Irrelevant rate is the share of injected context the grader marks off-topic. Knowledge coverage is the fraction of ground-truth facts that made it into the package.
Five metrics make it much harder to cherry-pick. A system can win one number by accident. Winning across success, precision, freshness, relevance, and coverage at once is a pattern, not a fluke.
What the numbers actually say
Task success goes zero, eight, ten. RAG closes most of the distance from nothing to working; Borg closes the rest, the last two tasks where similarity retrieval pulls back something plausible but wrong.
Retrieval precision: 6%, then 81%, then 91.3%. Stale-fact rate: 0%, 11.5%, 2.5%, the 78% reduction. Irrelevant content drops 61% versus vector RAG, and knowledge coverage improves to 90.8% from 78.2%.
The most important detail is what doesn’t explain the gain. Borg is not winning by spending fewer tokens. Context sizes are comparable between RAG and Borg, roughly 2,806 tokens against 3,026. The improvement comes from what’s included, not from a smaller or larger budget. Same order of tokens, better tokens.
What the bench does not prove
This is the part most product pages bury, and Borg’s puts it in a labeled section. Stated plainly, the caveats are:
The results are self-reported, from a single evaluator model, on a single domain. They have not been externally reproduced. Ten tasks is a small sample, so confidence intervals are wide and the absolute numbers should be read as directional, not definitive. The same LLM family answers the tasks and grades them, which risks circularity; an independent judge model would make the claim stronger. And while the harness and task definitions are committed, the raw model outputs and grader artifacts are kept separately and not in the repo.
That candor is itself the credibility argument. A benchmark that lists its own weaknesses is telling you it’s trying to measure something rather than sell something. The seeds are public so you can run your own tasks and see whether the gap holds for your work. A benchmark’s job isn’t to end the argument, it’s to make the argument inspectable.
The final post steps back from measurement to the ideas underneath the whole system: the research compiled memory is actually built on.
Part 5 of a 6-part series on building memory for AI coding agents. Next: The science behind it.