06-reference

alphasignal memorytax agent memory study

2026-10-25·reference·source: AlphaSignal·by Ben Dickson
agent-memorycontext-engineeringllm-economicsbenchmarks

Why this is in the vault

A controlled benchmark (MemoryTax, Shanghai Jiao Tong University) quantifies when a dedicated agent-memory layer earns back its cost versus just feeding an agent raw conversation history — directly relevant to how RDCO already architects sub-agent context handling.

Mapping against Ray Data Co

The single most concrete connection: AGENTS.md hard rule #4 ("route long artifacts through subagents — don't read them into parent context") is RDCO's own prior commitment to exactly the condition this study found cheapest and most accurate — agent-directed search over raw history, rather than full-context stuffing or a precomputed memory layer (Mem0/Zep-style extraction). The task this very note was produced under — one sub-agent reading one newsletter's full raw body and handing back only a filed note — is a live instance of the "Pi harness" condition in the study: 99% accuracy at $6.95 by selectively reading ~10k tokens, versus 95% at $17.84 for stuffing the full ~28.9k-token history into every call. RDCO picked the cheap, accurate path by default, without ever benchmarking it against the alternatives this study tests.

The second finding is a sharper warning: memory quality is capped by whichever model wrote it, not by whoever reads it later — a knowledge graph built by the weak model stayed weak (43%→45%) even when a far stronger model read it, but shot to 87% once the strong model built and read its own graph. That is the empirical case underneath "Workflow agent output integrity" (scratchpad contamination, false "verified" stamps) and the rule that one gate per chain must hit the primary source — a sloppy or under-resourced sub-agent's condensed output is not recoverable by a smarter downstream reader. It also argues against ever letting a cheap fan-out agent (e.g. Haiku-tier) write the memory artifact that an Opus-tier agent will later rely on without re-touching the source.

The core argument

AI agents handling long-running sessions (the newsletter example: a support agent across a six-month customer relationship) face a design choice every time a prior fact is needed — re-search the raw conversation, or query a structured memory system built in advance. MemoryTax tests seven variants of that choice, grouped into four families, against three models (GPT-4o-mini, a stronger "GPT-6 Luna," and Claude Opus 5.5) on one long multi-turn conversation from the LoCoMo benchmark:

Three findings stood out. First, model strength changes the economics: with the weakest model, Mem0 won on accuracy; with the strongest open model tested, full context won; Claude Opus 5.5 hit 95% either way, meaning memory extraction only helped the weak model and was a pure cost with no accuracy upside for the strong one. Mem0's upfront build cost (roughly comparable to one full-context pass) only pays back after several hundred repeated questions against the same stored history — a system queried once doesn't recoup it.

Second, memory is a lossy compression bounded by its author: reading someone else's extracted graph, even with a much stronger model, couldn't recover detail the weaker extractor dropped (a missing purchase date blocking a later warranty question, in the study's framing). Accuracy only jumped when the same strong model both wrote and read the memory.

Third, harness and model-directed search can beat both full context and precomputed memory on cost and accuracy simultaneously — the Pi harness reached 99% for $6.95 against full context's 95% for $17.84 on the same model, by processing roughly a third of the tokens per question. Harness choice alone (not just model) moved cost materially: on identical underlying models, Codex cost $25.41 versus Claude Code's $28.41 for comparable accuracy — echoing AlphaSignal's own earlier HarnessTax coverage.

The authors flag real limits — one conversation, one run per configuration, a GPT-4o-mini judge, and a history that fit inside every model's context window anyway — so the paper reads as a testing framework, not a definitive ranking. Their practical sequence for developers: start with full context and measure; add retrieval once context gets unwieldy; precompute memory only where repeated-query reuse economics justify the build cost; and always benchmark the full stack (accuracy, latency, tokens, cost amortized across reuse) rather than any one layer in isolation.

⚠️ Sponsorship

Sponsor: Datalab. The "From Datalab" item promoting the open-source Indra OCR 1.1 release (1M-parameter model, claimed 44 pages/sec on 32-core CPU, 15x faster than Tesseract) is a paid placement — confirmed by a second, separately-boxed "Sponsor Logo" ad unit later in the same email linking to the identical tracked URL. The speed/accuracy claims in that item are vendor-supplied and unverified independently in this note. The sponsorship has no bearing on the Sunday Deep Dive (MemoryTax) content, which is unrelated to Datalab's product and reads as the newsletter's standard editorial long-form piece.

Related

No deep-fetches performed — this issue classified as single-thread-deep-dive, not curation, so the "max 2 deep-fetches, curation-mode only" step does not apply; the study's own content and the newsletter's framing of it were sufficient for the mapping above.