06-reference

alphasignal latent sparse attention 10m context

2026-11-08·reference·source: AlphaSignal·by Ben Dickson
long-contextattention-mechanismsagent-harnessai-infrastructure

Why this is in the vault

A "Sunday Deep Dive" single-article issue on Latent Sparse Attention (LSA) — a new technique from Prefrontal AI and ETH Zurich that cuts prefill/decode cost for million-token contexts, aimed squarely at long-running coding agents — worth keeping as primary-ish technical grounding for why context cost is non-linear, not just "more tokens = more context rot."

The core argument

Full attention cost scales roughly quadratically with context length (double the context, roughly quadruple the compute), which is brutal for agents that accumulate millions of tokens across long codebases and document histories. Existing sparse-attention methods (e.g., DeepSeek Sparse Attention) reduce this by scoring candidate keys with a cheaper indexer, but the indexer itself still examines every prior key or block — so its own prefill cost keeps growing quadratically. In a 10M-token test on GLM-5.3-Flash, token selection alone consumed 98% of an attention layer's prefill time.

LSA fixes this by organizing attention keys into learned clusters with representative "centroid" vectors (like grouping library books before searching), so a query compares against cluster centroids instead of every key. Two integration paths: Distilled LSA (D-LSA) bolts a trained indexer onto a frozen existing model; Native LSA (N-LSA) learns the clustering during pretraining. Results: at 10M tokens on H200s, D-LSA sped up prefill by 22.7x (vs. sparse baseline) on GLM and 72.4x (vs. full attention) on Qwen-3.8-27B; end-to-end time-to-first-token improved 9.2x on Qwen. Tradeoff: GLM lost up to 8 points on the RULER long-context benchmark at 2-5M tokens relative to its sparse baseline, so the technique isn't free of retrieval-accuracy cost. Code, distilled weights, and an SGLang integration are open (Apache 2.0).

Mapping against Ray Data Co

Directly load-bearing for AGENTS.md hard rule #4 (route long artifacts through subagents, context rot degrades reasoning as context grows) and the harness-thesis work tracked in 01-projects/ — this is primary-source evidence for why that rule exists at the inference-compute layer, not just the "model gets worse" layer we usually cite. The piece makes explicit that the cost of long context isn't linear: full attention is quadratic, and even sparse-attention indexers inherit quadratic prefill cost unless they cluster first. That's a sharper technical version of the subagent-isolation argument RDCO already runs operationally (per-article/per-question fan-out, scratch-dir isolation in the brigade stations) — the model-side fix (LSA) and the harness-side fix (don't load 100KB artifacts into parent context) are solving the same problem from opposite ends of the stack. Relevant watch item if/when Claude or a coding-agent harness RDCO depends on adopts LSA-style sparse attention: the RULER accuracy tradeoff at 2-5M tokens is a real caveat worth remembering before assuming "longer context window" automatically means "equally reliable retrieval" at that length.

⚠️ Sponsorship

"From Decodo" block pitches Decodo's Web Data API (proxy rotation across 125M+ IPs, anti-bot bypass, JS rendering) for AI-agent web scraping, with a "Claim 10K requests" CTA. Decodo is a confirmed AlphaSignal sponsor-pool member (first logged 2026-10-30, pool now 39+) — standalone "Presented by"-style block, not integrated into the LSA technical content. Treat as a paid placement; no bearing on the LSA findings themselves, which come from the Prefrontal AI/ETH Zurich paper, not Decodo.

Related