Memory Caching: The Most Principled RNN Long-Context Fix Yet — And Why You Can't Use It Tomorrow
What Google Actually Built
The team behind Titans and MIRAS has published Memory Caching, a framework that attacks the oldest problem in recurrent architectures: as sequences grow, early tokens get progressively overwritten in the fixed-size state vector. Memory Caching segments the input sequence, saves intermediate RNN states as a cache, then retrieves relevant cached states at inference time. The complexity lands at O(NL) — where N is the configurable number of segments — sitting precisely between RNNs' O(L) and Transformers' O(L²).
This isn't just another architectural trick. The N parameter is a tunable knob: set N=1 and you have a standard RNN; push N toward L and you approach Transformer-like global attention. For a workload running at L=8K tokens with N=16 segments, the paper implies a roughly 500x FLOP reduction compared to quadratic attention. But FLOPs ≠ wall-clock time — memory bandwidth and implementation details determine actual latency gains.
Why GRM Wins and What That Tells You
Four retrieval strategies were tested. Gated Residual Memory (GRM) — which uses input-dependent gates to soft-weight each cached segment's relevance per token — won consistently across all benchmarks. This is a notable result because it inverts the MoE intuition: in parameter space, sparse top-k routing outperforms dense mixing, but in this temporal cache regime with small N, dense gating dominates. When your cache has 16 states, the overhead of learning a router function doesn't justify itself vs. attending softly to everything.
The Memory Soup approach — treating cached states as model parameters to merge rather than activations to aggregate — is architecturally creative and borrows from model merging literature, but doesn't consistently beat GRM's simpler mechanism. Sparse Selective Caching (SSC), the MoE-style approach, underperforms both.
Dense gating beats sparse routing when your cache is small — the opposite of what parameter-space MoE research would predict. Watch whether this pattern holds as cache sizes grow.
The Unification Claim — and Its Limits
The paper's most ambitious claim: under simplifying assumptions, hybrid RNN-attention architectures (interleaved recurrent and attention layers, à la Griffin or Jamba) are a special case of Memory Caching. If this holds at scale, it provides a principled design framework for hybrid architectures rather than the current practice of hand-tuning layer interleaving patterns. The simplifying assumptions required for this equivalence are not fully detailed, so treat this as a theoretical direction.
Where It Breaks
Transformers still dominate on the hardest exact-retrieval tasks — UUID lookup at long contexts, the kind of needle-in-haystack matching requiring global attention over every token. Memory Caching helps with holistic understanding (summarization, classification, conversational context) but not precise lookup. If your workload requires exact retrieval from long sequences, this won't replace attention.
The Scale Question
Every experiment caps at 1.3B parameters. This is the critical constraint. We've watched enough architectural innovations fail the scaling test — early linear attention variants, certain SSM configurations — to know that 1.3B results are necessary but not sufficient for production relevance. The Titans team has access to Google's compute. If frontier-scale results don't appear within 6 months, that absence is itself diagnostic.
What to do
Profile your top 3 long-context inference workloads by sequence length and retrieval-type requirements (holistic understanding vs. exact lookup) this sprint
Set a calendar reminder to check Google's Titans/MIRAS/Memory Caching publication line in October 2025 for >10B parameter results
Benchmark a GRM-augmented RNN against your current Transformer on one representative long-context task at your working parameter scale