According to @_avichawla on X, expanding a transformer's context window by 8x can balloon memory by 64x due to quadratic attention, and according to the original transformer paper by Vaswani et al. (2017) this O(n^2) scaling is fundamental to full self‑attention. As reported by Meta AI and OpenAI research blogs, practical long‑context systems use sparse or compressed attention to control costs: 1) sliding window and dilated attention reduce kv cache growth (according to Longformer, Beltagy et al., 2020), 2) blockwise and local‑global patterns bound complexity (according to BigBird, Zaheer et al., 2020), 3) low‑rank projections compress keys and queries (as reported by Linformer, Wang et al., 2020), 4) recurrent state summarization avoids quadratic memory (according to RWKV and RetNet papers by authors on arXiv), 5) retrieval‑augmented generation restricts attention to retrieved chunks (as reported by Meta’s RAG and OpenAI cookbook), 6) segment‑level recurrence and memory tokens extend context efficiently (according to Transformer‑XL, Dai et al., 2019; Memorizing Transformers, Wu et al., 2022), and 7) grouped and multi‑query attention shrink KV cache at inference (as reported by Google’s multi‑query attention and OpenAI inference docs). According to Anthropic’s Claude long‑context evaluations and Google’s Gemini technical reports, business impact includes lower serving latency, reduced GPU memory per token, and higher accuracy on long‑document tasks when using retrieval plus local attention. For builders, the opportunity is to combine multi‑query attention with sliding‑window attention and retrieval to fit 200K–1M token contexts on commodity GPUs while maintaining quality, as reported by Mistral’s inference notes and open‑source frameworks like FlashAttention and vLLM.