According to @_avichawla on Twitter, attention workloads in transformers repeatedly shuttle large matrices between on-chip SRAM and high bandwidth memory to compute QK products and softmax, which creates significant memory bandwidth pressure across layers. As reported by the tweet thread, Q and K matrices are distributed to threads for parallel compute, with the QK product written back to HBM; the softmax stage similarly redistributes the product to threads, computes, and writes outputs to HBM, then repeats per layer. According to this description, the bottleneck implies business opportunities for kernel-level optimizations like FlashAttention, fused attention, and recompute-aware tiling, as well as hardware strategies such as larger SRAM, better tensor core utilization, and near-memory compute. As noted by the source, the repeated SRAM-HBM traffic underscores why IO-aware attention kernels, KV cache compression, and sequence-parallelism are key levers for reducing latency and cost in LLM serving and training.

According to @_avichawla on Twitter, FlashAttention uses on-chip SRAM to cache intermediate attention blocks, cutting redundant HBM transfers and delivering up to 7.6x speedups over standard attention. As reported by the FlashAttention paper from Dao et al. (Stanford), the IO-aware tiling algorithm keeps queries, keys, and values in fast SRAM, minimizing memory bandwidth bottlenecks and improving throughput on GPUs. According to the authors’ benchmarks, FlashAttention accelerates training and inference for Transformer models, enabling lower latency, higher tokens-per-second, and reduced cost per token in production LLM serving. For businesses, this translates to more efficient RAG pipelines, faster streaming responses, and better GPU utilization without accuracy loss, as reported by the original paper and follow-up engineering notes.

According to @_avichawla on X, a thread is the smallest unit of execution, multiple threads form a block, threads within a block share fast but limited on‑chip SRAM, and all blocks access abundant but slower global HBM; as reported by the post, understanding this hierarchy is key to optimizing AI kernels through shared memory tiling, reducing global memory traffic, and improving throughput on modern GPUs. According to NVIDIA developer documentation cited in industry practice, placing reused tensors in shared memory can cut HBM reads and boost occupancy for transformer attention and convolution workloads, creating practical speedups for inference and training. As reported by practitioners, aligning thread blocks to data tiles and coalescing HBM accesses enables higher effective bandwidth and lower latency in production ML pipelines.

According to Jeff Dean on X, TPU v8i is co-designed with Google’s Gemini research team to deliver low-latency inference by incorporating large on-chip SRAM that reduces trips to HBM for model weights and KV cache state, enabling more computations to stay on chip. As reported by Jeff Dean, these memory locality improvements target transformer serving bottlenecks—specifically attention KV cache bandwidth and latency—helping accelerate token generation and lower tail latency in LLM inference. According to Jeff Dean, the design focus implies better cost efficiency for enterprise-scale Gemini deployments, higher throughput per watt, and improved responsiveness for real-time applications such as chat, code assistance, and multimodal agents.