NVIDIA Co-Design Enhances AI Factory Memory Efficiency and Throughput

NVIDIA has unveiled a co-design strategy aimed at maximizing the efficiency of high-bandwidth memory (HBM) in AI factories. By optimizing how memory is utilized, the company has achieved significant throughput gains, with the NVIDIA Vera Rubin NVL72 delivering up to 30x the throughput per megawatt compared to the GB300 NVL72, despite both GPUs maintaining the same 288 GB HBM capacity. This improvement highlights the importance of maximizing the useful work each byte of memory performs, rather than merely increasing capacity.

NVFP4: Compressing Data Without Compromising Accuracy

One of the key innovations driving these gains is NVFP4, a 4-bit precision format that combines block-level and tensor-level scaling to store more intelligence in fewer bytes. NVFP4 reduces the model footprint by approximately 3.5x compared to FP16 and 1.8x compared to FP8, freeing up memory for critical tasks such as increasing KV-cache capacity and supporting larger batches. Benchmarks like AA-AgentPerf and MLPerf Endpoints have validated NVFP4's ability to deliver up to 4x the throughput of FP8 while maintaining strict accuracy requirements. Additionally, NVFP4 reduces KV cache footprints by up to 50%, enabling higher cache-hit rates and lower bandwidth consumption during decoding.

NVLink: Creating a Unified Memory Pool

Another critical component of NVIDIA's co-design approach is the NVLink scale-up domain, which effectively pools memory across multiple GPUs. For instance, in wide expert parallelism (Wide-EP) configurations, NVLink enables GPUs to share memory resources, reducing the weight-loading costs associated with mixture-of-experts (MoE) models. On GB200 NVL72, NVLink provides 130 TB/s of aggregate bandwidth, allowing EP32 configurations to deliver up to 1.8x more output tokens per second per GPU compared to EP8 configurations. The Vera Rubin NVL72 extends this capability with 3.0 TB/s of NVLink 6 bandwidth and inline memory tensor acceleratory (TMA) descriptor updates, further optimizing memory utilization across racks.

NVIDIA Dynamo: Optimizing KV Cache Utilization

To ensure that cached data is effectively reused, NVIDIA has developed Dynamo, a distributed serving layer that treats the KV cache as a fleet-wide resource. Dynamo employs KV-aware routing to track where cache blocks reside, minimizing the need for recomputation. This approach is particularly beneficial for agentic workloads, which often involve reusable system prompts, tool schemas, and shared documents. By balancing cached-prefix overlap with worker load, Dynamo reduces the inefficiencies associated with misplaced or evicted cache data.

Through these innovations, NVIDIA's co-design strategy demonstrates how a holistic approach to GPU architecture, interconnects, and software can significantly enhance the performance and efficiency of AI factories, maximizing the value of every byte of memory.