NVIDIA NeMo RL Achieves 48% Speedup with End-to-End FP8 Precision Training
Jessie A Ellis Apr 20, 2026 23:41
NVIDIA's new FP8 recipe for reinforcement learning delivers 48% faster training while matching BF16 accuracy, cutting AI infrastructure costs significantly.
NVIDIA has released a comprehensive FP8 precision recipe for reinforcement learning that delivers up to 48% faster training throughput while maintaining accuracy parity with traditional BF16 approaches—a development with significant implications for AI infrastructure costs and GPU compute economics.
The technique, detailed in a technical blog post from NVIDIA's Guyue Huang, addresses one of RL training's thorniest problems: the numerical disagreement between generation and training phases when using different precision levels across separate engines.
The Technical Breakthrough
Traditional RL pipelines use vLLM for rollouts and Megatron Core for training—each with unique CUDA kernels that introduce cumulative numerical differences. These discrepancies magnify at lower precision levels, historically limiting FP8 adoption.
NVIDIA's solution? Apply FP8 consistently across both generation and training rather than mixing precision levels. Testing on Llama 3.1 8B Instruct showed validation accuracy of 0.613 with end-to-end FP8 versus 0.616 for BF16—effectively closing the gap. Meanwhile, using FP8 for generation only dropped accuracy to 0.586.
The recipe uses block-wise quantized FP8 (E4M3 format) with 128x128 granularity for weights and 1x128 for activations. Linear layers run FP8 math at 2x theoretical peak throughput versus BF16, while attention, normalization, and non-linear functions stay in BF16.
Real-World Performance Gains
For linear layers alone, the FP8 recipe delivers consistent 15-25% throughput improvements. The gap between theoretical 2x speedup and actual gains comes from attention layers remaining in BF16 plus quantization kernel overhead.
Extending FP8 to KV cache and attention operations pushes total speedup to approximately 48% over BF16 baselines. The catch: RL's constantly updating policy weights require dynamic recalibration of quantization scales after each training step. NVIDIA's approach adds roughly 2-3% overhead for this recalibration—a minor cost for substantial acceleration.
Testing on Qwen3-30B (a mixture-of-experts model) showed matching accuracy curves between FP8 and BF16 configurations, suggesting the technique scales across architectures.
Why This Matters for AI Economics
RL training for reasoning-capable models like those behind advanced AI assistants requires massive compute. A 48% speedup translates directly to reduced GPU-hours and lower electricity bills for organizations training these systems.
The importance sampling technique that enables accuracy preservation could prove equally valuable. By correcting distribution mismatches between generation and training models on a per-token basis, it allows aggressive precision reduction without sacrificing model quality.
The full implementation is available in NVIDIA's open-source NeMo RL library, with pre-configured recipes for Llama 3.1 8B and Moonlight 16B models. Advanced users can fine-tune the approach—keeping specific transformer layers in BF16 or switching to power-of-2 scaling factors for additional optimization.
For AI infrastructure operators watching compute costs climb alongside model complexity, this represents a meaningful efficiency lever that doesn't require hardware upgrades—just smarter use of existing H100 capabilities.
Image source: Shutterstock.jpg)