Transformer Engine
Transformer Engine
NVIDIA's Transformer Engine Boosts MoE Training in JAX by 10x
NVIDIA's Transformer Engine accelerates Dropless Mixture-of-Experts (MoE) training in JAX, achieving a 10x performance gain and 97% scaling efficiency.
Transformer Engine
Nvidia's New MoE Kernels Promise 93% Speedup for AI Training
Nvidia unveils advanced MoE training kernels, boosting AI model throughput by up to 93% in GPT pre-training and redefining large-scale efficiency.