NVIDIA and Princeton Unveil ReplaySSM to Accelerate Hybrid-Model Inference
ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes.
NVIDIA and Princeton Introduce ReplaySSM to Accelerate Hybrid AI Model Decoding
ReplaySSM addresses state-space model bottlenecks by caching inputs instead of state, delivering up to 1.8× faster decoding and 2.1× speculative serving throughput on production models.
NVIDIA Blackwell GPUs Achieve 15x AI Inference Boost With DFlash
NVIDIA's DFlash speculative decoding delivers 15x faster AI inference on Blackwell GPUs, revolutionizing multiagent workflows and boosting throughput.
Reducing AI Inference Latency with Speculative Decoding
Explore how speculative decoding techniques, including EAGLE-3, reduce latency and enhance efficiency in AI inference, optimizing large language model performance on NVIDIA GPUs.
NVIDIA Enhances TensorRT-LLM with Speculative Decoding for Up to 3.6x Speed Boost
NVIDIA's TensorRT-LLM now includes speculative decoding, significantly improving inference throughput by up to 3.6x for large language models on NVIDIA GPUs.
IBM Research Unveils Cost-Effective AI Inferencing with Speculative Decoding
IBM Research has developed a speculative decoding technique combined with paged attention to significantly enhance the cost performance of large language model (LLM) inferencing.