State Space Models
NVIDIA and Princeton Unveil ReplaySSM to Accelerate Hybrid-Model Inference
ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes.
State Space Models
NVIDIA and Princeton Introduce ReplaySSM to Accelerate Hybrid AI Model Decoding
ReplaySSM addresses state-space model bottlenecks by caching inputs instead of state, delivering up to 1.8× faster decoding and 2.1× speculative serving throughput on production models.
State Space Models
Mamba-3 SSM Drops With Inference-First Design Beating Transformers at Decode
Together.ai releases Mamba-3, an open-source state space model built for inference that outperforms Mamba-2 and matches Transformer decode speeds at 16K sequences.