Replayssm
NVIDIA and Princeton Unveil ReplaySSM to Accelerate Hybrid-Model Inference
ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes.
Replayssm
NVIDIA and Princeton Introduce ReplaySSM to Accelerate Hybrid AI Model Decoding
ReplaySSM addresses state-space model bottlenecks by caching inputs instead of state, delivering up to 1.8× faster decoding and 2.1× speculative serving throughput on production models.