NVIDIA and Princeton Unveil ReplaySSM to Accelerate Hybrid-Model Inference
ReplaySSM, a joint project by NVIDIA and Princeton's Dao AI Lab, reduces state memory traffic in hybrid language models by caching recent inputs rather than full state matrices, delivering up to 1.80× kernel speedups at serving batch sizes.
NVIDIA Introduces Skip Softmax for Enhanced LLM Inference Efficiency
NVIDIA's Skip Softmax in TensorRT-LLM offers up to 1.4x faster inference for LLMs by optimizing attention computation, enhancing performance on Hopper and Blackwell architectures.
NVIDIA Enhances TensorRT-LLM with KV Cache Optimization Features
NVIDIA introduces new KV cache optimizations in TensorRT-LLM, enhancing performance and efficiency for large language models on GPUs by managing memory and computational resources.
NVIDIA Enhances LLM Inference with ReDrafter Support in TensorRT-LLM
NVIDIA's TensorRT-LLM now supports Apple's ReDrafter, optimizing large language model inference with speculative decoding, offering significant performance boosts on NVIDIA GPUs.
NVIDIA TensorRT-LLM Enhances Encoder-Decoder Models with In-Flight Batching
NVIDIA's TensorRT-LLM now supports encoder-decoder models with in-flight batching, offering optimized inference for AI applications. Discover the enhancements for generative AI on NVIDIA GPUs.
NVIDIA Enhances TensorRT-LLM with Speculative Decoding for Up to 3.6x Speed Boost
NVIDIA's TensorRT-LLM now includes speculative decoding, significantly improving inference throughput by up to 3.6x for large language models on NVIDIA GPUs.
NVIDIA's TensorRT-LLM Multiblock Attention Enhances AI Inference on HGX H200
NVIDIA's TensorRT-LLM introduces multiblock attention, significantly boosting AI inference throughput by up to 3.5x on the HGX H200, tackling challenges of long-sequence lengths.
NVIDIA NIM Revolutionizes AI Model Deployment with Optimized Microservices
NVIDIA NIM streamlines the deployment of fine-tuned AI models, offering performance-optimized microservices for seamless inference, enhancing enterprise AI applications.
NVIDIA's TensorRT-LLM Enhances AI Efficiency with KV Cache Early Reuse
NVIDIA introduces KV cache early reuse in TensorRT-LLM, significantly speeding up inference times and optimizing memory usage for AI models.
Enhancing Large Language Models with NVIDIA Triton and TensorRT-LLM on Kubernetes
Explore NVIDIA's methodology for optimizing large language models using Triton and TensorRT-LLM, while deploying and scaling these models efficiently in a Kubernetes environment.