tensorrt-llm
NVIDIA H100 GPUs and TensorRT-LLM Achieve Breakthrough Performance for Mixtral 8x7B
NVIDIA's H100 Tensor Core GPUs and TensorRT-LLM software demonstrate record-breaking performance for the Mixtral 8x7B model, leveraging FP8 precision.
NVIDIA TensorRT-LLM Boosts Hebrew LLM Performance
NVIDIA's TensorRT-LLM and Triton Inference Server optimize performance for Hebrew large language models, overcoming unique linguistic challenges.
Enhancing Large Language Models with NVIDIA Triton and TensorRT-LLM on Kubernetes
Explore NVIDIA's methodology for optimizing large language models using Triton and TensorRT-LLM, while deploying and scaling these models efficiently in a Kubernetes environment.
NVIDIA's TensorRT-LLM Enhances AI Efficiency with KV Cache Early Reuse
NVIDIA introduces KV cache early reuse in TensorRT-LLM, significantly speeding up inference times and optimizing memory usage for AI models.
NVIDIA's TensorRT-LLM Multiblock Attention Enhances AI Inference on HGX H200
NVIDIA's TensorRT-LLM introduces multiblock attention, significantly boosting AI inference throughput by up to 3.5x on the HGX H200, tackling challenges of long-sequence lengths.
NVIDIA NIM Revolutionizes AI Model Deployment with Optimized Microservices
NVIDIA NIM streamlines the deployment of fine-tuned AI models, offering performance-optimized microservices for seamless inference, enhancing enterprise AI applications.
NVIDIA TensorRT-LLM Enhances Encoder-Decoder Models with In-Flight Batching
NVIDIA's TensorRT-LLM now supports encoder-decoder models with in-flight batching, offering optimized inference for AI applications. Discover the enhancements for generative AI on NVIDIA GPUs.
NVIDIA Enhances TensorRT-LLM with KV Cache Optimization Features
NVIDIA introduces new KV cache optimizations in TensorRT-LLM, enhancing performance and efficiency for large language models on GPUs by managing memory and computational resources.