Anyscale Launches Cost-Effective Batch LLM Inference Solution
Ted Hisokawa Oct 02, 2024 13:38
Anyscale introduces a new batch LLM inference solution, reducing costs significantly compared to AWS Bedrock and OpenAI.
Anyscale has announced the launch of its new batch Large Language Model (LLM) inference solution, which promises significant cost reductions compared to leading online inference providers. According to Anyscale, the new solution can cut costs by up to 2.9 times compared to AWS Bedrock and OpenAI, and up to 6 times in shared prefix scenarios.
Core Differences Between Online and Batch Inference
While online inference is designed for low latency and real-time responses, batch inference focuses on higher throughput and cost-effectiveness by optimizing GPU resource utilization. This makes batch inference ideal for tasks that do not require immediate interaction, offering a more efficient and economical use of LLMs.
Online inference typically employs high-end hardware to meet Service Level Agreements (SLAs) and uses techniques like speculative decoding. In contrast, batch inference prioritizes throughput-per-dollar using more cost-effective GPUs and techniques like pipeline parallelism and KV cache offloading.
Advantages of LLM Batch Inference on Ray & Anyscale
For self-hosted inference, Ray and Anyscale offer unique advantages in optimizing batch LLM inference. Anyscale uses a proprietary inference engine based on vLLM, an open-source LLM inference engine, augmented with performance improvements like FP8 support and chunked prefill.
Ray Core and Ray Data, key components of Anyscale's stack, are built for data-intensive AI workloads. Ray Core’s flexible distributed primitives enable nested parallelism, while Ray Data supports Python async primitives for high GPU utilization and built-in fault tolerance.
The Anyscale Platform, a fully-managed Ray solution, provides a cost-effective way to run AI applications. Features like the Smart Instance Manager and accelerated model loading enhance the platform's efficiency and reliability.
Anyscale has also developed RayLLM-Batch, a library aimed at optimizing LLM batch inference workloads, offering a simplified API, an optimized inference engine, and job-level fault tolerance.
Benchmarking with LLaMA
In benchmarking experiments, Anyscale evaluated its solution on Llama 3.1 8b Instruct against AWS Bedrock and OpenAI GPT-4o mini with batch pricing. Using commonly available GPUs on AWS, Anyscale's FP8 deployment was found to be 2.2 times cheaper than OpenAI's batch pricing and 2.9 times cheaper than Amazon Bedrock.
In scenarios with shared prompts, Anyscale's solution was 6.1 times cheaper than AWS Bedrock and 4.78 times cheaper than OpenAI GPT-4o mini. Similar cost advantages were observed in large model use cases, with Anyscale's FP8 batch inference solution being 22% cheaper than Amazon Bedrock for Llama 3.1 70b Instruct models.
Conclusion
Anyscale's new batch LLM inference solution offers a powerful, cost-effective alternative for large-scale batch processing workloads. The RayLLM-Batch library simplifies the deployment and optimization of batch inference tasks, making it easier for users to benefit from the cost savings and performance enhancements.
For more information, visit the official announcement.
Image source: Shutterstock.jpg)