According to @_avichawla, continuous batching replaces static slots, lifting LLM throughput up to 23x in high variance workloads, per Anyscale tests.

According to @_avichawla, LMCache’s multiprocess caching and CacheBlend deliver 14x faster TTFT and 4x faster decode on Qwen3-235B, cutting costs by up to 90%.

According to @_avichawla, speculative decoding lets small models guess K tokens and big models verify at once, delivering 2–3x faster LLM inference.