Best LLM for transcript summaries depends on call length

AssemblyAI says there is no single best large language model for summarizing transcripts, arguing that the right choice changes with call length, cost and latency. In a post published Oct. 28, 2026, the company said its Voice Standard Bench ranks models on real speech-to-text output across 30-second voicemails, 2-minute calls, 15-minute support calls and 60-minute meetings.

Why transcript length changes the ranking

AssemblyAI said transcript summarization is not the same as document summarization because spoken input is disfluent, repetitive and often multi-speaker, and because transcript length can vary by two orders of magnitude within the same product. The company said short inputs can be handled well by many models, while coverage and faithfulness become more important as calls get longer.

On 30-second voicemails, AssemblyAI said 41 of the 52 models on its Summarization tab score 90 or higher, with differences driven mainly by speed, price and whether the model pads the answer. At 15 minutes, it said only 10 models score 90 or higher, and at 60 minutes the spread between the top and bottom rankings is widest.

How the bench is scored

The Voice Standard Bench is open source and tests Gateway models on summarization, information extraction, text correction and dictation cleanup, according to AssemblyAI. For summarization, each model receives the same prompt: write a plain-prose summary a busy person could read in place of the call, covering the topics discussed, then decisions, then action items, and keeping the length scaled to the call.

AssemblyAI said each output is scored by two LLM judges, Claude Opus 5.5 and GPT-5.6 Sol, on faithfulness, coverage and conciseness. The company said the benchmark uses ten real transcripts per length bucket, each within 15% of the target length, and that requests which error out are recorded but excluded from the score rather than counted as zero.

The company said it uses judges instead of ROUGE or BERTScore because token-overlap metrics can reward matching phrasing while missing factual errors, which is a particular problem for spoken transcripts.

Published picks by length

AssemblyAI listed different top performers depending on transcript length. For 30-second voicemails, it named gpt-5.6-terra as the highest-performance model with a score of 100, nemotron-3-nano-30b-a3b as the lowest-cost option at $0.0016 per hour of audio with a score of 76, and qwen3.5-4b-32k-fast as the fastest at 0.28 seconds median with a score of 76.

For 2-minute calls, AssemblyAI said claude-opus-5-5 led with a score of 99, while nemotron-3-nano-30b-a3b was the lowest-cost pick at $0.0016 per hour of audio and qwen3.5-4b-32k-fast was the fastest at 0.60 seconds median. For 15-minute support calls, it said claude-opus-5-5 again ranked highest at 97, with nemotron-3-nano-30b-a3b lowest cost at $0.0013 per hour of audio.

On 60-minute meetings, AssemblyAI said deepseek-v4.1-flash led with a score of 96, but it also logged 25 errored requests across its 60-minute runs. The company said its median response time at that length was 73.3 seconds and that the next-highest scores were claude-opus-5-5 and gpt-6-astra, both at 95.

Cost and operational trade-offs

AssemblyAI said summarization economics should be measured in cost per hour of audio rather than per token, because voice products are sold in hours and model bills depend on input tokens, output tokens and in some cases hidden reasoning tokens. It said the conversion can produce large gaps in cost, even when quality differences are smaller than the price spread.

As one example, AssemblyAI said the top scorer on 2-minute calls, claude-opus-5-5, costs about $0.30 per hour of audio, while the lowest-cost nemotron-3-nano-30b-a3b costs about $0.0016 per hour of audio. It said that is a 17-point quality gap at under 1% of the price.

What the company says to watch for

AssemblyAI said lower-cost and faster models can be competitive on short inputs, but that model choice matters most for longer meetings where coverage and reliability become more important. It also said users should check the score beside the lowest-cost and fastest cards, because those picks are raw options with no quality floor.

The company recommended passing speaker turns rather than a wall of text, asking for grounded quotes when needed, and chunking and merging summaries for 60-minute calls. It also said users can test models through its Gateway API, which is OpenAI-compatible and priced at provider list price with 0% markup.

Separately, recent product releases from other vendors underscore that transcript summarization remains an active area of development. OpenAI said on Oct. 6, 2026 that ChatGPT now supports audio uploads for transcription, recording summaries and questions about audio content, while Anthropic said on Sept. 28, 2026 that Claude Sonnet 5.5 was designed to be faster and lower cost than Sonnet 5. Those releases are not the same as AssemblyAI’s benchmark rankings, but they show how rapidly the surrounding product landscape is changing.

AssemblyAI said the live results page is the source of truth and may change as models change. It did not provide a fixed universal recommendation, saying the best model depends on transcript length and the trade-off between quality, speed and cost.