AssemblyAI benchmark ranks Gemini 4 Argon first for dictation cleanup

AssemblyAI said its Voice Standard Bench still puts different LLMs in different roles for dictation cleanup, with a frontier model leading the quality score, a smaller model leading speed, and another open model leading cost at the time of the Oct. 28, 2026 update. The company said the task should be judged separately from transcription because cleanup has a deterministic target and interactive latency matters as much as output quality.

What the benchmark measures

According to AssemblyAI’s Oct. 28, 2026 post, dictation cleanup in Voice Standard Bench uses a synthetic dataset with known answers and evaluates exact match, word error rate, punctuation accuracy, capitalization accuracy and formatting-command accuracy. The benchmark does not use an LLM judge for dictation, and it only tests two lengths, 30 seconds and 2 minutes, because longer dictation windows are less representative of the use case.

The company said the benchmark’s dictation prompt asks models to execute spoken commands such as “comma,” “new paragraph” and “all caps” while leaving spoken numbers spelled out as words. AssemblyAI also said the task is narrow by design: it does not score filler removal or self-corrections, even though those are part of production cleanup workflows.

Live picks at 30 seconds and 2 minutes

On the Dictation cleanup tab as of Oct. 28, 2026, AssemblyAI said Gemini 4 Argon was the Highest Performance pick at both lengths, scoring 100 at 30 seconds and 99 at 2 minutes. The company said Qwen3.5 4B Fast was the Fastest pick at both lengths, with a median response time of 0.28 seconds at 30 seconds of audio and 0.60 seconds at 2 minutes, but with lower scores of 81 and 75 respectively. Nemotron 3 Nano 30B A3B was the Lowest cost pick, at $0.0013 per hour of audio for the 30-second bucket and $0.0017 per hour at 2 minutes, with scores of 81 and 67.

AssemblyAI’s live table also included a middle option that was close to the top score at a low price: Gemini 2.5 Flash Lite scored 97 at 30 seconds at about $0.002 per hour of audio and a 0.70-second median response time, but fell to 82 at 2 minutes, according to the post.

Why the trade-off matters

The company argued that dictation cleanup should be optimized for latency first, then exactness, then cost. In a text-entry workflow, the user stops speaking and waits for finished text, so even a small delay is visible, while models that over-edit can fail mechanically even if the result reads more smoothly. AssemblyAI said cost differences are real, but the dollar impact is usually small at dictation volumes because the token counts are modest.

AssemblyAI also cautioned that the results are only a shortlist. The company said its own methodology page advises testing candidate models on a user’s own recordings, and the post repeated that the synthetic dictation dataset is not a substitute for real transcripts.

Two implementation paths

For teams using AssemblyAI’s Dictation API, the post said the service can transcribe and rewrite in one call, returning both the verbatim transcript and cleaned text. It described two rewrite modes: fast, the default with lower latency, and smart, which follows detailed formatting instructions more reliably. The API also accepts an llm_instruction field for custom cleanup rules, and the Python SDK exposes final_text for the rewritten output when available.

For teams that already have transcripts or want to choose a specific model, AssemblyAI said LLM Gateway is OpenAI-compatible and can run the same cleanup prompt across multiple models. The post provided a workflow for comparing shortlisted models on a team’s own data and said the cleanup prompt should explicitly list supported spoken commands and instruct the model not to output those command words literally.

AssemblyAI did not supply a next benchmark refresh date in the post, and the company said the live page remains the source of truth as models change.