Word Error Rate Is Not Enough: 10 ASR Models Tested

Word error rate (WER) measures speech-to-text accuracy by counting substitutions, deletions, and insertions against a reference transcript. Lower is better. But WER treats every word error as if it matters equally. A missing comma, a broken drug name, and a dose changed from 2.5 mg to 25 mg can each count as one error even though their consequences are completely different.

I saw that problem clearly when I tested 10 automatic speech recognition models on three French medical dictations. Google Chirp 3 produced the lowest WER at 10.10%, yet it still made four medically meaningful errors and failed two of the three cases under a strict semantic review. No ASR model passed all three.

I then tested eight language models on a separate task: turning corrected transcripts into structured reports without changing the facts. GPT-5.6 SOL was the only model to complete all nine repeated runs cleanly. Gemini 3.6 Flash reached eight out of nine.

The conclusion is not that WER is useless. It is the right starting metric. It is simply not enough on its own when some words matter far more than others.

What is word error rate?

Word error rate is the standard edit-distance metric used to evaluate automatic speech recognition. It compares a model transcript with a trusted reference transcript and asks how many word-level edits are needed to make them match. Google Cloud describes WER as the common industry metric for speech-recognition accuracy, while Microsoft uses the same three error types in its evaluation guidance. source source

The three errors are:

• Substitution: the system outputs the wrong word.
• Deletion: a spoken word is missing.
• Insertion: the system adds a word that was not spoken.

How is word error rate calculated?

The formula is:

WER = (substitutions + deletions + insertions) / words in the reference transcript × 100

If a 100-word reference needs five substitutions, three deletions, and two insertions, the WER is 10%.

Normalization matters. Capitalization, punctuation, spacing, written numbers, and filler words can change the score even when the audible content is similar. A fair model comparison therefore needs one fixed normalization policy, the same reference transcript, and the same source audio for every contender. Google and Microsoft both recommend reviewing more than one aggregate number because averages can hide file-level or category-level failures. source source

What is a good WER?

There is no universal cutoff. A good WER is one that is materially better than the alternatives on your audio, vocabulary, language, and recording conditions while still passing the error checks your workflow requires.

A score from a clean English audiobook cannot be compared honestly with a score from noisy French technical dictation. Even results on the same dataset can move when the reference or normalization changes. Artificial Analysis, for example, documents its dataset cleanup and normalization rules alongside its AA-WER benchmark because those choices are part of the result. source

Why WER can hide the errors that matter

WER counts edits. It does not understand meaning.

Changing éversion to et version may look ugly while leaving the surrounding report understandable. Changing Bisoprolol 2.5 mg to Bisoprolol 25 mg is also a small textual edit, but it changes a dose by a factor of ten. Losing pas from pas d’allergie connue reverses the statement. An aggregate percentage gives those failures no special weight.

That creates three common blind spots:

1. Severity: one critical error can matter more than many harmless ones.
2. Distribution: a respectable average can hide one disastrous recording.
3. Error type: names, units, negations, and numbers may fail more often than ordinary prose.

The answer is not to replace WER with subjective impressions. It is to keep WER for lexical comparison and add a second scoring layer for meaning.

The large scale in the cover represents that distinction: dozens of ordinary text errors can still be outweighed by one fact that cannot be wrong.

What happened when I tested 10 ASR models

The ASR benchmark used three French emergency-department dictations totaling about 9 minutes 11 seconds. Six models ran locally and four used cloud services. Each model transcribed each recording once, producing 30 transcripts.

The recordings increased in difficulty. The first was a short ankle-injury report. The second included hydration, oral intake, urine output, and one uncertain fluid name. The third was a dense cardiovascular emergency with symptoms, medications, doses, oxygen, ECG findings, treatment, and transfer instructions.

Full speech-to-text accuracy results

Google Chirp 3 was the clear winner in this set: lowest WER, lowest CER, and fewest reviewed medical errors. It was also the only model to transcribe diurèse conservée correctly. Yet it changed the ankle mechanism, weakened a pain finding, lost the location of an absent pulse, and corrupted a medication name.

AssemblyAI ranked second and produced the only clean version of the first case. Cohere Transcribe was the best local model at 15.46% WER, but it still made nine reviewed medical errors. “Best local” and “safe to trust without review” are not the same claim.

The errors behind the percentages

This is the benchmark’s main finding. Chirp’s 10.10% WER was genuinely better than the alternatives here, but the metric did not tell me whether the remaining 10.10% included punctuation noise or a changed dose. The manual review did.

Speed was a separate result

The timing below covers all 9 minutes 11 seconds of audio. Local timings include cold model loading. Cloud timings include the provider operation needed to obtain the transcript, so the two groups are useful but not perfectly equivalent.

GPT-4o Transcribe was the fastest cloud system but ranked sixth on WER. Faster-Whisper Turbo was the fastest local system but ranked last on WER. Google Chirp 3 was slower and produced the strongest transcripts.

Slow does not mean accurate. Fast does not mean careless. It means latency and factual integrity need separate columns.

Why I tested LLM post-processing separately

A modern dictation workflow often has two models doing different jobs:

1. The ASR model turns audio into text.
2. A language model cleans, restructures, or rewrites that text.

Testing them as one black box makes the source of an error impossible to identify. For the second phase, I therefore gave eight language models the corrected human reference transcripts, not the ASR outputs. The task was to create a concise structured report while preserving every number, uncertainty, medication, finding, and disposition.

Each model received the same frozen French prompt and ran three times on each of the three references. That created 72 scheduled outputs; 71 completed because one GPT-OSS run failed. Temperature was submitted as zero, top_p as one, and each request started without conversation history or examples.

Repeated LLM results

GPT-5.6 SOL was the only strict pass. Gemini 3.6 Flash was close, but on one of three runs it guessed physio for an uncertain fluid that the prompt required it to preserve as [terme incertain].

That single variation is why repeated runs matter. Temperature zero did not make every output identical. Claude repeatedly converted an uncertain penicillin allergy into a confirmed allergy. GPT-OSS changed 2.5 mg to 25 mg twice and failed to return one report. DeepSeek and Ministral often added plausible but unsupported care details.

The best model for this job was not the one that wrote the most impressive report. It was the one that resisted filling gaps.

If you want the broader distinction between raw transcription and AI transformation, read From Dictation to Clean, Structured Text.

How the benchmark was controlled

Every ASR model received the same mono, 16 kHz, signed 16-bit PCM audio. There was no denoising, gain adjustment, silence trimming, or external voice-activity deletion. The MP3 sources were decoded once, hashed, and reused.

The local models ran on a Windows 11 laptop with an NVIDIA RTX 3070 Laptop GPU, AMD Ryzen 7 5800H, and 32 GB of RAM. Each started in a fresh process. Timing began before model construction and stopped after the final transcript was written.

Some engines accepted the full recording; others needed segments. Granite, Cohere, and Chirp shared one deterministic segmentation rule: a maximum of 35 seconds, split near the lowest-energy point in the final five seconds, with no gap and no overlap. Qwen used CrispASR’s fixed internal slicing. The remaining models received each full file. No ASR model received a cleanup prompt.

WER normalization lowercased text, normalized Unicode, converted punctuation and line breaks to spaces, and collapsed repeated whitespace. It did not apply stemming, phonetic matching, vocabulary repair, or medical synonym replacement. Accents, digits, negations, and dictated formatting words remained significant.

The manual review then classified additions, omissions, alterations, doses, medications, negations, uncertainty, chronology, diagnosis, treatment, and disposition. Harmless punctuation, capitalization, line breaks, and section placement did not fail an output.

One ground-truth correction was necessary. The original second reference contained Du reste Conservée; context and audio review showed the intended phrase was Diurèse conservée. I corrected the reference, changed its hash, and regenerated every affected score. Ground truth can be wrong too. Treating a reference as sacred can reward the wrong output.

How to benchmark speech-to-text models for your own work

Public leaderboards can narrow a shortlist. They cannot tell you which model preserves your customers’ names, product codes, jargon, units, commands, or accents.

Provider choice matters because the winning combination can change with language, microphone, privacy limits, hardware, latency, and task. Local models change the risk profile, but “local” is not automatically more accurate. For the larger product decision, see the Windows dictation software buyer guide.

Limits of this ASR benchmark

This was a strict, auditable, and deliberately small experiment.

• It used three French medical recordings, not a broad multilingual corpus.
• Each ASR model ran once per recording, so ASR run-to-run variation was not measured.
• Each LLM ran three times per reference, which exposes some variation but remains a small sample.
• The LLM phase used corrected references, not ASR outputs. It was not an end-to-end safety test.
• Timings measured cold local execution and complete cloud operations, not live streaming latency.
• Cost, microphone quality, accents, background noise, network reliability, and other specialties were outside scope.
• The strict pass applies only to these inputs and settings. It does not establish general reliability, regulatory compliance, or clinical safety.

The source recordings and references are not public article assets. The local benchmark package retains hashes, model revisions, settings, sanitized requests, raw outputs, annotations, timings, and generated scorecards so the published tables can be traced back to evidence without exposing the source material.

Word error rate FAQ

Can WER be higher than 100%?

Yes. Insertions can make the number of errors larger than the number of words in the reference transcript. A system that repeats or hallucinates long passages can therefore exceed 100% WER.

Is a lower WER always better?

It is better when the models were tested on the same audio, reference, and normalization rules. It still does not prove that the remaining errors are harmless. Compare the score, the per-file distribution, and the error categories together.

What was the best ASR model in this test?

Google Chirp 3 ranked first with 10.10% WER, 3.77% CER, four reviewed medical errors, and one clean case out of three. It was the strongest model here, not a universal winner for every language or workflow.

What was the best local speech-recognition model?

Cohere Transcribe ranked first among the six local models at 15.46% WER. It still made nine reviewed medical errors and produced no fully clean case.

Can an LLM fix ASR errors automatically?

Sometimes, but this experiment did not test that claim. A language model can also guess, remove uncertainty, or invent a plausible detail. The LLM phase used corrected references specifically to measure rewriting errors without confusing them with ASR errors.

Is MachinesFluent medical software?

No. MachinesFluent is a general-purpose Windows dictation and AI workflow app. Medical dictation made factual failures easy to identify and consequential; it did not turn this experiment into clinical certification or establish safe unsupervised use.

If you want to compare local and cloud speech engines on the words that matter in your own work, try MachinesFluent for Windows. Use your own vocabulary and failure cases, and keep human review wherever a changed fact has real consequences.

Sources checked

• Google Cloud: Measure and improve speech accuracy
• Microsoft: Test recognition quality and evaluate accuracy
• Artificial Analysis: AA-WER v2 methodology
• Original MachinesFluent benchmark artifacts: locked audio hashes, reference hashes, raw outputs, timings, annotations, and generated scorecards retained locally.