Hugging Face evaluated 11 open-source automatic speech recognition models and found that top-scoring systems frequently reproduce erroneous reference transcripts and predict silenced audio rather than transcribing what is spoken.
Researchers tested the models against the VoxPopuli English dataset to measure how systems handle reference errors. In one sample clip containing the audible phrase "Thank you, Mr. President," the reference transcript omitted "Thank you." Six of the 11 tested models dropped the courtesy phrase and copied the reference transcript's punctuation formatting. Hugging Face flagged reference errors in 40 percent of analyzed VoxPopuli test clips, covering roughly 3 percent of reference words. Models reporting the lowest word error rates reproduced those errors 18 to 30 percent of the time.
Silenced numbers and spelling switches
Hugging Face also tested models by removing numbers from test audio clips. On LibriSpeech datasets, several leading models still output the removed numbers in roughly 30 to 40 percent of examples. In another test involving European Parliament audio, a model autocompleted the silenced year "2011" while matching other errors present in the benchmark reference.
A third probe measured whether models altered spelling conventions to match specific benchmark styles, such as switching between "any one" and "anyone" within LibriSpeech or choosing between "Mr." in VoxPopuli and "Mister" in LibriSpeech. Several models exceeded the 50 percent random baseline and reached approximately 90 percent switch accuracy. When presented with fresh recordings collected after model training cutoffs, the systems stopped matching benchmark errors and returned to audio-faithful transcriptions.
Leaderboard updates
Hugging Face added a dedicated "Benchmark fitting" tab to its Open ASR Leaderboard to track VoxPopuli reference error rates and orthographic switching across public benchmarks. The organization also open-sourced its evaluation scripts and un-normalized model outputs on GitHub.
