Hugging Face Finds Top Speech Models Optimizing for ASR Benchmarks
Hugging Face tests 11 ASR models and finds benchmark cues can trigger wrong transcripts, masked-number recovery and dataset-specific spelling choices.
Summary
Published August 21, 2026, research by Theo Lebryk, Eric Bezzam, Alice, David Ayllon, Jakub Piotr Cłapa, Jens Madsen and Panagiotis Tzirakis introduced consensus disagreement, masked entity retrieval and orthographic switching tests for VoxPopuli English and LibriSpeech clean and other. The study evaluated CohereLabs/cohere-transcribe-03-2026, nvidia/canary-qwen-2.5b, ibm-granite/granite-speech-4.1-2b, microsoft/Phi-4-multimodal-instruct, nvidia/parakeet-tdt-0.6b-v2, bosonai/higgs-audio-v3-8b-stt-v2, Qwen/Qwen3-ASR-0.6B-hf, mistralai/Voxtral-Mini-3B-2507, moonshotai/Kimi-Audio-7B-Instruct, openai/whisper-large-v3 and moonshine-ai/moonshine-streaming-medium.
A low phoneme error rate ensemble flagged potential reference mistakes in 40% of analyzed VoxPopuli clips, covering roughly 3% of reference words. Benchmark-optimized models repeated wrong references 18 to 30% of the time, with the lowest word error rate models most prone. Six of 11 omitted an audible “Thank you” on one original clip, five did so on a same-speaker clone, one on a post-training parliamentary clone, and none on generic synthetic speech. Strong LibriSpeech performers recovered silenced numbers in roughly 30 to 40% of examples. Orthographic tests found several models above the 50% random baseline and some near 90%, matching dataset-specific spellings despite identical sounds.
Fresh European Parliament and LibriVox recordings, translations, restricted attention, trimmed context and added conversational audio often restored faithful transcription, while appended VoxPopuli audio could reverse it. Hugging Face added a Benchmark fitting tab covering VoxPopuli reference errors and cross-dataset orthographic switching to the Open ASR Leaderboard, releasing scripts and raw outputs on GitHub. The findings support held-out evaluation in Real World VoiceEQ, the Open ASR Leaderboard and Far-field ASR Leaderboard, plus temporal, speaker or metadata-based splits and greater training-data transparency.
Positives
- All 11 models restored the audible courtesy when the parliamentary sentence was rendered in a generic synthetic voice.
- Fresh European Parliament and LibriVox recordings reduced benchmark-matching behavior for many models despite coming from the same source domains.
- Translation, restricted attention, trimmed context and appended conversational audio often restored audio-faithful transcripts.
- The Open ASR Leaderboard now includes a Benchmark fitting tab measuring VoxPopuli reference errors and orthographic switching.
- Hugging Face released the testing scripts and unnormalized model outputs on GitHub for independent examination.
Risks & concerns
- Potential reference errors appeared in 40% of analyzed VoxPopuli clips and affected roughly 3% of its reference words.
- Benchmark-optimized models reproduced erroneous VoxPopuli references 18 to 30% of the time.
- Top LibriSpeech performers recovered numbers removed from the audio in roughly 30 to 40% of examples.
- Some models approached 90% orthographic switch accuracy, suggesting recognition of dataset-specific spelling conventions from acoustic context.
- The models with the lowest VoxPopuli word error rates were also most likely to reproduce incorrect benchmark references.
- Appending VoxPopuli audio made otherwise faithful synthetic or mined samples more likely to follow the benchmark reference.