audio: unpinned default ASR model changed on the Hub, breaking test_asr_ctc_default_gpu.py #155

Where it lives: main (43ccd29) and PyPI 0.1.1. src/granite_switch/vllm/audio/asr.py loads DEFAULT_ASR_MODEL_ID = "ibm-granite/granite-speech-5.0-470m-turboctc" by name, with no revision.

Problem

tests/integration/test_asr_ctc_default_gpu.py fails on both GPU CI legs (vllm26, vllm30). The vllm30 logs of both runs below show the same 4 of its 6 tests failing (the vllm26 logs show only that the file failed):

• test_transcribes_correctly_and_ignores_decode_kwargs
• test_float16_override_still_transcribes
• test_window_decides_single_pass_vs_chunked[100-1]
• test_window_decides_single_pass_vs_chunked[240-None]

E   AssertionError: assert 'What is the ...al of Israel?' == 'what is the ...tal of israel'
E   AssertionError: assert 'israel' in 'What is the capital of Israelrael? What is the What is the capital of Israel What is the capital of Israel?'

Seen in GPU Tests run 39792219145 (target 861b917, finished 2026-10-22) and run 39913176918 (target aa7dd88, 2026-10-23).

Cause

The default model is loaded by name, so it is whatever the Hub repo holds that day. On 2026-10-21 the repo was replaced in place by a multilingual release:

• 121ff64 "Upload 13 files": new weights and tokenizer. The pad token changed from <|blank|> to <unk>.
• 621ff2e "Remove legacy custom-code files".

The model card now says the model "outputs punctuated and capitalized transcripts". It names the previous English-only release as revision 947f59a (947f59af40db9791170a0628cf0f3f4812d720f1).

Evidence

• Same code, different result: 861b917 passed this file on an A100 outside CI on 2026-10-20, then failed it in CI on 2026-10-22.
• 861b917 and aa7dd88 are PR heads on top of main 7e72456. Neither touches the audio code, this test, uv.lock or pyproject.toml compared with main, so main is affected the same way.
• Reproduced on CPU with CI's library versions (torch/torchaudio 2.11.0, transformers 5.17.0), through our own ASRTranscriber:

At 947f59a, every assertion in the test passes on CPU.

Impact beyond CI

Since 2026-10-21, anyone serving an --enable-audio checkpoint with the default ASR model gets the new model, with no code change on our side: cased, punctuated, multilingual transcripts. PyPI 0.1.1 loads the default the same way. Workaround for an existing checkpoint, in its config.json:

"asr_pipeline_kwargs": {"revision": "947f59af40db9791170a0628cf0f3f4812d720f1"}

revision reaches the model, the tokenizer and the feature extractor alike. That revision loads natively, with no trust_remote_code.

Other findings

• "Israelrael" comes from the model, not from the chunk merge. It shows up in the 100 s case, which runs in one pass with no seams. The new release drops most copies of a phrase repeated back to back (about 22 copies in, 3 recognized), with or without pauses between copies. On real speech it is fine: on 73 distinct LibriSpeech utterances (503 s), word recall (ignoring case and punctuation) is 98.0% with the new release and 95.9% with the old one.
• <|nospeech|> would reach the prompt. On noise or a tone, the new release outputs the literal text <|nospeech|>. The transformers CTC pipeline decodes with skip_special_tokens=False, so that text gets spliced into the LLM prompt. This affects anyone on the new release, which today includes every user of the unpinned default.
• A separate, older seam-merge bug, on both releases. merge_transcripts needs an exact word match at each seam. When the next window starts mid-word (aut hypnotic vs autohypnotic), or the two windows disagree on one word, it finds no overlap and repeats about 10 words. On repetitive input it can cut real words instead. The Hub change didn't cause this.

Fix

1. Remove the tests that check the ASR output (the four above). Those are quality tests, not unit tests: a test here should only check that nothing throws. Note that test_window_decides_single_pass_vs_chunked also checks single-pass vs chunked call counts, which don't depend on the transcript. tests/unit/test_asr.py::TestSinglePassCeiling already covers that boundary with a fake pipeline.
2. Pin the ASR model version, so there are no surprises in the future.

If it helps, branch bugfix/pin-default-asr-revision (97d5c45) has a pin of the default, with unit tests and docs.