Repository navigation
feat: place a transcript in its recording by listening to it (M11, 2/4) - #36
Conversation
M11, second of four. Forced alignment: the words are already known, so what the recogniser supplies is timing, and the characters it gets right carry the clock across the ones it does not. The approach changed after checking what was actually stopping it. The missing Chinese model was not an un-downloaded one — this machine's MDM profile sets `allowDictation = false`, so Dictation cannot be switched on, its languages cannot be added, and `SFSpeechRecognizer(zh-CN) .supportsOnDeviceRecognition` is false with *No Assistant asset for language zh-CN* in the log. `en-US` and `ru-RU` report true, being the two assets that happen to be present. macOS 26's `SpeechTranscriber` manages its own models and is not governed by that restriction: on the same machine it reports zh-CN supported, offers an installation request, and installs. So automatic alignment has a floor of macOS 26 / iOS 26, and older systems place boundaries by hand instead. A second implementation on `SFSpeechRecognizer` would fail precisely where the first was needed, which is not a fallback. Network recognition is refused rather than merely avoided. It would send the reader's own recordings to a server in an app whose premise is that it works offline, and it is capped at about a minute of audio besides. Measured on a real 147-second recording: transcription takes 1.2 seconds, and the transcriber hands back one run per character — 0.24 s for a typical one. Slicing the audio at the resulting boundaries and transcribing each slice on its own reproduces the phrase text, which is the only non-circular way to check that times are right. That run also showed the character budget to be a poor proxy for duration on material with pauses: eighteen characters came to ten seconds, and six phrases in eighteen ran past the seven-second cap with the longest at thirty. Over-long phrases are now cut again once the clock is known. The pauses come free. A recogniser folds the silence before a character into that character's own span, so a character running far longer than its neighbours is one that followed a gap — 1.8 seconds against a typical 0.24. That is silence detection without decoding a sample, and it is what rescues this material, because recognised punctuation is sparse and cutting at punctuation alone left six phrases over. With both: longest 29.8 s to 8.0 s, over-cap six of eighteen to three of twenty-six, slices still 97–100%. `Speech` is fenced to `HanReaderPlayback` by a SwiftLint rule, as `NaturalLanguage` and GRDB already are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR Summary by QodoAdd on-device forced alignment for transcript phrases
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
`SpeechAlignerTests` asserted that the system can transcribe Chinese. True here, where the model had just been installed; false on a headless CI runner, which reports no supported languages at all. So the test passed locally and failed in CI for the only reason it could — it was checking the environment rather than the code. It is conditional now, on something that *is* a property of the code: wherever the system can transcribe anything, it can transcribe Chinese, because that is the language this app exists for and the one the availability check names. Where it transcribes nothing there is nothing to assert, and saying so is better than a green tick meaning "this machine happens to be set up". `SpeechAligner.supportedLanguages()` comes with it, because the same distinction matters above: "this Mac has no speech support" and "it has speech support but not Chinese" need different things said to the reader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Misheard characters at a phrase's edge were cut out of its audio. A phrase anchored on its first and last *matched* characters begins after a mis-recognised opening character has already been spoken — the character is in the recording, the reader simply does not hear it. A phrase now reaches out to where the neighbouring content ends and begins. Stretching a short final phrase backwards to meet the minimum ignored the phrase before it, so two could replay the same audio and the reader would transcribe the same words twice without being told why. A cut point offered immediately after a terminator stranded the closing quote following it — the same defect as one found in the first part of this milestone, reproduced in new code. Re-cutting a long sentence could hand back a practice phrase that was one 」. Re-cutting rebuilt the document's metrics once per over-long phrase, which is also the same performance mistake as last time in new code: the whole transcript scanned again for an answer concerning one range of it. And listening never said it had finished. The fraction came from the last recognised segment, so a recording trailing into silence stopped at eight-tenths and looked stuck, and one the recogniser heard nothing in never reported at all. **A correction, from testing rather than reasoning.** The 97% slice in the previous commit was attributed to the boundary clipping the onset of 白, and a lead-in was proposed for the transport. Measured: with a 0.25 s lead-in the slice reads 黑白电影里, because the boundary falls *inside the word* 黑白 — the recogniser had put a full stop there, 「只有在黑。白电影里」. So it is neither clipping nor a timing error but a misplaced stop in an ASR-derived transcript, which is an artefact of verifying against a transcript the recogniser wrote. A human transcript punctuates correctly, and the transport needs no lead-in on that evidence. Re-measured after the fixes: progress reaches 1.0 exactly, mean slice accuracy 99% over six phrases, twenty-six phrases with a median of 5.4 s and a longest of 8.0 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
All five addressed in 1 — misheard edge characters were cut out of their phrase. Anchoring on first and last matched characters means a phrase whose opening character was misheard begins after that character has already been spoken. It is in the recording; the reader simply does not hear it. A phrase now reaches out to where the neighbouring content ends and begins. 2 — the last-resort stretch could overlap its neighbour. Pulling a short final phrase's start backwards to meet the minimum ignored the previous phrase's end, so two phrases could replay the same audio. 3 — cut points stranded closing quotes. The same defect found in PR 1, reproduced in new code: re-cutting a long sentence could hand back a practice phrase that was one 4 — re-cutting rebuilt document-sized metrics per phrase. Also the same performance mistake as PR 1, in new code. Built once now. 5 — listening never said it had finished. The fraction came from the last recognised segment, so a recording trailing into silence stopped at 0.8 and looked stuck; one with nothing recognised never reported at all. RetractionThe PR description said a slice scored 97% because the boundary clipped the onset of 白, and proposed a lead-in for the transport in PR 3. That was wrong. Tested: with a 0.25 s lead-in the slice reads 黑白电影里 — the boundary falls inside the word 黑白, because the recogniser had put a full stop there: 「只有在黑。白电影里」. So it is neither clipping nor a timing error. It is a misplaced full stop in an ASR-derived transcript, which is an artefact of verifying against a transcript the recogniser itself wrote. A human transcript — the actual use case — punctuates correctly. The transport needs no lead-in on that evidence, and PR 3 will not add one. Re-measured after the fixes
465 core tests. Four more mutants planted and killed. 🤖 Generated with Claude Code |
1 participant
Second of four for M11. Forced alignment: the words are already known, so what the recogniser supplies is timing, and the characters it gets right carry the clock across the ones it does not.
The approach changed, because the obstacle was not what the plan assumed
The plan said the reader installs the Chinese Dictation language. On the machine this is built for, they cannot:
An MDM profile. Dictation cannot be switched on, so its languages cannot be added, so
SFSpeechRecognizer(locale: "zh-CN").supportsOnDeviceRecognitionis false and the system logs No Assistant asset for language zh-CN.en-USandru-RUreporttrue— precisely the two assets present.macOS 26's
SpeechTranscribermanages its own models and is not governed by that restriction. On the same machine it reports zh-CNsupported, offers an installation request, and installs successfully.So:
SpeechTranscriberonly, floor of macOS 26 / iOS 26 for automatic alignment. Older systems place boundaries by hand — slower, never wrong. A second implementation onSFSpeechRecognizerwould fail exactly where the first was needed, which is not a fallback.Speechis fenced toHanReaderPlaybackby a SwiftLint rule, asNaturalLanguageand GRDB already are.Verified on a real recording
147 seconds of 听力 material:
The round trip is the part that matters: if the times were wrong, a slice would hold the neighbour's words. It is the only non-circular check available when the transcript is derived from the same recogniser.
What the real data caught
The character budget is a poor proxy for duration on material with pauses. Eighteen characters came to ten seconds; six phrases in eighteen ran past the seven-second cap, the longest at 29.8 s. No unit test would have found this — the synthetic recognisers all speak at a constant rate.
The fix came free from data already in hand. A recogniser folds the silence before a character into that character's own span, so a character running far longer than its neighbours followed a gap — 1.8 s against a typical 0.24 s. That is silence detection without decoding a sample, and it is what rescues this material, because recognised punctuation is sparse and cutting at punctuation alone left six phrases over.
Tests
459 core, 46 playback. Ten mutants planted and killed across the alignment and the re-cutting.
The audio path is not covered by an automated test, deliberately: a test that downloaded a speech model would fail on a fresh clone and on CI.
SpeechAlignerTestscovers what can be checked without one — language support, the refusal to fetch a model unasked, unreadable recordings, and which failures are worth offering a download for. The audio behaviour is what the measurements above record.Known
🤖 Generated with Claude Code