Repository navigation
Translate manga pages inside the reader (on-device OCR + existing translation backend) - #61
Closed
professorDeveloper wants to merge 2 commits into
Closed
professorDeveloper wants to merge 2 commits into
professorDeveloper wants to merge 2 commits into
Conversation
On-device OCR with ML Kit (Android and iOS), bubbles merged from nearby text blocks, one translateLines request per page with docKey manga:<chapterUrl>:<page>. A preview offers Apply, Show original and Cancel; applied overlays are kept per chapter and page in memory and on disk. The reader settings gain a page translation language and an optional translate-as-I-read switch.
Owner
Author
|
Combined into one PR against release/3.2.1. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An Arabic reader asked to translate the English text in manga speech bubbles into Arabic. This adds that to the comic reader on Android and iOS.
What it does
TextBlocks are merged into bubbles: stacked blocks within 0.8 line heights that share a column. Blocks with no letters are dropped. ALL-CAPS lettering is sent in sentence case, and a word hyphenated across lines is joined again.POST /contents/subtitles/translate/linescall withdocKey = manga:<chapterUrl>:<pageIndex>. The chapter url is hashed if the key would pass the server's 200-character cut, so pages never collapse into one document. Nofromis sent, so the providers auto-detect the source language.manga_translations/. Re-reading costs nothing.SubtitleTranslationService.translateLinesnow throws the typedTranslationOffline/TranslationNeedsSignIn. The subtitle player still catches them as plain exceptions, as before.manga.*in all 16 locale files.Backend
No backend change was needed.
translateSubtitleLinesalready charges the daily allowance once perdocKeyper user per day, refunds it when every provider fails, and caches by text hash. Each newly translated page uses one daily translation, and a cached page uses none. With a smalldailyLimitPerUser, "translate every page" will use up the allowance within one chapter. That is why it is off by default and labelled as such.Dependency and platform impact
google_mlkit_text_recognition: ^0.16.0(withgoogle_mlkit_commons0.12.0). I picked it deliberately:kotlin { compilerOptions }without the Kotlin plugin), and this app is on AGP 8.11.1.kotlin-android2.2.20, the same Kotlin version as the app.flutter.minSdkVersion, so there is no minSdk change. The Latin model is bundled, so expect the APK to grow by a few MB per ABI.ios/Podfile, the post_install floor and the threeIPHONEOS_DEPLOYMENT_TARGETentries in the Xcode project.Podfile.lockwas not regenerated, because nothing was run here. The nextpod installwill add the MLKit pods. Devices on iOS 15.0–15.4 can no longer install updates.pubspec.lockwas edited by hand with the pub.dev archive hashes. CI'sflutter pub getwill correct it if anything is off.Not verified
Nothing was run locally: no build, no tests, no device, no simulator. CI covers analyze, translation keys, unit tests, and the Android and Linux debug builds. Not verified on a device:
pod installat all.Known limits:
TiledZoompaints its sharp tiles over the overlay.Follow-up (not in this PR)
Tests
test/features/manga/page_translation_test.dartcovers:PageOverlaysstate (store, apply, show original, remove, language change, notifications).