Skip to content

Translate manga pages inside the reader (on-device OCR + existing translation backend) - #61

Closed
professorDeveloper wants to merge 2 commits into
next/3.3.0from
feat/manga-translate
Closed

professorDeveloper wants to merge 2 commits into
next/3.3.0from
feat/manga-translate

Conversation

@professorDeveloper

Copy link
Copy Markdown
Owner

An Arabic reader asked to translate the English text in manga speech bubbles into Arabic. This adds that to the comic reader on Android and iOS.

What it does

  • Translate page in the reader's top bar (comics only, Android/iOS only; hidden on desktop, Linux and for novels).
    • ML Kit text recognition (Latin script) runs on the device over the page's cached image file. A page that is not in the image cache yet is fetched with its headers and cookies.
    • Nearby ML Kit TextBlocks are merged into bubbles: stacked blocks within 0.8 line heights that share a column. Blocks with no letters are dropped. ALL-CAPS lettering is sent in sentence case, and a word hyphenated across lines is joined again.
    • All bubbles of a page go in one POST /contents/subtitles/translate/lines call with docKey = manga:<chapterUrl>:<pageIndex>. The chapter url is hashed if the key would pass the server's 200-character cut, so pages never collapse into one document. No from is sent, so the providers auto-detect the source language.
    • A full-screen preview shows the page with each bubble covered by a white rounded box holding the translation. The text shrinks until it fits, and RTL target languages such as Arabic are laid out right to left. Buttons: Apply, Show original (toggle) and Cancel.
    • Applied overlays stay on the page in all three layouts (continuous, paged, spread). The top-bar icon turns accent-coloured on a translated page; tapping it reopens the preview from cache without a new request.
    • Translations are cached per chapter, page and language: in memory, and as small JSON files under app support manga_translations/. Re-reading costs nothing.
  • Reader settings (comics, mobile): a "Translate pages into" language, which defaults to the subtitle translation language, and an optional "Translate every page as I read" switch, off by default. It only fires for a page that stays on screen for 700 ms, so flicking through pages does not spend the allowance. On any failure other than "no text" it shows one message and turns itself off for the session.
  • Messages: no text found, daily limit (429), offline, not signed in (401) and a generic error. SubtitleTranslationService.translateLines now throws the typed TranslationOffline / TranslationNeedsSignIn. The subtitle player still catches them as plain exceptions, as before.
  • 12 new keys under manga.* in all 16 locale files.

Backend

No backend change was needed. translateSubtitleLines already charges the daily allowance once per docKey per user per day, refunds it when every provider fails, and caches by text hash. Each newly translated page uses one daily translation, and a cached page uses none. With a small dailyLimitPerUser, "translate every page" will use up the allowance within one chapter. That is why it is off by default and labelled as such.

Dependency and platform impact

  • google_mlkit_text_recognition: ^0.16.0 (with google_mlkit_commons 0.12.0). I picked it deliberately:
    • 0.17.x uses AGP 9's built-in Kotlin (kotlin { compilerOptions } without the Kotlin plugin), and this app is on AGP 8.11.1.
    • 1.x is Swift Package Manager only, and this app uses CocoaPods.
    • 0.16.0 applies kotlin-android 2.2.20, the same Kotlin version as the app.
  • Android: the plugin's minSdk is 21, and the app uses flutter.minSdkVersion, so there is no minSdk change. The Latin model is bundled, so expect the APK to grow by a few MB per ABI.
  • iOS: GoogleMLKit/TextRecognition 9.0 needs iOS 15.5. I raised the deployment target from 15.0 to 15.5 in ios/Podfile, the post_install floor and the three IPHONEOS_DEPLOYMENT_TARGET entries in the Xcode project. Podfile.lock was not regenerated, because nothing was run here. The next pod install will add the MLKit pods. Devices on iOS 15.0–15.4 can no longer install updates.
  • pubspec.lock was edited by hand with the pub.dev archive hashes. CI's flutter pub get will correct it if anything is off.

Not verified

Nothing was run locally: no build, no tests, no device, no simulator. CI covers analyze, translation keys, unit tests, and the Android and Linux debug builds. Not verified on a device:

  • OCR quality on real pages;
  • how well bubbles merge;
  • overlay alignment, including EXIF-rotated images and very tall webtoon strips, where ML Kit may downscale;
  • the RTL rendering;
  • the iOS build and pod install at all.

Known limits:

  • When a page is zoomed past the sharpening threshold in paged mode, TiledZoom paints its sharp tiles over the overlay.
  • The disk cache is not pruned.

Follow-up (not in this PR)

  • "Use my own AI key", so heavy readers are not bound by the shared daily allowance.

Tests

test/features/manga/page_translation_test.dart covers:

  • block merging (stacking, gaps, side-by-side bubbles, transitive merges, reading order, dropping non-letters);
  • OCR text normalisation;
  • font fitting (binary search against a fake measurer);
  • docKey building, including the long-url hash;
  • RTL detection;
  • JSON round trip;
  • the PageOverlays state (store, apply, show original, remove, language change, notifications).

On-device OCR with ML Kit (Android and iOS), bubbles merged from nearby
text blocks, one translateLines request per page with docKey
manga:<chapterUrl>:<page>. A preview offers Apply, Show original and
Cancel; applied overlays are kept per chapter and page in memory and on
disk. The reader settings gain a page translation language and an
optional translate-as-I-read switch.
@professorDeveloper

Copy link
Copy Markdown
Owner Author

Combined into one PR against release/3.2.1.

@professorDeveloper
professorDeveloper deleted the feat/manga-translate branch November 7, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant