Repository navigation
fix(pdf): route text rhwp cannot measure (Cyrillic, Greek, Latin Extended) around it - #378
Conversation
… exists rhwp lays text out with built-in width tables generated only for ASCII, Latin-1, general punctuation, math, CJK symbols, Hangul and the fullwidth forms (edwardkim/rhwp#8143). Every other letter is placed at a fixed half em while its glyph is drawn at full width, so Russian, Greek, Polish, Czech, Turkish or Vietnamese text in a DOCX/Markdown/... -> PDF overlaps. Measured with rhwp 0.8.7 and Arial: every Cyrillic advance is 0.500 em against Arial's 0.458-0.778 em; z-dot, e-ogonek, z-acute, S-caron as well. ITextShapingCheck now answers two questions from one scan of the source text (ScanScripts): a script that needs shaping (unchanged: route around rhwp, or refuse) and letters rhwp has no widths for. The second is derived from rhwp's measured ranges, not from a list of scripts: a letter or combining mark outside those ranges and outside the full-width CJK rhwp estimates at an em. Such a file takes a route without rhwp (LibreOffice) when one exists and keeps rhwp otherwise; it is never refused. Shaping keeps precedence. Character references outside Unicode are skipped and variation selectors are ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you @anatolykoptev, this is a great find and a careful fix. I reproduced it locally with only rhwp installed: Russian, Polish and Greek Markdown → PDF overlap exactly as you describe ("Отчёт", "ОПТ", "Αναφορά"), and with your branch they go through LibreOffice and come out right. Deriving the rule from rhwp’s measured ranges rather than a script list, keeping shaping first, never refusing for widths, and the variation-selector and out-of-range reference cases are all spot on. Format, build, Core (455) and the ComplexScript/Rhwp/HWPX engine tests (108) pass here. On your policy question, one case matters a lot to us: Korean HWPX documents. A Korean report or paper with a single "α", "Ω" or "μ" now goes to LibreOffice + H2Orestart instead of rhwp, which is our most faithful HWP renderer. I checked it: a lone Greek letter between spaces draws fine in rhwp, while the LibreOffice page comes out with different margins. We’ll make two small changes on your branch before merging, unless you’d like to do them:
Also a heads-up: #351 adds an .xls branch to ComplexScripts.FindInFile that calls Find(cell.Text). Once both are in, it has to use the classifier, otherwise .xls sheets skip the widths check. We’ll take care of that when merging. CI for fork runs is waiting for approval; we’ll start it together with the change. Thanks again! |
…h rhwp - A symbol or unit on its own (α, Ω, μm in a Korean report) draws fine in rhwp and no longer sends a document to LibreOffice: an extended Latin letter counts next to any letter (Polish, Vietnamese), a letter of another script only next to one of its own script (Отчёт, Αναφορά). Character references count as inside a word. - An HWPX keeps rhwp, its most faithful renderer, for missing widths; only text that needs shaping sends it elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KnifeLemon
left a comment
There was a problem hiding this comment.
Thanks again @anatolykoptev! I pushed the two changes we talked about: an HWPX stays with rhwp for missing widths, and only letters inside words count (an extended Latin letter next to any letter, another script's letter next to one of its own), so α, Ω or μm in a Korean report keep rhwp while Russian, Greek, Polish and Vietnamese words still reroute. Tests for both are in. Merging.
- The .xls branch of ComplexScripts.FindInFile takes the scan's classifier, so old Excel files are also checked for words rhwp has no widths for (KnifeLemon#378), with a test. - Nepali strings for the trim preset and its error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What does this change?
Documents with Cyrillic, Greek, or Latin letters outside Latin-1 (Polish, Czech, Turkish, Vietnamese, …) come out of DOCX / Markdown / TXT / HTML / XLSX / PPTX / HWPX → PDF with overlapping letters. The cause is in rhwp: its built-in width tables are generated only for ASCII, Latin-1, general punctuation, math, CJK symbols, Hangul and the fullwidth forms, so every other letter is placed at a fixed half em while its glyph is drawn at full width. Reported upstream as edwardkim/rhwp#8143.
Measured with rhwp 0.8.7 (Filee 2.0.0, macOS), Arial in every slot, advance per glyph:
This PR treats that like the existing complex-shaping case, but softer:
ITextShapingChecknow answers both questions from one scan of the source text (ScanScripts→TextScripts(NeedsShaping, WithoutWidths)).WithoutWidthsis derived from rhwp's measured ranges instead of a list of scripts: a letter or combining mark outside those ranges and outside the full-width CJK rhwp estimates at an em. Latin-1-only Western European text, Korean, Chinese and Japanese are not affected. Variation selectors (emoji ❤️, IVS) are ignored, and character references outside Unicode are skipped.JobQueue: such a file takes a route without theUnshapededge when one exists (LibreOffice). Otherwise it keeps rhwp and converts. It is never refused. Shaping scripts keep precedence and their behaviour is unchanged.docs/ENGINES.mdand the comments describe the rule.Policy point for you to decide: one such letter anywhere in the scanned text sends the whole document to LibreOffice. That includes a stray Greek μ (U+03BC) or a French œ in an otherwise Latin-1 document. This matches how the shaping check treats one Arabic letter, and rhwp really does draw those letters at a half em. Happy to add a threshold if you prefer.
How was it tested?
dotnet format --verify-no-changesdotnet test:Filee.Core.Tests456 passed / 2 skipped (main: 454 / 2).Filee.Engines.Testsfiltered toComplexScript|Rhwp: 59 passed / 1 skipped. I did not run the full Engines suite locally, so I'm relying on CI for it.JobQueuebranch →Text_the_engine_cannot_measure_takes_another_route_but_is_never_refused.Text_with_shaping_and_unmeasured_scripts_is_refused_for_shaping.Scripts_without_widths_are_found_in_documents.A_file_is_scanned_again_once_its_text_changed.A_character_reference_outside_unicode_is_skipped.Variation_selectors_are_not_unmeasured_text.Not in this PR, noticed on macOS:
RhwpFonts.Planreplaces a font that is installed but unusable, but not one that is missing. On a Mac without Malgun Gothic, Filee-written HWPX text falls back to rhwp's own pick (Apple SD Gothic Neo). That's cosmetic next to the width issue.Checklist
src/Filee.App/Assets/i18n/*.jsonfiles (none added)THIRD-PARTY-NOTICES.md(none added)🤖 Generated with Claude Code