Repository navigation
Fold letters outside the BMP in OrdinalIgnoreCase [patch] - #78
Open
matt-edmondson wants to merge 1 commit into
Open
matt-edmondson wants to merge 1 commit into
matt-edmondson wants to merge 1 commit into
Conversation
OrdinalIgnoreCase folded one UTF-16 code unit at a time, and char.ToUpperInvariant leaves a surrogate unchanged, so a cased letter encoded as a surrogate pair (Adlam, Deseret, Osage, Warang Citi) was never folded: "\U00010428a" sorted after "\U00010400b", and names equal ignoring case did not sort next to each other. Each code unit is now folded as part of the code point it belongs to, with Rune.ToUpperInvariant on .NET and TextInfo.ToUpper on netstandard. Every text comparison still goes through the same fold, including text against a number's first digit and the prefix branch, so the order stays transitive. Folded text is still compared by code unit, the way Ordinal and the ordinal tiebreak order it, and the README says where that differs from OrdinalIgnoreCase (U+E000..U+FFFF against a pair). Fixes #77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vfJW3Ri1HBoN2geG4RgBi
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #77
What was wrong
NaturalStringComparer.OrdinalIgnoreCasefolded one UTF-16 code unit at a time.char.ToUpperInvariantreturns a surrogate unchanged, so cased letters encoded as surrogate pairs (Adlam, Deseret, Osage, Warang Citi, Medefaidrin) were never folded. As a result"\U00010428a"sorted after"\U00010400b", and𞤢file/𞤀filedidn't sort next to each other.Change
Fold(char)becomesFold(string, int). With ignore case on, each code unit is folded as part of the code point it belongs to (FoldIgnoringCase):Rune.ToUpperInvarianton .NET, andTextInfo.ToUpperon netstandard, which has noRune. On netstandard this allocates, but only for a surrogate pair in a text chunk.CompareFolded, text against a number's first digit, and the prefix branch ofCompareTextChunks. This keeps the order transitive. Folding never changes a code point's width, so a pair that straddles the shorter chunk's end still compares consistently.Ordinalmode and the ordinal tiebreak order text. This differs fromstring.Compare(..., OrdinalIgnoreCase)in one place: U+E000–U+FFFF against a surrogate pair. The code comments and the README document it.Tests
New tests in
NaturalStringComparerIgnoreCaseTests:Compare_IgnoringCase_FoldsLettersOutsideTheBmp: the Deseret and Adlam cases from the issue.Sort_IgnoringCase_KeepsSupplementaryNamesEqualIgnoringCaseTogether:𞤢fileand𞤀fileend up adjacent, and they still tie-break ordinally rather than comparing as 0.Compare_IgnoringCase_FoldsAsOrdinalIgnoreCaseDoes: agreement withOrdinalIgnoreCaseplus the ordinal tiebreak, over BMP and supplementary letters.Compare_IgnoringCase_IsTransitiveOverBmpAndSupplementaryLetters: a mixed BMP/supplementary set, including lone surrogates, U+E000+, digits, and pairs straddling chunk ends.Verification:
dotnet build -c Releaseof the library across all TFMs (net8/9/10, netstandard2.0/2.1) succeeds with no warnings.Generated by Claude Code