Skip to content

gh-153668: Replace Unicode numeric switch with binary-search tables - #156081

Open
scorpi023 wants to merge 1 commit into
python:mainfrom
scorpi023:gh-153668-msvc-arm64-numeric-lookup
Open

gh-153668: Replace Unicode numeric switch with binary-search tables#156081
scorpi023 wants to merge 1 commit into
python:mainfrom
scorpi023:gh-153668-msvc-arm64-numeric-lookup

Conversation

@scorpi023

@scorpi023 scorpi023 commented Aug 20, 2026

Copy link
Copy Markdown

Fixes #153668.

Summary

makeunicodedata.py currently emits _PyUnicode_ToNumeric() as a switch
with 2,023 case labels. On Windows ARM64, MSVC's /O2 optimizer consumes
excessive memory and the x86-hosted ARM64 compiler fails with C1002 while
compiling Objects/unicodectype.c.

Generate a sorted code-point table, a parallel value-index table, and a
table of the 151 UCD numeric-value entries instead. _PyUnicode_ToNumeric()
now uses binary search and still returns -1.0 for code points without a
numeric value. NUMERIC_MASK and _PyUnicode_IsNumeric() are unchanged.

This is a source-level alternative to the temporary build workaround in
#153669. That PR disables optimization for affected MSVC toolsets; this
change keeps optimization enabled by removing the compiler-stressing
generated switch.

Windows ARM64 result

VCTools 14.51.36231, Release /O2, on a Windows 11 ARM64 lab host:

Compiler host main This PR
HostX86 -> ARM64 C1002 after 11.487 s / 3.544 GB private success in 0.971 s / 0.037 GB
HostARM64 -> ARM64 success in 29.482 s / 25.196 GB private success in 0.598 s / 0.050 GB

Validation

  • Full native ARM64 Release build: success.
  • python -m test -j1 test_unicodedata test_str: 212 tests passed.
  • python -m test -j1 -u cpu test_unicodedata: 73 tests passed.
  • python -m test -j0 -x test_profiling: 49,207 tests passed across
    442 test files. test_profiling was excluded because its sampling
    coordinator hangs independently on this lab image.
  • Exhaustively compared all 1,114,112 Unicode code points with the previous
    generated switch; all 2,023 numeric mappings and non-numeric results match.
  • Tools\patchcheck\patchcheck.py: passed.

AI assistance: OpenAI Codex assisted with the investigation, implementation,
and validation. This PR is opened as a draft so the submitter can complete
the required human review before marking it ready.

@python-cla-bot

python-cla-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@scorpi023
scorpi023 marked this pull request as ready for review August 20, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows Arm64 release build fails

1 participant