Skip to content

Fix hnswlib SIGILL in stubtest by building without -march=native - #16244

Open
ekanshul wants to merge 1 commit into
python:mainfrom
ekanshul:hnswlib-no-native
Open

Fix hnswlib SIGILL in stubtest by building without -march=native#16244
ekanshul wants to merge 1 commit into
python:mainfrom
ekanshul:hnswlib-no-native

Conversation

@ekanshul

Copy link
Copy Markdown

Fixes #16100. Reverts the darwin-only workaround from #16125 so hnswlib is tested on Linux again.

What was happening

hnswlib is source-only and its setup.py adds -march=native on unix. CI restores pip's wheel cache across runners, keyed only on requirements-tests.txt and stubs/**/METADATA.toml. So a wheel compiled on one ubuntu-latest host gets restored onto another with a different CPU, and the first unsupported vector instruction is a SIGILL. That accounts for every symptom in the issue: exit -4, empty output (it dies on import before stubtest prints anything), passing locally (you build and run on one machine), and passing on darwin (uniform hardware).

I did not want to rest this on the build flags alone, so I pulled the logs for the three consecutive daily runs around the first failure. Same runner image (ubuntu-24.04 20260720.247.2) on all three, so the only variable is the physical host:

date hnswlib step outcome cache
07-27 25.27 s success hit, 630 MB
07-28 5.82 s SIGILL hit, 702 MB
07-29 8.89 s success hit, 716 MB

A clean source build of hnswlib takes about 25 s (I timed 25 s locally on Apple Silicon too), so 07-27 compiled fresh. 5.82 s and 8.89 s are far too fast for a compile: both runs used a cached wheel built on some earlier host. On 07-28 that host's instruction set did not match; on 07-29 it did. Same cached artifact, different CPU, different result.

The fix

hnswlib provides an HNSWLIB_NO_NATIVE opt-out in setup.py. stubtest_third_party.py ran pip install without an env=, and there was no per-distribution way to set one, so this adds a general [tool.stubtest] install-environment key (a table of environment variables applied to the pip install step), documents it in CONTRIBUTING.md, and uses it for hnswlib. I made it general rather than special-casing hnswlib since any source-only package with CPU-specific flags has the same exposure.

Because stubs/**/METADATA.toml is in the cache key, changing hnswlib's METADATA.toml also rotates the cache, so no stale native wheel can be restored after this merges.

Verification

  • Built hnswlib with pip -v with and without the variable: without it, -march=native appears in the actual c++ compile commands; with it, the flag is gone.
  • Ran tests/stubtest_third_party.py hnswlib end to end through the patched code: hnswlib... (44.51 s) success, a full source build.
  • Wrapped subprocess.run to confirm the env actually reaches pip: HNSWLIB_NO_NATIVE=1 is present for hnswlib and absent for an unrelated distribution (six).
  • read_stubtest_settings("hnswlib") parses to {"HNSWLIB_NO_NATIVE": "1"} and ci_platforms falls back to the default ["linux"].
  • check_typeshed_structure.py, typecheck_typeshed.py (mypy --strict on tests/ and scripts/, linux 3.13), black, flake8, and ruff check all pass.

hnswlib is source-only and compiles with -march=native by default. CI
restores pip's wheel cache across runners with different CPUs, so a
wheel built on one host can hit an illegal instruction on the next.
That is the SIGILL (exit -4) with empty output from python#16100.

Add a general `install-environment` key to [tool.stubtest] that sets
environment variables for the pip install step, and use it to pass
hnswlib's own HNSWLIB_NO_NATIVE opt-out. This lets the darwin-only
workaround from python#16125 be dropped so hnswlib is tested on Linux again.

Fixes python#16100

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hnswlib stubtest fails for unknown reasons

2 participants