Skip to content

QNN Qwen3 calibration fix, and every job on GitHub-hosted runners - #17

Merged
alpharomercoma merged 2 commits into
mainfrom
qnn-qwen3-bos
Oct 7, 2026
Merged

alpharomercoma merged 2 commits into
mainfrom
qnn-qwen3-bos

Conversation

@alpharomercoma

@alpharomercoma alpharomercoma commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Every Qwen3 QNN window failed about three minutes in. The failing runs were 37643657045 (Qwen3-0.6B, all three windows) and 37643663364 (Qwen3-1.7B). The error came from inside lm_eval during wikitext calibration: TypeError: 'NoneType' object cannot be interpreted as an integer.

The cause is in ExecuTorch 1.5.1. EagerEvalWrapper.prefix_token_id returns bos_id whenever the attribute exists, even when it is None, and Qwen3 has no BOS token. ExecuTorch main falls back to the end-of-text token in that case, as lm_eval's own HFLM does. pipeline/qnn_launch.py backports that one property and then runs the Qualcomm script unchanged, via runpy.

The tests cover the patched property against the real eager_eval (22 QNN tests pass with the export pins). CI skips that one test, since it doesn't install lm_eval or transformers.

Every job now runs on GitHub-hosted runners. The exports already run on Modal, but each export job held a Namespace runner the whole time it waited. The profile took six jobs at a time, so a five-model QNN dispatch queued behind itself. ubuntu-latest is free for this public repo and has no such cap. It does stop a job at 360 minutes, so:

  • Every Sandbox now ends by 330 minutes.
  • The longest windows get more cores instead: 64 for QNN models above 1.3B and for MediaTek at 32k.
  • .github/actionlint.yaml, which only declared the Namespace label, is removed.

🤖 Generated with Claude Code


View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

alpharomercoma and others added 2 commits October 7, 2026 23:35
…Qwen3 window calibrates

The Qualcomm script calibrates on wikitext through lm_eval with ExecuTorch's
EagerEvalWrapper, whose prefix_token_id in 1.5.1 returns bos_id whenever the attribute
exists, None included. Qwen3 has no BOS token, so every Qwen3-0.6B and Qwen3-1.7B window
failed three minutes in with "'NoneType' object cannot be interpreted as an integer".
ExecuTorch main falls back to the end-of-text token, as lm_eval's HFLM does;
pipeline/qnn_launch.py sets that property and runs the script unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on Modal

The Namespace profile took six jobs at a time, and each export job held one for its whole
run only to wait on a Modal Sandbox, so a QNN dispatch of five models queued behind itself.
ubuntu-latest, free for this public repository, has no such cap. It stops a job at 360
minutes, so the Sandboxes now end by 330 and the long ones get more cores instead: 64 for
QNN models above 1.3B and for MediaTek at 32k, 32 for MediaTek from 8k to 16k.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alpharomercoma alpharomercoma changed the title QNN export: backport the prefix-token fix Qwen3 calibration needs QNN Qwen3 calibration fix, and every job on GitHub-hosted runners Oct 7, 2026
@alpharomercoma
alpharomercoma merged commit 93f1acc into main Oct 7, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant