QNN Qwen3 calibration fix, and every job on GitHub-hosted runners

.github/actions/setup-light/action.yml

@@ -1,8 +1,8 @@
 name: Set up a light job
 description: >-
   Python from config/versions.env and the pipeline's plain requirements, for jobs that only read
-  Hub metadata and do arithmetic (the targets jobs). The Namespace runner's system Python
-  refuses pip installs (PEP 668: externally-managed-environment), so these jobs need their own.
+  Hub metadata and do arithmetic (the targets jobs), on the Python the exports pin rather than
+  the runner's system one (which may refuse pip installs, PEP 668).
 
 runs:
   using: composite

.github/workflows/audit-published.yml

@@ -28,7 +28,7 @@ jobs:
   audit:
     # x86: the gate runs ExecuTorch's C++ runner through the wheel, which fails on aarch64
     # the same way the verify job does (finding 30).
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}

.github/workflows/ci.yml

@@ -11,12 +11,12 @@ permissions:
 jobs:
   test:
     # Lint and unit tests only.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
 
       - name: Runner
-        # What the Namespace profile really has, read by config/pipeline.yaml's runner_tiers.
+        # What the runner really has, for the record (the exports run on Modal, not here).
         run: |
           echo "arch $(uname -m), $(nproc) CPUs"
           grep -E 'MemTotal|MemAvailable|SwapTotal' /proc/meminfo

.github/workflows/export-mtk.yml

@@ -51,7 +51,7 @@ concurrency:
 jobs:
   targets:
     # Hub metadata only, no model download: it reads a config.json and does arithmetic.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     env:
       # The matrix now reads the model's config to size each window's runner, and some of
       # the watched repos are private, so this job needs the token the export job has.
@@ -100,9 +100,10 @@ jobs:
     # export_mtk.MODAL_SHAPES, because streaming calibration's time grows with the window.
     # This job only sends it, waits, and publishes what comes back. Modal's hosts are x86,
     # which mtk_converter 8.13.0 (a cp310 x86 build) and the SDK's libraries need (finding 14).
-    runs-on: namespace-profile-execupack
-    # Past the longest Sandbox (export_mtk.MODAL_SHAPES), so the Sandbox times out first.
-    timeout-minutes: 630
+    runs-on: ubuntu-latest
+    # GitHub-hosted jobs stop at 360 minutes: past the longest Sandbox (export_mtk.MODAL_SHAPES),
+    # so the Sandbox is what times out and says so.
+    timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
       MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}

.github/workflows/export-qnn.yml

@@ -39,7 +39,7 @@ concurrency:
 jobs:
   targets:
     # Hub metadata only, no model download: a parameter count and arithmetic.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     env:
       # Llama 3.2 is gated, so even its metadata needs the token.
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
@@ -85,9 +85,10 @@ jobs:
     # and the QAIRT SDK is x86_64. The wheel downloads QAIRT inside the Sandbox on first
     # import (export_qnn.qnn_sdk), so the SDK is fetched from Qualcomm on every run and never
     # stored in an image, a cache or the Volume.
-    runs-on: namespace-profile-execupack
-    # Past the longest Sandbox (export_qnn.QNN_MODAL_SHAPES), so the Sandbox times out first.
-    timeout-minutes: 1410
+    runs-on: ubuntu-latest
+    # GitHub-hosted jobs stop at 360 minutes: past the longest Sandbox (export_qnn.QNN_MODAL_SHAPES),
+    # so the Sandbox is what times out and says so.
+    timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
       MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}

.github/workflows/export-vulkan.yml

@@ -35,7 +35,7 @@ concurrency:
 jobs:
   targets:
     # Hub metadata only, no model download: reads config.json and does arithmetic.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     env:
       # Sizing reads the model's config, and gated repos (Llama 3.2) need the token.
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
@@ -71,7 +71,7 @@ jobs:
     # the window squared, from ~5 GiB at 2k to ~134 GiB for Qwen3-4B at 32k. This job only
     # sends it, waits, and publishes what comes back. Modal has no GPU either, so the file
     # gets a structural check (findings 28, 30).
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}

.github/workflows/export-xnnpack.yml

@@ -43,7 +43,7 @@ concurrency:
 jobs:
   targets:
     # Hub metadata only, no model download: reads config.json and does arithmetic.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     env:
       # Sizing reads the model's config, and gated repos (Llama 3.2) need the token.
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
@@ -79,7 +79,7 @@ jobs:
     # Nothing to solve for when no window fits (an empty matrix).
     if: ${{ inputs.gptq && needs.targets.outputs.entries != '[]' }}
     # The solve runs on Modal with the memory targets asked for; this job waits on it.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
@@ -127,7 +127,7 @@ jobs:
     # test through ExecuTorch's C++ runner, then the decision gate against the fp32 model's own
     # answers, and only then does this job publish. They used to be two jobs because the
     # runner fails on aarch64 (finding 30) and the export ran on ARM; on Modal both are x86.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     timeout-minutes: 350
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}

.github/workflows/probe-runner.yml

@@ -25,9 +25,9 @@ jobs:
         context: ${{ fromJSON(inputs.contexts) }}
     # The causal masks grow with the window squared and have killed a 32k export before
     # (docs/research findings 2 and 28). x86, because the wheel's C++ TextLLMRunner raises
-    # `IndexError: stoi` on aarch64 (finding 30). The Namespace profile has no swap, so the
-    # swap asked for below is not added there.
-    runs-on: namespace-profile-execupack
+    # `IndexError: stoi` on aarch64 (finding 30). This probes the runner it lands on, a
+    # GitHub-hosted one; the exports themselves run on Modal.
+    runs-on: ubuntu-latest
     timeout-minutes: 120
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}
@@ -48,8 +48,8 @@ jobs:
           swap-gib: "24"
 
       - name: Export at a forced window
-        # Exit 4 is the gate refusing a window this runner cannot hold (16k and up for
-        # Qwen3-0.6B on the Namespace profile): a measurement in itself, not a failure.
+        # Exit 4 is the gate refusing a window this runner cannot hold: a measurement in
+        # itself, not a failure.
         run: |
           set +e
           python -m pipeline export-xnnpack "$MODEL_ID" --context "$CONTEXT" --out "$WORK_DIR/out" --work "$WORK_DIR/work"

.github/workflows/watch-hf.yml

@@ -40,7 +40,7 @@ concurrency:
 jobs:
   watch:
     # Hub metadata only, no model download.
-    runs-on: namespace-profile-execupack
+    runs-on: ubuntu-latest
     timeout-minutes: 30
     env:
       HF_TOKEN: ${{ secrets.HF_TOKEN }}

CLAUDE.md

@@ -6,7 +6,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 exe-expo exports small dense open-weight LLMs (≤4B, no MoE) from Hugging Face to ExecuTorch `.pte` files for the
 [openweights](https://github.com/alpharomercoma/openweights) Android app, running entirely on hosted CI runners
-(since 2026-10-07 every GitHub job runs on the Namespace profile `namespace-profile-execupack`, 4 x86 vCPUs, 15.7 GiB, no swap; the XNNPACK and Vulkan exports and the GPTQ solve run on **Modal**, each in a Sandbox sized to the step by `pipeline/remote.py` (`python -m pipeline modal-run`): the window's `sizing.host_need_bytes` plus `modal.memory_margin`, up to Modal's 344064 MiB, with the export, the smoke test through ExecuTorch's C++ runner and the decision gate in one Sandbox before the job publishes (finding 41). MediaTek and QNN still run on the Namespace profile, where a window that does not fit its RAM is skipped (finding 40).) `docs/PLAN.md` holds the decisions, measured costs, phase status and known
+(since 2026-10-07 every export runs on **Modal** and the GitHub jobs that drive them run on GitHub-hosted `ubuntu-latest`, which stops a job at 360 minutes; the XNNPACK, Vulkan, QNN and MediaTek exports and the GPTQ solve each run in a Sandbox sized to the step by `pipeline/remote.py` (`python -m pipeline modal-run`): for XNNPACK and Vulkan the window's `sizing.host_need_bytes` plus `modal.memory_margin`, up to Modal's 344064 MiB, with the export, the smoke test through ExecuTorch's C++ runner and the decision gate in one Sandbox before the job publishes (finding 41). QNN and MediaTek get memory, cores and minutes per window from `qnn-matrix` and `mtk-matrix`. The Namespace profile used briefly before that took six jobs at a time and 15.7 GiB (finding 40).) `docs/PLAN.md` holds the decisions, measured costs, phase status and known
 limits. Read it before changing sizing, backends or naming, and update it when a decision or measurement changes.
 
 ## Commands

README.md

@@ -1,19 +1,19 @@
 # exe-expo
 
 Exports small open-weight LLMs from Hugging Face to ExecuTorch `.pte` files for the
-[openweights](https://github.com/alpharomercoma/openweights) Android app. The GitHub jobs run
-on a [Namespace](https://namespace.so) runner and send each XNNPACK and Vulkan export, and the
-GPTQ solve, to a [Modal](https://modal.com) Sandbox with the memory that window needs (up to
-336 GiB), so every window from 2k to 32k can be built again; MediaTek and QNN still run on the
-Namespace runner. Design and decisions: [docs/PLAN.md](docs/PLAN.md).
+[openweights](https://github.com/alpharomercoma/openweights) Android app. Every export
+(XNNPACK, Vulkan, QNN and MediaTek) and the GPTQ solve runs on [Modal](https://modal.com), in a
+Sandbox sized to its model and window (up to 336 GiB and 64 cores); the GitHub jobs, on
+GitHub-hosted runners, only send the work, wait, and publish what comes back. Design and
+decisions: [docs/PLAN.md](docs/PLAN.md).
 
 What has shipped to [`experimentalmachines`](https://huggingface.co/experimentalmachines) on
 Hugging Face, and what is built but not published:
 
 | Backend | Status |
 |---|---|
 | XNNPACK (CPU) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes), Qwen3, Qwen2.5, Llama 3.2, SmolLM2, at every window from 2k to 32k the runner could build. Each window is now compared with its fp32 model before it publishes. Twelve models were published before that gate existed and are being re-checked by **Audit published**; the two audited so far, SmolLM2-135M and 360M, score 0.19 and 0.22 on its prompts against about 0.58 for their fp32 models |
-| MediaTek NeuroPilot (MT6991) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes) from 512 up to 8k, and the heretic finetunes at 16k; Qwen3, Llama 3.2 and SmolLM2 at the windows that finished before the 2026-10-04 cancel. Built on Blacksmith runners of up to 128 GB; the Namespace profile used since 2026-10-07 cannot build LFM2.5 at any window (finding 40). Checked structurally on the runner, which has no NPU |
+| MediaTek NeuroPilot (MT6991) | **Published**: LFM2.5 1.2B and 2.6B (and their heretic finetunes) from 512 up to 8k, and the heretic finetunes at 16k; Qwen3, Llama 3.2 and SmolLM2 at the windows that finished before the 2026-10-04 cancel. Built on Blacksmith runners of up to 128 GB, and on Modal since 2026-10-07. Checked structurally on the runner, which has no NPU |
 | Vulkan (GPU) | **Published**: Qwen3, Qwen2.5, Llama 3.2 and SmolLM2 at 2k to 32k, same recipe as XNNPACK through ExecuTorch's Vulkan delegate, checked structurally on the runner (no GPU) and run through both apps on a Mali and two Adreno GPUs (finding 38) |
 | Qualcomm QNN (SM8750; SM8650 optional) | Workflow in place for the checkpoints in ExecuTorch 1.4.0's Qualcomm registry (Qwen3, Qwen2.5 base, Gemma 3 1B, SmolLM2 135M, SmolLM3 3B, Llama 3.2). Nothing published yet |
 | Samsung Exynos (ENN) | Waiting for an LLM path in ExecuTorch (1.4.0 ships the delegate with CNN examples only) |
@@ -81,8 +81,9 @@ stages are 87-93% of it:
   736 s (43%), NeuroPilot lowering 472 s (28%), over 4 chunks. Its limit was calibration
   memory, which held every prompt at once and pinned every window at 512. Holding one prompt at
   a time moved the rest to disk, so the runner for each window is picked by RAM and disk
-  together. On the Blacksmith tiers that put 2k to 32k within the 32-vCPU one; on the one
-  Namespace profile used since 2026-10-07 (15.7 GiB, no swap) no LFM2.5 window fits (finding 40).
+  together. On the Blacksmith tiers that put 2k to 32k within the 32-vCPU one; on the
+  Namespace profile briefly used on 2026-10-07 (15.7 GiB, no swap) none fitted (finding 40), and since
+  then each window gets a Modal Sandbox sized by the same estimate.
 
 ## Locally
 

config/pipeline.yaml

@@ -176,8 +176,8 @@ export:
         disk_bytes: 549755813888
         swap_gib: 0
 
-# Modal: where the exports run since 2026-10-07 (pipeline/remote.py). The GitHub jobs stay on
-# the Namespace profile and drive a Sandbox per window, sized to that window, with the repo's
+# Modal: where the exports run since 2026-10-07 (pipeline/remote.py). The GitHub jobs run on
+# GitHub-hosted ubuntu-latest, which caps a job at 360 minutes, and drive a Sandbox per window, sized to that window, with the repo's
 # pipeline inside an image built from the pinned requirements. Limits measured on the
 # workspace that day: 344064 MiB of memory and 64 physical cores per Sandbox, refused above.
 modal:

pipeline/export_mtk.py

@@ -133,8 +133,9 @@ def state_layout(script: str) -> str:
 # window, physical cores, minutes). Streaming calibration runs a corpus that fills the window
 # through the model twice, so the time and not the memory grows with the window: the 16k
 # 8-chunk LFM2.5-1.2B build took 3,235 s on 8 physical cores (16 vCPUs, finding 39), and a
-# 32k calibration runs twice the tokens over twice the cache.
-MODAL_SHAPES = ((4096, 8, 240), (16384, 16, 330), (32768, 32, 600))
+# 32k calibration runs twice the tokens over twice the cache. The minutes stay inside the 360 a
+# GitHub-hosted job may wait, so the long windows get more cores instead.
+MODAL_SHAPES = ((4096, 8, 240), (16384, 32, 330), (32768, 64, 330))
 # The smallest Sandbox: lowering and MediaTek's compiler service on top of the estimate.
 MODAL_FLOOR_BYTES = 16 * 2**30
 # From this window up, a hybrid is cut so no chunk holds two attention layers (finding 39).

pipeline/export_qnn.py

@@ -58,9 +58,10 @@
 # The window enters through calibration and the two graphs' attention, at most half again by 8k.
 QNN_WINDOW_SCALE = 16384
 # Time grows with the model and faster than the window: Qwen3-0.6B took 2,434 s at 2k and
-# 5,987 s at 4k on a 4 vCPU runner (findings 8 and 10). (largest parameter count, physical
-# cores, Sandbox minutes); the minutes cost nothing unless a run hangs.
-QNN_MODAL_SHAPES = ((1_300_000_000, 16, 720), (4_500_000_000, 32, 1380))
+# 5,987 s at 4k on a 4 vCPU runner (findings 8 and 10), and SmolLM2-135M 1,041 s at 2k on 16
+# Modal cores. (largest parameter count, physical cores, Sandbox minutes). The minutes stay
+# inside the 360 a GitHub-hosted job may wait, so the larger models get more cores instead.
+QNN_MODAL_SHAPES = ((1_300_000_000, 16, 330), (4_500_000_000, 64, 330))
 
 
 def modal_request(params: int, window: int) -> tuple[int, int, int]:
@@ -94,7 +95,9 @@ def llama_command(
     command = [
         sys.executable,
         "-m",
-        "executorch.examples.qualcomm.oss_scripts.llama.llama",
+        # ExecuTorch's executorch.examples.qualcomm.oss_scripts.llama.llama, with the upstream
+        # fix for tokenizers without a BOS token (pipeline/qnn_launch.py says why).
+        "pipeline.qnn_launch",
         "--decoder_model",
         decoder,
         "--soc_model",
@@ -283,6 +286,8 @@ def run(
     if qairt != expected:
         raise ExportError(f"executorch's QAIRT is {qairt}, config/versions.env pins {expected}")
     env = qnn_env(sdk, work_dir / "qnn-libs", dict(os.environ))
+    # The script runs from work_dir, so `-m pipeline.qnn_launch` needs this repo on the path.
+    env["PYTHONPATH"] = os.pathsep.join(p for p in (str(settings.ROOT), env.get("PYTHONPATH", "")) if p)
 
     started = time.time()
     # The script fetches the weights itself (repo_id in its registry); only Llama 3.2 needs

pipeline/qnn_launch.py

@@ -0,0 +1,38 @@
+"""Runs ExecuTorch's Qualcomm llama script with the one upstream fix the pinned wheel lacks.
+
+    python -m pipeline.qnn_launch --decoder_model qwen3-0_6b ...   (export_qnn.llama_command)
+
+The script calibrates on wikitext through lm_eval, using ExecuTorch's EagerEvalWrapper as the
+model (examples/qualcomm/oss_scripts/llama/dataset/loaders.py, collect_lm_eval_tokens). In
+1.5.1, EagerEvalWrapper.prefix_token_id returns the tokenizer's bos_id whenever the attribute
+exists, even when it is None, and Qwen3's tokenizer_config has no bos_token. lm_eval then
+puts None at the head of the first rolling window and fails building the tensor ("'NoneType'
+object cannot be interpreted as an integer"): every Qwen3 window failed this way on
+2026-10-07. ExecuTorch main falls back to the end-of-text token when bos_id is None
+(examples/models/llama/evaluate/eager_eval.py), as lm_eval's own HFLM does; this is that
+property, set before the script runs. Drop this module when the ExecuTorch pin has the fix.
+"""
+
+from __future__ import annotations
+
+import runpy
+
+SCRIPT = "executorch.examples.qualcomm.oss_scripts.llama.llama"
+
+
+def _prefix_token_id(self):
+    bos_id = getattr(self._tokenizer, "bos_id", None)
+    if bos_id is not None:
+        return bos_id
+    return self.eot_token_id
+
+
+def patch() -> None:
+    from executorch.examples.models.llama.evaluate import eager_eval
+
+    eager_eval.EagerEvalWrapper.prefix_token_id = property(_prefix_token_id)
+
+
+if __name__ == "__main__":
+    patch()
+    runpy.run_module(SCRIPT, run_name="__main__", alter_sys=True)

pipeline/remote.py

@@ -4,7 +4,7 @@
         --put "$WORK_DIR/codes.pt" --get "$WORK_DIR/out" \\
         --script 'python -m pipeline export-xnnpack "$MODEL_ID" --context "$CONTEXT" --out "$OUT" --work "$WORK"'
 
-The GitHub job stays small (the Namespace profile) and waits here; the step itself runs in a
+The GitHub job stays small (GitHub-hosted, ubuntu-latest) and waits here; the step itself runs in a
 Sandbox with the memory its window needs, which a fixed runner cannot offer: an export's peak
 grows with the window squared, from a few GiB at 2k to 134 GiB for Qwen3-4B at 32k. Modal
 gives a Sandbox up to 344064 MiB and 64 cores (measured 2026-10-07), with no swap.