Skip to content

site-build: the GPU leg; programming joins the base environment - #87

Merged
mmcky merged 2 commits into
mainfrom
site-build/programming
Nov 22, 2026
Merged

mmcky merged 2 commits into
mainfrom
site-build/programming

Conversation

@mmcky

@mmcky mmcky commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

What

Step 3 of bringing the sites into one environment for every site (QuantEcon/project-monorepo#77), in the owner's order of 2026-11-22: intro (#84), advanced (#86), then the GPU sites, programming first. This writes the GPU leg and builds programming's site on it.

The GPU leg's shape is the owner's choice of 2026-11-22: "The scan's shape (Recommended)", that is, "setup-miniconda with python=3.13 + anaconda=2026.07 on the RunsOn machine, then the list, then jax[cuda13]==0.12.1 with a GPU check; the design note revised to match". The environment scan of 2026-11-19 built all three GPU sites in this shape with nothing new. It gives a GPU site the same environment as a CPU site: the RunsOn machine image's own Conda environment has no anaconda metapackage, so updating that one with the list (the design note's first plan) would not have.

File Change
tools/site-matrix The g4dn.2xlarge class has a leg. Each entry gains leg (container or gpu) and machine (RunsOn's spec: family=g4dn.2xlarge/ami=ami-0edec81935264b6d3/volume=80gb/spot=false, the AMI the GPU series' own .github/runs-on.yml names). A GPU site's runs_on is RunsOn's label, runs-on=<run ID>/<machine>, from the new --run-id, and its image is empty. A GPU site outside environment: base is refused.
.github/workflows/site-build.yml container: ${{ matrix.site.image }}, empty for a GPU site, so its job runs on the machine. Three GPU steps: setup-miniconda creates _base/environment.yml; conda env update applies _base/environment-update.yml; then nvidia-smi, jax[cuda13] at the list's jax version, and a check that jax.devices() is a GPU. A GPU site's cache key hashes the machine spec and _base/environment.yml in place of /etc/quantecon-container. Staging and unstaging run in a login shell, where the GPU machine has the environment's python. SITES: dp intro advanced programming. Comments updated.
_base/environment.yml New: the GPU leg's base, Python 3.13 and anaconda=2026.07, as the full image carries it.
sites/programming/site.yml environment: base
tools/tests/test_site_matrix.py The GPU entry and its refusals; the workflow passes the run's ID.
sites/README.md, _base/README.md, README.md The GPU leg, and programming. Three stale lines from the reviews of #85 and #86 are corrected too: "the other four sites" (two now), the root README's "dp alone" (lines 49, 94 and 209), and the workflow's "(dp publishes HTML alone)" (now "HTML alone for now").

_base/environment-update.yml is not touched, so intro's and advanced's caches keep their keys. The job limit for programming is 130 minutes (cell_timeout 600 s plus 120). It builds HTML alone for now.

Validation

Done on 2026-11-22; the evidence is in the comment below. Programming's leg builds green and strict on the RunsOn GPU machine, with no container, in 7 minutes. jax 0.12.1 finds the T4, every page with code is executed, and all 39 pins are installed with jax[cuda13] on top. Against two fresh builds of programming's own at bae31ee, every difference beyond their noise is install output or the working directory a cell prints. The cell-output warnings are the same, and the build log has no Sphinx warning. The container legs still build: intro and dp restored their caches under unchanged keys, and advanced built cold and green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WT8bMacV9Lmy1VLUc6XJoL

Step 3 of bringing the sites into one environment for every site
(QuantEcon/project-monorepo#77), by the owner's order of 2026-11-22:
intro, then advanced, then the GPU sites, programming first.

A site on the g4dn.2xlarge class now builds on a RunsOn machine with one
NVIDIA T4, from QuantEcon's GPU machine image (by the AMI ID the GPU
series' own runs-on.yml gives), with no container, and in the base
environment alone. In the shape the owner chose on 2026-11-22, the one the
environment scan of 2026-11-19 proved: setup-miniconda creates
_base/environment.yml (Python 3.13 and anaconda=2026.07, the full image's
base), conda env update applies _base/environment-update.yml, and
jax[cuda13] at the list's jax version is added, with a check that jax finds
the GPU.

tools/site-matrix gives each entry its leg and RunsOn machine spec, builds
RunsOn's label from --run-id, and refuses a GPU site outside the base
environment. The job's container is empty for a GPU site. Its cache key
hashes the machine spec and _base/environment.yml in place of the image's
marker file. Stage and unstage run in a login shell, where the GPU machine
has the environment's python. _base/environment-update.yml is untouched,
so intro's and advanced's caches keep their keys.

The READMEs name programming and the GPU leg. Three stale lines from the
reviews of #85 and #86 are corrected on the way: the
count of sites still to move in sites/README.md, the root README's "dp
alone", and the workflow's "(dp publishes HTML alone)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMacV9Lmy1VLUc6XJoL
@mmcky

mmcky commented Nov 22, 2026

Copy link
Copy Markdown
Contributor Author

Validation: programming built from the pool on the GPU leg, against programming's own builds

Builds compared, all of programming's lectures/ at bae31ee, the pool's pin and programming's main:

Build Run What
This pull request, programming leg 45009278207 at eb99d27 from the pool on the RunsOn GPU machine (Tesla T4, driver 580.105.08, CUDA 13.0) with no container, strict (-n -W), cold: 26 pages executed and 1 taken from the cache (troubleshooting, which has no code), none failed; 7 minutes for the whole job
programming's own, fresh QuantEcon/lecture-python-programming 45009276381 cache.yml on main, 2026-11-22
programming's own, fresh again QuantEcon/lecture-python-programming 45009934467 the same, minutes later, for the noise baseline

The GPU leg's environment. setup-miniconda created _base/environment.yml (Python 3.13 and anaconda=2026.07), conda env update applied the list, and jax[cuda13]==0.12.1 was added. The check printed jax 0.12.1 [CudaDevice(id=0)]. The installed jax packages are jax, jaxlib, jax-cuda13-plugin and jax-cuda13-pjrt, all at 0.12.1. All 39 pins in _base/environment-update.yml are installed at their versions: 38 by pip, and plotly-orca 1.3.1 by conda. Adding jax[cuda13] moved none of them.

Noise baseline. programming's two own builds differ on 13 of 31 pages: random draws, memory addresses and object identities, timings and download progress. This pull request's build differs from the first on 15 pages: the same 13, plus two where every difference is explained.

  • jax_intro: install output only. The list already holds quantecon, so pip prints "Requirement already satisfied", where programming's build downloads it.
  • python_essentials: the working directory a cell prints, /home/runner/_work/lectures/lectures/lectures against /home/runner/_work/lecture-python-programming/lecture-python-programming/lectures.

On the 13 noise pages, every line that differs is a random value, an address, a timing or install output, as between programming's own two builds. The status page differs only in its times.

One file only programming's build has: _admonition/gpu.html. That is the GPU admonition, an include fragment, which Sphinx builds as an orphan page when it sits in the source directory; no table of contents links it. The pool's site does not stage it as a page. The pages that include it (jax_intro and others) carry the same text in both builds.

Warnings. The build log has no Sphinx warning; its only warning lines are Node.js deprecation notices from GitHub's own actions. The cell outputs carry the same warnings and tracebacks as programming's own build, page by page. These are the lectures' own demonstrations: names lists the built-in names, the warning classes among them, and debugging, numpy, oop_intro, python_advanced_features and python_essentials raise errors on purpose.

The other legs. intro and dp restored their execution caches under unchanged keys, with every page taken from the cache. So the string form of container: and the login-shell staging work in the containers. advanced built cold, because this run started before main's first build of advanced had saved its cache: 77 pages executed and 3 from the cache, none failed, in 45 minutes, the same as in #86.

@mmcky
mmcky marked this pull request as ready for review November 22, 2026 10:33
Copilot AI adaptive review requested due to automatic review settings November 22, 2026 10:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the GPU build and shared-environment changes.

No open issues 🎉

What changed in this PR

Adds a GPU build leg to the shared site-build workflow and moves programming into the common base environment, following intro and advanced.

Changes:

  • Adds RunsOn GPU runner support, Conda environment setup, and CUDA-enabled JAX with a GPU check.
  • Enables programming’s strict HTML build and GPU-specific execution cache keys while preserving CPU cache keys.
  • Extends matrix tests and updates documentation for the four enabled sites.
File Description
tools/​tests/​test_site_matrix.py Tests GPU matrix entries, invalid configurations, and workflow run-ID forwarding.
tools/​site-matrix Adds GPU runner settings and run-ID validation.
sites/​README.md Documents GPU builds, environment selection, and caching.
sites/​programming/​site.yml Selects the shared base environment for programming.
README.md Updates the build overview to include all four enabled sites.
.github/​workflows/​site-build.yml Adds GPU setup, CUDA verification, cache handling, and programming builds.
_base/​README.md Explains the GPU base environment and shared package list.
_base/​environment.yml Defines the GPU base with Python 3.13 and Anaconda 2026.07.

Review effort: Balanced (adaptive)

The organiser's review of #87; the owner chose "Fix the
channel, then merge". _base/environment.yml named the channel `default`,
copied from programming's environment.yml. That is the anaconda.org user
channel "default", empty today, which conda would search first; the
Anaconda channels are `defaults`. The GPU leg's cache key hashes this
file, so programming's key changes.

Wording: the base environment's container and root-user notes say they
are the container leg's; extra_packages go into "the environment jb runs
in"; sites/README's environment row names the GPU leg; _base/README no
longer hard-codes the jax version the workflow reads from the list. The
cache-key step tests the leg, as the other steps do.
_base/environment-update.yml's own comment still names 0.12.1: the base
cache key hashes that whole file, so it waits for the plotly-orca pull
request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMacV9Lmy1VLUc6XJoL
@mmcky

mmcky commented Nov 22, 2026

Copy link
Copy Markdown
Contributor Author

Validation of 02101d7: the channel fix

_base/environment.yml now names the conda channel defaults; default was the empty anaconda.org user channel of that name. Run 45014389106 is green on every leg.

  • programming (GPU): conda now lists one channel, defaults. The installed packages (pip list, 415 of them) are identical to those of the first run, 45009278207. The leg's cache key changed with the file, so the leg built cold again: 26 pages executed and 1 from the cache, none failed. jax 0.12.1 found the T4.
  • advanced, intro and dp restored their caches. advanced's comes from main's first base build of advanced, 45008401723.

One correction to the evidence above: the build log's warning lines are not only Node.js notices. On the GPU leg, setup-miniconda also prints its deprecation notices (auto-activate-base is deprecated), and conda prints a conda-pypi tip as a ::warning::. Neither comes from the lectures or Sphinx, and the GPU series' own workflows print the same.

@mmcky
mmcky merged commit 066c271 into main Nov 22, 2026
10 checks passed
@mmcky
mmcky deleted the site-build/programming branch November 22, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants