Skip to content

Commit 804a70f

Browse files
ellainngellainng
authored
perf(PERF-01): measure compiled payload and browser worker costs (#37)
1 parent 35fbe6e commit 804a70f

8 files changed

Lines changed: 5982 additions & 0 deletions

File tree

‎.github/workflows/ci.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,16 +57,21 @@ jobs:
5757
with:
5858
version: '0.12.19'
5959
enable-cache: true
60+
- uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
61+
with:
62+
node-version: '24.19.0'
6063
- run: uv python install
6164
- run: uv sync --locked --group browser-tests
6265
- run: uv run --no-sync python -m playwright install --with-deps chromium
6366
- run: uv run --no-sync python scripts/compiler_conformance.py --browser --report build/compiler-parity-report.json
6467
- run: uv run --no-sync python scripts/compiler_browser_smoke.py --report build/compiler-browser-report.json
68+
- run: uv run --no-sync python scripts/measure_compiler.py --browser --repeats 3 --output build/compiler-measurements
6569
- uses: actions/upload-artifact@v5
6670
if: always()
6771
with:
6872
name: compiler-browser-report
6973
path: |
7074
build/compiler-browser-report.json
7175
build/compiler-parity-report.json
76+
build/compiler-measurements/report.json
7277
if-no-files-found: error

‎IMPLEMENTATION_PLAN.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,3 +481,11 @@ Shimify's exact MIT notice, including the runnable browser example. The versione
481481
[compatibility matrix](docs/compiler-conformance.md) records the adopted scope
482482
and remaining differences; full Python tracebacks and arbitrary dynamic Python
483483
remain outside this compiler profile. No release was published.
484+
485+
PERF-01 now provides [reproducible compiler measurements](docs/compiler-measurements.md):
486+
required helpers and full helpers for the same profile pass the shared reference
487+
corpus in Node/browser workers. Reports separate source, generated modules,
488+
helpers, maps, notices, metadata/harness and compressed payloads, with raw warm
489+
build, fresh/warm context startup and separate dedicated-worker V8 heap samples.
490+
These synchronous compiler fixtures do not include the browser-host/PWA shell;
491+
complete application and future interpreter measurements remain separate scopes.

‎docs/compiler-measurements.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Reproducing compiler measurements
2+
3+
PERF-01 measures the existing helper selection in the development compiler. It
4+
compares required helpers with every helper available to the same language
5+
profile; it does not add an optimization or enable otherwise unsupported syntax.
6+
The complete positive compiler corpus must match CPython in both variants, in
7+
Node and in a real browser worker. Rejected/deferred features remain covered by
8+
the separate compiler conformance gate. Published 0.1.0 predates these APIs.
9+
10+
```sh
11+
uv sync --locked --group browser-tests
12+
uv run --no-sync python -m playwright install chromium
13+
uv run --no-sync python scripts/measure_compiler.py --browser --repeats 5 \
14+
--output build/compiler-measurements
15+
```
16+
17+
Choose a new output directory. Omit --browser for the Node/reference and build/
18+
size gates. --channel chrome selects an installed Chrome and records its actual
19+
version. The reference and Node versions in CI are 3.14.7 and 24.19.0; the locked
20+
Playwright 1.64.0 installs its Chromium build. Hardware, OS caches, CPU frequency
21+
and background load are not controlled. Compare reports with matching recorded
22+
inputs and versions; the command accepts other development environments and
23+
reports them rather than silently treating them as identical.
24+
25+
The output contains schema-1 compiler-footprint/1 report.json and twelve
26+
reproducible .tar.gz browser-fixture archives. Reports record the repository
27+
revision/modified state, actual installed compiler source hashes with canonical
28+
LF line endings, measurement/corpus hashes, Python/Node/browser/host versions,
29+
every output file/hash, raw build/startup/heap samples and reference outcomes.
30+
No host-specific absolute source paths are included. The initial observations
31+
were collected before their commit; the recorded script and compiler hashes
32+
identify the actual measurement inputs.
33+
34+
## Payload accounting and behavior
35+
36+
Three fixtures exercise numeric/Unicode values, the shared text-metrics callable,
37+
and its invalid-input error. Each has required/all-profile helpers and included/
38+
omitted maps. The application source and generated executable module statements
39+
are the same within a helper comparison. The baseline includes all synchronous
40+
helpers for a synchronous profile, or all non-browser helpers for an async corpus
41+
case. Injecting browser/async adapters into a synchronous application would change
42+
its requirements and is not a comparable baseline.
43+
44+
The report separates original input source, generated executable modules,
45+
compatibility helpers, source maps, notices, compiler metadata and the small
46+
measurement HTML/worker harness. The physical buckets sum to complete payload
47+
bytes. Original source is a separate input metric; source maps can embed those
48+
same bytes. Interpreter assets are zero in this compiled profile. The browser-host
49+
SDK and pocket-python PWA shell are outside these synchronous fixtures and need
50+
separate complete-application measurements.
51+
52+
Per-file gzip transfer bytes sum gzip level-9, mtime-zero streams for every file;
53+
HTTP headers, TLS framing and server content-encoding policy are excluded. This
54+
is a reproducible transfer estimate, not captured network traffic. A separate
55+
canonical USTAR archive uses sorted paths, fixed modes/ownership/timestamps and
56+
includes archive padding before gzip. Compressed archive bytes and independently
57+
compressed files answer different questions. Record the zlib version when
58+
comparing exact compressed bytes.
59+
60+
The shared computation retains Shimify's exact MIT notice in every corresponding
61+
artifact. Other fixtures are original regression cases. Compiler provenance
62+
notices do not choose a license for the compiler itself. Only these approved
63+
fixtures are executed; the script is not a general application benchmark runner.
64+
65+
## Timing and memory methods
66+
67+
Build samples measure compilation and manifest generation in an already imported
68+
Python process after warming the compiler's Unicode repr table. They exclude
69+
Python startup, dependency installation and filesystem publication. The required
70+
and all-profile variants must emit identical bytes on every repeated build.
71+
72+
Cold browser samples create a fresh storage/HTTP-cache context and a fresh module
73+
worker for each observation. The running browser process and OS caches are
74+
shared. Warm samples use one context, perform an explicit untimed warm-up, then
75+
create a new worker for each observation. HTTP cache is shared; V8 decides whether
76+
its code cache is reusable. Every served file is immutable during the run and
77+
uses max-age=3600. No service worker or server computation enters this measurement.
78+
79+
The page timer starts before worker creation and ends after its result message.
80+
It includes worker bootstrap, application import, module initialization and one
81+
complete call; document navigation is outside that timer. Worker observations
82+
also separate dynamic import from initialization/call time. Every observed result
83+
and original error span must match CPython. Both helper modes additionally run
84+
all 24 positive corpus cases in Node and the browser (48 observations per target).
85+
These include precision/Unicode, side effects, imports, original errors and async
86+
cases; the measured synchronous fixtures do not imply broader browser-host support.
87+
88+
Memory uses separate fresh-context workers, outside startup samples. The command
89+
selects the dedicated worker by URL and browser-context identity, forces a V8 GC,
90+
and reads CDP Runtime.getHeapUsage before importing the application and after
91+
retaining one module/result. It records used/total V8 heap, embedder heap and
92+
backing storage in bytes, plus the used-heap difference. These figures exclude
93+
process RSS, other processes and native/code allocations outside that API. A
94+
post-GC retained heap difference is not peak application memory or a hard memory
95+
limit. Memory samples must also produce the same reference outcome.
96+
97+
The script refuses a measurement if helper variants disagree with the reference,
98+
any timed/memory call changes its result, browser scripts raise unhandled errors,
99+
or requests leave the local static GET fixture. Tests also deliberately corrupt
100+
an arithmetic helper to demonstrate that agreement between two wrong variants
101+
cannot satisfy the CPython gate. Neither smaller source nor a single run's timing
102+
establishes a general speed or memory improvement.
103+
104+
## Recorded observations
105+
106+
[The 2026-10-20 report](measurements/2026-10-20-compiler.json) records five samples
107+
per condition on one Darwin arm64 host, CPython 3.14.7, Node 24.19.0 and Chromium
108+
156.0.8078.4. The tables below describe that run only; raw samples and exact file
109+
identities remain in the report.
110+
111+
| Fixture / variant | Input source B | Helper B | Complete payload B | Per-file gzip B |
112+
| --- | ---: | ---: | ---: | ---: |
113+
| numeric-and-unicode / required-maps | 138 | 9,796 | 17,855 | 6,522 |
114+
| numeric-and-unicode / all-profile-maps | 138 | 23,547 | 31,665 | 11,668 |
115+
| numeric-and-unicode / required-no-maps | 138 | 9,796 | 17,379 | 6,219 |
116+
| numeric-and-unicode / all-profile-no-maps | 138 | 23,547 | 31,189 | 11,367 |
117+
| text-metrics / required-maps | 702 | 10,357 | 25,152 | 8,947 |
118+
| text-metrics / all-profile-maps | 702 | 23,547 | 38,378 | 13,996 |
119+
| text-metrics / required-no-maps | 702 | 10,357 | 23,612 | 8,146 |
120+
| text-metrics / all-profile-no-maps | 702 | 23,547 | 36,838 | 13,194 |
121+
| text-metrics-error / required-maps | 651 | 10,357 | 24,614 | 8,815 |
122+
| text-metrics-error / all-profile-maps | 651 | 23,547 | 37,840 | 13,868 |
123+
| text-metrics-error / required-no-maps | 651 | 10,357 | 23,146 | 8,053 |
124+
| text-metrics-error / all-profile-no-maps | 651 | 23,547 | 36,372 | 13,103 |
125+
126+
| Text-metrics variant | Warm build ms | Cold context ms | Warm context ms | Retained used-heap delta B |
127+
| --- | ---: | ---: | ---: | ---: |
128+
| required-maps | 1.290 | 10.400 | 4.200 | 67,152 |
129+
| all-profile-maps | 1.359 | 10.600 | 4.600 | 107,824 |
130+
| required-no-maps | 1.165 | 10.100 | 4.300 | 67,088 |
131+
| all-profile-no-maps | 1.256 | 10.100 | 4.500 | 107,760 |
132+
133+
Helper selection reduces these fixture payloads relative to their full-profile
134+
baselines. The text-metrics required runtime is 10,357 bytes versus 23,547 bytes
135+
for all synchronous helpers. These observations do not compare full browser
136+
applications, MicroPython, Pyodide or the independent interpreter, and they do not
137+
establish a general startup or memory improvement.

0 commit comments

Comments
 (0)