[S4] Make source builds reproducible and add pinned consumer canaries #141

Why S4 now

S1 (#131), S2 (#134 / #132), and S3 (#140 / #137) characterize the existing runtime before additive API work. Next, make consumption of a selected source revision reproducible and verifiable. Package-version synchronization/publication was already retired in #136 / #135; fixed local npm/setuptools metadata is not provenance.

Implementation baseline: d70cc1960af694d4a2abec101c9118b9e6d42a67. Preserve the TypeScript lifecycle suite and real Python stdio/settings characterization.

Concrete problems

• Shared npm run build runs tsc without cleaning dist: deleted/renamed source can leave obsolete JS/declarations.
• Pinned Flake8 2eeefe33258bb9b5ed6a790f74cd99ef70cfe7c3, shared gitlink ea47d519798d121fea262aaae23f0571e8f3d6da, uses a file: TypeScript dependency but postinstall accepts any existing dist/index.js. Moving its submodule does not invalidate that output. noxfile.py independently installs Python with --no-deps into bundled/libs.
• A manifest containing only git rev-parse HEAD can falsely describe dirty source, pick up a consumer's ancestor repository, or accompany mismatched/stale artifacts. Metadata alone does not establish artifact contents. Consumer dependency closure and actual bundled/imported paths require separate validation.

Bounded first increment: shipped scope

Scope update: Per direct user direction, the proposed implementation excludes Flake8-canary code, workflows, and documentation, and does not add a generic .s4-* ignore rule. Any pinned consumer comparison is LOCAL-ONLY validation, staged under ignored test-submodules/, not part of the proposed change. Checked-in consumer canaries, shared consumer-check workflows, and consumer-pipeline adoption are follow-ups.

1. Make ordinary shared TypeScript builds clean dist before compilation.
2. Add a narrow standard-library source-build CLI under scripts/, using existing npm/TypeScript and setuptools tooling, not a generic orchestration framework. Build TS and a Python wheel from one isolated source snapshot, never reuse existing dist, and refuse pre-existing output destinations.
3. Record the exact shared checkout root/commit, clean-vs-dirty identity and a deterministic input-content fingerprint; dirty builds require explicit opt-in and must never claim clean commit identity. Require an explicit expected shared commit and reject ancestor-repository resolution.
4. Bind provenance to complete artifact file hashes and check it against the selected current source before consumption; detect input edits/removals, moved pins, missing/extra/tampered artifacts and TS/Python mismatches. Reproducible here means locked TS dependency resolution and deterministic input/output verification, not a promise of byte-identical wheels across timestamps/toolchains.
5. Add focused source-build regression tests, integrate their script checks into existing nox lint configuration, and document generic build/verify usage and limitations. Keep staging/dependencies local and isolated.

Expected shipped touchpoints: typescript/package.json, scripts/source_build.py, scripts/source_build_test.py, noxfile.py, .gitignore (only /test-submodules/), and generic README/CONTRIBUTING build guidance. No consumer-specific script, workflow, or documentation is included. No runtime source, lifecycle, or layout changes are needed.

Shipped acceptance and validation

• python -m unittest discover -s scripts -p "*_test.py"; targeted new-script lint/format checks and git diff --check.
• With the runtime from .nvmrc: cd typescript && npm ci && npm run build && npm test && npm run lint.
• Document exact source-build and verify CLI commands, including explicit expected commit and dirty opt-in.
• Demonstrate the stale/deleted-output failure before the fix; afterward, ordinary shared builds remove obsolete artifacts and verification rejects edited/deleted/added source, wrong pin, mixed artifacts, and tampering.
• Build and verify paired TS/wheel artifacts from the selected source. Dirty source must remain explicitly dirty in provenance; source and artifact hashes must agree.
• Existing settings/environment/lifecycle/tool behavior remains unchanged. Record actual test/runtime versions and known unrelated formatting failures rather than broadening this task.

LOCAL-ONLY pinned consumer validation (not shipped)

The local validation compares immutable Flake8 extension revision 2eeefe33258bb9b5ed6a790f74cd99ef70cfe7c3 against its own clean shared baseline ea47d519798d121fea262aaae23f0571e8f3d6da and the candidate d70cc1960af694d4a2abec101c9118b9e6d42a67 plus its honest dirty-source fingerprint. Consumer source and tool policy remain fixed. The local harness/evidence live only under ignored test-submodules/; no harness, consumer documentation, or shared consumer CI is proposed.

Evidence covers real production extension JS and VSIX packaging, actual shared webpack inputs and packaged Python wheel contents, consumer hash-pinned Python requirements, and isolated packaged-server imports. Real stdio initialization, representative F401 diagnostics, shutdown, and exit compare equally. This is a bounded packaged-artifact/stdio check, not VS Code extension-host integration, all-five compatibility, or a native consumer-pipeline pass.

Dependency/integration caveat: Native candidate-root npm ci fails against this older consumer lock, including changed shared requirements such as the candidate's dotenv ^18 versus the consumer's older selection. The local smoke installs the original frozen consumer toolchain with the baseline shared manifest, restores the exact candidate manifest, then installs the candidate's own exact runtime lock separately under its shared package. No consumer source or root lock changes are retained. The staged runtime closure is audited, but this workaround does not demonstrate native consumer-lock/install compatibility. Adoption requires a separately reviewed consumer lockfile/build integration change. The existing packaged smoke used Node 26.13.0 and Python 3.13.14; focused shared source-build tests and TS build/test/lint were additionally verified with .nvmrc Node 24.17.0 / npm 11.13.0, without relabeling or rebuilding the earlier consumer artifacts.

Larger S4 follow-ups (not this initial shipped change)

• Review and adopt source invalidation/provenance checks in the five consumer pipelines; their postinstall scripts and lockfile updates are outside this repository.
• Separately propose checked-in pinned-consumer validation scripts/workflows, including all-five, per-consumer baseline comparisons for Flake8, Black, Pylint, isort and Mypy. Preserve each tool's own diagnostic/formatting and execution modes rather than homogenizing policy.
• Treat Black's npm-to-source dependency migration as a separate adoption dependency, not proof of equivalence with a historical npm tarball.
• Add VS Code-host checks where needed; packaging/stdio smoke is deliberately narrower.

Non-goals

No synchronized versions, registry publishing, new wire-version/LSP fields, dependency modernization, physical client/server moves, scheduler rewrite, plugin/executor framework, strict schema/codegen, S5 typed configuration or S6 facade implementation. Do not homogenize tool policy or depend on concurrent lifecycle fixes.