From 9fa7c03ef91456eb7685d2a6730690dd724b0a2f Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 19 Nov 2026 16:57:26 -0800 Subject: [PATCH 1/5] docs(openspec): propose shipping Vale's grammars as their own package Vale 3.27.0 moved its tree-sitter grammars out of the binary. Without them it reads code comments by delimiters, which fails 6 vendor-contract cases on #703. Propose @taskless/vale-grammars, pinned and released with the platform packages, named to Vale through VALE_GRAMMARS_PATH. --- .../vale-grammars-package/.openspec.yaml | 2 + .../changes/vale-grammars-package/design.md | 89 +++++++++++++++++++ .../changes/vale-grammars-package/proposal.md | 49 ++++++++++ .../specs/cli-runtime-binary-export/spec.md | 21 +++++ .../specs/cli-vale-rule-engine/spec.md | 52 +++++++++++ .../specs/vale-binary-packages/spec.md | 64 +++++++++++++ .../changes/vale-grammars-package/tasks.md | 47 ++++++++++ 7 files changed, 324 insertions(+) create mode 100644 openspec/changes/vale-grammars-package/.openspec.yaml create mode 100644 openspec/changes/vale-grammars-package/design.md create mode 100644 openspec/changes/vale-grammars-package/proposal.md create mode 100644 openspec/changes/vale-grammars-package/specs/cli-runtime-binary-export/spec.md create mode 100644 openspec/changes/vale-grammars-package/specs/cli-vale-rule-engine/spec.md create mode 100644 openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md create mode 100644 openspec/changes/vale-grammars-package/tasks.md diff --git a/openspec/changes/vale-grammars-package/.openspec.yaml b/openspec/changes/vale-grammars-package/.openspec.yaml new file mode 100644 index 00000000..ce56180a --- /dev/null +++ b/openspec/changes/vale-grammars-package/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-11-19 diff --git a/openspec/changes/vale-grammars-package/design.md b/openspec/changes/vale-grammars-package/design.md new file mode 100644 index 00000000..846a5e9c --- /dev/null +++ b/openspec/changes/vale-grammars-package/design.md @@ -0,0 +1,89 @@ +## Context + +See proposal.md, "Why", for the regression. The facts below were measured on the pinned `vale 3.27.0` (darwin-arm64) and decide the approach: + +- Vale looks for a grammar in `/config/grammars/.wasm`, then in the directory named by the `VALE_GRAMMARS_PATH` environment variable. The fallback works **one grammar at a time**: a StylesPath holding only `go.wasm`, plus `VALE_GRAMMARS_PATH` holding every grammar, parsed Go, CSS and Python with real grammars and emitted no `W101`. An empty `config/grammars/` directory behaves the same. +- Upstream publishes 22 `grammar-.zip` assets on each release. Each zip holds `styles/config/grammars/.wasm`. They total about 15 MB zipped and 65 MB unpacked. `vale__checksums.txt` covers the six binary archives and `vale__wasm.tar.gz`, and **not** the grammar zips. +- Probing every comment-tier extension in `VALE_FORMAT_TIERS` against the bare binary names 20 grammars: bash, c, cpp, csharp, css, elixir, elm, go, java, javascript, julia, kotlin, lua, php, proto, python, ruby, rust, tsx, typescript. `.clj`, `.ps1` and `.qdoc` name none, because Vale reads them by delimiters with or without a grammar, as on 3.26.0. `toml` and `yaml` are unused today. +- With all 22 grammars supplied through `VALE_GRAMMARS_PATH`, all 373 Vale tests pass on 3.27.0, the six failing vendor-contract cases included. +- The CLI spawns Vale in exactly one place, `spawnVale` in `src/rules/vale/run.ts`. `verify` reaches it through `runVale`. Every assembled config sets `StylesPath` to the Vale rules tree (`.taskless/rules/vale`, `valeHeader` in `src/rules/assemble.ts`), and `verify` uses a scratch StylesPath of its own. +- The platform packages' release (`release-vale.yml`) already downloads, verifies against the manifest, stamps, packs, and publishes every tarball `vale-prepare.cjs` emits. The publish loop is generic over tarballs. + +## Goals / Non-Goals + +**Goals:** + +- Vale 3.27.0 behaves for a Taskless user exactly as 3.26.0 did, on every OS, with no network at lint time. +- Grammar bytes pass through the same review-gated manifest as the binaries. +- A future grammar gap fails CI instead of quietly falling back. + +**Non-Goals:** + +- Supporting Vale's `Lang` models (French, German, Spanish). Nothing the CLI assembles sets `Lang`. +- Honoring a user's own `vale sync`. Grammars outside the Vale rules tree are never read; the one inside it is refused. +- Shrinking the install by shipping a subset of grammars (see D2). + +## Decisions + +### D1. One platform-independent package, not grammars inside each platform package + +Grammars are WebAssembly, so the bytes are the same on every host. A single `@taskless/vale-grammars` package is installed once per host, at one version, and cannot differ between macOS and Linux. + +_Alternative: add the grammars to each of the six platform packages._ It needs no new npm name and no manual first publish. It was rejected because it publishes the same 15 MB six times, and turns "every OS has the same grammars" from a structural fact into something only the pipeline's discipline maintains. + +### D2. Ship every grammar upstream publishes + +The package carries all of upstream's grammars for the release, not the 20 the comment tier needs today. The extra two cost about 1 MB. Shipping the whole set means the manifest's grammar list is upstream's list, with no hand-maintained subset to drift. A new comment-tier row then needs no packaging change, and the coverage check (D6) is a check on upstream, not on our arithmetic. + +_Alternative: ship only the probed subset._ It is smaller by about 7%, but it adds a table that has to be re-derived on every bump and that only fails after it is already wrong. + +### D3. Digests are recorded at detect time and reviewed, since upstream does not sign them + +The manifest gains a `grammars` section: the package name and directory, the asset template `grammar-{name}.zip`, the archive member `styles/config/grammars/{name}.wasm`, and one `{ name, sha256 }` entry per grammar. `vale-detect.cjs` already runs without credentials. When upstream moves, it lists the release's `grammar-*.zip` assets, downloads each one, hashes it, and writes the list and digests into the manifest pull request. A human approves those digests exactly as they approve the binary digests. `vale-prepare.cjs` then verifies each archive against the committed digest before unzipping, and extracts only the named member, flattened to `grammars/.wasm`. + +This is trust-on-first-sight for the grammars, where the binaries are cross-checked against upstream's own checksums file. The review gate is the same; what is weaker is that a reviewer cannot compare the grammar digests with anything upstream published. That is the best upstream allows today, and it is stated in the manifest comment. + +### D4. One stamp, one gate, one upgrade + +`vale-prepare.cjs` emits the grammar tarball beside the six platform tarballs, at the same stamp, and the existing publish loop publishes all seven. `vale-gate.cjs` reports "already published" only when the platform packages and the grammar package both have a version for the pinned Vale release. `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs` treat the grammar package as a seventh member of the set: the upgrade pull request is opened only when all seven are published, and it moves all seven pins together. + +The CLI takes the grammar package as a regular dependency at an exact version, not an optional one, because there is no host that should go without it. `engine-version-consistency` asserts that the grammar pin's Vale component equals `VALE_VERSION` and the binary pins' Vale component. It does not require equal stamps: the manual first publish (Migration, step 2) stamps the grammar package once on its own. + +### D5. `VALE_GRAMMARS_PATH` is set at the single spawn site, from a resolver beside `findValeBinary` + +A sibling of `findValeBinary` resolves `@taskless/vale-grammars/package.json` and returns `/grammars`. It is cached per process, and a miss is reported as a value with the locations searched, as the binary resolver does. `spawnVale` adds `VALE_GRAMMARS_PATH` to the child environment it already builds for `PATH`. When the grammar package does not resolve, the variable is left unset and Vale runs. Its `W101` warnings then reach the user as notices, by the existing "diagnostics on a successful run" requirement. That is loud enough for a broken install, and it does not take the Markdown rules down with the code rules. + +It is set whichever tier resolved the binary. A `vale` from `PATH` older than 3.27.0 ignores the variable and uses its built-in grammars, so setting it costs nothing there. + +The resolver and the package spec are exported from `@taskless/cli/node/runtimes` next to `VALE_BINARY`, so the rule generator's verifier runs Vale with the same grammars. + +_Alternative: copy the grammars into `/config/grammars/` at assembly._ That copies 65 MB into the user's `.taskless/` tree, and makes our own grammars indistinguishable from the third-party ones D7 refuses. + +### D6. Coverage is asked of the binary, in the vendor contract + +The vendor contract runs every comment-tier extension in `VALE_FORMAT_TIERS` through the pinned binary with the pinned grammars. It fails on any `grammar "" is not installed` warning, naming the extension and the grammar. The binary is the authority on which grammar an extension needs, so the test asks it rather than keeping a mapping. The six cases failing on #703 stay as they are, and pass again once the grammars are present. + +### D7. The rules-tree grammar directory is refused at assembly + +`assembleValeConfig` checks for `/config/grammars/` before writing anything. When it exists, the result is a refusal naming the directory. That refusal is a tree-level entry beside the existing per-rule ones, reported the same way, so `validate` and `check` both refuse through the code they already share. Its presence is refused, not its contents, because an empty directory today becomes a populated one after the next `vale sync`, and an unvetted grammar would win over ours for that language. The rest of `config/` (vocabularies, for example) is not touched by this change. + +## Risks / Trade-offs + +- [Install grows by about 15 MB compressed and 65 MB unpacked] → Accepted; it is the price of offline, identical behavior. D2 keeps the cost of the two unused grammars at about 1 MB. +- [Upstream grammar digests have no upstream checksum to compare against] → Same review gate as the binaries, stated in the manifest comment. Revisit if upstream starts covering them. +- [A grammar ABI mismatch between a `PATH` Vale and our grammars] → Only reachable when the pinned platform package did not install, which the binary resolver already reports by tier (`source`). Vale's own error would surface as a notice or failure. +- [The publish gate fires on the PR A merge, and the grammar name does not exist yet] → The first publish happens before PR A merges (Migration, step 2), so the gate sees all seven published and does nothing. +- [Upstream renames or drops a grammar a tier extension needs] → D6 fails the upgrade pull request, naming the extension. + +## Migration Plan + +1. **PR A** (bottom, to `main`): this change's artifacts, the manifest `grammars` section for 3.27.0 with digests computed from upstream's current assets, `packages/vale-grammars/` metadata, and the detect, prepare, gate and upgrade script changes with their tests. It changes no CLI behavior and carries no changeset. +2. **Manual, before merging PR A**: a maintainer runs `vale-prepare.cjs` on PR A's branch, publishes the `@taskless/vale-grammars` tarball by hand with `--tag latest`, and registers its npm trusted publisher against `release-vale.yml` and the `npm-autopublish` environment. +3. Merge PR A. The gate finds 3.27.0 published for all seven packages and publishes nothing. `vale-upgrade.yml` rewrites #703 to include the grammar pin. +4. **PR B**, cut from #703's head and targeting `main`: the grammar dependency pin, the resolver and `VALE_GRAMMARS_PATH`, the runtime export, the D7 refusal, the D6 coverage check, the `update` ledger entry, and the grown `vale-3-27-0` changeset (patch). It archives this change. #703 is closed with "Superseded by #" once PR B lands. + +Rollback: revert PR B. The 3.26.0 pins come back and the grammar package becomes inert, since nothing pins it. PR A can stay. + +## Open Questions + +- Whether `vale-upgrade.yml`'s rolling branch rewrites #703 after step 3 or opens a fresh pull request. Either way, PR B is cut from whichever head carries the seven pins. diff --git a/openspec/changes/vale-grammars-package/proposal.md b/openspec/changes/vale-grammars-package/proposal.md new file mode 100644 index 00000000..55032d92 --- /dev/null +++ b/openspec/changes/vale-grammars-package/proposal.md @@ -0,0 +1,49 @@ +## Why + +Vale 3.27.0 moved its tree-sitter grammars out of the binary and into WebAssembly modules that `vale sync` downloads. The CLI never runs `vale sync`, so on 3.27.0 Vale finds comments in source files by their delimiters instead of parsing them. Measured on PR #703, which moves the pins to 3.27.0 and fails 6 vendor-contract cases: + +- linter directive comments (`//nolint`, `# noqa`, `// eslint-disable…`, `// ReSharper`, `// NOSONAR`) and a Python docstring's `:param:` lines are linted as prose: new false positives; +- a one-line CSS block comment (`/* … */`) is not linted at all: a new false negative; +- every run that touches a code file emits one `W101 grammar "" is not installed` warning per language, which `check` surfaces as notices. + +With the upstream grammars available through `VALE_GRAMMARS_PATH`, all 373 Vale tests pass on 3.27.0 unchanged. Shipping the grammars therefore turns a silent behavior change into no change at all. + +## What Changes + +- Publish one platform-independent package, `@taskless/vale-grammars`, carrying the grammar modules from the same upstream Vale release as the binaries. One package rather than a copy per platform package, so a grammar can never be one version on macOS and another on Linux. +- Pin the grammar archives' SHA256 digests in `.github/scripts/vale-manifest.json`. Upstream's `vale__checksums.txt` does not cover them, so the detect phase records the digests it downloads and the manifest pull request is where a human reviews them, the same trust boundary as the binaries. +- `vale-prepare.cjs` packs the grammar package beside the six platform packages, and `release-vale.yml` publishes all seven at one stamp. The first publish of the new name is a manual one by a maintainer, as npm trusted publishing requires, after which the trusted publisher is registered against the existing `release-vale.yml`. +- The CLI depends on `@taskless/vale-grammars` at an exact version whose Vale component matches the binary pins, and `vale-upgrade.yml` moves both pins together. +- Every Vale invocation sets `VALE_GRAMMARS_PATH` to the resolved grammar package directory. Vale reads that directory for any grammar the StylesPath lacks, one grammar at a time. +- A grammar directory inside the Vale rules tree (`.taskless/rules/vale/config/grammars/`, which Vale would read in preference to ours) is refused when the Vale config is assembled, so both `validate` and `check` reject grammars the CLI did not ship and pin. +- The runtime export (`@taskless/cli/node/runtimes`) publishes the grammar package spec and resolver beside `VALE_BINARY`, so a consumer that verifies Vale rules runs them with the same grammars the CLI does. +- A contract check fails when any comment-tier extension in `VALE_FORMAT_TIERS` needs a grammar the pinned package does not carry. + +Not breaking. The bump is `patch`: users see 3.26.0's comment handling on 3.27.0. + +## Delivery shape + +**Two landings**, because a pull request cannot pin a package that is not on npm yet: + +1. **PR A, single PR to `main`:** this proposal, the manifest's grammar section, packing, publishing, and the detect-phase digest recording. It changes no consumer and ships no release note. After it merges, a maintainer publishes `@taskless/vale-grammars` by hand and registers its trusted publisher. +2. **PR B, the lateral vendor-bump merge** (AGENTS.md, "Landing a machine-generated vendor bump"): cut from #703's head, targeting `main`. It carries the CLI pins, `VALE_GRAMMARS_PATH`, the rules-tree refusal, the runtime export, the coverage check, the `update` ledger entry, and the grown changeset. It may be a stack if the review size calls for one. It archives this change and supersedes #703. + +## Capabilities + +### New Capabilities + +None. + +### Modified Capabilities + +- `vale-binary-packages`: adds the grammar package, its committed digests, and its release beside the platform packages. +- `cli-vale-rule-engine`: Vale reads code with the pinned grammars, and a grammar directory in the rules tree is refused. +- `cli-runtime-binary-export`: the grammar package spec and resolver are published beside the binary resolver. + +## Impact + +- `.github/scripts/vale-manifest.json`, `vale-detect.cjs`, `vale-release.cjs`, `vale-prepare.cjs`, `vale-gate.cjs` and their tests; `.github/workflows/release-vale.yml`, `vale-upgrade.yml`. +- New `packages/vale-grammars/` (metadata only; modules are placed by the pipeline and never committed). +- `packages/cli`: `package.json` dependency, `src/rules/vale/run.ts` (spawn environment), `src/rules/vale/binary.ts` or a sibling grammar resolver, `src/rules/assemble.ts` (refusal), `src/node/runtimes/index.ts`, `src/agent/update.md`, the vendor-contract and engine-version-consistency tests. +- Install size: about 15 MB compressed for the grammar package, once per install regardless of platform. +- Manual, one time: the first publish of `@taskless/vale-grammars` and its trusted-publisher registration. diff --git a/openspec/changes/vale-grammars-package/specs/cli-runtime-binary-export/spec.md b/openspec/changes/vale-grammars-package/specs/cli-runtime-binary-export/spec.md new file mode 100644 index 00000000..568a10e1 --- /dev/null +++ b/openspec/changes/vale-grammars-package/specs/cli-runtime-binary-export/spec.md @@ -0,0 +1,21 @@ +## ADDED Requirements + +### Requirement: The Vale grammar resolver is published beside the binary resolver + +The runtimes entry SHALL publish the resolver the CLI uses to locate the pinned Vale grammar directory, together with the grammar package's spec, from the same modules the CLI dispatches on. + +A consumer that verifies a Vale rule over source code has to run it with the grammars the CLI runs it with: without them Vale reads comments by delimiters and reports different findings for the same rule. Publishing the resolver makes the grammar version a consequence of the CLI pin, as the binary version already is. + +Resolution SHALL report a miss as a value, returning an absent path alongside the locations searched, and SHALL NOT throw, for the reason the binary resolver does: each caller decides whether a miss is fatal. + +#### Scenario: A consumer runs Vale with the CLI's grammars + +- **WHEN** a consumer imports the runtimes entry and resolves the grammar directory +- **THEN** it SHALL receive the absolute path of the directory the CLI names to Vale +- **AND** the resolver SHALL be the module the CLI itself uses, not a copy kept in step + +#### Scenario: The grammar package does not resolve + +- **WHEN** the grammar package is not installed +- **THEN** resolution SHALL return an absent path and the locations it searched +- **AND** it SHALL NOT throw diff --git a/openspec/changes/vale-grammars-package/specs/cli-vale-rule-engine/spec.md b/openspec/changes/vale-grammars-package/specs/cli-vale-rule-engine/spec.md new file mode 100644 index 00000000..34b27b15 --- /dev/null +++ b/openspec/changes/vale-grammars-package/specs/cli-vale-rule-engine/spec.md @@ -0,0 +1,52 @@ +## ADDED Requirements + +### Requirement: Vale reads source code with the pinned grammars + +Every Vale invocation the CLI makes, for `check` and for `verify`, SHALL name the directory of the pinned grammar package as Vale's grammar directory. Vale reads that directory for any grammar its StylesPath does not hold. + +Every comment-tier extension the CLI lists SHALL be covered: run against the pinned binary with the pinned grammars, no listed extension SHALL produce a warning that its grammar is not installed. A listed extension Vale reads by delimiters with no grammar at all, as it did before grammars were separate, is not a gap. + +When the grammar package cannot be resolved, Vale SHALL still run, and its warnings that a grammar is not installed SHALL reach the user as notices, as any other zero-exit diagnostic does. + +#### Scenario: A directive comment is not linted as prose + +- **WHEN** a rule over comments runs on a Go file whose first line is `//nolint:simply` and whose second line is `// simply here` +- **THEN** the rule SHALL report line 2 and SHALL NOT report line 1 + +#### Scenario: A one-line CSS block comment is linted + +- **WHEN** a rule over comments runs on a CSS file containing `/* simply here */` +- **THEN** the rule SHALL report that line + +#### Scenario: No grammar warning on a covered extension + +- **WHEN** `check` runs Vale over a file with any comment-tier extension the CLI lists +- **THEN** Vale SHALL write no warning that a grammar is not installed, and the result SHALL carry no such notice + +#### Scenario: An extension needing an unshipped grammar fails the contract + +- **WHEN** a comment-tier extension is listed whose grammar the pinned grammar package does not carry +- **THEN** the vendor contract SHALL fail, naming the extension and the grammar + +### Requirement: A grammar directory in the Vale rules tree is refused + +The Vale rules tree is the StylesPath of every assembled config, and Vale reads a grammar from the StylesPath in preference to the pinned grammar directory. The system SHALL therefore refuse a grammar directory inside the Vale rules tree, whatever it contains, because its grammars were not shipped or pinned by the CLI. + +The refusal SHALL be made where the Vale config is assembled, so that `validate` and `check` both refuse it. Under `check` it SHALL refuse the Vale engine as a rejected rule config does: the engine reports a failure naming the directory, the failure reaches the exit code, and other engines still run. + +#### Scenario: validate refuses an unvetted grammar directory + +- **WHEN** `.taskless/rules/vale/config/grammars/` exists and `validate` runs +- **THEN** `validate` SHALL report a failure naming that directory and SHALL exit non-zero + +#### Scenario: check refuses the Vale engine over an unvetted grammar directory + +- **WHEN** `.taskless/rules/vale/config/grammars/` exists and `check` runs +- **THEN** the Vale engine SHALL report a failure naming that directory without running Vale +- **AND** the exit code SHALL be non-zero +- **AND** ast-grep results for the same run SHALL still be reported + +#### Scenario: A grammar directory elsewhere is not the CLI's concern + +- **WHEN** the project's own Vale StylesPath, outside the Vale rules tree, holds grammars from a user's `vale sync` +- **THEN** `validate` and `check` SHALL neither refuse nor read them diff --git a/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md b/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md new file mode 100644 index 00000000..9dca8ca7 --- /dev/null +++ b/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md @@ -0,0 +1,64 @@ +## ADDED Requirements + +### Requirement: Vale's grammars are published as one platform-independent package + +The repository SHALL publish one npm package, `@taskless/vale-grammars`, carrying every tree-sitter grammar module that upstream publishes for the Vale release the platform packages carry. It SHALL declare no `os` or `cpu`, so every host installs the same bytes. + +The package SHALL follow the platform packages' rules for contents and versioning: grammar modules and package metadata only, no `bin` entry, no executable JavaScript, no lifecycle script, and a version of `-` whose Vale component is the release the grammars came from. + +#### Scenario: Every host installs the same grammars + +- **WHEN** the grammar package is installed on two hosts with different operating systems or architectures +- **THEN** both hosts SHALL receive byte-identical grammar modules + +#### Scenario: The grammar version names its Vale release + +- **WHEN** a grammar package version is read +- **THEN** its `major.minor.patch` component SHALL be the upstream Vale release whose grammars it carries + +#### Scenario: Grammars are usable without an install step + +- **WHEN** the grammar package is installed with dependency lifecycle scripts disabled +- **THEN** the grammar modules SHALL be present and readable, and the package SHALL declare no `bin` and no `scripts` entry + +### Requirement: Fetched grammars are verified against committed checksums + +The repository SHALL commit a SHA256 checksum for each upstream grammar archive. Upstream's checksums file does not cover these archives, so the detect phase SHALL compute each digest from the archive it downloads and SHALL propose it in the same manifest pull request as the binary digests, where a human reviews it before anything can be published. + +The release pipeline SHALL verify every fetched grammar archive against its committed checksum before unpacking it, SHALL refuse to publish on a mismatch, and SHALL refuse to publish when an archive the manifest names is missing upstream. Grammar modules SHALL NOT be stored in version control. + +#### Scenario: Grammar checksum mismatch aborts the release + +- **WHEN** a fetched grammar archive's SHA256 does not match its committed checksum +- **THEN** the release SHALL fail and no package, platform or grammar, SHALL be published + +#### Scenario: A new upstream release proposes grammar digests for review + +- **WHEN** the detect phase finds upstream Vale ahead of the manifest +- **THEN** the pull request it opens SHALL update every grammar digest alongside the binary digests, and SHALL publish nothing + +#### Scenario: A clean checkout contains no grammar modules + +- **WHEN** the repository is cloned +- **THEN** no grammar module SHALL be present in the grammar package directory + +### Requirement: The grammar package is released with the platform packages + +The workflow that releases the platform packages SHALL also release the grammar package, from the same reviewed manifest, stamped with the same version as the platform packages in the same run. A run SHALL be considered already published only when both the platform packages and the grammar package have a version for the pinned Vale release. + +The CLI's dependencies on the grammar package and on the platform packages SHALL be moved together, by the same pull request, and SHALL name the same Vale release. Publishing the grammar package SHALL change no consumer until that pin moves. + +#### Scenario: Merging the update publishes binaries and grammars together + +- **WHEN** a manifest update pull request is merged +- **THEN** every platform package and the grammar package SHALL be stamped with one version and published in the same run + +#### Scenario: Missing grammars keep the gate open + +- **WHEN** the platform packages are published for the pinned Vale release and the grammar package is not +- **THEN** the publish gate SHALL NOT report the release as already published + +#### Scenario: The pins never straddle two Vale releases + +- **WHEN** the pull request that moves the CLI's Vale pins is opened +- **THEN** it SHALL move the grammar package pin with the platform pins, and both SHALL name the same Vale release diff --git a/openspec/changes/vale-grammars-package/tasks.md b/openspec/changes/vale-grammars-package/tasks.md new file mode 100644 index 00000000..9ba6d786 --- /dev/null +++ b/openspec/changes/vale-grammars-package/tasks.md @@ -0,0 +1,47 @@ +Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manual step between the two landings. Groups 5–8 are PR B, cut from #703's head and targeting `main`; it archives this change. + +## 1. Manifest and grammar package (PR A) + +- [ ] 1.1 Add a `grammars` section to `.github/scripts/vale-manifest.json` (package, directory, asset template `grammar-{name}.zip`, archive member `styles/config/grammars/{name}.wasm`, one `{ name, sha256 }` per upstream grammar for 3.27.0), and extend the manifest comment to say upstream's checksums file does not cover these digests; verify every digest by re-downloading the 22 assets and comparing `shasum -a 256`. +- [ ] 1.2 Create `packages/vale-grammars/` with `package.json` (no `os`/`cpu`, no `bin`, no `scripts`, placeholder version `0.0.0`, `files: ["grammars"]`), `README.md`, and upstream `LICENSE`; ignore `packages/vale-grammars/grammars/` in git and add the package to the changesets `ignore` list; verify `git status` stays clean after a prepare run. +- [ ] 1.3 Extend the pure functions in `vale-release.cjs` (manifest validation, stamping, digest comparison) to cover the grammar section; verify with new cases in `vale-release.test.cjs` under `node --test`. + +## 2. Pipeline (PR A) + +- [ ] 2.1 `vale-prepare.cjs`: download each grammar archive, verify it against its committed digest before unzipping, extract only the named member flattened to `grammars/.wasm`, stamp at the run's version, and `npm pack` beside the six platform tarballs; verify locally that `--out` yields seven tarballs and the grammar tarball lists 22 `.wasm` files. +- [ ] 2.2 `vale-detect.cjs`: when upstream moves, list the release's `grammar-*.zip` assets, download and hash each, and rewrite the grammar list and digests in the manifest pull request; verify with `vale-detect.test.cjs` cases for an added, a removed, and an unchanged grammar. +- [ ] 2.3 `vale-gate.cjs`: report "already published" only when the platform packages and `@taskless/vale-grammars` both have a version for the pinned Vale release; verify with `vale-gate.test.cjs` cases for grammars missing, binaries missing, and both present. +- [ ] 2.4 `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs`: treat the grammar package as a member of the set, wait for all seven, and move the CLI's grammar pin with the six platform pins; verify with their `.test.cjs` suites, including the case where binaries are published and grammars are not. +- [ ] 2.5 `release-vale.yml` and `vale-upgrade.yml`: update the header comments and any step that names "six" packages; confirm the workflow filename, workflow name, and `environment:` are unchanged (the OIDC binding); verify with `pnpm lint` (actionlint). + +## 3. PR A checks + +- [ ] 3.1 Run `pnpm typecheck`, `pnpm lint`, and `node --test .github/scripts/` and confirm they pass; open PR A to `main` with the `skip-changeset` label and request `@claude /review`. + +## 4. Bootstrap (maintainer, between PR A and PR B) + +- [ ] 4.1 Run `vale-prepare.cjs` on PR A's branch, publish the `@taskless/vale-grammars` tarball by hand with `npm publish --access public --tag latest`, and register its npm trusted publisher against `release-vale.yml` and `npm-autopublish`; verify with `npm view @taskless/vale-grammars versions`. +- [ ] 4.2 Merge PR A, then confirm the `Release Vale` run's gate publishes nothing and that `vale-upgrade.yml` produces a head carrying all seven pins. + +## 5. CLI wiring (PR B) + +- [ ] 5.1 Add `@taskless/vale-grammars` to `packages/cli/package.json` `dependencies` at the exact published version, if the upgrade workflow has not already, and regenerate `pnpm-lock.yaml`; verify with `pnpm install --frozen-lockfile`. +- [ ] 5.2 Add the grammar resolver beside `findValeBinary` (spec, per-process cache, miss as `{ path: undefined, tried }`); verify with unit tests for a resolved package and a missing one. +- [ ] 5.3 Set `VALE_GRAMMARS_PATH` in `spawnVale`'s child environment when the resolver returns a path; verify that the six #703 vendor-contract failures pass with no `VALE_GRAMMARS_PATH` in the test shell. +- [ ] 5.4 Export the grammar spec and resolver from `src/node/runtimes/index.ts` with a note in the entry's documentation; verify the build's export classification passes and `dist/node/runtimes/index.d.ts` declares both. + +## 6. Refusal and coverage (PR B) + +- [ ] 6.1 In `assembleValeConfig`, refuse when `/config/grammars/` exists, with a tree-level refusal that `validate` and `check` both report; verify with tests for the `validate` failure, the `check` failure with non-zero exit and ast-grep results still reported, and an empty directory. +- [ ] 6.2 Add the D6 coverage case to `test/vale-vendor-contract.test.ts`: every comment-tier extension in `VALE_FORMAT_TIERS`, run with the pinned grammars, produces no `grammar "" is not installed` warning; verify it fails when one grammar is withheld and passes with the full set. +- [ ] 6.3 Extend `engine-version-consistency.test.ts` so the grammar pin's Vale component equals `VALE_VERSION` and the platform pins'; verify it fails on a mismatched pin. + +## 7. Release notes (PR B) + +- [ ] 7.1 Add the Vale 3.27.0 entry to the `update` ledger in `src/agent/update.md`, stating that behavior is unchanged and that a `.taskless/rules/vale/config/grammars/` directory is now refused; verify the reconcile-marker and prompt parity tests pass. +- [ ] 7.2 Grow `.changeset/vale-3-27-0.md` (patch) to say the grammars ship as their own package and an unvetted grammar directory in the rules tree is refused; verify the `Changeset` check stays green. + +## 8. PR B checks and archive + +- [ ] 8.1 Run `pnpm build`, `pnpm typecheck`, `pnpm lint`, and the full CLI suite and confirm they pass on every branch of PR B. +- [ ] 8.2 Archive with `pnpm openspec archive vale-grammars-package -y` after confirming every prior scenario in the three standing specs survives (the pre-archive check in AGENTS.md); open PR B to `main` with `Refs #703` and request `@claude /review`. From cc20dae92226a5e905480e571a32e949c4ea2e9f Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 19 Nov 2026 17:02:05 -0800 Subject: [PATCH 2/5] feat(vale): pin Vale's grammars in the manifest and add their package Vale 3.27.0 ships its tree-sitter grammars as separate WebAssembly modules. Add a grammars section to vale-manifest.json pinning all 22 upstream grammar archives for 3.27.0. Upstream's checksums file does not cover them, so each digest is the one GitHub records for the release asset; all 22 match a local download. vale-release.cjs validates the section, lists the seven packages a release covers, reads grammar digests from a release's asset list, and accepts an explicit stamp for the grammar package's bootstrap publish. packages/vale-grammars holds only metadata; the pipeline places the modules. The design and tasks now record the GitHub digest source and that all seven pins share one exact stamp, as vale-upgrade-detect already requires. --- .changeset/config.json | 1 + .github/scripts/vale-manifest.json | 107 ++++++++- .github/scripts/vale-release.cjs | 226 +++++++++++++++++- .github/scripts/vale-release.test.cjs | 179 +++++++++++++- README.md | 6 +- .../changes/vale-grammars-package/design.md | 26 +- .../specs/vale-binary-packages/spec.md | 6 +- .../changes/vale-grammars-package/tasks.md | 20 +- packages/vale-grammars/.gitignore | 7 + packages/vale-grammars/LICENSE | 21 ++ packages/vale-grammars/README.md | 55 +++++ packages/vale-grammars/package.json | 17 ++ 12 files changed, 633 insertions(+), 38 deletions(-) create mode 100644 packages/vale-grammars/.gitignore create mode 100644 packages/vale-grammars/LICENSE create mode 100644 packages/vale-grammars/README.md create mode 100644 packages/vale-grammars/package.json diff --git a/.changeset/config.json b/.changeset/config.json index a0c0eb35..cb39993a 100644 --- a/.changeset/config.json +++ b/.changeset/config.json @@ -10,6 +10,7 @@ "ignore": [ "@taskless/vale-darwin-arm64", "@taskless/vale-darwin-x64", + "@taskless/vale-grammars", "@taskless/vale-linux-arm64", "@taskless/vale-linux-x64", "@taskless/vale-win32-arm64", diff --git a/.github/scripts/vale-manifest.json b/.github/scripts/vale-manifest.json index 72ad7780..cc11fc11 100644 --- a/.github/scripts/vale-manifest.json +++ b/.github/scripts/vale-manifest.json @@ -13,7 +13,14 @@ "", "`asset` is a template; {version} is replaced with valeVersion. The detect", "phase of .github/workflows/release-vale.yml rewrites valeVersion and every", - "sha256 from upstream's checksums file, and leaves the templates alone." + "sha256 from upstream's checksums file, and leaves the templates alone.", + "", + "`grammars` is the platform-independent @taskless/vale-grammars package: one", + "tree-sitter module per upstream grammar-.zip, unpacked from", + "`archiveMember`. Upstream checksums.txt does NOT cover these archives, so each", + "sha256 is the digest GitHub records for the release asset at upload time, read", + "from the releases API by the detect phase; it is reviewed in the same pull", + "request as the binary digests and verified the same way before unpacking." ], "valeVersion": "3.27.0", "upstream": { @@ -77,5 +84,101 @@ "archiveMember": "vale.exe", "sha256": "26e61cc58ac4180866d068dc2325d2625560c8cc518de7e7b5523211a5ab0cbc" } - ] + ], + "grammars": { + "package": "@taskless/vale-grammars", + "directory": "packages/vale-grammars", + "asset": "grammar-{name}.zip", + "archiveMember": "grammar-{name}/styles/config/grammars/{name}.wasm", + "modules": [ + { + "name": "bash", + "sha256": "383f41dac7ba7ff6b62310ae90fbecad5872b421e16362428e85f6d3383282a4" + }, + { + "name": "c", + "sha256": "0e4d8e14893f8f20dccf7ee85c79c6beeeac4c77f032551a237067a93b450b0d" + }, + { + "name": "cpp", + "sha256": "0f550858aa68c02663362631dc64d40e7b1a28aa48379fa5d55c0dfc3bf905f7" + }, + { + "name": "csharp", + "sha256": "1c5ef2dc517dcd7fad2c7e6d1e2b5c8a60690b10d34a22cb3b108d98b3b73898" + }, + { + "name": "css", + "sha256": "efc982ba928828df69d3a0a8d17ff4095b5890d9f918c41ce34adff940d31529" + }, + { + "name": "elixir", + "sha256": "3beddaad99e1f3069948d5233709feb1ce8aaeab5b5a8333de3dc5e32659ecb0" + }, + { + "name": "elm", + "sha256": "baedd9bdc51c76883a7d5f81e340e210f117e4355bc2fd479ae4b11ce8041ff1" + }, + { + "name": "go", + "sha256": "c188c53747864bff9849cc39eaac51393c3264be8ae3c67c12edb2ccc94eb130" + }, + { + "name": "java", + "sha256": "fc65e8426fe7116ef941b227520beb542cd2f6c8cf0789d28b63b370e8d36827" + }, + { + "name": "javascript", + "sha256": "448d9a94d558b4e2e62c929b815378337ea5a151fc12e1f8277c853314c9e1d4" + }, + { + "name": "julia", + "sha256": "edc4dbc9cfb3dc06f72c1820a7bd16d481a4c492efa35f9dd7a23a7c876d29f8" + }, + { + "name": "kotlin", + "sha256": "58b5e2d01f1920cb2fb908863a477b6cd14cc0e3cf8a36fc4413191b709f4926" + }, + { + "name": "lua", + "sha256": "10c94b27ea4d6cf4779be2bcf57accf256b1c752b57fcf27ff56be297ad9178f" + }, + { + "name": "php", + "sha256": "326f84f19873498972a9b606b4af4c5e4cfa77a75a45bb1ac6c29801bf9c16f2" + }, + { + "name": "proto", + "sha256": "4262870472d181b4bbc6afb3df39d81de13f84910d6f87a4032d66d346c17bef" + }, + { + "name": "python", + "sha256": "cd369ca78cb327534bfab0eefb710b38115e0644cda04709da174f675c28a1b2" + }, + { + "name": "ruby", + "sha256": "a31c8e1f88bab718794f281cbf6b7b4c5cfaf3526e379da1ba53ca7d3cfe6548" + }, + { + "name": "rust", + "sha256": "040788500c7414b3e789f08f0281e4d12fdcf197720beb238e27139bde511631" + }, + { + "name": "toml", + "sha256": "8a66d9450aba2f7d1423d16ed6bdc84a2d14dbb6eb26cd75a476128d5bdd2912" + }, + { + "name": "tsx", + "sha256": "b051dedfe0ad634a6689c7d328f5df61b2a44fe120cf93146c2d4185fb43f5bc" + }, + { + "name": "typescript", + "sha256": "d127e82362d35ec0d8945049b9f51735e0db10091b7d884f47ce2f6aebe7a162" + }, + { + "name": "yaml", + "sha256": "8184a6cf2329f6cbc6748fbf8807ae5b58de7ab20f9a19f1301efd1c36a4b971" + } + ] + } } diff --git a/.github/scripts/vale-release.cjs b/.github/scripts/vale-release.cjs index af79e34c..6a1d8b26 100644 --- a/.github/scripts/vale-release.cjs +++ b/.github/scripts/vale-release.cjs @@ -341,9 +341,175 @@ function assertManifest(manifest) { } seen.add(platform.package); } + assertGrammars(manifest.grammars, seen); return manifest; } +// --------------------------------------------------------------------------- +// Grammars +// +// Vale 3.27.0 moved its tree-sitter grammars out of the binary into +// WebAssembly modules, published upstream as one `grammar-.zip` per +// language. They are the same bytes on every host, so they ship as ONE +// platform-independent package rather than a copy inside each platform +// package: a grammar then cannot be one version on macOS and another on Linux. +// +// Upstream's checksums file does not cover these archives. Each committed +// digest is the one GitHub records for the release asset at upload, which the +// detect phase reads from the releases API and a reviewer can check the same +// way. See openspec/changes/vale-grammars-package/design.md, D3. +// --------------------------------------------------------------------------- + +const MANIFEST_GRAMMAR_FIELDS = [ + "package", + "directory", + "asset", + "archiveMember", +]; + +/** + * A grammar name is substituted into an asset name and an archive member + * path, so it is restricted to a single safe path segment. Upstream's names + * (`csharp`, `typescript`, `tsx`) all fit. + */ +const GRAMMAR_NAME_PATTERN = /^[a-z\d][\w-]*$/; + +/** The asset-name shape that marks an upstream release asset as a grammar. */ +const GRAMMAR_ASSET_PATTERN = /^grammar-([a-z\d][\w-]*)\.zip$/; + +function assertGrammars(grammars, seenPackages) { + if (typeof grammars !== "object" || grammars === null) { + throw new Error("manifest.grammars is missing"); + } + for (const field of MANIFEST_GRAMMAR_FIELDS) { + if (typeof grammars[field] !== "string" || grammars[field].length === 0) { + throw new Error(`manifest.grammars is missing ${field}`); + } + } + if (seenPackages.has(grammars.package)) { + throw new Error(`manifest lists ${grammars.package} more than once`); + } + // The member is never extracted onto disk (vale-prepare.cjs streams it), so + // a nested path is allowed here where a platform's is not. It is still held + // to relative, `..`-free segments, so a hand edit cannot name anything but a + // file inside the archive. + const member = applyTemplate(grammars.archiveMember, { name: "x" }); + if ( + member.startsWith("/") || + member.split(/[/\\]/).some((segment) => segment === ".." || segment === "") + ) { + throw new Error( + `manifest.grammars.archiveMember is not a relative path inside the archive: ${grammars.archiveMember}` + ); + } + if (!Array.isArray(grammars.modules) || grammars.modules.length === 0) { + throw new Error("manifest.grammars.modules must be a non-empty array"); + } + const names = new Set(); + for (const module_ of grammars.modules) { + if (!GRAMMAR_NAME_PATTERN.test(String(module_?.name ?? ""))) { + throw new Error( + `manifest grammar ${JSON.stringify(module_?.name ?? "?")} is not a valid grammar name` + ); + } + if (!SHA256_PATTERN.test(String(module_.sha256 ?? ""))) { + throw new Error( + `manifest grammar ${module_.name} has a sha256 that is not 64 lowercase hex characters` + ); + } + if (names.has(module_.name)) { + throw new Error(`manifest lists grammar ${module_.name} more than once`); + } + names.add(module_.name); + } +} + +/** + * Every package the manifest releases, platform packages first, then the + * grammar package. The publish gate and the stamp plan iterate this rather + * than `platforms`, so the grammar package cannot be left out of either. + */ +function manifestPackages(manifest) { + return [ + ...manifest.platforms.map(({ package: name, directory }) => ({ + package: name, + directory, + })), + { package: manifest.grammars.package, directory: manifest.grammars.directory }, + ]; +} + +/** The release asset filename for one grammar. */ +function resolveGrammarAsset(manifest, module_) { + return applyTemplate(manifest.grammars.asset, { + name: module_.name, + version: manifest.valeVersion, + }); +} + +/** The full download URL for one grammar's archive. */ +function resolveGrammarDownloadUrl(manifest, module_) { + return applyTemplate(manifest.upstream.downloadUrl, { + version: manifest.valeVersion, + asset: resolveGrammarAsset(manifest, module_), + }); +} + +/** The path of the grammar module inside its archive. */ +function resolveGrammarMember(manifest, module_) { + return applyTemplate(manifest.grammars.archiveMember, { name: module_.name }); +} + +/** + * The grammar modules an upstream release publishes, read from its asset + * list: `[{ name, sha256 }]`, sorted by name. + * + * Every `grammar-.zip` asset is taken, not a chosen subset (design D2), + * so the manifest's list is upstream's list and nothing hand-kept can drift. + * An asset without a GitHub-recorded sha256 throws: carrying it with no digest + * would leave nothing for a reviewer to check and nothing to verify against. + */ +function planGrammarModules(assets, version) { + const modules = []; + for (const asset of assets ?? []) { + const match = GRAMMAR_ASSET_PATTERN.exec(String(asset?.name ?? "")); + if (!match) { + continue; + } + const digest = /^sha256:([\da-f]{64})$/i.exec(String(asset.digest ?? "")); + if (!digest) { + throw new Error( + `upstream ${version} asset ${asset.name} has no recorded sha256 digest` + ); + } + modules.push({ name: match[1], sha256: digest[1].toLowerCase() }); + } + if (modules.length === 0) { + throw new Error(`upstream ${version} publishes no grammar-*.zip assets`); + } + return modules.sort((a, b) => (a.name < b.name ? -1 : 1)); +} + +/** The four bytes every WebAssembly module starts with: `\0asm`. */ +const WASM_MAGIC = Buffer.from([0x00, 0x61, 0x73, 0x6d]); + +/** + * Throw unless `bytes` starts like a WebAssembly module. The archive was + * verified against its digest before this runs; this catches a manifest whose + * `archiveMember` names the wrong file, which would otherwise pack a README or + * a symlink target into the published package. + */ +function assertWasmModule(bytes, label) { + if ( + !Buffer.isBuffer(bytes) || + bytes.length < WASM_MAGIC.length || + !bytes.subarray(0, WASM_MAGIC.length).equals(WASM_MAGIC) + ) { + throw new Error(`${label} is not a WebAssembly module`); + } + return true; +} + /** Substitute `{version}` (and optionally `{asset}`) into a manifest template. */ function applyTemplate(template, values) { return String(template).replaceAll(/{(\w+)}/g, (whole, key) => @@ -439,7 +605,12 @@ function assertChecksum({ asset, expected, actual }) { * rather than silently carrying a stale digest forward, which would otherwise * produce a manifest that fails verification only later, inside the publish run. */ -function planManifestUpdate({ manifest, upstreamTag, checksumsText }) { +function planManifestUpdate({ + manifest, + upstreamTag, + checksumsText, + releaseAssets, +}) { assertManifest(manifest); const upstreamVersion = parseReleaseTag(upstreamTag); if (!isUpstreamAhead(manifest.valeVersion, upstreamVersion)) { @@ -469,11 +640,24 @@ function planManifestUpdate({ manifest, upstreamTag, checksumsText }) { return { ...platform, sha256 }; }); + // Grammars are re-read from the release's asset list, not carried forward: + // a grammar upstream added appears and one it dropped disappears, both in + // the pull request a human reviews. + const grammars = { + ...manifest.grammars, + modules: planGrammarModules(releaseAssets, upstreamVersion), + }; + return { update: true, pinnedVersion: manifest.valeVersion, upstreamVersion, - manifest: { ...manifest, valeVersion: upstreamVersion, platforms }, + manifest: { + ...manifest, + valeVersion: upstreamVersion, + platforms, + grammars, + }, }; } @@ -488,19 +672,41 @@ function planManifestUpdate({ manifest, upstreamTag, checksumsText }) { * disagreed about their version would make the CLI's exact pins unresolvable on * some hosts and resolvable on others. */ -function planStamp({ manifest, date }) { +function planStamp({ manifest, date, version: requested }) { assertManifest(manifest); - const version = stampVersion(manifest.valeVersion, date); + const version = + requested === undefined + ? stampVersion(manifest.valeVersion, date) + : assertRequestedStamp(manifest, requested); return { version, - packages: manifest.platforms.map((platform) => ({ - package: platform.package, - directory: platform.directory, + packages: manifestPackages(manifest).map((entry) => ({ + ...entry, version, })), }; } +/** + * Accept an explicit stamp only when it names the manifest's Vale release. + * + * This exists for one job: publishing a package at a stamp the rest of the set + * already carries, as the grammar package's manual first publish does, so the + * CLI's `@taskless/vale-*` pins can stay at one exact version. A stamp for a + * different Vale release would pack this manifest's bytes under another + * release's name. + */ +function assertRequestedStamp(manifest, version) { + assertStampedVersion(version); + const [base] = version.split("-"); + if (base !== manifest.valeVersion) { + throw new Error( + `--version ${version} does not name the manifest's Vale ${manifest.valeVersion}` + ); + } + return version; +} + /** * Apply a stamped version to a parsed `package.json`, returning a new object. * Refuses to stamp a package.json whose version is neither the committed @@ -523,18 +729,24 @@ module.exports = { assertChecksum, assertManifest, assertStampedVersion, + assertWasmModule, compareStampedVersions, compareValeVersions, formatStampTimestamp, isUpstreamAhead, + manifestPackages, parseChecksumsFile, parseReleaseTag, parseValeVersion, + planGrammarModules, planManifestUpdate, planStamp, rangeMatches, resolveAssetName, resolveChecksumsUrl, resolveDownloadUrl, + resolveGrammarAsset, + resolveGrammarDownloadUrl, + resolveGrammarMember, stampVersion, }; diff --git a/.github/scripts/vale-release.test.cjs b/.github/scripts/vale-release.test.cjs index 4c242fc8..33c1fc17 100644 --- a/.github/scripts/vale-release.test.cjs +++ b/.github/scripts/vale-release.test.cjs @@ -12,19 +12,25 @@ const { assertChecksum, assertManifest, assertStampedVersion, + assertWasmModule, compareStampedVersions, compareValeVersions, formatStampTimestamp, isUpstreamAhead, + manifestPackages, parseChecksumsFile, parseReleaseTag, parseValeVersion, + planGrammarModules, planManifestUpdate, planStamp, rangeMatches, resolveAssetName, resolveChecksumsUrl, resolveDownloadUrl, + resolveGrammarAsset, + resolveGrammarDownloadUrl, + resolveGrammarMember, stampVersion, } = require("./vale-release.cjs"); @@ -66,9 +72,28 @@ function fixtureManifest() { sha256: DIGEST_B, }, ], + grammars: { + package: "@taskless/vale-grammars", + directory: "packages/vale-grammars", + asset: "grammar-{name}.zip", + archiveMember: "grammar-{name}/styles/config/grammars/{name}.wasm", + modules: [ + { name: "go", sha256: DIGEST_A }, + { name: "python", sha256: DIGEST_B }, + ], + }, }; } +/** Release assets as the GitHub releases API lists them for a new upstream. */ +function releaseAssets(version) { + return [ + { name: `vale_${version}_Linux_64-bit.tar.gz`, digest: `sha256:${DIGEST_B}` }, + { name: "grammar-rust.zip", digest: `sha256:${DIGEST_A}` }, + { name: "grammar-go.zip", digest: `sha256:${DIGEST_B}` }, + ]; +} + // --------------------------------------------------------------------------- // Version parsing and stamping (tasks 4.1, 4.2) // --------------------------------------------------------------------------- @@ -243,14 +268,14 @@ test("planStamp: the whole set shares one version", () => { plan.version, `${COMMITTED_MANIFEST.valeVersion}-20260806120000` ); - assert.equal(plan.packages.length, 6); + assert.equal(plan.packages.length, 7); assert.equal(new Set(plan.packages.map((p) => p.version)).size, 1); for (const entry of plan.packages) { assert.equal(entry.version, plan.version); } }); -test("planStamp: covers every committed platform package", () => { +test("planStamp: covers every committed platform package and the grammar package", () => { const plan = planStamp({ manifest: COMMITTED_MANIFEST, date: new Date("2026-08-06T12:00:00Z"), @@ -258,6 +283,7 @@ test("planStamp: covers every committed platform package", () => { assert.deepEqual(plan.packages.map((p) => p.package).sort(), [ "@taskless/vale-darwin-arm64", "@taskless/vale-darwin-x64", + "@taskless/vale-grammars", "@taskless/vale-linux-arm64", "@taskless/vale-linux-x64", "@taskless/vale-win32-arm64", @@ -438,6 +464,7 @@ test("planManifestUpdate: a newer upstream rewrites version and digests only", ( `${DIGEST_B} vale_3.18.0_Linux_64-bit.tar.gz`, `${DIGEST_A} vale_3.18.0_Windows_64-bit.zip`, ].join("\n"), + releaseAssets: releaseAssets("3.18.0"), }); assert.equal(result.update, true); assert.equal(result.pinnedVersion, "3.17.1"); @@ -479,3 +506,151 @@ test("planManifestUpdate: an unparseable checksums file aborts", () => { /parsed to no entries/ ); }); + +// --------------------------------------------------------------------------- +// Grammars (openspec/changes/vale-grammars-package) +// --------------------------------------------------------------------------- + +test("assertManifest: the committed manifest pins every grammar with a digest", () => { + const { grammars } = COMMITTED_MANIFEST; + assert.equal(grammars.package, "@taskless/vale-grammars"); + assert.equal(grammars.directory, "packages/vale-grammars"); + assert.ok(grammars.modules.length > 0); + // Sorted, so a detect rewrite produces a minimal, reviewable diff. + const names = grammars.modules.map((module_) => module_.name); + assert.deepEqual(names, names.toSorted()); +}); + +test("assertManifest: rejects a malformed grammars section", () => { + const missing = fixtureManifest(); + delete missing.grammars; + assert.throws(() => assertManifest(missing), /manifest\.grammars is missing/); + + const noAsset = fixtureManifest(); + delete noAsset.grammars.asset; + assert.throws(() => assertManifest(noAsset), /grammars is missing asset/); + + const empty = fixtureManifest(); + empty.grammars.modules = []; + assert.throws(() => assertManifest(empty), /non-empty array/); + + const badDigest = fixtureManifest(); + badDigest.grammars.modules[0].sha256 = "nope"; + assert.throws(() => assertManifest(badDigest), /64 lowercase hex/); + + const duplicate = fixtureManifest(); + duplicate.grammars.modules[1].name = "go"; + assert.throws(() => assertManifest(duplicate), /grammar go more than once/); + + const sharedPackage = fixtureManifest(); + sharedPackage.grammars.package = sharedPackage.platforms[0].package; + assert.throws(() => assertManifest(sharedPackage), /more than once/); +}); + +test("assertManifest: a grammar name and member cannot leave the archive", () => { + // The name is substituted into the member path, so it is held to one safe + // segment; the member template is held to relative, `..`-free segments. + for (const name of ["../go", "go/x", "", "Go"]) { + const bad = fixtureManifest(); + bad.grammars.modules[0].name = name; + assert.throws(() => assertManifest(bad), /not a valid grammar name/); + } + for (const member of ["/abs/{name}.wasm", "../{name}.wasm", "a//{name}.wasm"]) { + const bad = fixtureManifest(); + bad.grammars.archiveMember = member; + assert.throws(() => assertManifest(bad), /not a relative path/); + } +}); + +test("manifestPackages: the platform packages, then the grammar package", () => { + assert.deepEqual(manifestPackages(fixtureManifest()), [ + { package: "@taskless/vale-linux-x64", directory: "packages/vale-linux-x64" }, + { package: "@taskless/vale-win32-x64", directory: "packages/vale-win32-x64" }, + { package: "@taskless/vale-grammars", directory: "packages/vale-grammars" }, + ]); +}); + +test("resolveGrammar*: templates expand per grammar", () => { + const manifest = fixtureManifest(); + const [go] = manifest.grammars.modules; + assert.equal(resolveGrammarAsset(manifest, go), "grammar-go.zip"); + assert.equal( + resolveGrammarDownloadUrl(manifest, go), + "https://github.com/errata-ai/vale/releases/download/v3.17.1/grammar-go.zip" + ); + assert.equal( + resolveGrammarMember(manifest, go), + "grammar-go/styles/config/grammars/go.wasm" + ); +}); + +test("planGrammarModules: takes every grammar asset, sorted, with GitHub's digest", () => { + assert.deepEqual(planGrammarModules(releaseAssets("3.18.0"), "3.18.0"), [ + { name: "go", sha256: DIGEST_B }, + { name: "rust", sha256: DIGEST_A }, + ]); +}); + +test("planGrammarModules: an asset without a recorded digest aborts", () => { + assert.throws( + () => + planGrammarModules([{ name: "grammar-go.zip", digest: null }], "3.18.0"), + /grammar-go\.zip has no recorded sha256/ + ); +}); + +test("planGrammarModules: a release with no grammar assets aborts", () => { + assert.throws( + () => planGrammarModules([{ name: "vale_3.18.0_checksums.txt" }], "3.18.0"), + /publishes no grammar-\*\.zip assets/ + ); +}); + +test("planManifestUpdate: a newer upstream re-reads the grammar list", () => { + // python is dropped and rust added, exactly as upstream's assets say. + const result = planManifestUpdate({ + manifest: fixtureManifest(), + upstreamTag: "v3.18.0", + checksumsText: [ + `${DIGEST_B} vale_3.18.0_Linux_64-bit.tar.gz`, + `${DIGEST_A} vale_3.18.0_Windows_64-bit.zip`, + ].join("\n"), + releaseAssets: releaseAssets("3.18.0"), + }); + assert.deepEqual( + result.manifest.grammars.modules.map((module_) => module_.name), + ["go", "rust"] + ); + assert.equal(result.manifest.grammars.asset, "grammar-{name}.zip"); + assert.doesNotThrow(() => assertManifest(result.manifest)); +}); + +test("planStamp: an explicit stamp must name the manifest's Vale release", () => { + const plan = planStamp({ + manifest: fixtureManifest(), + version: "3.17.1-20261117230017", + }); + assert.equal(plan.version, "3.17.1-20261117230017"); + assert.ok(plan.packages.every((entry) => entry.version === plan.version)); + assert.throws( + () => + planStamp({ manifest: fixtureManifest(), version: "3.18.0-20261117230017" }), + /does not name the manifest's Vale 3\.17\.1/ + ); + assert.throws( + () => planStamp({ manifest: fixtureManifest(), version: "3.17.1" }), + /stamped/ + ); +}); + +test("assertWasmModule: accepts the wasm magic, rejects anything else", () => { + assert.equal( + assertWasmModule(Buffer.from([0x00, 0x61, 0x73, 0x6d, 1, 0, 0, 0]), "go"), + true + ); + assert.throws( + () => assertWasmModule(Buffer.from("../../etc/passwd"), "go"), + /go is not a WebAssembly module/ + ); + assert.throws(() => assertWasmModule(Buffer.alloc(0), "go"), /not a WebAssembly/); +}); diff --git a/README.md b/README.md index 37563552..1c08136d 100644 --- a/README.md +++ b/README.md @@ -126,9 +126,11 @@ Taskless runs prose rules with [Vale](https://vale.sh). The CLI lists one npm pa Each package holds the upstream `vale` binary, its MIT `LICENSE`, a README, and `package.json`. It has no JavaScript and no install script, so it works under package managers that refuse dependency build scripts. -The binaries aren't committed. The pipeline works in two halves: +A seventh package, `@taskless/vale-grammars`, holds the tree-sitter grammars Vale uses to find comments in source code. Since Vale 3.27.0 they ship apart from the binary. They are WebAssembly, the same bytes on every host, so there is one package rather than one per platform, and the CLI names its directory to Vale through `VALE_GRAMMARS_PATH`. -- **`release-vale.yml`** watches upstream Vale releases. When one appears it opens a pull request that pins the new version and its six archive digests in `.github/scripts/vale-manifest.json`. After a person reviews and merges it, the workflow downloads each archive, verifies it against the committed digest, and publishes the packages over OIDC as `-`. +The binaries and grammars aren't committed. The pipeline works in two halves: + +- **`release-vale.yml`** watches upstream Vale releases. When one appears it opens a pull request that pins the new version, its six binary archive digests, and a digest for each grammar archive in `.github/scripts/vale-manifest.json`. After a person reviews and merges it, the workflow downloads each archive, verifies it against the committed digest, and publishes all seven packages over OIDC as one `-`. - **`vale-upgrade.yml`** moves the CLI's pins onto the newly published set, as a separate pull request. The Vale packages are versioned outside Changesets, and publishing them changes nothing for a CLI user until the CLI's pins move. Each package's README covers versioning and provenance in detail. diff --git a/openspec/changes/vale-grammars-package/design.md b/openspec/changes/vale-grammars-package/design.md index 846a5e9c..bc55a2c3 100644 --- a/openspec/changes/vale-grammars-package/design.md +++ b/openspec/changes/vale-grammars-package/design.md @@ -37,17 +37,23 @@ The package carries all of upstream's grammars for the release, not the 20 the c _Alternative: ship only the probed subset._ It is smaller by about 7%, but it adds a table that has to be re-derived on every bump and that only fails after it is already wrong. -### D3. Digests are recorded at detect time and reviewed, since upstream does not sign them +### D3. Digests come from GitHub's record of each release asset, and are reviewed -The manifest gains a `grammars` section: the package name and directory, the asset template `grammar-{name}.zip`, the archive member `styles/config/grammars/{name}.wasm`, and one `{ name, sha256 }` entry per grammar. `vale-detect.cjs` already runs without credentials. When upstream moves, it lists the release's `grammar-*.zip` assets, downloads each one, hashes it, and writes the list and digests into the manifest pull request. A human approves those digests exactly as they approve the binary digests. `vale-prepare.cjs` then verifies each archive against the committed digest before unzipping, and extracts only the named member, flattened to `grammars/.wasm`. +The manifest gains a `grammars` section: the package name and directory, the asset template `grammar-{name}.zip`, the archive member `grammar-{name}/styles/config/grammars/{name}.wasm`, and one `{ name, sha256 }` entry per grammar, sorted by name. When upstream moves, `vale-detect.cjs` reads the release's asset list from the GitHub releases API. That list carries a `digest` (`sha256:`) for each asset, recorded by GitHub when upstream uploaded it. Detect takes every `grammar-*.zip` asset and its digest into the manifest pull request, and refuses an asset with no recorded digest. A human approves those digests exactly as they approve the binary digests. -This is trust-on-first-sight for the grammars, where the binaries are cross-checked against upstream's own checksums file. The review gate is the same; what is weaker is that a reviewer cannot compare the grammar digests with anything upstream published. That is the best upstream allows today, and it is stated in the manifest comment. +Upstream's checksums file does not cover the grammar archives, so GitHub's record is the independent source here. It is not upstream's signature, but it is not a digest of our own download either: anyone can read it back from the API. All 22 digests for 3.27.0 matched a local download when this was written. -### D4. One stamp, one gate, one upgrade +`vale-prepare.cjs` verifies each archive against the committed digest before reading anything out of it. It then streams the single named member to stdout with `unzip -p` rather than extracting it, so no archive path ever reaches the filesystem. That is why the grammar member may be nested, where a platform's `archiveMember` must stay a flat filename. The bytes must start with the WebAssembly magic (`\0asm`), which catches a member template that names the wrong file. Each module is written to `grammars/.wasm`. -`vale-prepare.cjs` emits the grammar tarball beside the six platform tarballs, at the same stamp, and the existing publish loop publishes all seven. `vale-gate.cjs` reports "already published" only when the platform packages and the grammar package both have a version for the pinned Vale release. `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs` treat the grammar package as a seventh member of the set: the upgrade pull request is opened only when all seven are published, and it moves all seven pins together. +### D4. One exact stamp, one gate, one upgrade -The CLI takes the grammar package as a regular dependency at an exact version, not an optional one, because there is no host that should go without it. `engine-version-consistency` asserts that the grammar pin's Vale component equals `VALE_VERSION` and the binary pins' Vale component. It does not require equal stamps: the manual first publish (Migration, step 2) stamps the grammar package once on its own. +`vale-prepare.cjs` emits the grammar tarball beside the six platform tarballs, at the same stamp, and the existing publish loop publishes all seven. `vale-gate.cjs` reports "already published" only when the platform packages and the grammar package both have a version for the pinned Vale release. + +`vale-upgrade-detect.cjs` already requires every `@taskless/vale-*` pin in the CLI, and the latest published version of each, to be one exact stamped version. The grammar package matches that prefix, so once the CLI pins it, it is a seventh member of that set with no script change. The upgrade pull request is then opened only when all seven are published at one version, and it moves all seven pins together. `vale-upgrade-wait.cjs` asks the same question through `vale-upgrade-detect.cjs --json`. + +That invariant is kept, not relaxed, which shapes the bootstrap. The binaries for 3.27.0 are already published at `3.27.0-20261117230017`, so the grammar package's manual first publish must carry that same stamp. `vale-prepare.cjs` takes `--version ` for this, and refuses a stamp whose Vale component is not the manifest's. Every later release stamps all seven in one run. + +The CLI takes the grammar package as a regular dependency at an exact version, not an optional one, because there is no host that should go without it. ### D5. `VALE_GRAMMARS_PATH` is set at the single spawn site, from a resolver beside `findValeBinary` @@ -78,12 +84,8 @@ The vendor contract runs every comment-tier extension in `VALE_FORMAT_TIERS` thr ## Migration Plan 1. **PR A** (bottom, to `main`): this change's artifacts, the manifest `grammars` section for 3.27.0 with digests computed from upstream's current assets, `packages/vale-grammars/` metadata, and the detect, prepare, gate and upgrade script changes with their tests. It changes no CLI behavior and carries no changeset. -2. **Manual, before merging PR A**: a maintainer runs `vale-prepare.cjs` on PR A's branch, publishes the `@taskless/vale-grammars` tarball by hand with `--tag latest`, and registers its npm trusted publisher against `release-vale.yml` and the `npm-autopublish` environment. -3. Merge PR A. The gate finds 3.27.0 published for all seven packages and publishes nothing. `vale-upgrade.yml` rewrites #703 to include the grammar pin. +2. **Manual, before merging PR A**: a maintainer runs `node .github/scripts/vale-prepare.cjs --only grammars --version 3.27.0-20261117230017` on PR A's branch, publishes the resulting `@taskless/vale-grammars` tarball by hand with `npm publish --access public --tag latest`, and registers its npm trusted publisher against `release-vale.yml` and the `npm-autopublish` environment. +3. Merge PR A. The gate finds 3.27.0 published for all seven packages and publishes nothing. The CLI does not pin the grammar package yet, so `vale-upgrade.yml` still sees six pins. 4. **PR B**, cut from #703's head and targeting `main`: the grammar dependency pin, the resolver and `VALE_GRAMMARS_PATH`, the runtime export, the D7 refusal, the D6 coverage check, the `update` ledger entry, and the grown `vale-3-27-0` changeset (patch). It archives this change. #703 is closed with "Superseded by #" once PR B lands. Rollback: revert PR B. The 3.26.0 pins come back and the grammar package becomes inert, since nothing pins it. PR A can stay. - -## Open Questions - -- Whether `vale-upgrade.yml`'s rolling branch rewrites #703 after step 3 or opens a fresh pull request. Either way, PR B is cut from whichever head carries the seven pins. diff --git a/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md b/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md index 9dca8ca7..e4608be9 100644 --- a/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md +++ b/openspec/changes/vale-grammars-package/specs/vale-binary-packages/spec.md @@ -23,7 +23,7 @@ The package SHALL follow the platform packages' rules for contents and versionin ### Requirement: Fetched grammars are verified against committed checksums -The repository SHALL commit a SHA256 checksum for each upstream grammar archive. Upstream's checksums file does not cover these archives, so the detect phase SHALL compute each digest from the archive it downloads and SHALL propose it in the same manifest pull request as the binary digests, where a human reviews it before anything can be published. +The repository SHALL commit a SHA256 checksum for each upstream grammar archive. Upstream's checksums file does not cover these archives, so the detect phase SHALL take each digest from the digest the upstream release records for that asset, SHALL refuse an asset that has none, and SHALL propose the digests in the same manifest pull request as the binary digests, where a human reviews them before anything can be published. The release pipeline SHALL verify every fetched grammar archive against its committed checksum before unpacking it, SHALL refuse to publish on a mismatch, and SHALL refuse to publish when an archive the manifest names is missing upstream. Grammar modules SHALL NOT be stored in version control. @@ -46,7 +46,7 @@ The release pipeline SHALL verify every fetched grammar archive against its comm The workflow that releases the platform packages SHALL also release the grammar package, from the same reviewed manifest, stamped with the same version as the platform packages in the same run. A run SHALL be considered already published only when both the platform packages and the grammar package have a version for the pinned Vale release. -The CLI's dependencies on the grammar package and on the platform packages SHALL be moved together, by the same pull request, and SHALL name the same Vale release. Publishing the grammar package SHALL change no consumer until that pin moves. +The CLI's dependencies on the grammar package and on the platform packages SHALL be moved together, by the same pull request, and SHALL all be the same exact stamped version. Publishing the grammar package SHALL change no consumer until that pin moves. #### Scenario: Merging the update publishes binaries and grammars together @@ -61,4 +61,4 @@ The CLI's dependencies on the grammar package and on the platform packages SHALL #### Scenario: The pins never straddle two Vale releases - **WHEN** the pull request that moves the CLI's Vale pins is opened -- **THEN** it SHALL move the grammar package pin with the platform pins, and both SHALL name the same Vale release +- **THEN** it SHALL move the grammar package pin with the platform pins, all to one exact stamped version diff --git a/openspec/changes/vale-grammars-package/tasks.md b/openspec/changes/vale-grammars-package/tasks.md index 9ba6d786..4fbb9d15 100644 --- a/openspec/changes/vale-grammars-package/tasks.md +++ b/openspec/changes/vale-grammars-package/tasks.md @@ -2,16 +2,16 @@ Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manua ## 1. Manifest and grammar package (PR A) -- [ ] 1.1 Add a `grammars` section to `.github/scripts/vale-manifest.json` (package, directory, asset template `grammar-{name}.zip`, archive member `styles/config/grammars/{name}.wasm`, one `{ name, sha256 }` per upstream grammar for 3.27.0), and extend the manifest comment to say upstream's checksums file does not cover these digests; verify every digest by re-downloading the 22 assets and comparing `shasum -a 256`. -- [ ] 1.2 Create `packages/vale-grammars/` with `package.json` (no `os`/`cpu`, no `bin`, no `scripts`, placeholder version `0.0.0`, `files: ["grammars"]`), `README.md`, and upstream `LICENSE`; ignore `packages/vale-grammars/grammars/` in git and add the package to the changesets `ignore` list; verify `git status` stays clean after a prepare run. -- [ ] 1.3 Extend the pure functions in `vale-release.cjs` (manifest validation, stamping, digest comparison) to cover the grammar section; verify with new cases in `vale-release.test.cjs` under `node --test`. +- [x] 1.1 Add a `grammars` section to `.github/scripts/vale-manifest.json` (package, directory, asset template `grammar-{name}.zip`, archive member `grammar-{name}/styles/config/grammars/{name}.wasm`, one `{ name, sha256 }` per upstream grammar for 3.27.0, taken from GitHub's recorded asset digests), and extend the manifest comment to say upstream's checksums file does not cover these digests; verify every digest by re-downloading the 22 assets and comparing `shasum -a 256`. +- [x] 1.2 Create `packages/vale-grammars/` with `package.json` (no `os`/`cpu`, no `bin`, no `scripts`, placeholder version `0.0.0`, `files: ["grammars"]`), `README.md`, and upstream `LICENSE`; ignore `packages/vale-grammars/grammars/` in git and add the package to the changesets `ignore` list; verify `git status` stays clean after a prepare run. +- [x] 1.3 Extend the pure functions in `vale-release.cjs` (manifest validation, stamping, digest comparison) to cover the grammar section; verify with new cases in `vale-release.test.cjs` under `node --test`. ## 2. Pipeline (PR A) -- [ ] 2.1 `vale-prepare.cjs`: download each grammar archive, verify it against its committed digest before unzipping, extract only the named member flattened to `grammars/.wasm`, stamp at the run's version, and `npm pack` beside the six platform tarballs; verify locally that `--out` yields seven tarballs and the grammar tarball lists 22 `.wasm` files. -- [ ] 2.2 `vale-detect.cjs`: when upstream moves, list the release's `grammar-*.zip` assets, download and hash each, and rewrite the grammar list and digests in the manifest pull request; verify with `vale-detect.test.cjs` cases for an added, a removed, and an unchanged grammar. +- [ ] 2.1 `vale-prepare.cjs`: download each grammar archive, verify it against its committed digest, stream only the named member with `unzip -p`, require the WebAssembly magic, write it to `grammars/.wasm`, stamp, and `npm pack` beside the six platform tarballs; accept `--only grammars` and `--version ` for the bootstrap; verify locally that `--only grammars --version 3.27.0-20261117230017` yields one tarball listing 22 `.wasm` files at that version. +- [ ] 2.2 `vale-detect.cjs`: when upstream moves, read the release's `grammar-*.zip` assets and their GitHub-recorded digests, and rewrite the grammar list and digests in the manifest pull request; verify with `vale-detect.test.cjs` cases for a rewritten grammar list and an asset with no digest. - [ ] 2.3 `vale-gate.cjs`: report "already published" only when the platform packages and `@taskless/vale-grammars` both have a version for the pinned Vale release; verify with `vale-gate.test.cjs` cases for grammars missing, binaries missing, and both present. -- [ ] 2.4 `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs`: treat the grammar package as a member of the set, wait for all seven, and move the CLI's grammar pin with the six platform pins; verify with their `.test.cjs` suites, including the case where binaries are published and grammars are not. +- [ ] 2.4 `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs`: confirm the grammar package joins the pinned set through the existing `@taskless/vale-*` prefix, and document it; verify with `vale-upgrade-detect.test.cjs` cases where a grammar pin moves with the platform pins, and where a grammar package published at a different version blocks the upgrade. - [ ] 2.5 `release-vale.yml` and `vale-upgrade.yml`: update the header comments and any step that names "six" packages; confirm the workflow filename, workflow name, and `environment:` are unchanged (the OIDC binding); verify with `pnpm lint` (actionlint). ## 3. PR A checks @@ -20,12 +20,12 @@ Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manua ## 4. Bootstrap (maintainer, between PR A and PR B) -- [ ] 4.1 Run `vale-prepare.cjs` on PR A's branch, publish the `@taskless/vale-grammars` tarball by hand with `npm publish --access public --tag latest`, and register its npm trusted publisher against `release-vale.yml` and `npm-autopublish`; verify with `npm view @taskless/vale-grammars versions`. -- [ ] 4.2 Merge PR A, then confirm the `Release Vale` run's gate publishes nothing and that `vale-upgrade.yml` produces a head carrying all seven pins. +- [ ] 4.1 Run `node .github/scripts/vale-prepare.cjs --only grammars --version 3.27.0-20261117230017` on PR A's branch, publish that tarball by hand with `npm publish --access public --tag latest`, and register its npm trusted publisher against `release-vale.yml` and `npm-autopublish`; verify `npm view @taskless/vale-grammars version` prints `3.27.0-20261117230017`. +- [ ] 4.2 Merge PR A, then confirm the `Release Vale` run's gate reports all seven packages published and publishes nothing. ## 5. CLI wiring (PR B) -- [ ] 5.1 Add `@taskless/vale-grammars` to `packages/cli/package.json` `dependencies` at the exact published version, if the upgrade workflow has not already, and regenerate `pnpm-lock.yaml`; verify with `pnpm install --frozen-lockfile`. +- [ ] 5.1 Add `@taskless/vale-grammars` to `packages/cli/package.json` `dependencies` at the same exact version as the six platform pins, and regenerate `pnpm-lock.yaml`; verify with `pnpm install --frozen-lockfile`. - [ ] 5.2 Add the grammar resolver beside `findValeBinary` (spec, per-process cache, miss as `{ path: undefined, tried }`); verify with unit tests for a resolved package and a missing one. - [ ] 5.3 Set `VALE_GRAMMARS_PATH` in `spawnVale`'s child environment when the resolver returns a path; verify that the six #703 vendor-contract failures pass with no `VALE_GRAMMARS_PATH` in the test shell. - [ ] 5.4 Export the grammar spec and resolver from `src/node/runtimes/index.ts` with a note in the entry's documentation; verify the build's export classification passes and `dist/node/runtimes/index.d.ts` declares both. @@ -34,7 +34,7 @@ Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manua - [ ] 6.1 In `assembleValeConfig`, refuse when `/config/grammars/` exists, with a tree-level refusal that `validate` and `check` both report; verify with tests for the `validate` failure, the `check` failure with non-zero exit and ast-grep results still reported, and an empty directory. - [ ] 6.2 Add the D6 coverage case to `test/vale-vendor-contract.test.ts`: every comment-tier extension in `VALE_FORMAT_TIERS`, run with the pinned grammars, produces no `grammar "" is not installed` warning; verify it fails when one grammar is withheld and passes with the full set. -- [ ] 6.3 Extend `engine-version-consistency.test.ts` so the grammar pin's Vale component equals `VALE_VERSION` and the platform pins'; verify it fails on a mismatched pin. +- [ ] 6.3 Extend `engine-version-consistency.test.ts` so the grammar pin is the same exact version as the platform pins and its Vale component equals `VALE_VERSION`; verify it fails on a mismatched pin. ## 7. Release notes (PR B) diff --git a/packages/vale-grammars/.gitignore b/packages/vale-grammars/.gitignore new file mode 100644 index 00000000..a9f8c4fc --- /dev/null +++ b/packages/vale-grammars/.gitignore @@ -0,0 +1,7 @@ +# The grammar modules are fetched and unpacked by the release pipeline +# (.github/workflows/release-vale.yml), never committed. A clean checkout of +# this repository holds no grammar modules. +grammars/ + +# npm pack output from a local dry run of that pipeline. +*.tgz diff --git a/packages/vale-grammars/LICENSE b/packages/vale-grammars/LICENSE new file mode 100644 index 00000000..8ed286bf --- /dev/null +++ b/packages/vale-grammars/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2016 Joseph Kato + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/packages/vale-grammars/README.md b/packages/vale-grammars/README.md new file mode 100644 index 00000000..d313e0a2 --- /dev/null +++ b/packages/vale-grammars/README.md @@ -0,0 +1,55 @@ +# @taskless/vale-grammars + +The tree-sitter grammar modules that [Vale](https://vale.sh) uses to read +comments in source code, redistributed as an npm package so that +`@taskless/cli` can hand them to Vale with no download step at install or lint +time. + +Since Vale 3.27.0 these grammars are WebAssembly modules that ship separately +from the Vale executable, and Vale normally installs them with `vale sync`. +Without them Vale finds comments by their delimiters, which lints linter +directives such as `//nolint` as prose and misses a one-line CSS `/* … */` +comment. `@taskless/cli` names this package's `grammars/` directory to Vale +through `VALE_GRAMMARS_PATH` instead. + +## What the published tarball holds + +`grammars/.wasm` for every grammar upstream publishes with the Vale +release, this README, Vale's upstream `LICENSE`, and `package.json`. Nothing +else: no `bin` entry, no JavaScript, and no lifecycle script. + +There is one package for every platform, with no `os` or `cpu`, because a +WebAssembly module is the same bytes on every host. A grammar therefore cannot +be one version on macOS and another on Linux. + +## Versioning + +Published versions are `-`, exactly as the +`@taskless/vale--` packages are, and the release workflow stamps all +of them with one version in one run. `@taskless/cli` pins this package and the +platform packages at that same exact version. + +The `version` field in the repository is the placeholder `0.0.0`. The release +workflow overwrites it with the stamped version immediately before packing, and +the placeholder is never itself published. + +## Where the grammars come from + +The modules are not stored in this repository. `.github/workflows/release-vale.yml` +downloads each `grammar-.zip` from the upstream Vale release, checks the +archive's SHA256 against the digest committed in +`.github/scripts/vale-manifest.json`, refuses to go further on a mismatch, and +only then reads the one `.wasm` member out of it. + +Upstream's `vale__checksums.txt` does not cover these archives. Each +committed digest is the one GitHub records for the release asset when it is +uploaded, which anyone can read back from the releases API to check it. + +## License + +Vale is MIT licensed, Copyright (c) 2016 Joseph Kato (errata-ai). The upstream +`LICENSE` ships in this package unmodified. Each grammar module is compiled by +the Vale project from the corresponding tree-sitter grammar, and upstream +distributes the modules without separate license files. This package +redistributes that build and is not affiliated with or endorsed by the Vale +project. diff --git a/packages/vale-grammars/package.json b/packages/vale-grammars/package.json new file mode 100644 index 00000000..d4ad577f --- /dev/null +++ b/packages/vale-grammars/package.json @@ -0,0 +1,17 @@ +{ + "name": "@taskless/vale-grammars", + "description": "Vale's tree-sitter grammar modules, packaged for @taskless/cli", + "version": "0.0.0", + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/taskless/cli.git", + "directory": "packages/vale-grammars" + }, + "files": [ + "grammars" + ], + "publishConfig": { + "access": "public" + } +} From e5977670a3b0fd9c5a533e09d6271e89031ff94d Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 19 Nov 2026 17:05:08 -0800 Subject: [PATCH 3/5] feat(vale): fetch, verify, gate, and publish the grammar package vale-prepare.cjs packs @taskless/vale-grammars beside the six platform packages. Each grammar archive is verified against its committed digest, then its one module is streamed out with unzip -p and must carry the WebAssembly magic, so nothing in the archive reaches the filesystem. --version stamps a package at a version the set already carries, for the grammar package's manual first publish. vale-detect.cjs reads the grammar list and digests from the release's asset list; vale-gate.cjs counts the grammar package as part of a published release. vale-upgrade-detect.cjs needs no change: the grammar pin matches @taskless/vale-* and is held to the same single version, which the new tests pin. Workflow comments and PR bodies now say seven. --- .github/scripts/release-notes.cjs | 9 ++ .github/scripts/vale-detect.cjs | 12 +- .github/scripts/vale-detect.test.cjs | 38 +++++ .github/scripts/vale-gate.cjs | 31 ++-- .github/scripts/vale-gate.test.cjs | 31 +++- .github/scripts/vale-prepare.cjs | 149 ++++++++++++++---- .github/scripts/vale-upgrade-detect.cjs | 7 + .github/scripts/vale-upgrade-detect.test.cjs | 64 ++++++++ .github/workflows/release-vale.yml | 49 +++--- .github/workflows/vale-upgrade.yml | 15 +- .../changes/vale-grammars-package/tasks.md | 10 +- pnpm-lock.yaml | 2 + 12 files changed, 340 insertions(+), 77 deletions(-) diff --git a/.github/scripts/release-notes.cjs b/.github/scripts/release-notes.cjs index e7493486..8acd02cf 100644 --- a/.github/scripts/release-notes.cjs +++ b/.github/scripts/release-notes.cjs @@ -83,6 +83,15 @@ async function fetchRelease(repository, reference) { tag: release.tag_name, notes: typeof release.body === "string" ? release.body : "", url: typeof release.html_url === "string" ? release.html_url : undefined, + // Each asset's `digest` is the `sha256:` GitHub records at upload. + // vale-detect.cjs pins Vale's grammar archives by it, since upstream's + // checksums file does not cover them. + assets: Array.isArray(release.assets) + ? release.assets.map((asset) => ({ + name: asset?.name, + digest: asset?.digest ?? undefined, + })) + : [], }; } diff --git a/.github/scripts/vale-detect.cjs b/.github/scripts/vale-detect.cjs index ffeb3d33..82c538ad 100644 --- a/.github/scripts/vale-detect.cjs +++ b/.github/scripts/vale-detect.cjs @@ -169,7 +169,12 @@ async function main({ console.log(`fetching ${checksumsUrl}`); const checksumsText = await text(checksumsUrl); - const plan = planManifestUpdate({ manifest, upstreamTag, checksumsText }); + const plan = planManifestUpdate({ + manifest, + upstreamTag, + checksumsText, + releaseAssets: release.assets, + }); console.log( `Upstream ${plan.upstreamVersion} is ahead of ${plan.pinnedVersion}.` ); @@ -178,6 +183,11 @@ async function main({ ` ${applyTemplate(platform.asset, { version: plan.upstreamVersion })} ${platform.sha256}` ); } + for (const module_ of plan.manifest.grammars.modules) { + console.log( + ` ${applyTemplate(plan.manifest.grammars.asset, { name: module_.name })} ${module_.sha256}` + ); + } if (write) { writeFileSync(MANIFEST_PATH, `${JSON.stringify(plan.manifest, null, 2)}\n`); diff --git a/.github/scripts/vale-detect.test.cjs b/.github/scripts/vale-detect.test.cjs index 27249b39..da144dd1 100644 --- a/.github/scripts/vale-detect.test.cjs +++ b/.github/scripts/vale-detect.test.cjs @@ -45,6 +45,17 @@ function checksumsFor(version) { .join("\n")}\n`; } +/** + * The release's asset list as the GitHub API returns it: every grammar the + * committed manifest pins, each with a recorded digest. + */ +function assetsFor() { + return MANIFEST.grammars.modules.map((module_, index) => ({ + name: `grammar-${module_.name}.zip`, + digest: `sha256:${digestFor(index)}`, + })); +} + /** * Run main() with both fetches stubbed and $GITHUB_OUTPUT pointed at a temp * file, then return the parsed step outputs plus which URLs were fetched. @@ -55,6 +66,7 @@ async function runDetect({ checksums, argv = [], wantNotes = false, + assets = assetsFor(), }) { const directory = mkdtempSync(join(tmpdir(), "vale-detect-test-")); const outputPath = join(directory, "github-output"); @@ -70,6 +82,7 @@ async function runDetect({ tag: upstreamTag, notes, url: `https://github.com/${MANIFEST.upstream.repository}/releases/tag/${upstreamTag}`, + assets, }), text: async (url) => { fetched.push(url); @@ -264,3 +277,28 @@ test("detect: a repository with no releases aborts rather than reporting current /has no published releases/ ); }); + +test("detect: the grammar list and digests come from the release's assets", async () => { + // main() hands the release's asset list to planManifestUpdate. Withholding + // the grammar assets proves it reaches there: the plan refuses a release + // with none, rather than carrying the old list forward unreviewed. + await assert.rejects( + runDetect({ + upstreamTag: "v3.99.0", + checksums: checksumsFor("3.99.0"), + assets: [], + }), + /upstream 3\.99\.0 publishes no grammar-\*\.zip assets/ + ); +}); + +test("detect: a grammar asset with no recorded digest aborts", async () => { + await assert.rejects( + runDetect({ + upstreamTag: "v3.99.0", + checksums: checksumsFor("3.99.0"), + assets: [{ name: "grammar-go.zip" }], + }), + /grammar-go\.zip has no recorded sha256 digest/ + ); +}); diff --git a/.github/scripts/vale-gate.cjs b/.github/scripts/vale-gate.cjs index 116ca1e8..e8299b98 100644 --- a/.github/scripts/vale-gate.cjs +++ b/.github/scripts/vale-gate.cjs @@ -35,7 +35,7 @@ const { appendFileSync, readFileSync } = require("node:fs"); const { join } = require("node:path"); -const { assertManifest } = require("./vale-release.cjs"); +const { assertManifest, manifestPackages } = require("./vale-release.cjs"); const MANIFEST_PATH = join(__dirname, "vale-manifest.json"); const REGISTRY = "https://registry.npmjs.org"; @@ -85,15 +85,17 @@ async function fetchPublishedVersions(packageName) { * forced, nothing published, everything published, and a partial set. */ function planPublish({ manifest, publishedByPackage, forced }) { - const missing = manifest.platforms + // Every package the manifest releases, the grammar package included: binaries + // published without their grammars are not a published release. + const missing = manifestPackages(manifest) .filter( - (platform) => + (entry) => !coversVersion( - publishedByPackage[platform.package] ?? [], + publishedByPackage[entry.package] ?? [], manifest.valeVersion ) ) - .map((platform) => platform.package); + .map((entry) => entry.package); if (forced) { return { shouldPublish: true, missing, reason: "forced" }; @@ -117,11 +119,12 @@ async function main({ // Concurrent, not sequential: the six lookups are independent, and this job // exists to decide cheaply BEFORE prepare downloads ~60 MB. Sequential awaits // would make the gate six round trips deep for no reason. + const packages = manifestPackages(manifest); const publishedByPackage = Object.fromEntries( await Promise.all( - manifest.platforms.map(async (platform) => [ - platform.package, - await published(platform.package), + packages.map(async (entry) => [ + entry.package, + await published(entry.package), ]) ) ); @@ -129,20 +132,18 @@ async function main({ const plan = planPublish({ manifest, publishedByPackage, forced }); console.log(`Vale ${manifest.valeVersion}`); - for (const platform of manifest.platforms) { - const covered = !plan.missing.includes(platform.package); - console.log( - ` ${covered ? "published" : "MISSING "} ${platform.package}` - ); + for (const entry of packages) { + const covered = !plan.missing.includes(entry.package); + console.log(` ${covered ? "published" : "MISSING "} ${entry.package}`); } if (plan.reason === "forced") { console.log( - `\nExplicitly dispatched — publishing regardless (${plan.missing.length} of ${manifest.platforms.length} not yet on the registry).` + `\nExplicitly dispatched — publishing regardless (${plan.missing.length} of ${packages.length} not yet on the registry).` ); } else if (plan.shouldPublish) { console.log( - `\n${plan.missing.length} of ${manifest.platforms.length} package(s) lack Vale ${manifest.valeVersion}. Publishing.` + `\n${plan.missing.length} of ${packages.length} package(s) lack Vale ${manifest.valeVersion}. Publishing.` ); } else { console.log( diff --git a/.github/scripts/vale-gate.test.cjs b/.github/scripts/vale-gate.test.cjs index 7a77b685..9d87aa4f 100644 --- a/.github/scripts/vale-gate.test.cjs +++ b/.github/scripts/vale-gate.test.cjs @@ -22,7 +22,10 @@ const MANIFEST = JSON.parse( readFileSync(join(__dirname, "vale-manifest.json"), "utf8") ); const PINNED = MANIFEST.valeVersion; -const PACKAGES = MANIFEST.platforms.map((platform) => platform.package); +const PACKAGES = [ + ...MANIFEST.platforms.map((platform) => platform.package), + MANIFEST.grammars.package, +]; /** A registry stub answering from a {package: [versions]} table. */ const registry = (table) => async (packageName) => table[packageName] ?? []; @@ -131,3 +134,29 @@ test("a 404 package reads as unpublished rather than failing the gate", async () assert.equal(plan.shouldPublish, true); assert.deepEqual(plan.missing, PACKAGES); }); + +test("publishes when the binaries are out and the grammars are not", () => { + // Binaries without their grammars are not a published release: a CLI + // pinned to that set would read code comments by delimiters. + const table = allPublished(); + delete table[MANIFEST.grammars.package]; + const plan = planPublish({ + manifest: MANIFEST, + publishedByPackage: table, + forced: false, + }); + assert.equal(plan.shouldPublish, true); + assert.deepEqual(plan.missing, [MANIFEST.grammars.package]); +}); + +test("publishes when the grammars are out and a binary is not", () => { + const table = allPublished(); + delete table[MANIFEST.platforms[0].package]; + const plan = planPublish({ + manifest: MANIFEST, + publishedByPackage: table, + forced: false, + }); + assert.equal(plan.shouldPublish, true); + assert.deepEqual(plan.missing, [MANIFEST.platforms[0].package]); +}); diff --git a/.github/scripts/vale-prepare.cjs b/.github/scripts/vale-prepare.cjs index b841c998..a5ca4add 100644 --- a/.github/scripts/vale-prepare.cjs +++ b/.github/scripts/vale-prepare.cjs @@ -19,7 +19,8 @@ * anything is unpacked, so unverified bytes are never even expanded onto * disk; * 3. unpack the single executable named by the manifest into its package - * directory at mode 0755; + * directory at mode 0755, or, for the grammar package, read each grammar + * module out of its archive into `grammars/.wasm`; * 4. stamp every package.json with one shared version; * 5. `npm pack` each package into --out. * @@ -30,11 +31,18 @@ * sealed into a tarball. * * Usage: - * node .github/scripts/vale-prepare.cjs [--out ] [--only ]... [--skip-pack] + * node .github/scripts/vale-prepare.cjs [--out ] [--only ]... + * [--version ] [--skip-pack] * * --out where to write the .tgz files (default: .vale-dist at the repo root) - * --only restrict to one platform package; repeatable. Accepts the full - * name (@taskless/vale-linux-x64) or the suffix (linux-x64). + * --only restrict to one package; repeatable. Accepts the full name + * (@taskless/vale-linux-x64, @taskless/vale-grammars) or the + * suffix (linux-x64, grammars). + * --version stamp with this exact version instead of one from the clock. + * Its Vale component must be the manifest's. It exists for + * publishing a package at a stamp the rest of the set already + * carries, which is how the grammar package's first, manual + * publish keeps every @taskless/vale-* pin at one version. * --skip-pack fetch, verify, unpack, and stamp, but do not run npm pack. * * Running this locally leaves the stamped version in each package.json and the @@ -63,9 +71,13 @@ const { applyStamp, assertChecksum, assertManifest, + assertWasmModule, planStamp, resolveAssetName, resolveDownloadUrl, + resolveGrammarAsset, + resolveGrammarDownloadUrl, + resolveGrammarMember, } = require("./vale-release.cjs"); const REPO_ROOT = resolve(__dirname, "..", ".."); @@ -95,6 +107,9 @@ function parseArguments(argv) { } else if (argument === "--only") { index += 1; options.only.push(requireValue(argv, index, "--only")); + } else if (argument === "--version") { + index += 1; + options.version = requireValue(argv, index, "--version"); } else if (argument === "--skip-pack") { options.pack = false; } else { @@ -105,16 +120,14 @@ function parseArguments(argv) { } /** Match `--only` against either the full package name or its suffix. */ -function selects(only, platform) { +function selects(only, entry) { if (only.length === 0) { return true; } - const suffix = platform.package.replace("@taskless/vale-", ""); + const suffix = entry.package.replace("@taskless/vale-", ""); return only.some( - (entry) => - entry === platform.package || - entry === suffix || - entry === `vale-${suffix}` + (name) => + name === entry.package || name === suffix || name === `vale-${suffix}` ); } @@ -209,6 +222,58 @@ function unpackMember(archivePath, member, destinationPath) { } } +/** + * Read one grammar module out of a verified zip and return its bytes. + * + * `unzip -p` writes the member's contents to stdout and creates nothing on + * disk, so unlike `unpackMember` there is no extracted path to contain: a + * nested member, a `../` name, or a symlinked directory inside the archive + * cannot place a file anywhere. That is why the grammar member may be nested + * where a platform's `archiveMember` must stay flat. What remains is reading + * the wrong member, and `assertWasmModule` refuses anything that is not a + * WebAssembly module, including a symlink entry, which `-p` emits as the + * link's target text. + */ +function readGrammarMember(archivePath, member) { + const result = spawnSync("unzip", ["-p", archivePath, member], { + encoding: "buffer", + // The largest 3.27.0 grammar is under 6 MB; the cap is headroom, not a + // measurement, and exceeding it fails the run rather than truncating. + maxBuffer: 64 * 1024 * 1024, + }); + if (result.error) { + throw result.error; + } + if (result.status !== 0) { + throw new Error( + `${basename(archivePath)} has no readable member ${member} (unzip exited ${result.status})` + ); + } + assertWasmModule(result.stdout, `${basename(archivePath)}:${member}`); + return result.stdout; +} + +/** + * Write a stamped package.json and pack the package into `out`. Shared by the + * platform packages and the grammar package, so both are stamped the same way. + */ +function stampAndPack(packageDirectory, version, options) { + const packageJsonPath = join(packageDirectory, "package.json"); + const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8")); + writeFileSync( + packageJsonPath, + `${JSON.stringify(applyStamp(packageJson, version), null, 2)}\n` + ); + console.log(` stamp ${version}`); + if (options.pack) { + run( + "npm", + ["pack", "--ignore-scripts", "--pack-destination", options.out], + { cwd: packageDirectory } + ); + } +} + function setOutput(key, value) { const file = process.env.GITHUB_OUTPUT; if (file) { @@ -224,12 +289,18 @@ async function main() { const selected = manifest.platforms.filter((platform) => selects(options.only, platform) ); - if (selected.length === 0) { - throw new Error(`--only matched no platform in ${MANIFEST_PATH}`); + const grammarsSelected = selects(options.only, manifest.grammars); + if (selected.length === 0 && !grammarsSelected) { + throw new Error(`--only matched no package in ${MANIFEST_PATH}`); } - // One version for the whole set, computed once from the run's start time. - const plan = planStamp({ manifest, date: new Date() }); + // One version for the whole set, computed once from the run's start time, + // unless --version names one the rest of the set already carries. + const plan = planStamp({ + manifest, + date: new Date(), + version: options.version, + }); console.log(`Vale ${manifest.valeVersion} → version ${plan.version}`); mkdirSync(options.out, { recursive: true }); @@ -258,29 +329,51 @@ async function main() { ` unpack ${platform.directory}/${platform.archiveMember} (0755)` ); - const packageJsonPath = join(packageDirectory, "package.json"); - const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8")); - writeFileSync( - packageJsonPath, - `${JSON.stringify(applyStamp(packageJson, plan.version), null, 2)}\n` - ); - console.log(` stamp ${plan.version}`); - + stampAndPack(packageDirectory, plan.version, options); if (options.pack) { - run( - "npm", - ["pack", "--ignore-scripts", "--pack-destination", options.out], - { cwd: packageDirectory } - ); packed.push(platform.package); } } + + if (grammarsSelected) { + const { grammars } = manifest; + const packageDirectory = join(REPO_ROOT, grammars.directory); + const modulesDirectory = join(packageDirectory, "grammars"); + // Emptied first, so a grammar upstream dropped cannot survive from an + // earlier local run into the tarball. + rmSync(modulesDirectory, { recursive: true, force: true }); + mkdirSync(modulesDirectory, { recursive: true }); + console.log(`\n${grammars.package}`); + + for (const module_ of grammars.modules) { + const asset = resolveGrammarAsset(manifest, module_); + const archivePath = join(downloadDirectory, asset); + const actual = await download( + resolveGrammarDownloadUrl(manifest, module_), + archivePath + ); + // Fatal on mismatch, before anything is read out of the archive. + assertChecksum({ asset, expected: module_.sha256, actual }); + const bytes = readGrammarMember( + archivePath, + resolveGrammarMember(manifest, module_) + ); + writeFileSync(join(modulesDirectory, `${module_.name}.wasm`), bytes); + console.log(` ${module_.name.padEnd(12)} sha256 ${actual}`); + } + + stampAndPack(packageDirectory, plan.version, options); + if (options.pack) { + packed.push(grammars.package); + } + } } finally { rmSync(downloadDirectory, { recursive: true, force: true }); } + const preparedCount = selected.length + (grammarsSelected ? 1 : 0); console.log( - `\nPrepared ${selected.length} package(s) at ${plan.version}` + + `\nPrepared ${preparedCount} package(s) at ${plan.version}` + (options.pack ? `; ${packed.length} tarball(s) in ${options.out}` : "") ); setOutput("version", plan.version); diff --git a/.github/scripts/vale-upgrade-detect.cjs b/.github/scripts/vale-upgrade-detect.cjs index 13c67fff..90f5cb23 100644 --- a/.github/scripts/vale-upgrade-detect.cjs +++ b/.github/scripts/vale-upgrade-detect.cjs @@ -105,6 +105,13 @@ const PIN_PREFIX = "@taskless/vale-"; * What counts as a platform pin. Used BOTH to enumerate the pins and to rewrite * them, so the two cannot drift apart — the trailing hyphen is the boundary * here, since there is no bare `@taskless/vale` package. + * + * `@taskless/vale-grammars` matches too, deliberately. Vale's grammars ship as + * their own package since 3.27.0 and are released with the binaries at one + * stamp, so the one-version rule below holds the grammar pin to the platform + * pins, and an upgrade waits until the grammars are published with them. It is + * a regular dependency rather than an optional one, which is why every + * dependency field is read. */ const PIN_PATTERN = /^@taskless\/vale-/; diff --git a/.github/scripts/vale-upgrade-detect.test.cjs b/.github/scripts/vale-upgrade-detect.test.cjs index 3edcd4a2..476999f1 100644 --- a/.github/scripts/vale-upgrade-detect.test.cjs +++ b/.github/scripts/vale-upgrade-detect.test.cjs @@ -338,3 +338,67 @@ test("base version strips the stamp", () => { assert.equal(baseVersion("3.21.0-20260914010203"), "3.21.0"); assert.throws(() => baseVersion("3.21.0"), /not a stamped version/); }); + +// --------------------------------------------------------------------------- +// The grammar package (openspec/changes/vale-grammars-package) +// +// @taskless/vale-grammars matches the same @taskless/vale-* prefix, so once +// the CLI pins it, it is a seventh member of the set with no change here. It +// is a regular dependency rather than an optional one, since every host needs +// it, which is the case these tests pin. +// --------------------------------------------------------------------------- + +const GRAMMARS = "@taskless/vale-grammars"; + +const sourceWithGrammarsAt = (version) => + `${JSON.stringify( + { + name: "@taskless/cli", + dependencies: { [GRAMMARS]: version, zod: "^4" }, + ...pinnedAt(version), + }, + undefined, + 2 + )}\n`; + +test("--write moves the grammar pin with the six platform pins", async () => { + const source = sourceWithGrammarsAt("3.27.0-20261117230017"); + const { written } = await run({ + packageJson: JSON.parse(source), + packageJsonSource: source, + published: "3.28.0-20261201000000", + argv: ["--write"], + }); + const pins = JSON.parse(written); + assert.equal(pins.dependencies[GRAMMARS], "3.28.0-20261201000000"); + assert.equal(written.match(/3\.28\.0-20261201000000/g).length, 7); + assert.doesNotMatch(written, /3\.27\.0-20261117230017/); + assert.equal(pins.dependencies.zod, "^4"); +}); + +test("grammars published at a different version block the upgrade", async () => { + // Binaries out at the new stamp, grammars still at the old one: upgrading + // would pin a grammar package npm does not serve at that version. + const source = sourceWithGrammarsAt("3.27.0-20261117230017"); + await assert.rejects( + run({ + packageJson: JSON.parse(source), + published: Object.fromEntries([ + ...PLATFORMS.map((name) => [name, "3.28.0-20261201000000"]), + [GRAMMARS, "3.27.0-20261117230017"], + ]), + }), + /not at one version, so it is mid-publish or partially failed/ + ); +}); + +test("a grammar pin that disagrees with the platform pins aborts", () => { + assert.throws( + () => + collectPins({ + dependencies: { [GRAMMARS]: "3.26.0-20261102164455" }, + ...pinnedAt("3.27.0-20261117230017"), + }), + /pins disagree/ + ); +}); diff --git a/.github/workflows/release-vale.yml b/.github/workflows/release-vale.yml index 56345592..f437b824 100644 --- a/.github/workflows/release-vale.yml +++ b/.github/workflows/release-vale.yml @@ -12,10 +12,12 @@ # detect Runs on a schedule with NO npm credential and NO OIDC identity. It # compares the latest upstream Vale release against the version # pinned in .github/scripts/vale-manifest.json. When upstream is -# ahead it opens a pull request that updates the pinned version and -# all six SHA256 digests, taken from upstream's own checksums file, -# and carries upstream's release notes so a reviewer can see what the -# digests are for. It publishes nothing. +# ahead it opens a pull request that updates the pinned version, the +# six binary SHA256 digests taken from upstream's own checksums file, +# and one digest per grammar archive taken from GitHub's record of +# each release asset (upstream's checksums file does not cover +# those). It carries upstream's release notes so a reviewer can see +# what the digests are for. It publishes nothing. # # That pull request lives on ONE ROLLING BRANCH, # `vendor/vale/republish`, rebuilt and retitled as upstream moves @@ -53,7 +55,7 @@ # or any 3.17.1-* stamp, which is exactly the set this workflow can mint for it. # `gate` runs that check, because the push trigger below fires on ANY edit to # vale-manifest.json and a `paths:` filter cannot see why the file changed — a -# reworded comment would otherwise publish six packages. An explicit dispatch +# reworded comment would otherwise publish seven packages. An explicit dispatch # passes --force and is never suppressed. # # WHY prepare AND publish ARE SEPARATE JOBS: `prepare` downloads third-party @@ -74,10 +76,13 @@ # short-lived OIDC-minted token, with no stored NPM_TOKEN anywhere. That binding # is registered PER PACKAGE on npmjs.com, and it names this workflow's FILENAME # and its environment — so renaming this file or changing `environment:` below -# breaks every publish until the six bindings are re-registered to match. There -# is nothing to bind until the package name exists, so the FIRST publish of each -# of the six names is a deliberate one-time manual step by a maintainer, who -# then registers the trusted publisher. This workflow assumes that has already +# breaks every publish until the seven bindings are re-registered to match. +# There is nothing to bind until the package name exists, so the FIRST publish +# of each of the seven names (the six platform packages and +# @taskless/vale-grammars) is a deliberate one-time manual step by a maintainer, +# who then registers the trusted publisher. For a name added to an existing set, +# that publish takes the set's current stamp (vale-prepare.cjs --version), so +# every @taskless/vale-* pin can stay at one version. This workflow assumes that has already # happened for every name in the manifest, and there is no fallback token path # here on purpose. # @@ -85,7 +90,7 @@ # reviewer — unlike `npm-production`, where @taskless/cli still waits for a # click. That is not a relaxation, because the review already happened # somewhere better: the manifest-update pull request IS the gate. A human reads -# the upstream Vale version and all six SHA256 digests there, and only merging +# the upstream Vale version and every SHA256 digest there, and only merging # it can start a publish at all. An environment approval would be a second copy # of that same gate, asked at a point where nothing is left to decide — the # bytes were fixed when the digests were reviewed, and a reviewer standing at @@ -171,7 +176,7 @@ jobs: # `--notes-out` writes upstream's release notes for the proposed version # to a file, which the next step appends to the pull request body. A # reviewer's job here is to decide whether these bytes should be - # publishable, and a bare version number with six digests does not say + # publishable, and a bare version number with a list of digests does not say # whether the release is a security fix or a docs pass. The notes arrive # as a FILE rather than a step output because they are third-party # Markdown: a $GITHUB_OUTPUT line is delimited text that a release body @@ -217,8 +222,12 @@ jobs: Upstream Vale is ahead of the version this repository packages. This updates `.github/scripts/vale-manifest.json` to the new upstream - version and replaces every platform's SHA256 with the digest from - upstream's `vale__checksums.txt`. + version, replaces every platform's SHA256 with the digest from + upstream's `vale__checksums.txt`, and rewrites the grammar + list and digests from the release's `grammar-*.zip` assets. Upstream's + checksums file does not cover those archives, so each grammar digest + is the one GitHub records for the asset, which you can read back from + the releases API. **Reviewing this is the trust boundary.** Merging it authorizes the publish phase to download those exact archives and package them; a @@ -261,7 +270,7 @@ jobs: # Is there anything to publish? The push trigger fires on ANY edit to # vale-manifest.json — a reworded comment, a reformat, a digest correction — # and it cannot tell those from a version bump. Without this gate each of them - # publishes six packages at a fresh - stamp, which + # publishes seven packages at a fresh - stamp, which # nothing downstream can absorb because every stamp is novel by construction. # # Cheap on purpose: it reads the registry and decides BEFORE `prepare` @@ -320,7 +329,7 @@ jobs: # dot-prefixed. upload-artifact has excluded hidden files by default since # v4.4, and a leading-dot DIRECTORY makes everything beneath it hidden — # so `.vale-dist/*.tgz` silently matched nothing even though `prepare` - # had just written six tarballs there. `if-no-files-found: error` is what + # had just written its tarballs there. `if-no-files-found: error` is what # turned that into a failed run rather than an empty artifact handed to # the publish job; keep both. - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 @@ -364,7 +373,7 @@ jobs: # publisher for each @taskless/vale-* package is bound. Reviewer-free by # design (see APPROVAL POLICY in the header): the manifest pull request is # the gate, and this environment's branch policy still confines the - # credential to `main`. Changing this name requires re-registering all six + # credential to `main`. Changing this name requires re-registering all seven # bindings on npmjs.com first — the environment is part of the binding. environment: npm-autopublish permissions: @@ -403,7 +412,7 @@ jobs: # these are not preview builds, they are the only builds, and the # prerelease component exists to defeat range matching, not to signal # instability. - - name: Publish every platform tarball + - name: Publish every tarball working-directory: tarballs env: STAMPED_VERSION: ${{ needs.prepare.outputs.version }} @@ -417,11 +426,11 @@ jobs: echo "Publishing ${#tarballs[@]} package(s) at ${STAMPED_VERSION}." # The loop deliberately does not stop at the first failure, and the - # step deliberately does not lean on `set -e` here. Six sequential - # publishes are six chances for a transient registry error, and + # step deliberately does not lean on `set -e` here. Seven sequential + # publishes are seven chances for a transient registry error, and # aborting midway leaves the set partially released — some platforms # resolvable at this version, others not, which is the one state the - # CLI's exact cross-package pins cannot tolerate. Attempting all six + # CLI's exact cross-package pins cannot tolerate. Attempting all seven # and failing at the end means one re-run has at most the stragglers # left to do. # diff --git a/.github/workflows/vale-upgrade.yml b/.github/workflows/vale-upgrade.yml index 67d1f1d9..551c252e 100644 --- a/.github/workflows/vale-upgrade.yml +++ b/.github/workflows/vale-upgrade.yml @@ -6,10 +6,10 @@ # # republish release-vale.yml. Watches upstream Vale, proposes a manifest # with new digests on `vendor/vale/republish`, and on merge -# publishes the six @taskless/vale-* packages at +# publishes the seven @taskless/vale-* packages at # -. Changes nothing for a consumer. # -# upgrade this workflow. Moves the six pins in packages/cli/package.json +# upgrade this workflow. Moves every @taskless/vale-* pin in packages/cli/package.json # to the published set on `vendor/vale/upgrade`. THIS is what # reaches a user. # @@ -73,7 +73,7 @@ on: # other's result. # # Starting together is also the race. release-vale.yml has to fetch, verify, - # pack, and publish six packages before the registry can answer "what is + # pack, and publish seven packages before the registry can answer "what is # published" with the new set, and the detect step below asks the registry. # Measured on 3.22.0: the manifest merged at 18:09:13Z, the publish run # started three seconds later, the packages were stamped 18:09:30 — and a run @@ -160,7 +160,7 @@ jobs: # pins. GITHUB_TOKEN is only for the rate limit on the release-notes # lookup. # - # `--write` moves the six pins and `VALE_VERSION` in capabilities.ts + # `--write` moves every @taskless/vale-* pin and `VALE_VERSION` in capabilities.ts # together, the latter to the base version the binary reports. It fails # the run if the declaration is not found exactly once. # @@ -179,7 +179,7 @@ jobs: if: steps.wait.outputs.ready == 'true' # Resolution, not editing. `--lockfile-only` skips linking, so nothing is - # downloaded and no lifecycle script runs. It fails if the six packages + # downloaded and no lifecycle script runs. It fails if the pinned packages # are not all resolvable at the new version — a second, independent check # on the half-published set the detect step already refuses. - name: Regenerate the lockfile @@ -314,11 +314,12 @@ jobs: The \`@taskless/vale-*\` packages are published ahead of what \`@taskless/cli\` pins. - This moves all six pins from \`${PINNED_VERSION}\` to + This moves every \`@taskless/vale-*\` pin from \`${PINNED_VERSION}\` to \`${VALE_VERSION}\` — Vale **${BASE_VERSION}** — and regenerates \`pnpm-lock.yaml\`. The pins move **together** on purpose: the platform packages are selected by optional dependency, so a straggler left at - the old version is a different Vale on one platform than on the others. + the old version is a different Vale on one platform than on the others, + and the grammar package must match the binaries that read it. It also moves \`VALE_VERSION\` in \`capabilities.ts\` to \`${BASE_VERSION}\` and regenerates \`src/generated/vale-vocabulary.ts\` diff --git a/openspec/changes/vale-grammars-package/tasks.md b/openspec/changes/vale-grammars-package/tasks.md index 4fbb9d15..a5e50da9 100644 --- a/openspec/changes/vale-grammars-package/tasks.md +++ b/openspec/changes/vale-grammars-package/tasks.md @@ -8,11 +8,11 @@ Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manua ## 2. Pipeline (PR A) -- [ ] 2.1 `vale-prepare.cjs`: download each grammar archive, verify it against its committed digest, stream only the named member with `unzip -p`, require the WebAssembly magic, write it to `grammars/.wasm`, stamp, and `npm pack` beside the six platform tarballs; accept `--only grammars` and `--version ` for the bootstrap; verify locally that `--only grammars --version 3.27.0-20261117230017` yields one tarball listing 22 `.wasm` files at that version. -- [ ] 2.2 `vale-detect.cjs`: when upstream moves, read the release's `grammar-*.zip` assets and their GitHub-recorded digests, and rewrite the grammar list and digests in the manifest pull request; verify with `vale-detect.test.cjs` cases for a rewritten grammar list and an asset with no digest. -- [ ] 2.3 `vale-gate.cjs`: report "already published" only when the platform packages and `@taskless/vale-grammars` both have a version for the pinned Vale release; verify with `vale-gate.test.cjs` cases for grammars missing, binaries missing, and both present. -- [ ] 2.4 `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs`: confirm the grammar package joins the pinned set through the existing `@taskless/vale-*` prefix, and document it; verify with `vale-upgrade-detect.test.cjs` cases where a grammar pin moves with the platform pins, and where a grammar package published at a different version blocks the upgrade. -- [ ] 2.5 `release-vale.yml` and `vale-upgrade.yml`: update the header comments and any step that names "six" packages; confirm the workflow filename, workflow name, and `environment:` are unchanged (the OIDC binding); verify with `pnpm lint` (actionlint). +- [x] 2.1 `vale-prepare.cjs`: download each grammar archive, verify it against its committed digest, stream only the named member with `unzip -p`, require the WebAssembly magic, write it to `grammars/.wasm`, stamp, and `npm pack` beside the six platform tarballs; accept `--only grammars` and `--version ` for the bootstrap; verify locally that `--only grammars --version 3.27.0-20261117230017` yields one tarball listing 22 `.wasm` files at that version. +- [x] 2.2 `vale-detect.cjs`: when upstream moves, read the release's `grammar-*.zip` assets and their GitHub-recorded digests, and rewrite the grammar list and digests in the manifest pull request; verify with `vale-detect.test.cjs` cases for a rewritten grammar list and an asset with no digest. +- [x] 2.3 `vale-gate.cjs`: report "already published" only when the platform packages and `@taskless/vale-grammars` both have a version for the pinned Vale release; verify with `vale-gate.test.cjs` cases for grammars missing, binaries missing, and both present. +- [x] 2.4 `vale-upgrade-detect.cjs` and `vale-upgrade-wait.cjs`: confirm the grammar package joins the pinned set through the existing `@taskless/vale-*` prefix, and document it; verify with `vale-upgrade-detect.test.cjs` cases where a grammar pin moves with the platform pins, and where a grammar package published at a different version blocks the upgrade. +- [x] 2.5 `release-vale.yml` and `vale-upgrade.yml`: update the header comments and any step that names "six" packages; confirm the workflow filename, workflow name, and `environment:` are unchanged (the OIDC binding); verify with `pnpm lint` (actionlint). ## 3. PR A checks diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index db4ba48f..d33f6460 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -186,6 +186,8 @@ importers: packages/vale-darwin-x64: {} + packages/vale-grammars: {} + packages/vale-linux-arm64: {} packages/vale-linux-x64: {} From de530ba3cad1c1e17c79fd553ad160e5e8cd115a Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 19 Nov 2026 17:06:15 -0800 Subject: [PATCH 4/5] docs(openspec): check off the PR A checks --- openspec/changes/vale-grammars-package/tasks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openspec/changes/vale-grammars-package/tasks.md b/openspec/changes/vale-grammars-package/tasks.md index a5e50da9..da7b60dc 100644 --- a/openspec/changes/vale-grammars-package/tasks.md +++ b/openspec/changes/vale-grammars-package/tasks.md @@ -16,7 +16,7 @@ Groups 1–3 are PR A, cut from `origin/main`. Group 4 is the maintainer's manua ## 3. PR A checks -- [ ] 3.1 Run `pnpm typecheck`, `pnpm lint`, and `node --test .github/scripts/` and confirm they pass; open PR A to `main` with the `skip-changeset` label and request `@claude /review`. +- [x] 3.1 Run `pnpm typecheck`, `pnpm lint`, and `node --test .github/scripts/` and confirm they pass; open PR A to `main` with the `skip-changeset` label and request `@claude /review`. ## 4. Bootstrap (maintainer, between PR A and PR B) From a83b34de63c0928b07f1b0f126b9e951fb0d3026 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 19 Nov 2026 17:19:50 -0800 Subject: [PATCH 5/5] docs(vale): say seven packages in the publish gate's comments --- .github/scripts/vale-gate.cjs | 13 +++++++------ .github/scripts/vale-gate.test.cjs | 2 +- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/.github/scripts/vale-gate.cjs b/.github/scripts/vale-gate.cjs index e8299b98..a31a9516 100644 --- a/.github/scripts/vale-gate.cjs +++ b/.github/scripts/vale-gate.cjs @@ -6,7 +6,7 @@ * vale-manifest.json (plus explicit dispatch). That `paths:` filter cannot see * WHY the file changed: correcting a typo in its comment block, reformatting * it, or fixing a digest all look identical to a version bump, and each one - * would publish six fresh packages. Publishing is not idempotent here — every + * would publish seven fresh packages. Publishing is not idempotent here — every * run stamps -, a version npm has never seen — so * nothing downstream can absorb the mistake. * @@ -19,9 +19,10 @@ * * WHAT IT DELIBERATELY DOES NOT DO. It never suppresses an explicit dispatch: * a human asking for a publish gets one, stamp collision or not. And it skips - * only when ALL six packages already carry the pinned base version. Checking a - * single package would silently skip a half-published set, so requiring all six - * makes the gate double as partial-release repair — it re-runs precisely the + * only when ALL seven packages (the six platform packages and the grammar + * package) already carry the pinned base version. Checking a single package + * would silently skip a half-published set, so requiring all seven makes the + * gate double as partial-release repair — it re-runs precisely the * case the publish loop's failure aggregation is there to report. * * Usage: @@ -116,9 +117,9 @@ async function main({ JSON.parse(readFileSync(MANIFEST_PATH, "utf8")) ); - // Concurrent, not sequential: the six lookups are independent, and this job + // Concurrent, not sequential: the seven lookups are independent, and this job // exists to decide cheaply BEFORE prepare downloads ~60 MB. Sequential awaits - // would make the gate six round trips deep for no reason. + // would make the gate seven round trips deep for no reason. const packages = manifestPackages(manifest); const publishedByPackage = Object.fromEntries( await Promise.all( diff --git a/.github/scripts/vale-gate.test.cjs b/.github/scripts/vale-gate.test.cjs index 9d87aa4f..fd08bf92 100644 --- a/.github/scripts/vale-gate.test.cjs +++ b/.github/scripts/vale-gate.test.cjs @@ -3,7 +3,7 @@ /** * Tests for vale-gate.cjs — the decision that keeps an unrelated edit to - * vale-manifest.json from publishing six packages. + * vale-manifest.json from publishing seven packages. * * The interesting cases are all "what does the registry already have", so the * registry is stubbed throughout and nothing here touches the network. The