Skip to content

feat(eql): absorb the EQL subsystem, and kill the version skew - #885

Merged
tobyhede merged 1912 commits into
mainfrom
feat/eql-1-vendor
Aug 21, 2026
Merged

feat(eql): absorb the EQL subsystem, and kill the version skew#885
tobyhede merged 1912 commits into
mainfrom
feat/eql-1-vendor

Conversation

@tobyhede

@tobyhede tobyhede commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Absorbs the EQL subsystem from cipherstash/encrypt-query-language into this repo, and closes the version skew between the Rust that emits EQL payloads and the SQL that stores them.

Ready for review. Phases 1–4 are done. Phase 3 keeps one open item that no machine without Docker and CS_* can close (below), and Phase 5 — the release cutover — has not started and is deliberately out of scope.

main has been merged in. The branch was CONFLICTING and is now MERGEABLE. The collision was two independent re-anchorings of the same content-addressed Prisma Next migration chain: main's 0.16 → 0.17 upgrade and this branch's eql-3.0.5 bump each rewrote the baseline's bytes, so the conflicting hashes could not be settled by picking a side — both were digests over content both sides had changed. The artefacts were regenerated instead, and only the baseline and the 3.0.5 edge: all four migration.ts files call readVerifiedInstallSql(), so re-emitting the 3.0.2 and 3.0.4 edges would have baked current SQL into artefacts describing historical releases. Three files merged cleanly with no conflict markers and were semantically wrong anyway (refs/head.json, the baseline migration.json, the baseline migration.ts); those are the ones worth a second look.

Land this with a merge commit, not a squash. The subtree lineage is the point of the import — a squash discards the second parent and with it the git-subtree-dir: packages/eql metadata that makes a future git subtree pull resolve. This repository permits all three merge methods, so the button can undo that silently.

#915 is stacked on this branch and carries the review fixes for the four blockers found here, the upstream 3.0.5 sync, and a second review round. Its 88c21c9b is itself a git subtree pull merge, so the same no-squash rule applies to it.

docs/plans/2026-08-13-eql-monorepo-absorption.md is the authoritative artifact. Every decision is recorded there with its reasoning, including the ones that turned out to be wrong. The commit messages carry the rest — each says what was done and, more usefully, what the instruction got wrong.


What lands

A verbatim-prefix subtree import at packages/eql/ (1844 commits, 975 files, ~13 MiB packed). No path rewriting: sync-generated.mjs, the 887-line mise.toml, the 47 scripts under tasks/, Doxyfile, docker-compose.yml and every workflow path filter are repo-root-relative, so a verbatim prefix keeps them resolving with a working-directory change and nothing else. The npm package therefore lands at packages/eql/packages/eql, on the packages/protect-ffi/platforms/* precedent.

eql-bindings now resolves by path, not from crates.io. This is the reason for the import. packages/protect-ffi pinned =3.0.2 while the EQL tree carried 3.0.4 — the emitting Rust on a different release from the storing SQL, with nothing asserting they agree. Today that skew is benign, which is exactly why now: 3.0.2, 3.0.4 and the tree are byte-identical across src/, bindings/ and schema/ (diff -rq); what 3.0.3 and 3.0.4 changed was SQL. The flip lands with zero behaviour change, and from here the skew is unrepresentable.

Three EQL workflows now run from the root. They arrived under packages/eql/.github/, a directory GitHub never reads — the same defect the protect-ffi absorption kept finding, where a check arrives as a file and executes on no event. test-eql.yml (the SQLx matrix), bench-eql.yml and macro-expand-eql.yml are ported, path-filtered to packages/eql/**, credentialed to this repo's vars/secrets split, and fork-PR-skipped.

Review order

The diff is 985 files, and almost all of it is the import. Suggested order:

  1. docs/plans/2026-08-13-eql-monorepo-absorption.md — the plan, with results recorded inline.
  2. 8fda740c — the workspace wiring. The load-bearing commit.
  3. 3830f066, e614cf72, 7f937286 — the CI port and the guard that keeps it honest.
  4. 7a9604c6, e275b093 — the path-dep flip and its linter.
  5. e54aa5b6 — the import itself. Verified faithful (975 in, 975 out) before anything else touched it; nothing to read line by line.

57dd5bf6 is a pure Biome 1.8.3 → 2.5.2 reflow, committed separately so it never confounds a semantic diff. It is ordered after f29fdfb0 deliberately: the imported tree carries 211 generated .ts files that Biome would rewrite into a shape the ts-rs generator never emits, which breaks mise run types:check until reverted.

Things worth a reviewer's attention

Root pnpm test now uses --filter './packages/**'. Turbo's one-level glob selected packages/eql and not packages/eql/packages/eql, so @cipherstash/eql's own Vitest suite would have run nowhere. build was already covered by ^build traversal; test was not. Verified safe because the six platforms/* manifests declare no scripts block. Mutation-checked both directions.

The lockstep version hook moved to the root. scripts/sync-lockstep-versions.mjs is what writes the computed npm version into Cargo.toml, and Changesets only runs the root version script. release.yml now passes version: pnpm run version — without that line the hook is present, plausible, and dead. Asserted by scripts/__tests__/release-version-hook.test.mjs.

The private @cipherstash/eql-workspace root manifest is deleted. packages/* already matches packages/eql, so leaving it enrols a second workspace root. Its ten scripts sort into four groups and only version survives the move; the subtree needs no node_modules for its mise tasks at all.

Four new guards, each mutation-checked. In this repo's house style — discovery over the tree rather than a hand-maintained list, with a floor so a scan that matches nothing fails instead of passing, and exemptions that carry a written reason and go stale loudly:

  • scripts/__tests__/eql-suite-ci.test.mjs — every mise task that invokes cargo must be reached by a root workflow, through depends or a tasks/*.sh hop. It also holds the un-ported packages/eql/.github/ deposit as an equality, so the final deletion in Phase 5 is forced rather than remembered.
  • scripts/lint-no-eql-registry-pins.mjs — no manifest may name eql-bindings or @cipherstash/eql by registry version.
  • The Dependabot cargo entry for the EQL workspace. supply-chain.e2e.test.ts asserts coverage per ecosystem, and cargo was already covered by protect-ffi — so packages/eql/Cargo.lock read as monitored while nothing proposed updates for it.
  • Two existing guards were extended rather than exempted: workflow-dispatch-job-conditions.test.mjs (EQL's jobs && the fork guard with a relevance gate, so the single-spelling rule now applies to the clause) and ffi-binding-step-order.test.mjs (EQL's credentialed jobs encrypt through the cipherstash-client crate and never load index.node, so they take a BINDING_EXEMPT_JOBS entry with that reason — but not a pre-flight exemption; they run require-cs-secrets).

Verification

Run locally, on this branch before the main merge — the counts have moved since (the scripts suite is larger, and code:check's warning baseline shifted), so treat these as the record of the import's own verification rather than as current numbers. CI on the merged head is the live answer:

Check Result
cargo build -p protect-ffi (host) clean, 4m 51s
cargo build -p protect-ffi --target wasm32-unknown-unknown clean, 1m 18s
pnpm --filter @cipherstash/protect-ffi test:cargo 310 passed, 0 failed; cargo fmt --check clean
pnpm run test:scripts 509 passed / 36 files
pnpm run code:check 0 errors, 192 warnings (the pre-existing baseline)
pnpm install --frozen-lockfile clean
pnpm test under a PATH trap for cargo zero cargo invocations, 14 tasks executed

The wasm32 build was not in the plan's list and should have been: the EQL workspace never builds for that target, and merging the two Cargo workspaces was rejected partly on that ground. It is where a cross-workspace path dep would break first. It does not break.

What is NOT verified

The credentialed integration-tests suite has not run against the path dep. It needs Docker and CS_*, and it is the only path in the repo that carries an EQL payload through Postgres — so it is what turns "no-op" from an argument about byte-identity into an observed result. The plan's box is deliberately split in two so the runnable half being green cannot be mistaken for the whole thing. Do not merge without it.

The SQLx matrix's first run in this repo is also unproven by construction — that is what this PR being open is for.

Deliberately out of scope

  • Phase 5, the release cutover. Nothing here publishes EQL. Trusted publishing still names cipherstash/encrypt-query-language, and four of the ten imported workflows are the release machinery — so packages/eql/.github/ is kept until Phase 5 ports them, rather than reconstructing a publish pipeline from git history at the one moment nobody wants to be doing that.
  • Merging the two Cargo workspaces. The path dep is what buys the anti-skew guarantee; unification buys one target/ directory, and it would make EQL's dead_code = "deny" and default-members repo-wide policy as a side effect of a move. The lockfiles already agree on cipherstash-client (=0.42.0, matching checksums).
  • Absorbing packages/protect-ffi/integration-tests into the workspace. It moves @cipherstash/auth, vitest and the last @cipherstash/eql registry pin at once, and only a credentialed run can show that is neutral. It carries a named, reasoned exemption in the new linter until then.

Changesets

Two, and through the Stack fixed group they release the whole set: eql-3-0-5-migration (@cipherstash/stack-prisma minor) and supabase-skill-eql-305 (stash patch). An earlier revision of this description said "None"; that was true when the branch was opened and stopped being true with the 3.0.5 work.

@cipherstash/eql itself is still not published from here — that is Phase 5. What holds the line is not a changeset lint but scripts/release-gate.mjs, which exits non-zero if a frozen package's committed version is missing from npm, or if any published package carries a runtime workspace: range that only that package could satisfy. Either condition fails the gate job and skips release entirely. Run node scripts/release-gate.mjs for what it blocks at any given moment — that answer comes from the registry, not from this description. It exits 0 today.

The .changeset/*.md.deferred parking convention referenced in earlier revisions is gone. e77bfcec on main retired the guard and activated both parked FFI changesets in the same commit, and they released in @cipherstash/protect-ffi@0.32.0. EQL's own rename-ste-vec-contains was deleted deliberately in 9b1c44d9 rather than re-parked: the 3.0.5 bump was entered by hand in the CHANGELOG, so re-activating that major changeset would bump the package a second time for a rename already released.

tobyhede and others added 30 commits July 8, 2026 14:41
Collapse the release surface onto the unified release.yml (production from
main via changesets; alpha/prerelease from eql_v3 via a chore(release):
commit). Remove the parallel machinery and roll the remaining release
artifacts into the one workflow.

- Remove the legacy alpha coordinator: release-alpha.yml, release-typescript.yml,
  the .github/scripts helpers (derive-identity, release-alpha-resolve,
  release-alpha-pin-bindings, release-alpha-verify-commit-signature,
  release-typescript-workflow guard) and their tests, and the tasks/release/*
  wrappers (all/eql/bindings/rust/typescript/resolve-alpha/pin-bindings).
- Remove release-eql.yml: the old manual final-release path. release.yml now
  builds + attaches the SQL + docs release itself; a GITHUB_TOKEN-created
  release never triggered its on: release fan-out, and changesets manages the
  changelog (its verify-changelog gate is obsolete).
- Restore the Multitudes production-deploy notification into the automated
  path: re-gate the _build-sql.yml step to fire on release.yml's non-prerelease
  SQL release (the old event_name == 'release' gate never matched under push).
- Roll the Postgres+EQL image build into release.yml: a build-image job
  dispatches release-postgres-eql-image.yml on production finals
  (update_floating_tags=true). Drop its dormant on: release trigger and the
  dead release-event branch; workflow_dispatch stays for manual/alpha images.
- Harden .changeset/config.json access to "public" (was "restricted"; inert for
  @cipherstash/eql today via publishConfig, but a footgun for future packages).
- Fix crates/eql-bindings/README.md (v3 envelope is v: 3, not v: 2) and align
  the release docs (CLAUDE.md, releasing-an-alpha.md, workflows/README.md,
  README badge, CHANGELOG) plus the lint-release gate to the single workflow.
- Mark the superseded 2026-07-04 coordinator design snapshots as historical.
Add docs/development/releasing.md — a single reference covering the whole
release architecture (one release.yml, both modes, four artifacts, one lockstep
version): production via changesets on main, prerelease/alpha on eql_v3, the
lockstep versioning mechanism (changesets -> sync-lockstep-versions.mjs ->
Cargo + bundled SQL), tag namespaces, the GITHUB_TOKEN inline-vs-dispatch model,
the changesets pre-mode footgun, and verify/smoke-test steps.

Delete docs/development/releasing-an-alpha.md: it was alpha-only and had gone
stale after the release.yml consolidation (referenced the deleted
verify-changelog job, server-side N derivation, and the coordinator). Repoint
CLAUDE.md, .changeset/README.md, and the superseded design-snapshot banners at
the new doc.
…elease tag

- Notify Multitudes (_build-sql.yml): add continue-on-error. It is the last
  step of build-sql and uses `curl --fail-with-body`; on the production path
  npm + SQL have already published upstream, so a Multitudes outage / empty
  token must not fail build-sql and skip build-docs / build-image.
- build-image (release.yml): dispatch release-postgres-eql-image.yml against
  the eql-<version> tag instead of the branch. build-sql (a `needs`) already
  created that tag at the release commit, so pinning to it builds the image
  from the exact released source even if the branch advanced since publish.
Way 2 is now canonical: contributors add a changeset (`.changeset/*.md`); Changesets
owns both the version bump and CHANGELOG.md generation. Retire the manual
`## [Unreleased]` promotion everywhere the docs still described it.

- CLAUDE.md "Release & changelog discipline": add a changeset (not a CHANGELOG.md
  entry); the entry is the changeset body; the bump type is the changeset
  frontmatter; "Cutting a release" is now the Changesets Version-PR flow.
- CHANGELOG.md preamble: state the file is Changesets-generated, not hand-edited.
- DEVELOPMENT.md "Releasing": replace the manual GitHub-release flow (and the
  stale release-eql.yml reference) with the release.yml + Changesets model.
- releasing.md: changelog section rewritten to the Changesets model; fix the
  prerelease changelog note.
- docs/upgrading/v3.0.md: drop the stale `[Unreleased]` phrasing.
Migrate the hand-written CHANGELOG.md `## [Unreleased]` block (the pending 3.0.0
notes) into 33 Changesets files, so Changesets — now the canonical owner —
generates the release section. De-duplicated in the process:

- dropped 6 exact duplicates and 7 stale pre-rename duplicates (the scalar types
  were renamed int4->integer, int2->smallint, int8->bigint, bool->boolean,
  float4->real, float8->double; both old- and new-name entries were present).
- bump per entry by nature: 4 major (eql_v2 removed, envelope v:3, eql_v3_internal
  move, sole installer), 24 minor, 5 patch -> aggregate resolves to 3.0.0.
- full dense prose + PR links preserved verbatim in each changeset body.

Remove the `[Unreleased]` block and its compare-URL link reference from
CHANGELOG.md; historical `## [2.x.y]` sections are untouched. `changeset status`
confirms a major bump for @cipherstash/eql.
Rebased onto eql_v3, which added the query-operand domain surface
(query_<name> in eql_v3, CIP-3432/3442), COMMENT ON DOMAIN, and the
conditional SEM opclass install. Reconcile the branch-local work:

- Regenerate eql-bindings + @cipherstash/eql bindings from the updated
  catalog (adds the query_<T> schemas/types; jsonb_query -> query_jsonb).
- Convert base's four new CHANGELOG [Unreleased] entries (#377, #373,
  CIP-3442, #375) into changesets and remove the reconstructed
  [Unreleased] block (Changesets owns the changelog).
types:check / typescript:generate / typescript:check invoked
'pnpm --filter @cipherstash/eql sync:generated', but the contexts that run
them — test-eql.yml's Rust workspace crates job and the pre-commit hook —
provision mise tools only, so CI failed with 'pnpm: command not found'
(exit 127).

sync-generated.mjs uses node builtins exclusively; it needs neither pnpm's
workspace resolution nor node_modules. Invoke it with node directly and add
node to mise [tools] so every mise-run context has it. pnpm remains a
package-workflow concern (lint-release.yml and release.yml set it up
explicitly).
…terministic no-op, not a conflict to resolve
…ning, cache lint coverage

Addresses the remaining items from the changes-requested review
(cipherstash/encrypt-query-language#374 (review)):

- DEV-placeholder publish guards (finding 6): release-plz.yml refuses to
  publish when the committed crates/eql-bindings/sql bundle wasn't prepared
  for the crate's version (enforced only when the version isn't already on
  crates.io, so routine no-op runs still pass); the npm package gains a
  prepublishOnly gate (scripts/verify-release-assets.mjs) that covers BOTH
  publish paths — changeset publish (production) and npm-publish.mjs
  (prerelease) shell out to npm publish, which runs it.
- Prerelease marker (finding 7): classify accepts exactly 'chore(release):'
  (a bare 'release:' doc commit can no longer trigger a publish) and
  short-circuits to skip when the identity's eql-typescript-v tag already
  exists.
- One-identity-one-commit (finding 8): prerelease-publish-rust dispatches
  release-plz.yml against the eql-<V> tag (same pattern as build-image), not
  the branch, so the crate publishes from the exact release commit.
- Supply chain (findings 9/10): npm pinned to an exact version in both
  publish jobs; mise-action cache disabled in release.yml and release-plz.yml
  and the no-workflow-caching lint extended to cover jdx/mise-action (with
  release-plz.yml added to its default file set, plus test cases).
- Changelog ownership (finding 5): CHANGELOG.md header, CLAUDE.md, and
  releasing.md now say what is true — Changesets maintains
  packages/eql/CHANGELOG.md (the lockstep release changelog); the root file
  is the frozen pre-3.0 archive.
- Access-model note (finding 12): releasing.md documents that prerelease
  publishes are gated by eql_v3 push access + branch protection.
- Tests (coverage finding): bumpCargoPackageVersion extracted from
  sync-lockstep-versions.mjs, now section-anchored (can never rewrite a
  column-0 version line outside [package]) with vitest coverage;
  prepare-bindings-assets.test.sh covers the version-validation gate (wired
  into lint-release.yml).
- Nits: no defaults.run.shell 'bash {0}' override in release.yml /
  release-plz.yml (GitHub's default bash is -eo pipefail); unused repoRoot
  dropped from copy-assets.mjs.
feat(release): publish EQL v3 language bindings (Rust + TypeScript) with bundled SQL
Lockstep prerelease pin via 'pnpm run version' (changesets pre-mode):
package.json, Cargo.toml, and the bundled exact-version SQL assets all
carry 3.0.0-alpha.3. Pushing this marker commit to eql_v3 triggers
release.yml's prerelease path: SQL + docs GitHub release (eql-3.0.0-alpha.3),
npm publish (@cipherstash/eql, dist-tag alpha, tag eql-typescript-v3.0.0-alpha.3),
and the dispatched crate publish (eql-bindings-v3.0.0-alpha.3).
First attempt (b5c586f) failed on two jobs:
- prerelease-publish-rust: 'gh workflow run' resolves the workflow via a
  local git repo, and the job has no checkout — fixed with --repo.
- prerelease-publish-npm: ENEEDAUTH — npm Trusted Publishing is not yet
  configured for @cipherstash/eql + release.yml on npmjs.com (alpha.2 was
  published manually). Needs the npmjs-side publisher config; will fail
  again on this retry, which is fine: the eql-typescript tag is only
  created after a successful npm publish, so classify keeps allowing
  retries for this identity until npm lands.

The marker subject re-triggers the prerelease path for 3.0.0-alpha.3:
build-sql/docs update the existing eql-3.0.0-alpha.3 release in place
(action-gh-release updates by tag), and the fixed dispatch publishes the
crate from the eql-3.0.0-alpha.3 tag.
GitHub only registers (and allows workflow_dispatch of) workflows whose
file exists on the default branch; release-plz.yml lived only on eql_v3
and had never run, so it had no workflow ID and release.yml's crate
publish dispatch 404'd. workflow_dispatch executes the file at the
dispatched ref, so this stub only provides registration — the real
publish job runs from the eql-<version> tag release.yml dispatches
against. Replace with the full workflow when eql_v3 merges to main.
…orkflow

ci: register release-plz.yml on the default branch (dispatch stub)
…red)

npm and crates.io OIDC trusted publishers are now configured
(@cipherstash/eql -> release.yml; eql-bindings -> release-plz.yml) and
release-plz.yml is registered on the default branch (PR #383). This
marker retry completes the two package legs: the eql-typescript tag
doesn't exist yet so classify proceeds; SQL/docs update in place.
…ranch dispatch)

Second retry failed on both package legs for new reasons:

- npm ENEEDAUTH persisted despite trusted publishing being configured:
  mise-action runs after the 'upgrade npm' step and prepends mise's node
  (from [tools]) to PATH — its bundled npm 10.x has no OIDC trusted
  publishing, so the upgraded npm 11.5.1 never ran. The upgrade step now
  runs AFTER mise-action in both publish jobs.

- release-plz errored 'cannot determine current branch': dispatching
  against the eql-<V> tag gives it a detached HEAD, which it refuses.
  prerelease-publish-rust now pins a release/eql-<V> branch at the
  release commit and dispatches against that — same exact-commit
  guarantee, real branch for release-plz.

classify proceeds for 3.0.0-alpha.3 (the eql-typescript tag is still
uncreated); SQL/docs update in place.
npm publish and the crate publish both succeeded on the previous retry
(@cipherstash/eql@3.0.0-alpha.3 with provenance; eql-bindings@3.0.0-alpha.3
via crates.io trusted publishing). The only remaining failure was the
eql-typescript tag push: with persist-credentials: false the git
extraheader hack didn't authenticate ('could not read Username'). Create
the tag ref via 'gh api' instead.

This retry converges: npm publish skips (already published), the tag is
created, release-plz no-ops (version already on crates.io).
Until the 3.0.0 final ships, the alphas are the package's only release
line, so a bare 'npm install @cipherstash/eql' should resolve to the
newest alpha rather than whichever version last happened to hold latest
(alpha.2 had it from a manual publish while alpha.3 sat under the alpha
tag). PRE_GA_LATEST in npm-publish.mjs flips the policy back to channel
dist-tags after GA. Policy per James, 2026-07-08.
…warning

Doxygen auto-links the bare word `text` (it collides with a documented
symbol) into a cross-reference — even inside inline code. In the jsonb
selector operator's @warning, `` `col -> 'sel'::text` `` became
`<computeroutput>col -> 'sel'<ref>text</ref></computeroutput>`: a link
nested inside inline code. Rendered to Markdown that surfaces as an
unbalanced `<tt>`, which fails the downstream MDX/docs build
(cipherstash/docs was broken by eql-3.0.0-alpha.3's API.md at line 2153,
`Expected a closing tag for <tt>`). It also drops the `::`, so the cast
read as `'sel'text`.

Prefix the two `text` occurrences with Doxygen's `%` no-autolink marker
(`::%text`, `-> %text`). Verified with doxygen: the nested <ref> is gone
and `col -> 'sel'::text` now renders as clean, balanced inline code.
…arse

fix(docs): skip name-dropped CREATE FUNCTION mis-parses
…tolink

fix(docs): suppress Doxygen auto-linking of `text` in jsonb selector @warning
…talls (CIP-3468)

CREATE OPERATOR CLASS requires superuser, so on cloud-hosted Supabase and
most managed Postgres the installer has always attempted the ORE operator
class and skipped it on insufficient_privilege — but the ORE-carrying
domains still installed half-working: </> comparisons ran as unindexable
seq scans while CREATE INDEX ... (eql_v3.ord_term(col)) and bare ORDER BY
failed with opaque Postgres errors.

The installer now capability-detects the skip (checking pg_opclass after
the attempt) and poisons all 38 ORE-carrying domains — _ord/_ord_ore on
every ordered scalar, text_search, and their eql_v3.query_* twins — with
an always-raising CHECK constraint: the first value cast or inserted
(including NULL) raises feature_not_supported (SQLSTATE 0A000) naming the
domain and the platform-supported alternatives (_ord_ope for indexed
CLLW-OPE ordering, _eq for equality, text_match for pattern match).
Superuser installs are unchanged: the operator class is created and
nothing is poisoned.

The fallback (src/v3/scalars/ore_fallback.sql) is generated from the
catalog by eql-codegen via a new ore_fallback.sql.j2 template, so new
ORE-carrying families are covered by construction. The poison function
honours the encrypted-domain footguns: LANGUAGE plpgsql (never inlined,
RAISE cannot be planned away) and not STRICT (NULLs cannot slip through).
Integration tests run the shipped installer under SET ROLE to a
NOSUPERUSER role and derive the poisoned/functional split from the same
catalog.
…IP-3468)

Addresses PR #388 review feedback (tobyhede + internal review).

NOT VALID (blocking finding): ALTER DOMAIN ... ADD CONSTRAINT validates
existing stored data, and the poison raises unconditionally — so re-running
the installer over a database already holding ORE values (written under an
earlier superuser install, before the installing role was demoted) aborted
inside the DO block, for exactly the users the fallback exists to help. For
domains, NOT VALID does not weaken enforcement (coercion applies every
constraint regardless of validation status, NULL included); it skips the
existing-data scan. Pinned by a codegen unit test and a new integration
test (reinstall_over_ore_data) that runs the shipped installer as a
superuser role, stores an ORE row, demotes the role, re-installs — old row
stays readable, new writes raise 0A000 — and re-installs once more to pin
non-superuser re-install idempotency.

Also from review:
- closing RAISE NOTICE no longer advertises "_match domains" generically
  (only text has one); count field dropped for entries|length
- domain name now sql_str-escaped in the poison CHECK literal, matching
  the alternatives string
- U-003: NOT VALID semantics, the text_search split collateral on
  superuser->managed migrations (equality/match lost too, re-encryption
  required), and a scope caveat that the SteVec ore_cllw path keeps silent
  degradation (follow-up CIP-3471)
- payload examples use the wire-contract integer "v":3 (the bindings
  reject a JSON string; only the domain CHECK's ->> coercion accepted it)
…l-ore-types-on-cloud-hosted-supabase

feat(install): disable ORE-backed domains loudly on non-superuser installs (CIP-3468)
Public-schema encrypted-domain type names now carry an eql_v3_ version
prefix (public.eql_v3_integer, public.eql_v3_text_eq, public.eql_v3_json,
...). The prefix stops EQL domains shadowing PostgreSQL built-in type
names and gives each EQL version a distinct column-type namespace so
multiple versions can coexist at runtime (future version migrations).
Query-operand domains (eql_v3.query_*) are unchanged — their schema
already versions them.

The rule lives in the catalog (eql_domains::PUBLIC_TYPNAME_PREFIX,
Domain::sql_typname); codegen renders SQL + bindings through it, the
hand-written jsonb (SteVec) surface and bindings are updated to match,
and every generated surface (src/v3/scalars, eql-bindings Rust/TS/JSON
schema, packages/eql) is regenerated — never hand-edited. dump-catalog
now emits the installed typname so the docs manifest derives names from
the same source.
SteVec entries now carry hm XOR op; entry ordering extracts
eql_v3.ord_ope_term(entry) — the eql_v3_internal.ope_cllw bytea domain the
scalar _ord_ope surface already uses, ordering under the DEFAULT btree
opclass with a fully inlinable comparison chain. Removes the
eql_v3_internal.ore_cllw composite, its per-byte comparator, six operators,
and the superuser-only operator class — the last EQL surface that could not
index on cloud-hosted Supabase / managed Postgres.

- eql-bindings: SteVecTerm::OpeCllw{op} replaces OreCllw{oc}; from_v2
  rejects oc sv entries (UnconvertibleOreTerm) — ORE bytes misorder under
  byte comparison, so re-encryption is the only conversion
- fixtures: SteVec generation pinned to SteVecMode::Compat with a
  fail-closed oc->op remap shim until cipherstash-client emits sv-level op
- upgrade note U-003; changeset (major, folds into 3.0.0 pre-mode)

CIP-3469
…IP-3472)

Sweep every non-generated consumer of the public domain names onto the
prefixed spellings: the SQLx suite (matrix/property/ope/fixture sources,
the clean-install and splinter scripts, the regenerated public-surface
golden), the doc tree (README, SUPABASE, reference/tutorial pages), and
the v3.0 upgrade guide (new U-003 note with the ALTER COLUMN recipe;
historical pre-release names in U-002 kept unprefixed).

Two places derived the eql_v3.query_<name> twin from the storage name
and would have produced query_eql_v3_*: property.rs::query_cast and the
ope fixture macro now strip PUBLIC_TYPNAME_PREFIX first — the twin joins
query_ to the bare catalog name (the eql_v3 schema already versions
query operands, so they carry no prefix).
tobyhede added a commit that referenced this pull request Aug 20, 2026
Audit of the #885 review (2026-08-14) and the #915 review (2026-08-19) against
the current tree, plus fixes for what was still open. Most of both was already
addressed by 4098a1f..00d685f and the uncommitted work this commit also
carries; what follows is only what was not.

STALE — no longer true, verified rather than assumed:

  * "eql_v3.ste_vec_contains is hard-removed (no alias)". Upstream restored the
    deprecated delegating aliases in 142f41d and 3.0.5 shipped with them; the
    subtree sync 88c21c9 brought that here and 2a140e8 re-emitted against it.
  * "protect-ffi's Cargo.lock is stale at 3.0.4". Both locks read 3.0.5, and
    cargo-lock-freshness.test.mjs holds them there.
  * All four blockers, which landed in ca2b0a7 / 4098a1f with guards.

REJECTED — the recommendation would make things worse:

  * "The re-emitted baseline keeps createdAt: 2026-07-14 — misleading
    provenance." createdAt is not provenance: it is the PRIMARY TIE-BREAK KEY in
    the migrator's neighbour ordering (createdAt -> to -> migrationHash), so a
    re-emit of the same logical migration must not shift its position in the
    graph. It is also hashed, so moving it would have surfaced as "re-pin this
    digest" and slipped through. Pinned explicitly in migration-v3.test.ts, with
    the reason written into the migration header.
  * "path filters miss packages/eql/docker/**". Its only CI consumer is
    test:docs_v3_grep reading docker/README.md, in the docs-static job — which
    carries no relevance gate and so runs on every pull request already.

FIXED:

  * 7a9604c moved eql-bindings from crates.io =3.0.2 to an in-tree path dep at
    3.0.5 — a change to what compiles into the published binding — and shipped
    with no changeset. Added, as a patch: the crate's Rust is byte-identical
    across 3.0.2/3.0.4/3.0.5 (only its bundled SQL moved), so emitted payloads
    do not change; what changes is that the emitter and the SQL are now the same
    commit and the skew is unrepresentable.
  * test-eql.yml's `splinter` job compiled Rust uncached. It looks Rust-free —
    its task is bash over psql — but its build step runs `mise run build`, and
    tasks/build.sh shells out to `cargo run -q -p eql-codegen` twice. New
    invariant in eql-suite-ci.test.mjs: every job invoking a mise task that
    REACHES cargo must restore the shared cache. It closes over the task graph
    rather than grepping the workflow, because the string `cargo` appears
    nowhere in it — and it found exactly this one job.
  * Two `.changeset/*.md.deferred` files were still sitting in this tree, and
    the obvious reading of them is wrong. The cutover did its job properly:
    e77bfce retired the guard AND renamed both files in one commit, and they
    released in @cipherstash/protect-ffi@0.32.0. This branch was cut before
    that, so it still carries them under the old suffix — where reactivating
    them would republish a shipped changelog entry and re-bump the package for a
    change two versions old. Deleted, not renamed.

    New guard scripts/__tests__/no-parked-changesets.test.mjs fails on the
    suffix either way and points the reader at the released CHANGELOG to decide
    which resolution is right. It also fails if the retired guard is reinstated
    beside it, since a half-retired convention is what produces a parked file.
    The retired guard, its self-test, its fixtures, the lint:ffi-changeset
    script and the tests.yml step are deleted here too, matching main.
  * packages/protect-ffi/README.md still taught the embargo and the .md.deferred
    parking, both over. It ships in the tarball, so the wrong instructions ship
    with it. AGENTS.md and the Phase-4 checklist corrected likewise.

Also carries the previously uncommitted work for #915's findings 1-4, 6 and 7:
the five workflows missing packages/eql/crates/** triggers, the exact runtime
pin (workspace:^ -> workspace:*) with frozen-publisher-runtime-pins.test.mjs,
the two skills corrections, the robust main-guard form with
script-main-guards.test.mjs, and macro-expand-eql.yml's rust-cache workspace.

Suites: 683 script tests, 363 stack-prisma, protect-ffi JS green; release-gate
exits 0; lint:eql-pins clean; `changeset status` resolves with the seven FFI
packages at patch.
@tobyhede
tobyhede requested a review from freshtonic August 20, 2026 00:56

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes — but narrowly, and with the path out fully mapped. Since my last review this branch has gained exactly one commit: the merge of main (8e9e10a8). All four blockers are still present at HEAD — I re-verified each mechanically rather than assuming:

  1. Version hook still unrunnable: .github/workflows/release.yml on this branch sets up pnpm/Node only — no mise, no Rust toolchain — while pnpm run version still chains to mise run … → cargo run -p eql-codegen.
  2. Release still armed with no mechanical guard: scripts/release-gate.mjs contains no workspace-range satisfiability check and nothing excludes the 3.0.5 @cipherstash/eql from changeset publish.
  3. Cache keys still blind to the path dep: both build-ffi-binding keys hash packages/protect-ffi/crates/** only.
  4. The PG matrix still has no trigger that fires in this repo.

Two things have changed around the PR that sharpen rather than soften this:

  • Blocker 2 is more live than when I filed it. The release train on main is no longer theoretical — #920/#921 re-armed it and Version Packages #922 has merged, so the machinery this PR would poison is actively firing. The first main push after this merges alone takes the changesets version branch into the blocker-1 ENOENT, and the armed-but-unpublishable @cipherstash/eql sits behind it.
  • A new finding from reading the stack: this branch's in-tree "3.0.5" is not the published 3.0.5. Upstream restored the deprecated ste_vec_contains aliases in the actual 3.0.5 release (142f41d8, flowing in via #915's subtree sync to 91ebf28d), and #915's 2a140e84 re-emits the stack-prisma artefacts against the published bytes for exactly that reason. Merging this branch alone ships a tree whose version claims a release it does not match — the precise class of skew this PR exists to kill.

None of this is a criticism of the work — the fixes exist and target the right things. 4098a1f2 (hook runnable + refuse to half-publish) and ca2b0a75 (hash the compile input + a firing trigger) on #915 address blockers 1–4 head-on, the subtree sync brings in the alias restoration that resolves my should-fix, and fd8c40b1/3b38d885 pick up the guard-coverage should-fixes too. But they live one PR up the stack, #915 carries its own unresolved CHANGES_REQUESTED (2026-08-19, with commits since), and nothing mechanical prevents this PR merging without it. An approval here is a green light on exactly that merge.

Path to approval, which should be quick:

  1. Land #915 into this branch — as a merge commit: its 88c21c9b is itself a git subtree pull merge, so the no-squash rule this PR's own body states for main applies to that hop too.
  2. Re-request review. With the fixes in this PR's own diff, the re-review is verifying the four closures plus the 3.0.5 reconciliation — the structural work (import faithfulness, CI port, path-dep flip, guard engineering) was already reviewed and stands.

For the one thing that is new on this branch: the main merge itself looks right. The resolution strategy described in the body — regenerate the doubly-rewritten baseline and 3.0.5 edge rather than pick a side, leave the 3.0.2/3.0.4 historical edges alone because readVerifiedInstallSql() would bake current SQL into historical artefacts — is the correct reasoning, and I found no conflict markers or stray artefacts in it. (Note #915's 2a140e84 supersedes part of that re-emit anyway, for the published-3.0.5 reason above — one more argument for landing the stack as a unit.)

Brings #885's merge of main (8e9e10a) down into this branch so the stack can
land as a unit. Fourteen conflicts; none resolved by picking a side blindly,
and three needed the artefacts regenerated rather than chosen.

REGENERATED, NOT PICKED. Both sides re-emitted the baseline and the 3.0.5
upgrade edge, and both hashes were wrong for the merged tree: this branch's
bake the published 3.0.5 SQL under the pre-0.17 `sha256:`-prefixed hash, #885's
bake 3.0.4-era SQL under 0.17's bare one. The right answer is neither — it is
this branch's BYTES under #885's HASH FUNCTION. Re-ran each migration's own
self-emit (`tsx migrations/<dir>/migration.ts`) and re-pinned
`migration-v3.test.ts` to what came out:

  baseline           bad30c9b… (was sha256:23c98b03… / 9447442333…)
  3.0.5 upgrade edge 8c47bd1d… (was sha256:3b2b838b… / 4050dee89e…)

`ops.json` is unchanged by the re-emit in both: `installSqlSha256` stays
accde0030…, byte-identical to the published `@cipherstash/eql@3.0.5` tarball
(verified by `npm pack`), and `createdAt` is preserved — it is the primary
tie-break key in the migrator's neighbour ordering, not provenance. The 3.0.2
and 3.0.4 edges were NOT re-emitted and still bake 05860ae4… and 63104a81…,
their own releases: `readVerifiedInstallSql()` reads the CURRENTLY installed
bundle, so re-emitting a historical edge would bake today's SQL into it.

THE 0.17 SCOPE RENAME BROKE TWO FILES THAT NOTHING WOULD HAVE CAUGHT TOGETHER.
main's c795225 retired the whole `@prisma-next/*` scope for `@prisma/orm-*`.

  * `stale-vendored-space.test.ts` (added here in 810b069) imported six
    modules from it — a hard failure, so it announced itself. Ported, plus the
    two API changes behind it: `emitContractSpaceArtefacts` →
    `…Artifacts`, and `readMigrationsDir` now requires `{ migrationsDir }`.

    Its fixture then failed for a REAL reason worth keeping. It pinned the
    hash 1.0.0 published (`sha256:fc495f7f…`); under 0.17 the same bytes hash
    to `1030654387…` — which is exactly what `@cipherstash/stack-prisma@1.1.0`
    ships, so the fixture is re-pinned to a published artefact rather than a
    computed one. The stale-but-SILENT case is now 1.1.0's, and a new case
    pins 1.0.0's: its prefixed digest cannot equal a bare recomputation, so
    `readMigrationsDir` reports `hashMismatch` and that space IS caught. The
    suite covers both vendored generations; 16 tests, all green.

  * `skills/stash-prisma/SKILL.md` took a section from this branch that rides
    `rawSql` imported from `@prisma-next/postgres/migration` — two dead package
    names in the file whose whole job is telling a customer what to type, and
    nothing compiles a skill. 0.17 also replaced that recipe with PSL
    expression indexes, so the section was redundant as well as wrong. Kept
    only what main's version lacks — the EQL-upgrade-drops-your-indexes warning
    — rewritten against the surface that ships, and the changeset with it.

    New guard `scripts/__tests__/skills-retired-package-scopes.test.mjs`:
    no `skills/*/SKILL.md` may name a member of a retired scope. The scope
    WILDCARD stays legal, because `stash-prisma` has to be able to say the
    scope is retired — so the rule is `@prisma-next/<pkg>`, never
    `@prisma-next/*`. Verified against the defect: reinstate the line and it
    fails naming file, line and specifier.

CORRECTED WHILE RESOLVING, both verified against the registry rather than
reasoned about:

  * AGENTS.md and `packages/protect-ffi/README.md` both said no FFI release had
    run from this repository and to "treat the path as configured rather than
    proven". False since 0.32.0: its SLSA provenance names
    github.com/cipherstash/stack and .github/workflows/release.yml. Written
    while 0.31.0 was newest and never revisited. Both now carry the fact and
    the one-line curl that re-checks it.

  * The plan doc's Phase-4 note warned that #905 had to rename its parked
    changeset before merging. It merged as f7abfec without doing so, so
    `.changeset/protect-ffi-jsonwebtoken-cve.md.deferred` is on main now — a
    protect-ffi patch for CVE-2026-25537 that `@changesets/read` cannot see.
    Committed version and npm newest are both 0.32.0, so it is genuinely
    unreleased and the fix there is `git mv` back to `.md`. Recorded, not
    fixed: this branch predates #905 and does not carry the file.

Other resolutions: pnpm-workspace.yaml keeps this branch's four re-selected
security overrides AND main's `sharp@<0.35.0` (disjoint sets); the lockfile was
regenerated rather than hand-merged and `--frozen-lockfile` passes; the
`release:gate` step in tests.yml and the corrected `.md.deferred` paragraph in
AGENTS.md are this branch's; the 1.1.0 mention in the 3.0.5 changeset is
main's. Root package.json's `lint:package-paths` line lost the stray two-space
indent 2e98298 gave it — a `code:check` error on this branch as it stood.

Suites: 688 script tests, 364 stack-prisma (16 in the ported suite), 1344 CLI,
94 protect-ffi JS. `biome check` error-free, `release-gate` exits 0,
`lint:eql-pins` clean, `changeset status` resolves with the seven FFI packages
at patch and no bump for the frozen `@cipherstash/eql`.
tobyhede added a commit that referenced this pull request Aug 20, 2026
Brings #885's merge of main (8e9e10a) down into this branch so the stack can
land as a unit. Fourteen conflicts; none resolved by picking a side blindly,
and three needed the artefacts regenerated rather than chosen.

REGENERATED, NOT PICKED. Both sides re-emitted the baseline and the 3.0.5
upgrade edge, and both hashes were wrong for the merged tree: this branch's
bake the published 3.0.5 SQL under the pre-0.17 `sha256:`-prefixed hash, #885's
bake 3.0.4-era SQL under 0.17's bare one. The right answer is neither — it is
this branch's BYTES under #885's HASH FUNCTION. Re-ran each migration's own
self-emit (`tsx migrations/<dir>/migration.ts`) and re-pinned
`migration-v3.test.ts` to what came out:

  baseline           bad30c9b… (was sha256:23c98b03… / 9447442333…)
  3.0.5 upgrade edge 8c47bd1d… (was sha256:3b2b838b… / 4050dee89e…)

`ops.json` is unchanged by the re-emit in both: `installSqlSha256` stays
accde0030…, byte-identical to the published `@cipherstash/eql@3.0.5` tarball
(verified by `npm pack`), and `createdAt` is preserved — it is the primary
tie-break key in the migrator's neighbour ordering, not provenance. The 3.0.2
and 3.0.4 edges were NOT re-emitted and still bake 05860ae4… and 63104a81…,
their own releases: `readVerifiedInstallSql()` reads the CURRENTLY installed
bundle, so re-emitting a historical edge would bake today's SQL into it.

THE 0.17 SCOPE RENAME BROKE TWO FILES THAT NOTHING WOULD HAVE CAUGHT TOGETHER.
main's c795225 retired the whole `@prisma-next/*` scope for `@prisma/orm-*`.

  * `stale-vendored-space.test.ts` (added here in 810b069) imported six
    modules from it — a hard failure, so it announced itself. Ported, plus the
    two API changes behind it: `emitContractSpaceArtefacts` →
    `…Artifacts`, and `readMigrationsDir` now requires `{ migrationsDir }`.

    Its fixture then failed for a REAL reason worth keeping. It pinned the
    hash 1.0.0 published (`sha256:fc495f7f…`); under 0.17 the same bytes hash
    to `1030654387…` — which is exactly what `@cipherstash/stack-prisma@1.1.0`
    ships, so the fixture is re-pinned to a published artefact rather than a
    computed one. The stale-but-SILENT case is now 1.1.0's, and a new case
    pins 1.0.0's: its prefixed digest cannot equal a bare recomputation, so
    `readMigrationsDir` reports `hashMismatch` and that space IS caught. The
    suite covers both vendored generations; 16 tests, all green.

  * `skills/stash-prisma/SKILL.md` took a section from this branch that rides
    `rawSql` imported from `@prisma-next/postgres/migration` — two dead package
    names in the file whose whole job is telling a customer what to type, and
    nothing compiles a skill. 0.17 also replaced that recipe with PSL
    expression indexes, so the section was redundant as well as wrong. Kept
    only what main's version lacks — the EQL-upgrade-drops-your-indexes warning
    — rewritten against the surface that ships, and the changeset with it.

    New guard `scripts/__tests__/skills-retired-package-scopes.test.mjs`:
    no `skills/*/SKILL.md` may name a member of a retired scope. The scope
    WILDCARD stays legal, because `stash-prisma` has to be able to say the
    scope is retired — so the rule is `@prisma-next/<pkg>`, never
    `@prisma-next/*`. Verified against the defect: reinstate the line and it
    fails naming file, line and specifier.

CORRECTED WHILE RESOLVING, both verified against the registry rather than
reasoned about:

  * AGENTS.md and `packages/protect-ffi/README.md` both said no FFI release had
    run from this repository and to "treat the path as configured rather than
    proven". False since 0.32.0: its SLSA provenance names
    github.com/cipherstash/stack and .github/workflows/release.yml. Written
    while 0.31.0 was newest and never revisited. Both now carry the fact and
    the one-line curl that re-checks it.

  * The plan doc's Phase-4 note warned that #905 had to rename its parked
    changeset before merging. It merged as f7abfec without doing so, so
    `.changeset/protect-ffi-jsonwebtoken-cve.md.deferred` is on main now — a
    protect-ffi patch for CVE-2026-25537 that `@changesets/read` cannot see.
    Committed version and npm newest are both 0.32.0, so it is genuinely
    unreleased and the fix there is `git mv` back to `.md`. Recorded, not
    fixed: this branch predates #905 and does not carry the file.

Other resolutions: pnpm-workspace.yaml keeps this branch's four re-selected
security overrides AND main's `sharp@<0.35.0` (disjoint sets); the lockfile was
regenerated rather than hand-merged and `--frozen-lockfile` passes; the
`release:gate` step in tests.yml and the corrected `.md.deferred` paragraph in
AGENTS.md are this branch's; the 1.1.0 mention in the 3.0.5 changeset is
main's. Root package.json's `lint:package-paths` line lost the stray two-space
indent 2e98298 gave it — a `code:check` error on this branch as it stood.

Suites: 688 script tests, 364 stack-prisma (16 in the ported suite), 1344 CLI,
94 protect-ffi JS. `biome check` error-free, `release-gate` exits 0,
`lint:eql-pins` clean, `changeset status` resolves with the seven FFI packages
at patch and no bump for the frozen `@cipherstash/eql`.
The #885 review found, by reading a diff, that this branch's in-tree
`@cipherstash/eql@3.0.5` was not the published 3.0.5: the subtree's install
bundle hashed `7ad9c9f8…` while npm's carried `accde0030…`, because upstream had
restored the deprecated `ste_vec_contains` aliases in the actual release. The
subtree sync fixed the bytes. Nothing was added to stop it recurring, and it is
a whole class rather than one slip.

WHY NOTHING IN THE TREE COULD SEE IT. `sql/release-manifest.json` is regenerated
alongside the SQL, so it agrees with whatever was generated — every digest
verifies, `migration-v3.test.ts`'s lockstep check passes, `lint:eql-pins` is
clean. The manifest is the artefact's statement about itself, and self-consistency
is exactly what a skew preserves. Only the registry disagreed, and nothing was
asking it.

WHY IT ONLY APPLIES TO A FROZEN PACKAGE. For one this repo publishes, in-tree
bytes differing from npm is an unreleased change — the normal state of every
pull request, and blocking it would freeze the repository. For a frozen one it
is a contradiction: the version cannot be released from here, so the tree is not
PROPOSING those bytes, it is asserting they are already on npm under that
number. That makes the comparison meaningful for exactly the set
`FROZEN_PUBLISHERS` already names, and meaningless outside it.

The reach is not theoretical: `packages/cli`'s installer calls `readInstallSql()`
with no digest check, so `stash eql install` would have put `jsonb_document_contains`
into a customer database under a version whose published SQL defines
`ste_vec_contains` — two databases both truthfully reporting 3.0.5 and disagreeing
on the function set.

CHECK C, in `scripts/release-gate.mjs` beside the two it already runs. For each
frozen package it `npm pack`s the committed version and compares the published
release manifest's digest with the in-tree one. Silent in three cases, each
tested: a package this repo can publish; a version npm does not carry (that is
`publishBlockers`' `frozen-publisher`, and two blockers for one fact make the
remedy ambiguous); a private manifest. It THROWS for a frozen publisher with no
artefact declared — a check with nothing to compare reads exactly like a check
that passed, which is the shape `lint-no-eql-registry-pins.mjs` exits 2 on.
`FROZEN_ARTEFACT_DIGESTS` holds the artefact paths, keyed identically to
`FROZEN_PUBLISHERS` and asserted so by equality, so a frozen publisher cannot
arrive without one.

THE REMEDIATION HAD TO BE WRITTEN SEPARATELY, and this is the part worth
reading. `reportBlockers`' first way out is "publish the frozen package" — which
for a bytes skew is precisely wrong: the version IS on npm, that is how the skew
was detected. An operator following it would try to release something already
released. The skew branch is checked first, names both digests, and says the
tree is what moves.

Costs one `npm pack` per frozen package (today: one) on a job that already talks
to the registry. `npm` and `tar` rather than a gzip+tar reader written here —
both are on every runner image and npm was already this script's one external
command, while a hand-rolled parser would be parsing code inside the script whose
job is to be trusted. Extraction is scoped to the single manifest path.

Verified end to end against the live registry, not just the injected-reader unit
tests: rewind the in-tree digest to `7ad9c9f8…` — the value this branch actually
shipped — and the gate exits 1 naming both hashes. 695 script tests green; `biome
check` error-free; the gate exits 0 on the tree as it stands.
tobyhede added a commit that referenced this pull request Aug 20, 2026
The #885 review found, by reading a diff, that this branch's in-tree
`@cipherstash/eql@3.0.5` was not the published 3.0.5: the subtree's install
bundle hashed `7ad9c9f8…` while npm's carried `accde0030…`, because upstream had
restored the deprecated `ste_vec_contains` aliases in the actual release. The
subtree sync fixed the bytes. Nothing was added to stop it recurring, and it is
a whole class rather than one slip.

WHY NOTHING IN THE TREE COULD SEE IT. `sql/release-manifest.json` is regenerated
alongside the SQL, so it agrees with whatever was generated — every digest
verifies, `migration-v3.test.ts`'s lockstep check passes, `lint:eql-pins` is
clean. The manifest is the artefact's statement about itself, and self-consistency
is exactly what a skew preserves. Only the registry disagreed, and nothing was
asking it.

WHY IT ONLY APPLIES TO A FROZEN PACKAGE. For one this repo publishes, in-tree
bytes differing from npm is an unreleased change — the normal state of every
pull request, and blocking it would freeze the repository. For a frozen one it
is a contradiction: the version cannot be released from here, so the tree is not
PROPOSING those bytes, it is asserting they are already on npm under that
number. That makes the comparison meaningful for exactly the set
`FROZEN_PUBLISHERS` already names, and meaningless outside it.

The reach is not theoretical: `packages/cli`'s installer calls `readInstallSql()`
with no digest check, so `stash eql install` would have put `jsonb_document_contains`
into a customer database under a version whose published SQL defines
`ste_vec_contains` — two databases both truthfully reporting 3.0.5 and disagreeing
on the function set.

CHECK C, in `scripts/release-gate.mjs` beside the two it already runs. For each
frozen package it `npm pack`s the committed version and compares the published
release manifest's digest with the in-tree one. Silent in three cases, each
tested: a package this repo can publish; a version npm does not carry (that is
`publishBlockers`' `frozen-publisher`, and two blockers for one fact make the
remedy ambiguous); a private manifest. It THROWS for a frozen publisher with no
artefact declared — a check with nothing to compare reads exactly like a check
that passed, which is the shape `lint-no-eql-registry-pins.mjs` exits 2 on.
`FROZEN_ARTEFACT_DIGESTS` holds the artefact paths, keyed identically to
`FROZEN_PUBLISHERS` and asserted so by equality, so a frozen publisher cannot
arrive without one.

THE REMEDIATION HAD TO BE WRITTEN SEPARATELY, and this is the part worth
reading. `reportBlockers`' first way out is "publish the frozen package" — which
for a bytes skew is precisely wrong: the version IS on npm, that is how the skew
was detected. An operator following it would try to release something already
released. The skew branch is checked first, names both digests, and says the
tree is what moves.

Costs one `npm pack` per frozen package (today: one) on a job that already talks
to the registry. `npm` and `tar` rather than a gzip+tar reader written here —
both are on every runner image and npm was already this script's one external
command, while a hand-rolled parser would be parsing code inside the script whose
job is to be trusted. Extraction is scoped to the single manifest path.

Verified end to end against the live registry, not just the injected-reader unit
tests: rewind the in-tree digest to `7ad9c9f8…` — the value this branch actually
shipped — and the gate exits 1 naming both hashes. 695 script tests green; `biome
check` error-free; the gate exits 0 on the tree as it stands.
fix(eql): the four blockers from #885, and the upstream 3.0.5 sync
tobyhede added a commit that referenced this pull request Aug 20, 2026
fix(eql): the four blockers from #885, and the upstream 3.0.5 sync
`stash eql install` read the v3 bundle out of the resolved
`@cipherstash/eql` with a bare `readFileSync` and executed it. Nothing
checked that those bytes were the bundle the release attests to, even
though `installSqlSha256` sat unused in the manifest `verify.ts` already
imported. This branch shipped the failure it protects against: a tree
declaring 3.0.5 whose bundle hashed `7ad9c9f8…` against npm's
`accde0030…`, which would have written functions into a customer
database that the reported version does not define.

The check lives in the CLI, not in `packages/eql/.../sql.ts`, and that
is not a convenience. `@cipherstash/eql` is a frozen publisher and
`workspace:*` packs as the exact published version, so every consumer
resolves 3.0.5 FROM NPM — a check added in the subtree would read as
fixed and ship nothing until the Phase-5 cutover.

Verification happens before `createPgClient`, so a refusal means nothing
was opened and nothing was sent, rather than something was rolled back;
the test asserts `mockQuery` was never called. All three readers are
covered: the install, the SQL baked into emitted migrations, and the
expected-surface baseline `eql verify` compares against.
`@cipherstash/stack-prisma` has verified against this same digest since
its v3 migrations landed — its comment pointed at a CLI
`readV3InstallSql` that never existed, back when the CLI verified
nothing. That is corrected here too.
`docs-static` runs `mise run test:docs_v3_grep`, whose `ROOT_DOCS` names
`README.md`, `SUPABASE.md` and `docker/README.md`, and which then reads
`git ls-files -- docs`. None of those four paths were in any of the
three copies of the EQL filter list, so a push to main touching only
documentation started no EQL workflow at all. `pull_request` was never
affected — it applies no `paths:` filter and `docs-static` is
deliberately not relevance-gated.

The review named `docker/**`. The guard found four, which is the point
of writing it as a derivation rather than a list: it walks each `mise
run` out to the task it names — through `[tasks."…"]` tables, file tasks
under `tasks/`, `depends`, and nested `mise run` — and fails when the
push filter does not select a path those bodies name. Its first
extractor was rejected for getting the right answer wrongly, scoring
`docker compose up` as the `docker/` directory; directories are now
derived only from `git ls-files -- <dir>`. It reads paths that are
WRITTEN DOWN, and says so in its header: `postgres:up` picking up
`tests/docker-compose.yml` from its working directory, and
`doc-anchors.sh`'s `git ls-files '*.md'` glob, are still read by hand.

Also asserts the three `jdx/mise-action` inputs in `release.yml` that
were load-bearing and unasserted. `working_directory` — without it mise
reads no config and the first `mise run` dies on a trust error that
reads like a broken toolchain. `add_shims_to_path: false` — the shim
directory is prepended to PATH and `packages/eql/mise.toml` pins node
22, so the default shadows `actions/setup-node` and `changeset publish`
reaches npm 10.x, which fails OIDC with E404. `env: false` — the default
exports that file's `[env]` block into `GITHUB_ENV` for the job holding
the publishing credential. Each is scoped to jobs carrying `id-token:
write`, with a floor so the scan cannot go empty, and each was proven by
mutating the real file and watching exactly one test fail.
Three defects in the lockstep path, one of them in the check that was
supposed to catch the other two.

`cargo-lock-freshness.test.mjs` guarded the Cargo half of the bump.
Nothing tied the SQL assets to the package version — a grep for
`eqlVersion` across `scripts/` found only release-gate. That matters
because `mise run build --version X` does not treat `--version` as a
cache-key input, so on unchanged SQL it re-serves whatever version the
previous build stamped. `eql-sql-asset-freshness.test.mjs` now compares
the npm package's version against every release manifest, the generated
TS mirror, the crate manifest, and the `COMMENT ON SCHEMA eql_v3` stamp
inside the bundle. All of those are needed: the digests are recomputed
over whatever bytes were served, so a stale bundle verifies perfectly
and only the stamp records which build produced it. Mutation-proven
against a copy of the real 2.6 MB bundle restamped to 3.0.6 — every
digest verified in that tree, and `FROZEN_ARTEFACT_DIGESTS` would have
passed it. Deliberately not keyed to `FROZEN_PUBLISHERS`: that check
compares the tree against npm and is deleted at the cutover; this
compares the tree against itself, which is a property of a lockstep
release rather than of who publishes it.

Step 4 of the version hook rewrote all four release manifests and both
copies of the bundle on EVERY release, including when the version had
not moved, against an unpinned `rust = "latest"`. A release that never
touched EQL could pick up a new digest under an unchanged version, fire
`frozen-bytes-skew` on the next gate run, and block the whole release —
after `changeset version` had already rewritten every manifest in the
tree. It now runs only when `eqlLockstepSkew` disagrees, and re-checks
that the copy landed. A real bump always disagrees, so the skip is not
reachable by bumping. Pinning rust in the subtree's `mise.toml` was
rejected: a divergence in the file upstream edits most, affecting the
whole EQL CI surface, to close a hole the skip already closes.

And the `npm pack` + `tar` extraction underneath `frozenBytesSkew` was
not merely untested — it was broken for the case it documents. Every
unit test injected both digests and the process test shimmed `npm` to
answer `view` only, so `pack` returned a synthetic E404 and the
comparison was skipped on every run. Driving it for real: `--silent`
suppressed the error text the function classified on, and npm answers a
missing VERSION of an existing package with `ETARGET`, not `E404` — so
`@cipherstash/eql@<next>` became an uncaught throw with an empty reason,
raised while building the blocker array, which meant `reportBlockers`
never printed the `frozen-publisher` remedy it had already computed. Now
driven by a PATH-shimmed npm producing genuine tarballs, with real tar.
SECURITY.md's publishing note claimed `@cipherstash/eql`, `eql-bindings`
AND the seven `@cipherstash/protect-ffi*` packages still publish from
their old repositories. The protect-ffi half was false, and the registry
settles it: the SLSA provenance on `@cipherstash/protect-ffi@0.32.0`
names `cipherstash/stack` and `.github/workflows/release.yml`, while
`@cipherstash/eql@3.0.5` still names `encrypt-query-language`. It also
contradicted AGENTS.md and `FROZEN_PUBLISHERS`, whose only entry is eql
— three in-tree sources said the cutover happened and this one said it
had not. The Scope bullet had the mirror-image error, claiming to cover
"all npm packages published from this repository", which excludes the
one package here that is NOT published from here.

`frozen-publisher-docs.test.mjs` covered AGENTS.md and the plan but not
SECURITY.md, so this sentence would have survived the Phase-5 cutover
that makes its remaining half wrong too. It is now held to the map by an
extractor rather than a parallel regex — a sentence carrying both a
"published from" verb and a foreign `cipherstash/…` repo must name only
packages the map freezes. Emptying `FROZEN_PUBLISHERS` now fails five
tests, two of them SECURITY.md's, so the corrected text goes stale
loudly at the cutover instead of quietly.

`integration-tests/package.json` gains `private: true`. It is inert
today only because pnpm's globs exclude it, and the real protection is
changesets — `publicPackages` filters on that flag, while npm's own
EPRIVATE throw is workspace-gated. `integration-tests` is a name npm
already serves, so enrolment would 403 mid-release, after the six
platform packages had published.

Plan lines 285 and 356 are ticked against runs 31759054388 (6m33s, head
ea9e140) and 32316091388 (5m55s, head 8e9e10a), both green with
`Require CipherStash secrets` and the live suite among the succeeding
steps — so neither was a credential skip. Line 285 warned against
ticking it from byte-identity, and that warning is honoured: this is a
live run against the path dep. The scope note records what it does not
prove — the suite installs EQL SQL from its own lockfile's 3.0.2 pin
while the Rust under test is the subtree's, so it covers a pairing, not
every pairing.
The cargo-task scan inlined one hop of `tasks/*.sh`, and depth-2
delegation already exists in this tree — `build.sh` sources
`build/ordering.sh` and shells to two `verify_*.sh`. A cargo helper
reached only at depth 2 dropped out of `CARGO_TASKS` entirely: no orphan
reported, no exemption demanded, and the job running it stopped counting
as a Rust job for the cache check. That is the silent direction. It is
now transitive, cycle-guarded, and throws with the unread frontier
rather than truncating past `MAX_SCRIPT_DEPTH`. Asserting "nothing cargo
below depth 1" was the cheaper option and was rejected: it would need an
allowlist of helpers permitted to be deep, and the next one lands with a
cargo line and gets added to it.

The same scan read `step.run` only, so a job reaching cargo through a
composite action read as carrying no Rust and its missing rust-cache was
never reported — also silent. Composite actions a workflow reaches by
`uses: ./…` are now read, transitively. `grep` finds no `mise run` under
`.github/actions/` today; the fixtures are what prove the closure,
because the fact being closed is about the future.

`lint-no-eql-registry-pins.mjs` never read the Cargo redirect tables. A
`[patch.crates-io]` re-pointing `eql-bindings` was not flagged, nor
`[replace]`, nor the dotted `[patch.crates-io.eql-bindings]` form — and
nor `.cargo/config.toml`, which cargo honours a `[patch]` from and which
a walk keyed on manifest filenames never opens. That last one is the
`pnpm-workspace.yaml` failure one ecosystem along: the quietest place to
re-point a dependency is the file the linter was not reading. Still not
read, and recorded rather than left to be rediscovered: `[source.*]
replace-with`, which redirects the whole registry rather than naming a
crate, so there is no declaration to classify.

`test:rust` gains `--locked`. Nothing in this repo passed it anywhere,
and the bill came due through the lockstep bump: `packages/protect-ffi`
depends on `eql-bindings` by path, its `Cargo.lock` records the version,
and nothing updated it — so `cargo metadata --locked` exited 101 while
every cargo command in CI regenerated the lock in memory, built against
the regenerated one, and threw it away with the runner. Nothing went
red. It is on the check and deliberately not on the builds: a
contributor who has just edited `Cargo.toml` regenerates the lock on
their next build, legitimately, and `--locked` there is a failure at the
end of a compile. `lintWiring` holds it, with the one exemption
(`cargo fmt`, which forwards the flag to rustfmt) expiring if it stops
applying.
… not

Six edits, all describing mechanisms this branch added or corrected.

The "one version, five artefacts" bullet ended on the hazard and never
said whether anything checked the result. It now names
`eql-sql-asset-freshness.test.mjs` and says why the stamp check is the
load-bearing one — the digests are recomputed over whatever bytes were
served, so a stale bundle verifies perfectly.

A new bullet on the version hook declining to regenerate a frozen
artefact on a release that does not bump EQL, including why declining is
the correct answer for in-tree SQL changed without a changeset, and why
pinning rust upstream is the right place for the remaining half.

The `FROZEN_ARTEFACT_DIGESTS` paragraph gains the two defects that
turned up when its `npm pack` path was finally executed — `--silent`
hiding the text it classified on, and `ETARGET` rather than `E404` for a
missing version.

The `.github` deposit bullet records that the cargo-task scan is now
transitive and reads composite actions, and names both failure
directions. A new bullet explains that the EQL path filters are three
copies of one list with no YAML anchors to hold them together, and that
the list is now derived — plus the two things the derivation cannot see.

The `test:cargo` bullet carries `--locked` and the reason it is on the
check and not the builds. The linter bullet gains the Cargo redirect
tables and `.cargo/config.toml`, and records the two things left open:
`[source.*] replace-with`, and `packages/eql`'s own cargo tasks, which
still pass no `--locked` and are the workspace whose lock the bump
actually moves.
One conflict, in the file this PR exists to change:
`packages/protect-ffi/crates/protect-ffi/Cargo.toml`. The two sides were
adjacent but did not overlap in intent — main bumped
`cipherstash-client`, `cts-common`, `stack-auth` and `stack-profile`
from `=0.42.0` to `=0.42.2` and left `eql-bindings` alone; this branch
replaced `eql-bindings = { version = "=3.0.2" }` with the in-tree path
dep and left the `0.42.x` pins alone. Resolved by taking both. The
lockfile auto-merged correctly (`cipherstash-client 0.42.2` from the
registry, `eql-bindings 3.0.5` with no `source` line) and `cargo
metadata --locked` exits 0.

The two Cargo workspaces now disagree on `cipherstash-client` —
`packages/eql/Cargo.lock` still records 0.42.0 — which is fine and worth
writing down rather than rediscovering: `eql-bindings` depends only on
`serde`, `serde_json`, `ts-rs` and `schemars`, so it never sees that
crate. The PR description's "the lockfiles already agree on `=0.42.0`"
is now stale.

The merge also surfaced a defect on main that main cannot see.
`.changeset/protect-ffi-jsonwebtoken-cve.md.deferred` was parked by
`415ab2d4` (18 Aug) citing `lint:ffi-changeset`, on the premise that
protect-ffi still published from `cipherstash/protectjs-ffi`.
`e77bfcec` (19 Aug) retired that guard and activated the files parked at
the time — but this one had been parked the day before and was missed,
and the guard that would have caught it was deleted in the same commit.
`@changesets/read` does not select the `.md.deferred` suffix, so the
CVE-2026-25537 fix has been invisible ever since: no other protect-ffi
changeset exists, so the bump would simply not happen.

It is genuinely unreleased — npm's newest is 0.32.0, the tree says
0.32.0, and no released CHANGELOG entry mentions 0.42.2, jsonwebtoken or
the advisory — so per the procedure in AGENTS.md it is renamed back to
`.md` rather than deleted. Blast radius is fixed group 1: the seven FFI
packages. It does not reach `stash` or `@cipherstash/stack`, which are
group 0 — `415ab2d4`'s "would bump all seven through the fixed group and
misattribute the next stack release" was right about the seven and wrong
about the stack release.

`scripts/__tests__/no-parked-changesets.test.mjs`, which does not exist
on main, is what caught it.
@coderdan

Copy link
Copy Markdown
Contributor

GPT-5.6-sol reviewed these changes and found no actionable correctness regressions.

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes — but this time on a single test-portability defect, not on the architecture. The path out of my last review has been walked: #915 landed into this branch as a genuine two-parent merge (71c87cf3, honouring the no-squash rule for the subtree hop), six follow-up commits closed the second review round, and main was merged cleanly (ca3d6a70 — no conflict markers, and the resolution includes the regenerated artefacts story). I re-verified all four blockers mechanically at HEAD:

  1. Version hook — closed. release.yml's version job now installs mise (SHA-pinned jdx/mise-action, working_directory: packages/eql) which also provides the Rust toolchain via [tools], so pnpm run version → sync-lockstep-versions.mjs → mise run → cargo run -p eql-codegen resolves. The three load-bearing inputs (cache: false, add_shims_to_path: false, env: false) each carry a written reason, and workflow-mise-setup.test.mjs asserts them.
  2. Release gate — closed, and deeper than asked. release-gate.mjs now carries FROZEN_PUBLISHERS (with @cipherstash/eql and a delete-at-cutover warning citing the protect-ffi precedent), the packed-range satisfiability check (packedRange, with the workspace:*-packs-exact insight stated), and FROZEN_ARTEFACT_DIGESTS comparing the tree's install-SQL digest against the published tarball's own manifest. I ran node scripts/release-gate.mjs at HEAD: exit 0.
  3. Cache keys — closed. Both build-ffi-binding keys (native and wasm) now hash packages/eql/crates/** and packages/eql/Cargo.toml.
  4. PG matrix — closed and observed. test-eql.yml runs on pull_request with no path filter (per-job relevance via the changes job), plus push and schedule, and all four PG17 shards are green on this head.

The 3.0.5 skew is reconciled. The in-tree release-manifest.json now reads installSqlSha256: accde0030… — I pulled the real @cipherstash/eql@3.0.5 tarball from npm and its manifest matches byte-for-byte; the restored ste_vec_contains aliases are in the in-tree bundle. And ac93104f goes further than the review asked: stash eql install now verifies the bundle digest before opening a connection, with the changeset and stash-cli skill updated in the same commit.

The "Do not merge without it" condition from the PR body is met: the credentialed protect-ffi integration suite passed on this head (6m20s), which is the one path that carries an EQL payload through Postgres against the path dep.

The blocker: CI is red on a tar fixture that only works on macOS

Three jobs fail — Run Tests (Node 22), Run Tests (Node 24), Verify no caching in release workflows — all on the same five tests in scripts/__tests__/release-gate.test.mjs ("publishedArtefactDigest, against a real tarball" and "blocks the release when npm's bytes are not the tree's bytes").

Root cause, verified: the fixture shim packs with

tar -czf fixture.tgz -C stage .

so every member is archived as ./package/dist/sql/release-manifest.json — with a leading ./. Production then extracts the exact member package/dist/sql/release-manifest.json. macOS bsdtar matches the two spellings; the runner's GNU tar does not, so extraction fails with Not found in archive and the success-path tests (plus the two that assert specific error messages) all miss. This is why the suite was green when authored locally and red in CI.

Production is unaffected — I verified the real npm tarball's members carry no ./ prefix (npm pack @cipherstash/eql@3.0.5package/dist/sql/release-manifest.json), so the gate itself works against the registry; only the fixture diverges from what npm actually produces. Which is itself the fix-shaped observation: the fixture should produce what npm produces. Change the shim's pack line to anchor at the directory rather than .:

tar -czf fixture.tgz -C stage package

(or --transform/member-list equivalents). One line, and the fixture stops asserting bsdtar semantics.

Worth a thought while you're in there: the fast local node scripts/release-gate.mjs run exits on "nothing to publish" before reaching CHECK C — correct behaviour (a frozen package at a published version has nothing to compare), but it means the digest check only ever executes in anger. The five tests being fixed here are the only rehearsal it gets; that's the right trade, just worth confirming it's the intended one.

Everything else from this round is good: 62186019's eql-sql-asset-freshness guard (the version-stamp-vs-digest distinction is exactly right — digests recomputed over stale bytes verify perfectly, only the stamp records provenance), dcb09e24's transitive task scan closing the depth-2 and composite-action blind spots, and 554025d9 finding four missed filter inputs where the review named one.

With the tar fixture fixed and CI green, this is an approval — nothing else stands.

publishedArtefactDigest's test fixture archived the staging dir with
`tar -czf ... -C stage .`, which GNU tar (the CI runner) writes as
`./package/...` members. The production extraction call matches an
exact member path and does not strip that prefix the way bsdtar
(macOS) does, so two tests failed on Linux CI while passing locally.
Archiving the staged entries by name instead of `.` avoids the
prefix on both tar implementations.
…U tar

Same defect as the last fix, in the `fakeRegistry` shim used by the
end-to-end `runGate` tests: it archived its staged tarball with
`tar -czf ... -C stage .`, which GNU tar (the CI runner) writes as
`./package/...` members that the production extraction call — an
exact member-path match — does not find, unlike bsdtar (macOS).
Archiving the staged entries by name instead of `.` fixes it on both.
@freshtonic
freshtonic self-requested a review August 21, 2026 01:13
tobyhede added a commit that referenced this pull request Aug 21, 2026
fix(eql): the four blockers from #885, and the upstream 3.0.5 sync
@tobyhede
tobyhede merged commit 33fbda4 into main Aug 21, 2026
41 checks passed
@tobyhede
tobyhede deleted the feat/eql-1-vendor branch August 21, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants