Skip to content

chore: release candidate for EQL v4 and Stack Encrypt Rust and Go support - #1229

Open
coderdan wants to merge 958 commits into
mainfrom
next
Open

coderdan wants to merge 958 commits into
mainfrom
next

Conversation

@coderdan

Copy link
Copy Markdown
Contributor

Summary

This is the integration branch for a release candidate of EQL v4 and of Stack Encrypt's Rust and Go support. EQL is the SQL that lets PostgreSQL store and search encrypted values; EQL v4 is a second copy of it for values written by Stack Encrypt, the Rust encryption engine behind the Go SDK. The open pull requests for both are merged here first, so they can be tested together before any of them reaches main.

The release version is not decided yet.

Why a branch, not main

Some of this work depends on vitaminc and ore.rs code that is not yet on crates.io, so it pins those crates to a git commit. stack-encrypt and eql-bindings are published crates, and cargo publish drops those git pins, so nothing on this branch can be released until vitaminc 0.6 is published (cipherstash/vitaminc#389, which waits on cipherstash/vitaminc#463). Once it is, the git pins become crates.io versions and this branch merges to main as a normal release.

What lands here, in order

  1. EQL v4: docs(adr): one value model, three encodings in vitaminc, EQL v4 by producer #1139, refactor(eql): write the SQL source with schema and version placeholders #1147, feat(eql): build the eql_v4 bundle beside eql_v3 #1161, feat(eql-bindings): generate EQL v4 bindings and derive Stack Encrypt traits on v4 only #1162, feat(eql-bindings): produce the unordered EQL v4 number, boolean and text types #1163, feat(eql-bindings): produce EQL v4 TextMatch from Stack Encrypt match terms #1164, test(golang): prove the EQL v4 types from Go, against Rust and PostgreSQL #1165. Uses vitaminc 0.5.1 from crates.io only.
  2. Stack Encrypt operators: docs(adr): ADR-0009, operators select the index and a query term is not the stored term #1177, feat(stack-encrypt)!: a query names an operator, not an index #1190, feat(stack-encrypt): verify a match query's rows after decrypt #1191, feat(stack-encrypt): size a match query's fetch with target_limit #1192, feat(stack-encrypt): store short TextMatch text as an empty filter, refuse only short queries #1206, feat(stack-encrypt): ordering operators Lt, Lte, Gt, Gte and Between, and remove query_index #1208, feat(stack-encrypt): JSON operators Contains, ContainedBy and AtPath #1209, feat(stack-encrypt)!: move the stack crates, the guests and the Go module onto vitaminc ADR 0001 #1213, feat(stack-encrypt)!: derive terms through vitaminc-normalize and walk dynamic values as Plaintext #1214, feat(stack-encrypt): query a plan field by operator through the guest, with verify and target_limit as exports #1215, feat(golang)!: generate one query method per operator, with Verify and TargetLimit on a matches query #1216. Before it lands, feat(stack-encrypt)!: move the stack crates, the guests and the Go module onto vitaminc ADR 0001 #1213 moves its vitaminc pin from the deleted branch feat/aead-remove-legacy-traits to one commit on vitaminc main, and takes vitaminc's breaking changes since then.
  3. Not planned yet: the KMS stack (feat(stack-kms): ZeroKMS as a vitaminc key provider (CIP-3965) #1084, feat(stack-encrypt)!: keysets resolve through a KeysetRegistry, and sealed leaves move to format 2 (CIP-4140) #1085, feat(stack-encrypt): run vendor KMS backends through a static keyset registry (CIP-3965) #1083). It is built on a vitaminc commit that is no longer on any branch, and it needs vitaminc's KMS work rebased first.

Rules for this branch

  • Pin vitaminc and ore.rs with rev, never branch. A branch is deleted when its stack merges.
  • Use one vitaminc commit in every Cargo workspace that pins one: the root, both Go guests, packages/eql and packages/stack-encrypt/fuzz. Two commits mean two copies of vitaminc's traits, and the code stops compiling.

Changes

  • One empty commit, so the branch has a pull request. The stacks above add the content.

Verification

  • next was created from main at 2a70a47. Nothing has been built yet, because nothing has changed.

Related

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Warning

Review details and warnings were omitted to fit the comment limit.

@changeset-bot

changeset-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c8a9c80

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 27 packages
Name Type
@cipherstash/protect-ffi Patch
@cipherstash/auth Patch
@cipherstash/stack-prisma Minor
@cipherstash/eql Minor
stash Minor
@cipherstash/stack-drizzle Minor
@cipherstash/stack-supabase Minor
@cipherstash/stack Minor
@cipherstash/ffi-integration-tests Patch
@cipherstash/wizard Minor
@cipherstash/prisma-example Patch
@cipherstash/bench Patch
@cipherstash/test-kit Patch
@cipherstash/basic-example Patch
@cipherstash/e2e Patch
@cipherstash/protect-ffi-darwin-arm64 Patch
@cipherstash/protect-ffi-darwin-x64 Patch
@cipherstash/protect-ffi-linux-arm64-gnu Patch
@cipherstash/protect-ffi-linux-x64-gnu Patch
@cipherstash/protect-ffi-linux-x64-musl Patch
@cipherstash/protect-ffi-win32-x64-msvc Patch
@cipherstash/auth-darwin-arm64 Patch
@cipherstash/auth-darwin-x64 Patch
@cipherstash/auth-linux-arm64-gnu Patch
@cipherstash/auth-linux-x64-gnu Patch
@cipherstash/auth-linux-x64-musl Patch
@cipherstash/auth-win32-x64-msvc Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught missed unviable timeout
436 0 203 0

cargo-mutants exited with no surviving mutants recorded. The run may not have completed cleanly; check the workflow run logs.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T13:51:48.267360Z a05affa Draft marked ready
🔒 Security Review ✅ Completed 2026-10-19T13:50:52.546472Z a05affa Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a05affaaa5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread languages/golang/encrypt/query.go Resolved
Comment thread languages/golang/encrypt/query.go Resolved
@coderdan
coderdan deployed to release-golang October 19, 2026 15:26 — with GitHub Actions Active
@coderdan
coderdan added this pull request to stack #1333 October 22, 2026 10:15

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I request changes for two release problems on the crates.io side. The code itself is in good shape, and every check I ran passes.

How I reviewed this
The diff is large because generated code is committed. I reviewed the generators and their templates, then I ran the drift checks to prove that the committed output matches them. I did not read each generated file.

  • EQL codegen: eql-codegen renders the version-neutral SQL source two times (eql_v3 and eql_v4) from the eql-domains catalogue. It also generates the Rust payload types and eql_gen.go.
  • Go: stashgen generates the *_stash.go files and the goldens.

Checks run (in a worktree at a05affaaa)

  • EQL:
    • cargo test -p eql-codegen (includes the Go parity test): pass.
    • mise run types:check, codegen:parity, test:schemas:parity: pass.
    • mise run build, with the installer completeness gate and the v4 isolation gate: pass.
    • git status was clean after all of these.
  • Rust and Go:
    • cargo fmt --check: pass.
    • cargo clippy --workspace --all-features --all-targets -D warnings: pass.
    • cargo test -p stack-guest-abi -p stack-encrypt --all-features: pass.
  • Go guests and SDK:
    • All five WASI guests build, and the import-surface checks pass.
    • go vet ./... and go test ./... in languages/golang: pass.
    • go generate ./... leaves no diff.
  • I built the v3 bundle and compared it with the @cipherstash/eql@3.0.6 tarball. Only lints() and comments changed, so the claim "v3 SQL unchanged" holds.
  • git merge-tree against current main: no conflicts.
  • cargo publish --dry-run -p stack-encrypt-derive -p stack-encrypt against crates.io: fails (see below).

Blocking

  1. stack-encrypt 0.3.0 cannot publish. The root release-plz line publishes every stack-* version that crates.io does not have, so merging this starts a publish of stack-encrypt 0.3.0. stack-encrypt requires stack-kms 0.1.0, and that is the only stack-kms version on crates.io. But src/lib.rs re-exports stack_kms::{diagnostic, ErrorPayload}, and these were added after the 0.1.0 release with no version bump. The packaged build fails with E0432: unresolved imports stack_kms::diagnostic, stack_kms::ErrorPayload.
    • Fix: bump stack-kms (for example to 0.2.0, with a CHANGELOG entry), and make stack-encrypt require that version.
    • Note that the publish dry run on main (tests-crates.yml) packages all five crates together, so stack-encrypt resolves the tree's stack-kms there. That step does not catch a stack-kms that crates.io lacks under the same version number.
  2. The stack-encrypt CHANGELOG has no 0.3.0 section. Cargo.toml says 0.3.0, but about 850 lines, including a ### Breaking section, are still under ## [Unreleased]. The root release-plz line only publishes and never rewrites the changelog, so 0.3.0 would ship under "Unreleased", as 0.1.0 did. Please rename the section to ## [0.3.0] - <date> in this PR.

Non-blocking (see inline)

  • SchemaVersion is now generic. Rust does not apply a default const parameter in an expression, so SchemaVersion::CURRENT and SchemaVersion::default() stop compiling (E0284). The changeset ships this as a minor bump and says "plain SchemaVersion still means v3".
  • The stored and single-bound query domain templates accept "v": null. The range template already guards against this.
  • stack-encrypt will publish with an exact pin on cllw-ore =0.5.2-rc.1.
  • There are some small nits: stale vitaminc version numbers in comments, a Go wipe path, an ADR reference, and the position of a use.

Before you merge
eql-bindings now requires stack-encrypt 0.3.0. Its crates.io publish (publish_all_features) succeeds only after release-crates has published stack-encrypt 0.3.0, so blocker 1 must be fixed first. Otherwise the EQL crate publish fails too.

Checked and fine

  • SQL renderer: it refuses literal eql_v3/eql_v4 names, unknown placeholders and unmatched blocks.
  • Range domains and between: correct, with STRICT/IMMUTABLE accessors and an inlinable between. On non-superuser installs, the ORE range domains are disabled with the same NOT VALID CHECK as the single-bound ORE domains.
  • v4 ore_block_v2 SQL: the layout offsets and the GF(2^128) doubling are correct, and errors do not echo the input bytes.
  • stack-guest-abi step and value: no new raw reads of host lengths, handles are never reused, MAX_OPEN is enforced, and secrets are Zeroizing with OpaqueDebug.
  • Go step loop: every early exit after a pause aborts the operation and wipes the buffers. maxPaused is tied to the Rust constant by a test.
  • Go public surface: QueryTerm, Verify and TargetLimit follow the existing Seal/Open pattern, and MatchQuery never prints its text.
  • CI: actions are pinned by SHA, both copies of the path filters were updated, and no release workflow changed.
  • Changesets: the EQL changesets are minor, which is correct apart from the SchemaVersion point. chacha20-0-10-2.md matches the lockfile moves.
  • Skills: no skill update is needed. stash eql install still installs v3 only, and no skill covers the Go SDK or Stack Encrypt.

Comment thread packages/stack-encrypt/Cargo.toml Resolved
Comment thread Cargo.toml Resolved
Comment thread .github/dependabot.yml Resolved
Comment thread packages/stack-encrypt/Cargo.toml Resolved
Comment thread languages/golang/encrypt/step.go Resolved
Comment thread packages/stack-guest-abi/src/call.rs Resolved
The doc of `Plan::with_scopes` said in one sentence what a scope does to a
field's context, and it was hard to follow, as the review of #1595 said.
It now says it in plain words, and a doc test shows the scope at the end
of the field's context through `Pending::context()`.

Refs #1486.
… Planned

A data plan was lowered by building a typed plan from it and running
that, so the plan builder sat in the dynamic path and could not be
removed. `compose` and `opening` in `dynamic::planned` now build each
field's combinator from the parsed plan, as ADR-0007's amendment of
9 November 2026 decided, and `encrypt_with`, `decrypt_with`,
`check_source`, `check_record` and `Plan::contexts` run them. The
whole-plan rules the builder checked for a data plan (names once, no
shared identity, plain labels) are checked where the plan is parsed.

No stored byte changes: the combinators, labels and order are the
builder's, and the conformance vectors and the equivalence suite pin
them. The builder, the chain and the old traits are removed in the
pull requests that follow.

Closes #1120. Refs #1488.
The review of #1597 found five in-diff mutants surviving. Three were
equivalent: the "keys nothing" distinction in `compose` under a label
context, where a passthrough ignores its context either way, and the
context-field disjunct of the predicate, dead because a context field
is always a passthrough. The distinction is removed rather than
excluded. Two were checks nothing reached: `check_stored` and
`check_names` are now reached by unit tests in `planned.rs`.

`validate` is deleted. Its duplicate-field and label rules were refused
by the parser first; the one live rule, shared identity between keyed
fields, is in `Plan::build`'s loop beside the target rule. The
builder's `join`, `field_label`, `FROM_PLAN` and `FROM_CALL` are shared
with the lowering rather than copied. A target field is skipped by
`operation_of` returning `None`, so no `unreachable!` sits in library
code. Docs that still described lowering through the builder, or that
said the checks run the lowering, are reworded: the checks share
`check_names`, `check_stored` and `opening_base` with it.

Tests added: a context field declared last, a batch with absent fields
under a context field opened with an expected context, the stored and
value checks refusing each kind of misfit, and the shared-identity
table with exact reasons.

Refs #1488.
…ule's docs

The crate's rustdoc is what a user reads; a design record is for the
people who maintain the crate. The two sentences state what the code
does and leave the record out. Refs #1598.
Each sealed field's closure read the record's context field by copying
the frame and parsing the copy, once per field per row. The frame is
now read in place through `IntoLabel for &ValueFrame`, so what each
field does per row is one parse of a short text and no copy. Parsing
once per row would need a slot the field closures share, which the
description has no place for: each field's `under_by` sees the row
alone.

Refs #1488.
…nators

The ADR listed the two check functions among the callers of `compose`
and `opening`. They run no combinator: they state the same rules through
the checks the lowering shares, as the CHANGELOG says.

Refs #1488.
…field markers

A field marker had operator methods only where the field's type was one
index. A field of several indexes, a nullable field and a field that is
a record had none, so those fields could be queried only through the
plan chain that #1488 removes.

A tuple of terms now answers each operator through the one term that
answers it. Rust's coherence rules allow no single impl that picks that
term, so each term states its answer to each scalar operator as a type
(op::OperatorAnswer, Yes or No), and op::AnsweredBy picks the one Yes.
An operator no term answers, or two answer, stays a compile error.
Option<T> answers what a present value answers; a test shows a present
value's term is a bare value's.

Marker::field reaches an inner record's markers under the outer field's
context. The marker now names where its field sits (marker::Root<R> or
marker::Nested<W>), so field refuses a marker of another record and the
marker stays Copy with no depth limit. A query wraps the caller's scope
in each outer field's context, outermost first, as the derive seals a
nested record.

(marker::Takes<T>), checked by the derive against the plaintext field,
so an equality marker no longer takes a value of any type.

Closes #1596
…ord field markers

OreBlockTerm had no LabelledTarget impl, so a block ORE field, alone or
beside an equality index, had a marker with no operator method. It is
sealed under the caller's context like the other order terms.

field() now bounds the inner record's scope with NestedScope, so an
inner record with a scope of its own is refused with a message that
says why, rather than a bare type mismatch. QueryValue is sealed for
every type parameter. Root's outer labels are pushed by an empty method,
so no equivalent mutant is left for the mutants gate.

The docs say that an untyped nullable marker takes an Option, and that
None derives a term no row holds; query = .. refuses it, and a ui case
pins that. The tests add a caller binding and stored bindings around a
nested record, an outer field under another identity, a nullable field
of two indexes, an empty nested scope, block ORE, and the chain's tuple
query without .via. The CHANGELOG describes the final Marker.

Refs #1596
…compiler now words it

The `Takes` doc quoted the error over `ToOwned::Owned`. The operator
methods take their value through `QueryInput` since the review of #1595,
so the compiler names `QueryInput::Value`, as the trybuild snapshot does.

Refs #1596.
@coderdan
coderdan removed this pull request from stack #1602 November 10, 2026 05:17
coderdan and others added 20 commits November 10, 2026 16:18
…text

feat(stack-encrypt): show the context a declaration or a call uses, and pin it in tests
…nd-bindings

feat(stack-encrypt): split the context into scopes and bindings
…y-preparation

feat(stack-encrypt)!: declare each operator once and prepare a query once
…rkers

feat(stack-encrypt): field markers and a value under a plan that is data
…lowering

refactor(stack-encrypt): lower a data plan to the combinators in dynamic::planned
…ields

feat(stack-encrypt): operator methods for tuple, nullable and record field markers
The rustdoc of stack-encrypt and stack-encrypt-derive is what a user
reads on docs.rs and in an editor. It named the crates' design records
(the ADRs under packages/stack-encrypt/docs/adr and the plans under
docs/plans) in 47 places. Those records are not published with the
crate, so a user could not follow the reference.

Each sentence now states the rule or the mechanism itself. Where a
maintainer benefits from the pointer, it moved to a plain // comment.
Three history references go too: a closed issue cited twice on
NullMode, and a closed tracking issue in the read docs, which now say
that query_term reads its input as the field's kind.

tests/doc_hygiene.rs reads every source file of both crates and fails,
naming file and line, on a /// or //! line that names an ADR number,
docs/adr or docs/plans.

Closes #1598.
The derive crate's attribute reference, pulled into its rustdoc with
include_str!, still named the API plan, so the guard now scans every
file a doc attribute includes. It also flags an ADR that ends a line,
whose number sits on the next one, and matches case-insensitively.

The JSON module's docs compared the code with its plan; they now
describe the behaviour, and the comparison is a maintainer comment. The
query docs said a type answers with one query domain per operator; EQL
v4 has one per type family and term, as the code binds it. A few more
history phrases go, and the row-binding sentence says what naming the
row costs.

Refs #1598.
… inner record with its own scope

A field marker is the typed way to query a field of a derived record.
Two kinds of record field had no marker query. The plan chain was the
only typed way to query them, and #1488 removes the chain.

Marker::field now takes a marker of the record a field holds or holds
an Option of (marker::Holds). The inner record may list a scope of its
own. The result's scope type is the outer record's scope joined with the
inner one (marker::JoinScope), and MarkedQuery::scope takes them as one
tuple, outermost first. The query extends the field's label as the
derive seals it: each record's scope goes before the context of the
field that holds the record.

marker::NestedScope is removed. It was added in this unreleased version.

Closes #1600
Clarify the explanation regarding row binding and its impact on ciphertext protection.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
JsonSelector::open was corrected in the previous commit. The same claim,
that a row binding makes selectors and terms that row's alone, stood in
Error::JsonWrongEntry. A binding protects the ciphertext; selectors and
terms derive under the query scopes alone, as
tests/scopes_and_bindings.rs pins.
A fourth record with a scope of its own now fails with the message about
the three-scope limit. Before, the three-scope tuple had one `JoinScope`
implementation, so rustc decided the inner marker's scope was `NoScope`
and reported a type mismatch that said the inner record must list no
scope. A second implementation stops that inference and carries the
message. Its bound is a trait whose one implementation is for a type no
caller can name. A new trybuild case pins it.

Tests now cover a binding beside a nested scope: an inner record that
stores a scope and a binding, under a record with a caller scope and a
caller binding, as a field and as a nullable field, and a nullable record
that stores a binding and no scope. A nullable record inside a nullable
record is covered too. `contexts()` is checked for the nullable record
with a scope and for three scopes. The empty-scope tests match
`PlanError::EmptyPart`, and the absent-record test compares the query
with a present record's term.

`extension` checks in debug builds that it has one scope for each record
that lists one. `GivenScopes` is implemented for flat tuples of `Scope`
only, the same types as `ScopeInput`.

The docs no longer say that a marker's scope is always the record's
caller scope: a stored scope gives the same type, and a nested marker can
take a tuple. The CHANGELOG no longer says a nested marker takes only the
outer record's scope.
…no-adrs

docs(stack-encrypt): state the rules the rustdoc cited records for
…in plain language

The rustdoc for a field that is a record, and for a query that needs
more than one scope, now says who does what in short sentences, and
lists the context parts and the mistakes that do not compile.

The three compiler messages state one thing each. The message for a
fourth scope names the limit, its label names the call that adds the
scope, and its note names the module to use instead.

The docs now also say that two scopes of the same type, passed in the
wrong order, compile and match no row.
A query on a field of a nested record took its scopes as a tuple:
.scope((Scope::new(outer), Scope::new(inner))). The caller input
everywhere else is a chain that starts at Scope::new, such as
Scope::new(tenant).bind(row). A query now takes the same form:
.scope(Scope::new(outer).scope(inner)).

Scope::scope returns the new target::Scopes<Outer, Inner>, and
Scopes::scope adds the scope of a record one level further in. Scopes
nests, so a query no longer has a limit of three scopes. The
FourthScope trait and its compile error, which only reported that
limit, are removed.

Refs #1600
Stack Encrypt's dynamic interface runs a plan that arrives as data. A
field of that plan can name an EQL type, and a program gives the engine
the EQL types through the TargetResolver trait. Nobody shipped one, so
the Go guest and three EQL encryption tests each wrote the same
resolver over encryption::targets, and the copies had drifted: two test
copies named fewer of a JSON document's query domains than the guest.

encryption::EqlTargets is the guest's resolver, moved into eql-bindings
behind the targets feature. The row-to-descriptor and refusal
conversions are From impls beside it. The guest and the tests use it
and their copies are gone, so a change to the trait's signature is made
once.

Closes #1571
- Name the three refusals that convert to `Other` (`DocumentQuery`,
  `NoIndex`, `Options`) instead of a wildcard, so a variant added later
  must be placed by hand. A test shows the engine refuses options on a
  type with no JSON index at plan time, before the resolver is called.
- Move the explanation and the example onto `EqlTargets`. The `resolver`
  module is private, so rustdoc did not show its module doc.
- Test the `Unproducible` and `Json` conversions and the `Other` arms.
- Drop the braces around a single `self as eql_targets` import.
- The API plan no longer says each program writes its own resolver.
The docs for EqlTargets and its two conversions, and the module docs of
the Go guest's resolver, now use short sentences that name who does
what. They say Stack Encrypt where they said the engine.
…cope-markers

feat(stack-encrypt): typed queries for a nullable record field and an inner record with its own scope
…solver

feat(eql-bindings): ship one TargetResolver for the EQL types

This branch was successfully deployed

1 active (outdated) deployment
release-golang — 929451b4 Deployed Nov 2, 2026 by coderdan via Tag and release #5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants