diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index c528f23..095b28a 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -10,6 +10,10 @@ https://semver.org/spec/v2.0.0.html[Semantic Versioning]. ==== Fixed +* Example truthfulness (H10, 2026-11-19): `examples/quintessence/README.adoc` no + longer prints output that no tool produced; it shows real CLI output. + `examples/quintessence/mix/tasks/verify.ex` is marked as a design sketch + that does not run (it calls modules that do not exist; plan stage Q2). * Trust level (2026-11-15): a `coqc` and a `lean` binary on PATH raised the reported trust level to 5, and no kernel check ran. The maximum is now 4, and `--require-level-5` always reports `Q-504` in this version. Suite case diff --git a/docs/plan/ULTRAPLAN.adoc b/docs/plan/ULTRAPLAN.adoc index 05d259a..5656603 100644 --- a/docs/plan/ULTRAPLAN.adoc +++ b/docs/plan/ULTRAPLAN.adoc @@ -44,7 +44,7 @@ all feature work. `H0` is closed. The others are open. | H7 | `Qir.idr` is not in `abi.ipkg`. No tool type-checks it. | `src/interface/abi.ipkg` | H8 | Code and README need Zig 0.17. `mise.lock` pins 0.16.0. Z3 and cvc5 are not pinned. | `mise.toml`, `mise.lock` | H9 | No CI workflow builds or tests the engine. | `.github/workflows/` -| H10 | `examples/quintessence/mix/tasks/verify.ex` calls modules that do not exist. `examples/quintessence/README.adoc` prints output that no tool produced. | read +| H10 (partly closed) | `examples/quintessence/mix/tasks/verify.ex` calls modules that do not exist. `examples/quintessence/README.adoc` printed output that no tool produced. | Closed 2026-11-19: the README now shows only output of the built CLI (passing module, K-CUT falsifier, vacuity), with versions and the known failures of the full example stated, and `verify.ex` is marked at its top and in the README as a design sketch that does not run. Still open: `verify.ex` is not real code (it waits for Q2), and the full example run exits 4 on a fresh build (H3). |=== == Definition of v1.0.0 diff --git a/examples/quintessence/README.adoc b/examples/quintessence/README.adoc index 62e3c8a..2d9a011 100644 --- a/examples/quintessence/README.adoc +++ b/examples/quintessence/README.adoc @@ -8,10 +8,18 @@ [IMPORTANT] ==== -This is a *reference tree*, not a buildable application. No Elixir, Zig, -Idris2, Agda, or Why3 toolchain is invoked by the repository that contains it. -The code fixes interfaces and proof obligations; the JSON hashes are -illustrative. The specification it illustrates is +This is a *reference tree*, not a buildable application. The Elixir sources are +read by the `quintessence` CLI (`src/interface/ffi`, built with Zig 0.17.x); no +Elixir, Idris2, Agda, or Why3 toolchain is invoked. The code fixes interfaces +and proof obligations; the JSON hashes are illustrative. + +*`mix/tasks/verify.ex` is a design sketch that does not run yet.* It calls +modules (`Quintessence.Project`, `Frontend`, `Echidna`, `Batch`, +`Mutation`, `Gate`, `Report`) that exist nowhere in this repository. Plan +stage Q2 (`docs/plan/ULTRAPLAN.adoc`) is the stage that would make it real. +The working entry point today is the CLI shown below, not `mix verify`. + +The specification this tree illustrates is link:../../docs/architecture/QUINTESSENCE.adoc[the architecture document]. ==== @@ -69,7 +77,8 @@ small enough to read and large enough to exercise all three layers: mode. | `mix/tasks/verify.ex` -| The `mix verify` orchestrator sketch. +| Design sketch of a `mix verify` orchestrator. *Does not run yet*: it calls + modules that do not exist (Q2). |=== == The protocol, and the boundary it respects @@ -130,50 +139,104 @@ receipt carries the proof hash. == A passing run +Real output, from the repository root. Build the CLI and put it on `PATH` first +(the build writes `src/interface/ffi/zig-out/bin/quintessence` and does not +install it): + +[source,sh] +---- +(cd src/interface/ffi && zig build) +export PATH="$PWD/src/interface/ffi/zig-out/bin:$PATH" +---- + [source,text] ---- -$ mix verify --module Verisim.Entity --receipt -==> quintessence 0.1.0 (qir/v1, elixir 1.19.0, OTP 28) -[g0] firewall: 2 modules, 11 macro calls, allowlist ok, expansion hash 7c3a... -[g1] subset: 9 functions, 2 handlers, 1 protocol, 0 rejected -[qir] sha256:9ab0... -[layer a] Verisim.Entity.handle_call/3: 9 VCs -> 9 proved (why3 1.7.2) - Verisim.Spec.drift_of/2: 4 VCs -> 4 proved - Verisim.Spec.depth/1: 2 VCs -> 2 proved (measure: length(history)) - Verisim.Spec.digest_of/1: 1 VC -> 1 proved -[layer b] protocol Verisim.Entity: 3 roles, 3 messages, 2 knowledge claims - B1 client<-entity: call/commit + reply/committed -> K-CUT-WARRANT OK - B2 auditor<-entity: cast/attest -> receiver-knows-sent OK - B3 entity->auditor: no claim; forbid check OK - B4 mailbox HWM: 64 <= 128 (bound 128) -[layer c] Verisim.Nif.hash/2: receipt sha256:dad8..., MaxPlus 180us -> normal - Verisim.Nif.drift/2: receipt sha256:dad8..., MaxPlus 240us -> normal -[echidna] 23 goals -> 23 warrants; 23 dk checked; batch: 23 Coq QED, 23 Lean QED -[receipt] sha256:5e11... trust_level 5, status complete -[mutate] 18 mutants, 18 caught, 0 survived (100%), vacuity V1-V4 clean -[gate] PASS +$ quintessence verify examples/quintessence --module Verisim.Entity +Quintessence Verification Suite v0.1.0 +------------------------------------------------------------ + Module Verisim.Entity (layer :b) [examples/quintessence/lib/verisim/entity.ex] + +Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel): + [QED] Verisim.Entity_genserver_state_invariant z3=unsat cvc5=unsat alethe=QED (344 steps, 76 res) + [QED] Verisim.Entity_bmc_mailbox_bound_k12 z3=unsat cvc5=unsat alethe=QED (8824 steps, 1848 res) + [QED] Verisim.Entity_bmc_no_unwarranted_claim_k12 z3=unsat cvc5=unsat alethe=QED (13936 steps, 3167 res) + [QED] Verisim.Entity_bmc_epoch_consistent_k12 z3=unsat cvc5=unsat alethe=QED (1828 steps, 218 res) + +ECHIDNA 6-Kind Mutation Report: + Caught 2/2 proof-relevant mutants (0 guard mutant(s) excluded) -> Score: 100.00% (gate >= 95.00%) +------------------------------------------------------------ +VERIFIED TrustLevel=4 Obligations=4 MutationScore=100.00% Exit=0 + Warrant: /tmp/quintessence-verify-out/warrant.json + Receipt: /tmp/quintessence-verify-out/receipt.json ---- -The output is illustrative, but every line corresponds to an obligation in the -specification. The `[layer b]` lines are the ones that no existing BEAM tool -would print. +Measured 2026-11-19 with Zig 0.17.1, Z3 4.8.12 and cvc5 1.1.2. This module +verifies at Trust Level 4 (Level 5 needs kernel checks that this version does +not run). Running the whole example (`quintessence verify examples/quintessence`) +does *not* pass on that setup: it exits 4 with `Q-303` (the committed NIF +receipt hash does not match a fresh build) and two `Q-503` Alethe failures on +`Verisim.Spec`. These are known holes (H3 in `docs/plan/ULTRAPLAN.adoc`), not +shown as a pass here. == The failing variant that matters -Change the client to `GenServer.cast(Verisim.Entity, {:commit, payload})` and -keep the `knowledge` claim: +A client that claims knowledge across a `cast` is rejected. The fixture below is +the one the test suite uses (`tests/quintessence/run_suite.sh`, case 9); it is a +minimal module, not a copy of `Verisim.Entity`. Write it to a scratch directory, +then run the CLI on it (with the CLI on `PATH` as above): + +[source,sh] +---- +cat > bad_kcut_cast.ex <<'EX' +defmodule BadKCutCast do + use GenServer + use Quintessence.Face, layer: :b + + @choreography [ + roles: [:client, :entity], + messages: [ + {:client, :entity, :commit, [:payload]} + ], + knowledge: [ + {:client, :entity, :committed, [:version], {:cast, :commit}} + ] + ] + + def handle_cast({:commit, _payload}, state), do: {:noreply, state} +end +EX +---- + +Real output: [source,text] ---- -$ mix verify --strict -[layer b] B1 client->entity: cast/commit has no causal reply - Q-401 UNWARRANTED_REMOTE_CLAIM - falsifier: AsyncKnowledge.sender-cannot-know-receipt - remedy: use GenServer.call/3 and claim after the reply, or - drop the knowledge claim and keep only the send fact -[gate] FAIL +$ quintessence verify bad_kcut_cast.ex +Quintessence Verification Suite v0.1.0 +------------------------------------------------------------ + Module BadKCutCast (layer :b) [bad_kcut_cast.ex] + +Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel): + [QED] BadKCutCast_bmc_mailbox_bound_k12 z3=unsat cvc5=unsat alethe=QED (8824 steps, 1848 res) + [QED] BadKCutCast_bmc_no_unwarranted_claim_k12 z3=unsat cvc5=unsat alethe=QED (13936 steps, 3167 res) + [QED] BadKCutCast_bmc_epoch_consistent_k12 z3=unsat cvc5=unsat alethe=QED (1828 steps, 218 res) + +Diagnostics: + ERROR [Q-401] bad_kcut_cast.ex:5:4: K-CUT violation (`AsyncKnowledge.sender-cannot-know-receipt`): `client` claims `entity.committed` across asynchronous `cast` without a causal reply + Counterexample / Witness: + Trace refuting `client` claiming `entity` across async `commit`: + Step 0: entity.version = 0, client.knows_committed = false + Step 1: client sends async `commit` and transitions client.knows_committed := true (no reply wait) + Step 2: Async channel drops `commit` (`msg_dropped = true`); entity.version remains 0 + Violated Invariant: `NoUnwarrantedRemoteClaim` (`client` claims commit while `entity.version == 0`) + Rule: K-SEND / K-CAST / K-REPLY (`choreographic-types` Y-6 & Y-8) + Remedy: Replace `cast`/`send` with synchronous `call -> commit -> reply` carrying correlation tag, monotonic version, and matching epoch under `K-REPLY`. +------------------------------------------------------------ +FAILED Diagnostics=1 Exit=1 ---- +The process exits with status 1. + There is no `--allow-unwarranted` flag. The two legal repairs are: . Change the protocol to `call` and claim after the reply. The claim is then the @@ -186,21 +249,50 @@ K-CUT proof and is not accepted by `--strict`. == A vacuity report -Suppose the postcondition were weakened to `result.digest != the empty binary`. The mutation -engine replaces the equality with the weak predicate; every non-empty digest -satisfies it, so the mutant survives: +A postcondition that every input satisfies leaves mutants alive. The fixture is +the one in `tests/quintessence/run_suite.sh` case 14 (a minimal module with +`@pre spec(x >= 10)` and `@post spec(result >= 0)`; not `Verisim.Spec`). Write it +to a scratch directory, then run the CLI on it: + +[source,sh] +---- +cat > weak_spec.ex <<'EX' +defmodule WeakSpec do + use Quintessence.Face, layer: :a + @pre spec(x >= 10) + @post spec(result >= 0) + def increment(x), do: x + 1 +end +EX +---- + +Real output: [source,text] ---- -[mutate] 18 mutants, 17 caught, 1 survived (94.4%) < 95.0% - survivor: WeakenPostcondition - original: new_state.digest == Verisim.Spec.digest_of(payload) - weakened: new_state.digest != <<>> - goal: Verisim.Entity.handle_call/3.post.2 - remedy: restore the equality; the weak predicate is type-only -[gate] FAIL +$ quintessence verify weak_spec.ex +Quintessence Verification Suite v0.1.0 +------------------------------------------------------------ + Module WeakSpec (layer :a) [weak_spec.ex] + +Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel): + [QED] WeakSpec_increment_vc_post_0 z3=unsat cvc5=unsat alethe=QED (166 steps, 36 res) + +ECHIDNA 6-Kind Mutation Report: + Caught 0/2 proof-relevant mutants (1 guard mutant(s) excluded) -> Score: 0.00% (gate >= 95.00%) + [SURVIVED] mut_1_op_swap (OperatorSwap) on `WeakSpec.increment/1` + [SURVIVED] mut_2_boundary_shift (BoundaryShift) on `WeakSpec.increment/1` + +Diagnostics: + ERROR [Q-603] weak_spec.ex:1:1: Mutation score 0.00% (0/2 proof-relevant mutants caught) is below required 95.00% threshold + Rule: Vacuity Probe V3 (ECHIDNA 6-Kind Mutation Gate >= 95%) + Remedy: Tighten `@post`, `@invariant`, or protocol assertions to kill the surviving mutants listed in the mutation report. +------------------------------------------------------------ +FAILED Diagnostics=1 Exit=5 ---- +The process exits with status 5. + If the precondition were `byte_size(payload) <= 1_048_576 and byte_size(payload) > 1_048_576`, V1 would find no model and report `Q-601 VACUOUS` before any proof was attempted. A proof of a vacuous spec is not a diff --git a/examples/quintessence/mix/tasks/verify.ex b/examples/quintessence/mix/tasks/verify.ex index 207dc52..a148be6 100644 --- a/examples/quintessence/mix/tasks/verify.ex +++ b/examples/quintessence/mix/tasks/verify.ex @@ -1,7 +1,13 @@ # SPDX-License-Identifier: MPL-2.0 # Copyright (c) 2026 Jonathan D.A. Jewell # -# Reference sketch. Not compiled by this checkout. See README.adoc. +# DESIGN SKETCH. THIS FILE DOES NOT RUN. +# It calls Quintessence.Project, .Frontend, .Echidna, .Batch, .Mutation, .Gate +# and .Report, and none of them exists in this repository (nor does +# Quintessence.Doctor; only Quintessence.Error is defined, in lib/). It is not +# compiled by this checkout. Plan stage Q2 (docs/plan/ULTRAPLAN.adoc) is the +# stage that would make it real. Today the working entry point is the +# `quintessence` CLI (src/interface/ffi). See README.adoc. defmodule Mix.Tasks.Verify do @moduledoc """