Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ https://semver.org/spec/v2.0.0.html[Semantic Versioning].

==== Fixed

* Example truthfulness (H10, 2026-11-19): `examples/quintessence/README.adoc` no
longer prints output that no tool produced; it shows real CLI output.
`examples/quintessence/mix/tasks/verify.ex` is marked as a design sketch
that does not run (it calls modules that do not exist; plan stage Q2).
* Trust level (2026-11-15): a `coqc` and a `lean` binary on PATH raised the
reported trust level to 5, and no kernel check ran. The maximum is now 4,
and `--require-level-5` always reports `Q-504` in this version. Suite case
Expand Down
2 changes: 1 addition & 1 deletion docs/plan/ULTRAPLAN.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ all feature work. `H0` is closed. The others are open.
| H7 | `Qir.idr` is not in `abi.ipkg`. No tool type-checks it. | `src/interface/abi.ipkg`
| H8 | Code and README need Zig 0.17. `mise.lock` pins 0.16.0. Z3 and cvc5 are not pinned. | `mise.toml`, `mise.lock`
| H9 | No CI workflow builds or tests the engine. | `.github/workflows/`
| H10 | `examples/quintessence/mix/tasks/verify.ex` calls modules that do not exist. `examples/quintessence/README.adoc` prints output that no tool produced. | read
| H10 (partly closed) | `examples/quintessence/mix/tasks/verify.ex` calls modules that do not exist. `examples/quintessence/README.adoc` printed output that no tool produced. | Closed 2026-11-19: the README now shows only output of the built CLI (passing module, K-CUT falsifier, vacuity), with versions and the known failures of the full example stated, and `verify.ex` is marked at its top and in the README as a design sketch that does not run. Still open: `verify.ex` is not real code (it waits for Q2), and the full example run exits 4 on a fresh build (H3).
|===

== Definition of v1.0.0
Expand Down
186 changes: 139 additions & 47 deletions examples/quintessence/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,18 @@

[IMPORTANT]
====
This is a *reference tree*, not a buildable application. No Elixir, Zig,
Idris2, Agda, or Why3 toolchain is invoked by the repository that contains it.
The code fixes interfaces and proof obligations; the JSON hashes are
illustrative. The specification it illustrates is
This is a *reference tree*, not a buildable application. The Elixir sources are
read by the `quintessence` CLI (`src/interface/ffi`, built with Zig 0.17.x); no
Elixir, Idris2, Agda, or Why3 toolchain is invoked. The code fixes interfaces
and proof obligations; the JSON hashes are illustrative.

*`mix/tasks/verify.ex` is a design sketch that does not run yet.* It calls
modules (`Quintessence.Project`, `Frontend`, `Echidna`, `Batch`,
`Mutation`, `Gate`, `Report`) that exist nowhere in this repository. Plan
stage Q2 (`docs/plan/ULTRAPLAN.adoc`) is the stage that would make it real.
The working entry point today is the CLI shown below, not `mix verify`.

The specification this tree illustrates is
link:../../docs/architecture/QUINTESSENCE.adoc[the architecture document].
====

Expand Down Expand Up @@ -69,7 +77,8 @@ small enough to read and large enough to exercise all three layers:
mode.

| `mix/tasks/verify.ex`
| The `mix verify` orchestrator sketch.
| Design sketch of a `mix verify` orchestrator. *Does not run yet*: it calls
modules that do not exist (Q2).
|===

== The protocol, and the boundary it respects
Expand Down Expand Up @@ -130,50 +139,104 @@ receipt carries the proof hash.

== A passing run

Real output, from the repository root. Build the CLI and put it on `PATH` first
(the build writes `src/interface/ffi/zig-out/bin/quintessence` and does not
install it):

[source,sh]
----
(cd src/interface/ffi && zig build)
export PATH="$PWD/src/interface/ffi/zig-out/bin:$PATH"
----

[source,text]
----
$ mix verify --module Verisim.Entity --receipt
==> quintessence 0.1.0 (qir/v1, elixir 1.19.0, OTP 28)
[g0] firewall: 2 modules, 11 macro calls, allowlist ok, expansion hash 7c3a...
[g1] subset: 9 functions, 2 handlers, 1 protocol, 0 rejected
[qir] sha256:9ab0...
[layer a] Verisim.Entity.handle_call/3: 9 VCs -> 9 proved (why3 1.7.2)
Verisim.Spec.drift_of/2: 4 VCs -> 4 proved
Verisim.Spec.depth/1: 2 VCs -> 2 proved (measure: length(history))
Verisim.Spec.digest_of/1: 1 VC -> 1 proved
[layer b] protocol Verisim.Entity: 3 roles, 3 messages, 2 knowledge claims
B1 client<-entity: call/commit + reply/committed -> K-CUT-WARRANT OK
B2 auditor<-entity: cast/attest -> receiver-knows-sent OK
B3 entity->auditor: no claim; forbid check OK
B4 mailbox HWM: 64 <= 128 (bound 128)
[layer c] Verisim.Nif.hash/2: receipt sha256:dad8..., MaxPlus 180us -> normal
Verisim.Nif.drift/2: receipt sha256:dad8..., MaxPlus 240us -> normal
[echidna] 23 goals -> 23 warrants; 23 dk checked; batch: 23 Coq QED, 23 Lean QED
[receipt] sha256:5e11... trust_level 5, status complete
[mutate] 18 mutants, 18 caught, 0 survived (100%), vacuity V1-V4 clean
[gate] PASS
$ quintessence verify examples/quintessence --module Verisim.Entity
Quintessence Verification Suite v0.1.0
------------------------------------------------------------
Module Verisim.Entity (layer :b) [examples/quintessence/lib/verisim/entity.ex]

Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel):
[QED] Verisim.Entity_genserver_state_invariant z3=unsat cvc5=unsat alethe=QED (344 steps, 76 res)
[QED] Verisim.Entity_bmc_mailbox_bound_k12 z3=unsat cvc5=unsat alethe=QED (8824 steps, 1848 res)
[QED] Verisim.Entity_bmc_no_unwarranted_claim_k12 z3=unsat cvc5=unsat alethe=QED (13936 steps, 3167 res)
[QED] Verisim.Entity_bmc_epoch_consistent_k12 z3=unsat cvc5=unsat alethe=QED (1828 steps, 218 res)

ECHIDNA 6-Kind Mutation Report:
Caught 2/2 proof-relevant mutants (0 guard mutant(s) excluded) -> Score: 100.00% (gate >= 95.00%)
------------------------------------------------------------
VERIFIED TrustLevel=4 Obligations=4 MutationScore=100.00% Exit=0
Warrant: /tmp/quintessence-verify-out/warrant.json
Receipt: /tmp/quintessence-verify-out/receipt.json
----

The output is illustrative, but every line corresponds to an obligation in the
specification. The `[layer b]` lines are the ones that no existing BEAM tool
would print.
Measured 2026-11-19 with Zig 0.17.1, Z3 4.8.12 and cvc5 1.1.2. This module
verifies at Trust Level 4 (Level 5 needs kernel checks that this version does
not run). Running the whole example (`quintessence verify examples/quintessence`)
does *not* pass on that setup: it exits 4 with `Q-303` (the committed NIF
receipt hash does not match a fresh build) and two `Q-503` Alethe failures on
`Verisim.Spec`. These are known holes (H3 in `docs/plan/ULTRAPLAN.adoc`), not
shown as a pass here.

== The failing variant that matters

Change the client to `GenServer.cast(Verisim.Entity, {:commit, payload})` and
keep the `knowledge` claim:
A client that claims knowledge across a `cast` is rejected. The fixture below is
the one the test suite uses (`tests/quintessence/run_suite.sh`, case 9); it is a
minimal module, not a copy of `Verisim.Entity`. Write it to a scratch directory,
then run the CLI on it (with the CLI on `PATH` as above):

[source,sh]
----
cat > bad_kcut_cast.ex <<'EX'
defmodule BadKCutCast do
use GenServer
use Quintessence.Face, layer: :b

@choreography [
roles: [:client, :entity],
messages: [
{:client, :entity, :commit, [:payload]}
],
knowledge: [
{:client, :entity, :committed, [:version], {:cast, :commit}}
]
]

def handle_cast({:commit, _payload}, state), do: {:noreply, state}
end
EX
----

Real output:

[source,text]
----
$ mix verify --strict
[layer b] B1 client->entity: cast/commit has no causal reply
Q-401 UNWARRANTED_REMOTE_CLAIM
falsifier: AsyncKnowledge.sender-cannot-know-receipt
remedy: use GenServer.call/3 and claim after the reply, or
drop the knowledge claim and keep only the send fact
[gate] FAIL
$ quintessence verify bad_kcut_cast.ex
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Quintessence Verification Suite v0.1.0
------------------------------------------------------------
Module BadKCutCast (layer :b) [bad_kcut_cast.ex]

Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel):
[QED] BadKCutCast_bmc_mailbox_bound_k12 z3=unsat cvc5=unsat alethe=QED (8824 steps, 1848 res)
[QED] BadKCutCast_bmc_no_unwarranted_claim_k12 z3=unsat cvc5=unsat alethe=QED (13936 steps, 3167 res)
[QED] BadKCutCast_bmc_epoch_consistent_k12 z3=unsat cvc5=unsat alethe=QED (1828 steps, 218 res)

Diagnostics:
ERROR [Q-401] bad_kcut_cast.ex:5:4: K-CUT violation (`AsyncKnowledge.sender-cannot-know-receipt`): `client` claims `entity.committed` across asynchronous `cast` without a causal reply
Counterexample / Witness:
Trace refuting `client` claiming `entity` across async `commit`:
Step 0: entity.version = 0, client.knows_committed = false
Step 1: client sends async `commit` and transitions client.knows_committed := true (no reply wait)
Step 2: Async channel drops `commit` (`msg_dropped = true`); entity.version remains 0
Violated Invariant: `NoUnwarrantedRemoteClaim` (`client` claims commit while `entity.version == 0`)
Rule: K-SEND / K-CAST / K-REPLY (`choreographic-types` Y-6 & Y-8)
Remedy: Replace `cast`/`send` with synchronous `call -> commit -> reply` carrying correlation tag, monotonic version, and matching epoch under `K-REPLY`.
------------------------------------------------------------
FAILED Diagnostics=1 Exit=1
----

The process exits with status 1.

There is no `--allow-unwarranted` flag. The two legal repairs are:

. Change the protocol to `call` and claim after the reply. The claim is then the
Expand All @@ -186,21 +249,50 @@ K-CUT proof and is not accepted by `--strict`.

== A vacuity report

Suppose the postcondition were weakened to `result.digest != the empty binary`. The mutation
engine replaces the equality with the weak predicate; every non-empty digest
satisfies it, so the mutant survives:
A postcondition that every input satisfies leaves mutants alive. The fixture is
the one in `tests/quintessence/run_suite.sh` case 14 (a minimal module with
`@pre spec(x >= 10)` and `@post spec(result >= 0)`; not `Verisim.Spec`). Write it
to a scratch directory, then run the CLI on it:

[source,sh]
----
cat > weak_spec.ex <<'EX'
defmodule WeakSpec do
use Quintessence.Face, layer: :a
@pre spec(x >= 10)
@post spec(result >= 0)
def increment(x), do: x + 1
end
EX
----

Real output:

[source,text]
----
[mutate] 18 mutants, 17 caught, 1 survived (94.4%) < 95.0%
survivor: WeakenPostcondition
original: new_state.digest == Verisim.Spec.digest_of(payload)
weakened: new_state.digest != <<>>
goal: Verisim.Entity.handle_call/3.post.2
remedy: restore the equality; the weak predicate is type-only
[gate] FAIL
$ quintessence verify weak_spec.ex
Quintessence Verification Suite v0.1.0
------------------------------------------------------------
Module WeakSpec (layer :a) [weak_spec.ex]

Discharged Obligations (Z3 + CVC5 + Native Alethe Proof Kernel):
[QED] WeakSpec_increment_vc_post_0 z3=unsat cvc5=unsat alethe=QED (166 steps, 36 res)

ECHIDNA 6-Kind Mutation Report:
Caught 0/2 proof-relevant mutants (1 guard mutant(s) excluded) -> Score: 0.00% (gate >= 95.00%)
[SURVIVED] mut_1_op_swap (OperatorSwap) on `WeakSpec.increment/1`
[SURVIVED] mut_2_boundary_shift (BoundaryShift) on `WeakSpec.increment/1`

Diagnostics:
ERROR [Q-603] weak_spec.ex:1:1: Mutation score 0.00% (0/2 proof-relevant mutants caught) is below required 95.00% threshold
Rule: Vacuity Probe V3 (ECHIDNA 6-Kind Mutation Gate >= 95%)
Remedy: Tighten `@post`, `@invariant`, or protocol assertions to kill the surviving mutants listed in the mutation report.
------------------------------------------------------------
FAILED Diagnostics=1 Exit=5
----

The process exits with status 5.

If the precondition were `byte_size(payload) <= 1_048_576 and
byte_size(payload) > 1_048_576`, V1 would find no model and report `Q-601
VACUOUS` before any proof was attempted. A proof of a vacuous spec is not a
Expand Down
8 changes: 7 additions & 1 deletion examples/quintessence/mix/tasks/verify.ex
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
# SPDX-License-Identifier: MPL-2.0
# Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
#
# Reference sketch. Not compiled by this checkout. See README.adoc.
# DESIGN SKETCH. THIS FILE DOES NOT RUN.
# It calls Quintessence.Project, .Frontend, .Echidna, .Batch, .Mutation, .Gate
# and .Report, and none of them exists in this repository (nor does
# Quintessence.Doctor; only Quintessence.Error is defined, in lib/). It is not
# compiled by this checkout. Plan stage Q2 (docs/plan/ULTRAPLAN.adoc) is the
# stage that would make it real. Today the working entry point is the
# `quintessence` CLI (src/interface/ffi). See README.adoc.

defmodule Mix.Tasks.Verify do
@moduledoc """
Expand Down