fix(engine): verify SLSA build provenance via sigstore-verify (ADR-0020)

docs/adr/0020-signature-continuity.md

@@ -288,7 +288,8 @@ new egress path.
    default-on, not opt-in — it was never a new egress destination, so gating it behind a
    flag was detection proliferation, not an egress control.)*
 
-**Known limitation (DECISION NEEDED — recorded for the architect).** The pinned
+**Known limitation (DECISION NEEDED — recorded for the architect).** *(RESOLVED — see the
+"provenance is verified via `sigstore-verify`" addendum at the end of this file.)* The pinned
 `sigstore` crate (0.14) verifies DSSE bundle referrers only for the cosign `sign/v1`
 predicate — `from_sigstore_bundle` rejects any other predicate type, so a **SLSA
 provenance** attestation is currently *not surfaced* by `trusted_signature_layers` end to
@@ -298,7 +299,10 @@ the `Verified` path is real, correct code that activates the moment the verifier
 SLSA layer; until then the production observer yields `Absent`/`Checking` — the safe,
 honest degradation. Closing the gap (compose sigstore's lower-level DSSE + Fulcio + Rekor
 primitives, or an upgraded `sigstore` release) is a follow-up that does not change this
-addendum's contract.
+addendum's contract. *(The optimism above was half-right: the pipeline/parser were correct,
+but the `Verified` path did NOT just activate — the resolving addendum documents why both of
+the `sigstore` crate's verify paths are dead ends for GitHub attestations, and what replaced
+them.)*
 
 ## Addenda (— the rendered "if enforced" is CONTINUITY, not keyless-identity)
 
@@ -422,6 +426,10 @@ not egress, and is retired.
    contacted; the only change is that the SLSA in-toto/DSSE layer `trusted_signature_layers` already
    fetched is now also classified. So flipping the default costs nothing egress-wise: any image the
    signing sweep already observes is now also read for provenance off the identical bytes.
+   *(Superseded in mechanism by the "provenance is verified via `sigstore-verify`" addendum below:
+   provenance no longer rides `fetch_layers` — it makes its OWN referrer fetch to the same registry.
+   The zero-egress conclusion still holds — same registry host, no new destination, offline
+   verification — but it is now a **separate** round trip, not the shared one described here.)*
 2. **The provenance sweep is now built unconditionally** (`build_provenance_scanner`, mirroring
    `build_signing_observer` exactly), bounded by the same `PROTECTOR_MAX_IMAGES` cap and TTL cache as
    before. `PROTECTOR_PROVENANCE_ENABLE` is removed; there is no replacement flag and no migration
@@ -432,3 +440,59 @@ not egress, and is retired.
    admission. The genuinely opt-in Rekor lane (`PROTECTOR_REKOR_ENABLE`) is UNCHANGED by this
    addendum — it remains gated because it is a real second egress destination, the case this
    principle's "egress" carve-out exists for.
+
+## Addenda (— provenance is verified via `sigstore-verify`; the `sigstore` crate cannot)
+
+The "Known limitation" note above assumed the `Verified` path would light up the moment the
+verifier surfaced a SLSA layer. It did not, and observation confirmed it in production: **every**
+image — including protector's own, which ships a real `actions/attest-build-provenance` attestation
+— read `Absent`, so the inventory's provenance column was uniformly blank. Root-causing it turned up
+**two** independent dead ends in the pinned `sigstore` crate (0.14, the newest release), not one:
+
+1. **The predicate guard (already known).** `trusted_signature_layers` fetches the SLSA bundle as an
+   OCI referrer, then `from_sigstore_bundle` rejects it — it hardcodes the cosign `sign/v1` predicate
+   and errors on `https://slsa.dev/provenance/v1`. So the shared signing fetch can never yield a SLSA
+   layer; the `Verified` path was unreachable, not merely dormant.
+2. **The DSSE verifier is also broken (newly found).** The crate's OTHER public path,
+   `bundle::verify::Verifier`, verifies a bundle directly — but its **offline** DSSE check recomputes
+   the Rekor `envelopeHash` from a proto round-trip of the envelope, which never equals the hash Rekor
+   stored over the originally-submitted bytes (a canonicalization bug), so it always fails; its
+   **online** DSSE path is an unimplemented stub. Both were confirmed against protector's live agent
+   image (`envelopeHash mismatch`). No published `sigstore` version fixes either, and the lower-level
+   Fulcio-chain / SCT primitives needed to hand-roll a correct verifier are `pub(crate)`.
+
+**Decision — add the `sigstore-verify` crate (prefix-dev) for the provenance verify step only.** It
+is a maintained, published verifier purpose-built for GitHub artifact attestation: it handles the
+DSSE Rekor-entry consistency correctly (matches the payload hash + signature, not the broken envelope
+recompute) and ships a built-in `SIGSTORE_PRODUCTION_TRUSTED_ROOT`. The signing axis stays on the
+`sigstore` crate untouched; only provenance uses the new stack. This carries a second sigstore
+dependency stack, accepted deliberately as the alternative to vendoring a patched `sigstore` fork of
+security-critical verification code.
+
+Mechanics ([`policies::signature::provenance_observer`]):
+
+1. **Its own referrer fetch, same registry.** Because the shared `fetch_layers` drops the SLSA layer
+   (dead end 1), provenance now fetches the image's OCI referrers directly via `oci-client`, selects
+   the `slsa.dev/provenance/*` bundle(s), and verifies each with `sigstore-verify`. This is a
+   **separate** round trip from the signing fetch (superseding the "one round trip, two postures"
+   claim in the default-ON addendum above), but to the **same registry host** — no new egress
+   destination, and the ADR-0015 zero-egress default holds. Verification is fully **offline**
+   (built-in trust root; the bundle's own embedded inclusion proof + checkpoint), so it adds no
+   transparency-log call.
+2. **The verified facts feed the unchanged classifier.** A verified bundle's `(predicate, keyless?)`
+   is turned into the same `ProvenanceFacts` the pure `classify_provenance` + `parse_slsa_predicate`
+   already consume, so the four-state precedence (verified / unverifiable / absent / checking), the
+   TOFU baseline, the drift finding, and the render are all unchanged — only the *source* of the
+   facts moved. Observation stays permissive on identity (no configured signer — the Fulcio/Rekor
+   chain is the anchor; the builder is learned, not gated), matching the signing sweep.
+3. **Referrers-unsupported is `Absent`, not `Checking`.** The manifest fetch runs first and proves
+   the image is reachable + authorized; a subsequent failure to LIST referrers therefore means the
+   registry does not support the OCI referrers API (common on mirrors, which return an
+   unparseable/empty referrers tag), which is the calm `Absent` — never a perpetual `Checking` that
+   would leave every mirrored base image stuck showing the transient glyph. Only an unreachable
+   *image* is `Checking`.
+4. **Guarded by a live end-to-end test.** The failure that hid here for so long was invisible to unit
+   tests — the pipeline was green on synthetic fixtures while dead in production. An `#[ignore]`d
+   integration test (`engine/tests/provenance_live.rs`) now verifies the whole chain against a real
+   attested image (`Verified`, right source + builder) and a real unattested mirror image (`Absent`),
+   so a future regression to the silent-blank state is catchable with one command.

engine/Cargo.toml

@@ -15,6 +15,7 @@ jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
 k8s-openapi = { version = "0.28.0", features = ["v1_33"] }
 kube = { version = "4.0.0", features = ["runtime", "client", "config", "derive", "admission"] }
 maud = "0.27.0"
+oci-client = { version = "0.17.0", default-features = false, features = ["rustls-tls"] }
 opentelemetry = "0.32.0"
 opentelemetry-otlp = { version = "0.32.0", default-features = false, features = ["http-proto", "reqwest-blocking-client", "reqwest-rustls", "trace", "metrics"] }
 opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
@@ -28,6 +29,7 @@ serde = { version = "1.0.228", features = ["derive"] }
 serde_json = "1.0.150"
 sha2 = "0.11.0"
 sigstore = { version = "0.14.0", default-features = false, features = ["cosign", "sigstore-trust-root", "rustls-tls"] }
+sigstore-verify = "0.11.0"
 subtle = "2.6.1"
 thiserror = "2.0.18"
 tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "net", "time", "signal", "sync", "process"] }

engine/src/policies/signature/auth.rs

@@ -88,6 +88,17 @@ impl RegistryAuth {
             None => Auth::Anonymous,
         }
     }
+
+    /// The resolved `(username, password)` for `image`, or `None` for `Anonymous` — the same
+    /// precedence as [`for_image`](Self::for_image), but auth-library-agnostic so the provenance
+    /// referrer fetch (which drives oci-client directly, not sigstore's wrapper) can authenticate
+    /// private images identically without this module depending on either client's auth enum.
+    pub(super) fn basic_for_image(&self, image: &str) -> Option<(String, String)> {
+        if let Some((user, pass)) = &self.env_override {
+            return Some((user.clone(), pass.clone()));
+        }
+        self.entries.get(&image_registry_key(image)).cloned()
+    }
 }
 
 /// Parse the whole dockerconfigjson `auths` map into a `host key → (user, pass)` table. Each entry

engine/src/policies/signature/cosign.rs

@@ -31,10 +31,7 @@ use tokio::sync::OnceCell;
 use super::SignatureChecker;
 use super::auth::RegistryAuth;
 use super::posture::{SignatureObserver, Signer, SigningPosture};
-use super::provenance::{
-    ProvenanceFacts, ProvenanceObserver, ProvenancePosture, classify_provenance,
-    is_slsa_predicate_type,
-};
+use super::provenance::{ProvenanceObserver, ProvenancePosture};
 
 /// The production [`SignatureChecker`] / [`SignatureObserver`]: verifies keyless cosign
 /// signatures with sigstore-rs against the public-good sigstore TUF root.
@@ -244,13 +241,27 @@ impl SignatureObserver for CosignChecker {
 #[async_trait]
 impl ProvenanceObserver for CosignChecker {
     async fn observe_provenance(&self, image: &str) -> ProvenancePosture {
-        // Reuse the SAME sanctioned registry/Rekor round trip as signature verification (ADR-0015):
-        // `trusted_signature_layers` already returns any attached in-toto/DSSE attestation layer.
-        // No second verifier, no new egress path. An infra error is the transient "checking" state.
-        match self.fetch_layers(image).await {
-            Ok(layers) => classify_provenance(&provenance_facts(&layers)),
-            Err(err) => {
-                tracing::debug!(%image, error = %err, "build-provenance: registry/Rekor unreachable — checking");
+        // NOT `trusted_signature_layers`: the `sigstore` crate can't observe SLSA provenance at all
+        // (it drops non-cosign predicates on the referrer path, and its DSSE bundle verifier has a
+        // Rekor envelope-hash bug), so that call can only ever yield `Absent`. Fetch + verify the
+        // provenance referrer directly instead (super::provenance_observer, via `sigstore-verify`).
+        // Same sanctioned registry egress (ADR-0015), offline transparency-log check.
+        //
+        // Bound the whole registry round trip so a slow/hung registry can't stall the sweep — the
+        // same budget the signing fetch uses. A timeout or infra error is the transient "checking".
+        match tokio::time::timeout(
+            self.verify_timeout,
+            super::provenance_observer::observe_provenance(&self.auth, image),
+        )
+        .await
+        {
+            Ok(Ok(posture)) => posture,
+            Ok(Err(err)) => {
+                tracing::debug!(%image, error = %err, "build-provenance: registry unreachable — checking");
+                ProvenancePosture::Checking
+            }
+            Err(_) => {
+                tracing::debug!(%image, "build-provenance: fetch timed out — checking");
                 ProvenancePosture::Checking
             }
         }
@@ -358,63 +369,6 @@ pub(super) fn classify_facts(facts: &[LayerFacts]) -> SigningPosture {
     }
 }
 
-/// Project the SLSA build-provenance facts off fetched layers: one [`ProvenanceFacts`]
-/// per layer whose in-toto predicate type is a SLSA provenance type (a plain signature layer never
-/// produces one). `keyless_verified` mirrors the signing axis — sigstore populates
-/// `certificate_signature` ONLY when the attestation's cert chained to the trusted Fulcio root AND
-/// its Rekor bundle verified, so an attacker-attached, unverifiable attestation comes back with
-/// `keyless_verified: false` (which classifies as [`Unverifiable`](ProvenancePosture::Unverifiable),
-/// never trusted). The predicate is decoded from the layer's DSSE PAE payload (the in-toto
-/// statement), which the classifier reads for the source repo + builder identity.
-pub(super) fn provenance_facts(layers: &[SignatureLayer]) -> Vec<ProvenanceFacts> {
-    layers
-        .iter()
-        .filter(|layer| is_slsa_predicate_type(&layer.simple_signing.critical.type_name))
-        .map(|layer| ProvenanceFacts {
-            predicate_type: layer.simple_signing.critical.type_name.clone(),
-            predicate: predicate_from_pae(&layer.raw_data),
-            keyless_verified: layer.certificate_signature.is_some(),
-        })
-        .collect()
-}
-
-/// Decode the SLSA `predicate` object out of a layer's DSSE PAE-encoded `raw_data`. sigstore stores
-/// the DSSE Pre-Authentication-Encoding (`DSSEv1 <len> <payloadType> <len> <payload>`) in
-/// `raw_data`; the `<payload>` is the in-toto Statement JSON, whose `predicate` field carries the
-/// SLSA provenance. Returns `None` when the PAE is malformed or the payload isn't the expected
-/// in-toto shape — a present-but-opaque attestation (never a fabricated predicate).
-fn predicate_from_pae(raw_data: &[u8]) -> Option<serde_json::Value> {
-    let payload = pae_payload(raw_data)?;
-    let statement: serde_json::Value = serde_json::from_slice(payload).ok()?;
-    statement.get("predicate").cloned()
-}
-
-/// Extract the `<payload>` bytes from a DSSE PAE header
-/// (`DSSEv1 <len(type)> <type> <len(payload)> <payload>`, all lengths ASCII decimal). The type and
-/// payload can themselves contain spaces, so this reads by the declared lengths rather than
-/// splitting on whitespace. Returns `None` on any malformed field.
-fn pae_payload(raw: &[u8]) -> Option<&[u8]> {
-    let rest = raw.strip_prefix(b"DSSEv1 ")?;
-    // <len(type)> up to the next space.
-    let sp = rest.iter().position(|&b| b == b' ')?;
-    let type_len: usize = std::str::from_utf8(&rest[..sp]).ok()?.parse().ok()?;
-    let rest = &rest[sp + 1..];
-    // Skip the type itself (type_len bytes) then a single space.
-    let rest = rest.get(type_len..)?;
-    let rest = rest.strip_prefix(b" ")?;
-    // <len(payload)> up to the next space.
-    let sp = rest.iter().position(|&b| b == b' ')?;
-    let payload_len: usize = std::str::from_utf8(&rest[..sp]).ok()?.parse().ok()?;
-    let payload = rest.get(sp + 1..)?;
-    // The declared length must match exactly what remains — a defensive check against a truncated
-    // or over-long PAE.
-    if payload.len() == payload_len {
-        Some(payload)
-    } else {
-        None
-    }
-}
-
 /// The per-image verification budget was exhausted before the registry/Rekor round trip returned
 /// . A typed error (rather than a formatted string) so [`classify_checking_reason`] can
 /// tell a spent timeout apart from a reachability failure via `downcast_ref`, robust to message
@@ -452,59 +406,6 @@ impl VerificationConstraint for IdentityVerifier {
 #[path = "cosign_tests.rs"]
 mod cosign_tests;
 
-#[cfg(test)]
-mod provenance_pae_tests {
-    use super::*;
-
-    /// Build a DSSE PAE the way sigstore's `compute_pae` does, so the extractor is tested against
-    /// the exact on-the-wire shape (`DSSEv1 <len> <type> <len> <payload>`).
-    fn pae(payload_type: &str, payload: &[u8]) -> Vec<u8> {
-        let mut out = format!(
-            "DSSEv1 {} {} {} ",
-            payload_type.len(),
-            payload_type,
-            payload.len()
-        )
-        .into_bytes();
-        out.extend_from_slice(payload);
-        out
-    }
-
-    #[test]
-    fn extracts_predicate_from_a_well_formed_pae() {
-        let statement = br#"{"_type":"https://in-toto.io/Statement/v1","predicateType":"https://slsa.dev/provenance/v1","predicate":{"runDetails":{"builder":{"id":"https://github.com/org/app/.github/workflows/x.yml@refs/heads/main"}}}}"#;
-        let raw = pae("application/vnd.in-toto+json", statement);
-        let predicate = predicate_from_pae(&raw).expect("predicate decoded");
-        assert_eq!(
-            predicate.pointer("/runDetails/builder/id").unwrap(),
-            "https://github.com/org/app/.github/workflows/x.yml@refs/heads/main"
-        );
-    }
-
-    #[test]
-    fn payload_with_embedded_spaces_is_read_by_length() {
-        // The in-toto statement JSON can contain spaces; the extractor must read by the declared
-        // length, not split on whitespace.
-        let statement = br#"{ "predicate": { "buildType": "a b c" } }"#;
-        let raw = pae("application/vnd.in-toto+json", statement);
-        let predicate = predicate_from_pae(&raw).expect("predicate decoded");
-        assert_eq!(predicate.pointer("/buildType").unwrap(), "a b c");
-    }
-
-    #[test]
-    fn malformed_pae_yields_none() {
-        assert!(pae_payload(b"not a dsse pae").is_none());
-        assert!(predicate_from_pae(b"garbage").is_none());
-    }
-
-    #[test]
-    fn a_truncated_payload_is_rejected() {
-        // Declared length longer than the actual bytes must not silently succeed.
-        let raw = b"DSSEv1 4 json 999 {}".to_vec();
-        assert!(pae_payload(&raw).is_none());
-    }
-}
-
 #[cfg(test)]
 mod checking_reason_tests {
     use super::*;

engine/src/policies/signature/mod.rs

@@ -14,6 +14,7 @@ pub mod continuity;
 mod cosign;
 pub mod posture;
 pub mod provenance;
+mod provenance_observer;
 pub mod rekor;
 pub mod tuf_tmpdir;