Add reusable Fastly delivery-compression support #362

Description

Add reusable delivery-compression support to EdgeZero so applications can emit identity response bodies and delegate final client compression to a capable adapter. Extract the generic mechanism from Trusted Server's SSAT implementation rather than moving its ad-serving policy into EdgeZero.

Priority remains P3 / low. The extraction now has paired draft PRs for joint review. It does not enable compression globally or expand the application's existing staging acceptance criteria.

Related application work: IABTechLab/trusted-server#858, implemented in trusted-server#884. The original description referenced trusted-server#853.

Draft implementation

Framework draft #452 publishes exact commit 157f091e42b5c584898a5d0f8a0d3ddc3a4dd5ec. Consumer draft trusted-server#884 now uses that commit in six declarations and eight locked packages, with no registry/SDK version drift. The consumer head is 85a9b9184a67a9c827be9b59ead95a801fb07faf, rebased onto current main 51730bd.

The implementation adds unavailable-by-default request capability, checked identity-response marking and opaque intent in core, plus a pure Fastly finalizer shared by standard conversion and custom senders. Marking follows installation of final identity bytes or an identity-producing stream. It does not decode or inspect the body. Caller-owned body transformers must remove intent before re-encoding.

Trusted Server removes its local mechanism, retains one pending output plan and calls the shared finalizer after terminal effects/privacy without collecting lazy bodies. Eligibility, reader negotiation, codecs, auctions, templates and privacy remain application-owned.

Original framework 63c7b63 and consumer db27e738 integration checks and hosted CI passed. Local review R1 is now fixed in the current framework head with literal Vary expectations; the consumer pin is refreshed. Fresh 13-test framework, six-test consumer delivery and ten-test build-digest checks passed with formatting and SDK scoped Clippy. Review reruns also found local SDK and consumer Fastly aliases link-blocked; identical-tree hosted runs pass, but the local discrepancy remains unexplained. The paired PRs record exact evidence and updated CI. No deployment or live Fastly performance claim was made.

Both PRs stay draft. Merge EdgeZero first and move Trusted Server to a merged release before merge-ready. Real Fastly staging remains open below.

Historical findings

• The pre-extraction Trusted Server implementation defined a private DynamicDeliveryCompression response extension, an explicit adapter capability, and enforce_delivery_compression() in publisher.rs. Those mechanisms do not inherently depend on publishers, ads, consent, or auctions.
• Its Fastly entry point reasserts the selected compression headers after late response mutations, before both buffered and streaming sends. The generic encoding/framing responsibility fits a platform adapter better than application publisher code.
• EdgeZero already has portable response bodies and shared gzip/Brotli stream decoders. The inspected implementation does not yet expose an equivalent delivery-compression intent/capability contract. Decoder reuse is separate from this delivery feature.
• EdgeZero's standard Fastly from_core_response() collects stream chunks into a Fastly body before returning. Trusted Server instead uses its own lazy streaming sender. Adding compression handling only to that standard converter would miss Trusted Server's actual send path. Offer a reusable finalization operation or equivalent API that custom streaming senders can call without collecting the body.

Source inspection covered Trusted Server commit 5b663ee2d3041c750ac090493c756b2335cea75e and the relevant EdgeZero response code through upstream main 87456c708ffe9edf86b3a0f8b06dfd15d60b07b5. This was an architecture/source check, not a live compression or performance measurement.

Suggested change

1. Define a small portable delivery-compression intent in edgezero-core, such as a typed response extension, and expose adapter capabilities independently of cache-header selection. The application must know whether offload is supported before choosing to omit its own encoder.
2. Implement the Fastly behavior in edgezero-adapter-fastly: honor an explicitly selected intent on identity output, set X-Compress-Hint: on, maintain Vary: Accept-Encoding without losing other fields or duplicating wildcard coverage, and prevent stale encoding/length metadata at final send. Integrate after late response mutations and make the same operation available to custom buffered and streaming senders.
3. Keep codec negotiation and fallback explicit. Fastly dynamic delivery supports gzip/Brotli, not deflate. Do not promise unsupported encodings or overwrite the original reader preferences. Preserve unsupported-adapter behavior rather than silently dropping application compression.
4. Update Trusted Server to consume the released API and remove its duplicate generic marker, capability plumbing where replaced, and header-finalization helper. Keep its decision to use offload, its input/output codec separation, and its SSAT tests in Trusted Server.

Do not merely remove Content-Encoding from an already compressed body. The application must hand over identity bytes before requesting delivery offload. Do not move SSAT eligibility, consent, auction timing, HTML rewriting/EOF processing, template caching, or private/no-store policy into EdgeZero. Do not change origin encoding negotiation or add a publisher configuration flag as part of this extraction.

Done when

• Portable intent and capability APIs have a documented identity-body and unsupported-adapter contract.
• Fastly applies the delivery contract at final send for both buffered and custom streaming paths, after late header effects, without introducing whole-body collection.
• Generic tests cover marked versus unmarked responses, other adapters, framing/body agreement and repeated/wildcard Vary. Consumer tests cover application-owned reader negotiation/fallback.
• Applications without the explicit intent retain their existing response behavior, including already encoded responses.
• Trusted Server's integration removes the duplicate generic glue while preserving its SSAT eligibility, codec/finalizer behavior, template neutrality, and privacy regressions. Coordinate this part through trusted-server#858.
• Real Fastly negotiation is verified separately from local/Viceroy guest-output tests. Do not claim wire-size or vCPU improvements without fresh platform measurements.

Affected area

Core response contracts, Adapter -- Fastly, adapter contract tests, and documentation. Axum, Cloudflare, and Spin need only the explicit unsupported/default contract unless a separately verified delivery facility is added later.

CI smoke fixture fix

Framework 157f091e42b5c584898a5d0f8a0d3ddc3a4dd5ec fixes the failed reusable HTTP smoke probe. Delivery contracts had passed 12/12 and Fastly library tests 103/103; uncontrolled origin EOF could race downstream receipt.

The post-commit origin now waits for the existing client first-body-byte signal before closing short. Plain immediate aborts still serve pre-commit fault tests. Status 200, exact partial, interrupted-response type and matching guest error remain mandatory. No sleeps, retries, SDK stubs, trap overrides or runtime changes were added. Cleanup releases waiters after failed probes.

A regression failed before the fix. Final checks pass 2 core + 27 harness tests, formatting, strict Clippy and the full pinned-Viceroy Fastly smoke. All three custom variants observed the required bytes and correlated error. Two independent source investigations/reviews found no blocking issue.

Consumer 85a9b9184a67a9c827be9b59ead95a801fb07faf refreshes all six declarations/eight lock sources. Six delivery tests, ten build-digest tests and formatting pass, with no registry/SDK drift. GitHub's test workflow now passes at 157f091e, including the previously failing Fastly job. Retained CI artifacts confirm 200/partial and matching guest errors for all three custom variants. Both PRs remain draft; consumer CI runs separately.

This does not resolve the separate local debug/WASM loader discrepancy or prove live Fastly negotiation/performance.