Skip to content

feat(types): keep wire bytes on parsed URLs, datetimes, UUIDs and emails so digests round-trip - #1617

Merged
bokelley merged 5 commits into
mainfrom
conductor/1610-digest-round-trip-anyurl-awaredatetime
Oct 26, 2026
Merged

bokelley merged 5 commits into
mainfrom
conductor/1610-digest-round-trip-anyurl-awaredatetime

Conversation

@bokelley

@bokelley bokelley commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1610.

The defect

core/audience-evidence.json defines content_digest as SHA-256 over the RFC 8785 (JCS) form of the entry as declared. The generated AudienceEvidence parsed baseline.system, methodology_url and attestation origins as pydantic AnyUrl, which re-serializes a bare origin with a trailing slash, and last_updated as AwareDatetime, which rewrites +00:00 as Z and drops .000. A parsed-then-dumped entry no longer matched its own digest: a seller validating with the SDK published digests buyers could not verify, and a buyer parsing with the SDK could not verify a valid one.

Root cause

The SDK had no invariant that a schema-valid wire document survives parse → dump unchanged, and several mechanisms rewrite documents silently:

  • The generator maps each JSON Schema format to the pydantic type that parses it best (AnyUrl, AwareDatetime, UUID, EmailStr), and those types serialize the value they parsed, not the text they were given.
  • The runtime adds its own rewrites: model_dump emits every field, so a schema default the generator wrote into the field appears though the document omitted it; exclude_none (the default) drops an explicit null the schema admits; by_alias is off, so $schema dumps as field_schema; extra='ignore' drops a property the pinned model does not declare.

Each is reasonable for an application type and wrong for an object whose bytes are an identity. Four earlier fixes were point patches of this one gap: #1384 typed one field (agent_url) as a wire string because an option ID hashed it; #1449 (item 3) reported that dumping a validated ProductFormatDeclaration "adds every branch default to the published format"; #1587 made nested models leave unset fields off from any parent; #1575 kept a ContextObject's null keys. #1610 is the next instance.

The invariant that prevents a recurrence: a value the model did not change is dumped as it was received. Two mechanisms carry it (below), and a conformance suite measures it for every bundle schema, so a regenerated schema or a new normalizing type cannot reintroduce the class silently. The full write-up is in docs/validation-contract.md ("Wire round trip").

Sweep

tests/conformance/_round_trip.py takes schema-valid documents for every bundle schema that has a model (the bundle's examples, completed over a synthesized base where they are fragments; a synthesized document per schema; the differential fixtures; a checked-in corpus for the digest-covered schemas the instantiator cannot derive), derives variants exercising each form a normalizing parser rewrites (bare origin, mixed-case host and scheme, default port, percent-encoding; +00:00 with .000, a negative offset, .5, nanoseconds, lower-case t/z; upper-case UUID; mixed-case email), runs each through Model.model_validate_json(doc).model_dump(mode="json") with the SDK's defaults, and compares to the input under JCS. 6,085 documents graded across 999 base documents and 766 schemas. python -m tests.conformance._round_trip --variants prints the report.

Distinct schemas affected per class, before and after this PR, under the default dump:

class what happens before after matters for
url_normalized AnyUrl re-serializes (trailing slash, host/scheme case, default port) 249 schemas; 7 digest-covered 0 every digest-covered object with a URL (baseline.system, issuer origin, evaluated_by, …)
datetime_reformatted AwareDatetime writes Z, drops .000, truncates nanoseconds 249 schemas; 6 digest-covered 0 last_updated, evaluated_at, expires_at, validity_hint, …
email_normalized EmailStr lower-cases the domain 56 schemas; 3 digest-covered 0 provider.data_subject_contestation.email via brand-ref
uuid_normalized UUID lower-cases 3 schemas 0 none digest-covered at this pin
default_emitted a schema default the document omitted is materialized 60 schemas, 143 fields 60 / 143 digest-covered: commercial_terms → terms_digest (6 defaults reachable), package-format-snapshot (267), check-governance-request (7). Filed #1613.
field_dropped extra='ignore' / a union arm narrower than the schema drops a property 5 schemas, 120 fields 5 / 120 no digest-covered schema at this pin; a forward-compat hazard. Filed #1614.
alias_emitted model_dump() writes pass_, field_schema, from_, media_buy 14 schemas 14 not digest-covered; wire-invalid output for anyone dumping without by_alias. Filed #1615.
null_dropped exclude_none drops an admitted explicit null 5 schemas 5 targeting_overlay.signal_targeting_groups; not digest-covered. Filed #1615.

Under model_dump_as_received() (below) the residue is field_dropped only. Digest- and echo-covered schemas: no difference of any class on that path, in every form. context/ext round-trip their nested nulls on both paths.

Also found: 40 schema-valid documents (core/mcp-webhook-payload.json's four bundle examples, core/async-response-data.json) make the model raise TypeError from a min_length applied to a Losses model instead of validating. Filed #1616; the suite records refusing schemas as a shrink-only set.

Inventory of digest- and signature-covered objects (pin 3.2.4)

Read off the bundle descriptions; listed in _round_trip.DIGEST_COVERED with the digest each declares:

core/audience-evidence.json (content_digest minus content_digest, attestation_refs), core/canonical-audience-evidence.json, core/audience-evidence-pin.json, core/audience-evidence-selection.json, core/canonical-audience-evidence-selection.json (the same digest, repeated and matched), core/rights-constraint.json (minus content_digest, attestation_refs, verification_url), core/rights-attestation-evaluation.json, core/creative-representation-set.json (revision_content_digest), core/representation-selection.json + core/creative-manifest.json (selected_output_digest over the derived manifest), media-buy/acceptance-policy-profile.json (content_digest), core/canonical-proposal.json + media-buy/commercial-terms.json (terms_digest), core/package-format-snapshot.json (tracker_execution_contract_digest, product_snapshot_digest), core/attestation-reference.json + core/attestation-evaluation.json (reference_digest), governance/check-governance-request.json (statement_digest), governance/check-governance-response.json (runtime_attestation_binding_digest), governance/get-plan-audit-logs-response.json (plan_hash). The reporting canonical_content_digest family is computed by adcp.reporting.canonical_json over logical rows, not over the carrying model's fields, so it is not affected. Echo-covered: core/context.json, core/ext.json.

Normalizing fields inside those subtrees (from the schema graph): audience-evidence reaches 44 uri, 23 date-time, 4 email, 2 date fields; rights-constraint 39/24/4; creative-representation-set 57/39/5; check-governance-request 69/46/6. date fields do not rewrite for schema-valid input (YYYY-MM-DD is the only form). Floats do not rewrite under JCS (1.0 and 1 canonicalize alike); enums dump their value.

The fix

1. Wire-preserving scalar types, applied by a generator rule. src/adcp/types/_wire.py defines WireDatetime(datetime), WireAnyUrl(AnyUrl), WireUuid(UUID) and WireEmail (a validated str). The text is kept only when it passes the strict wire check the bundled schema validator applies (adcp._formats, shared with schema_loader): RFC 3339 for a date-time, RFC 3986 syntax for a URI whose text names the host the parser reports, the hyphenated 8-4-4-4-12 form for a UUID, and for an email the trimmed address itself (case-insensitively equal to the normalized one). A string pydantic's lenient parser accepts but the schema rejects — "2026-10-01 00:00:00+00:00", "...+0000", "...T00:00Z", an epoch string, " https://x.example ", "https:x.example", "https://x.example/a b", a braced or urn:uuid: UUID, "Bob <bob@example.com>" — is normalized exactly as main did, so nothing invalid is ever echoed. str(url) and .host cannot disagree: "https://good.example\\@evil.example" (a backslash the WHATWG parser reads as a slash) falls back to https://good.example/@evil.example, as on main; a strict text whose host differs from the parsed host is not kept either. Each validates exactly as the pydantic type did (same core schema, same errors), keeps the attribute API (.year, .tzinfo, .host, .path, .hex), compares and hashes as the same value (WireAnyUrl == AnyUrl holds both ways), survives deepcopy/pickle/model_copy(deep=True)/revalidation, advertises the same JSON Schema (format: date-time / uri / uuid / email), and in JSON mode serializes the text it was parsed from — including under serialize_as_any=True, which AdCPBaseModel.model_dump passes and which otherwise bypasses the field serializer (a class-level __pydantic_serializer__, the same device AnyUrl uses).

scripts/post_generate_fixes.py::preserve_wire_text_on_normalizing_fields renames the identifier tokens AwareDatetime/AnyUrl/UUID/EmailStr in every generated module (tokenizer-based, so a description that says "UUID v4" is untouched) and rewrites the imports; the effect manifest records it. 301 generated files change, every change a type rename or import line (checked: no renumber churn). make validate-generated passes.

Behaviour change, stated plainly. For every parsed format: uri, date-time, uuid and email field in the ~300 generated modules, JSON output and str() now carry the spelling received instead of pydantic's normalized one (https://x.example stays without its trailing slash; +00:00 is not rewritten as Z; an upper-case UUID or email domain keeps its case). type(x) is datetime / is AnyUrl / is UUID is now false for a parsed value while isinstance still holds. A cache or map keyed by str(url) sees a different spelling for the same URL than before (https://x.example vs https://x.example/); key by url.unicode_string() or url.host for a normalized key. Values built from Python objects are unaffected.

Non-breaking for 9.x. At runtime every attribute is still a datetime/AnyUrl/UUID. To a type checker the names re-exported from adcp.types.base are AwareDatetime/AnyUrl/UUID (a TYPE_CHECKING alias), so an adopter's assert_type, assignment, or constructor call keeps its verdict — the adopter type-check suite (make typecheck-all, mypy strict + pyright) passes unchanged. wire_text(value) is the typed way to read the received text; .wire is a runtime property.

Constructed values (a Python datetime passed in rather than parsed) serialize exactly as before: RFC 3339 with Z for UTC and ±HH:MM otherwise, fractional seconds only when non-zero and then six digits; pydantic's normalized URL string; lower-case hyphenated UUID. A value derived from a parsed one (replace(), astimezone(), arithmetic) drops the wire text and takes that form.

Scope. The rule covers the generated AdCP schema types (adcp.types.domains). adcp.types.registry, the OpenAPI-generated registry client, keeps pydantic's AnyUrl/AwareDatetime; no registry object is digest-covered, and the docs say so.

Every format field, not only digest-covered subtrees. Recommended and implemented. A shared class (AttestationReference, Provenance, Targeting) sits inside digested and undigested objects alike, so a per-subtree rule would have to split classes; and the invariant worth having is that the SDK never rewrites a wire value it did not change. The blast radius is one behaviour: a parsed URL/datetime/UUID/email dumps as received instead of normalized. No SDK code depended on the normalized spelling (the suite and the full test run show none).

2. An explicit "dump as received" path. AdCPBaseModel.model_dump_as_received() = model_dump(mode="json", exclude_unset=True, exclude_none=False, by_alias=True): leaves off every field the document did not set (where a generated default would otherwise appear), keeps an admitted explicit null, writes wire names. This is the verification path for a digest until #1613 changes the default dump. For a model built in Python it emits exactly the fields the caller set.

Close-out for #1610. With this PR, content_digest(model.model_dump(mode="json")) == model.content_digest holds for AudienceEvidence and for every digest-covered schema in the pin except two: terms_digest on core/canonical-proposal.json and media-buy/commercial-terms.json still does not round-trip under the default dump, because the generator materializes targeting_overlay.age_restriction.verification_required and five other defaults inside commercial_terms; only model_dump_as_received() recovers those bytes until #1613 lands. The conformance suite names both schemas explicitly.

Why the default dump was not changed for defaults in 9.x. Leaving unset defaults off every model_dump() is wire-equivalent for a schema-applying reader but visible to any code that indexes a dumped dict (payload["sandbox"]), including a counterparty that does not apply schema defaults. That is a 10.0 decision; the evidence and two designs are in #1613. The suite declares the two digest-covered schemas where the default dump still materializes a default (canonical-proposal, commercial-terms) as a shrink-only set, so the gap is visible and a third cannot join it.

3. AudienceEvidence exported from adcp.types, with AudienceEvidencePin, AudienceEvidenceSelection, CanonicalAudienceEvidence, CanonicalAudienceEvidenceSelection. The bare name resolves to core.audience_evidence (the capabilities-response object of the same name is only reachable by its domain path, as before); public API snapshot regenerated.

Tests

  • tests/test_wire_types.py: the issue's repro (content_digest(model.model_dump(mode="json")) == model.content_digest, both model_validate and model_validate_json); every RFC 3339 and URL spelling round-trips; attribute API, equality, hashing, copies, constructed-value forms, validation unchanged, JSON Schema unchanged; model_dump_as_received on a CanonicalProposal whose terms_digest verifies only on that path; an admitted null and a wire alias.
  • tests/conformance/test_wire_round_trip.py (795 cases, ~90 s): every bound schema has a document; every bound object round-trips byte-for-byte as received, in every form; the default dump never rewrites a bound object; every other schema's differences are in ACCEPTED (with reason and issue) — a rewritten URL/datetime/UUID/email/value/shape fails anywhere; every accepted class still occurs (shrink-only); the as-received residue is dropped-unknown-fields only; refusing schemas are declared (shrink-only).
  • tests/fixtures/round_trip/corpus.json: hand-written, validator-confirmed documents for the seven digest-covered schemas the instantiator cannot derive, plus context/ext with nested nulls.

Existing tests touched by the behaviour change (each asserted the old normalization): test_type_aliases and test_governance_authorization pinned str(agent_url) == "https://agent.example.com/" for a model built from "https://agent.example.com"; they now expect the caller's bytes. test_code_generation pins the committed annotation spelling (WireDatetime | None). test_composability_invariant reviews the new AudienceEvidence public/capabilities name split. test_reporting_adjustment_evidence asserted that a typed dump differs from the raw canonical JSON; it now shows that a typed view parsed from a differently spelled document hashes the same raw bytes. One SDK producer changed: build_reporting_adjustment_receipt wrote observed_at with isoformat() and relied on the model to rewrite +00:00 as Z; it now writes the Z form itself (rfc3339), so receipts keep their shape. canonical_creative._has_boundary_serialization_metadata treats the wire types as plain (their python-mode dump is the stored value), so the direct boundary scan still applies.

Performance (median of 7–9 runs, same machine; main measured in a worktree). A 2,000-product GetProductsResponse with 24k URLs and 18k datetimes:

main this PR
model_validate_json 5.2–5.5 s 5.2–5.3 s
model_dump_json 3.0–3.1 s 2.8–2.9 s
model_dump(mode="json") 3.1–3.2 s 3.0 s
model_dump() 3.2–3.3 s 3.1–3.3 s

A scalar-only model (2,000 rows × 12 URLs + 9 datetimes, nothing else), which isolates the per-value cost:

main first push this push
model_validate_json 42 ms 104 ms 108–114 ms
model_dump_json 27 ms 25 ms 22–23 ms
model_dump(mode="json") 30 ms 27 ms 24–25 ms
model_dump() 17 ms 31 ms 22–23 ms

What changed: the datetime and UUID serializers run in JSON mode only (a python-mode dump returns the value with no Python call, as pydantic does for a datetime); the datetime is built from its packed pickle state in one C call instead of nine arguments through an overriding __new__ (3× cheaper); the URL validator is one Python layer around the core URL parser instead of AnyUrl's wrap plus another; and the strict URI check takes a fast regex subset of RFC 3986 for the common case, falling back to the full validator only for userinfo, IP literals, non-ASCII or escapes. Validation of a scalar-only model stays ~2.6× main's 42 ms (≈1.6 µs per value: the strict check plus one Python call), which is the floor for a Python-level validator; on the real response it is within run-to-run noise. The 1.4–1.7× model_dump_json slowdown the review measured does not reproduce here on either corpus; if its corpus differs from these, I will rerun it.

make lint, make typecheck-all, make validate-generated pass. make test: 17,135 passed, 2,684 skipped; the 12 that failed on the first full run are the updates above plus one A2A integration test that passed on rerun, and all pass now. Pre-commit could not build its python3.11 environment in this sandbox, so the commit skipped hooks; black, ruff, mypy and bandit were run directly.

Issues filed

🤖 Generated with Claude Code

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 25, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean, well-tested fix for #1610 with no blocking findings.

What I checked:

  • The high_risk flag is true only because two existing type files were modified (src/adcp/types/__init__.py, src/adcp/types/_eager.py), both (modified) — not deleted. No medium-or-higher findings on either, so the modification is presumed safe (no escalation on the flag alone).
  • gated_paths: false, so despite review_decision: REVIEW_REQUIRED, the hard path gate (row 2) does not apply.
  • No author no-auto-approve team match.
  • No prior decision (first review).

Findings: Reviewer reports zero Critical/High/Medium findings. Wire-preserving scalar types make parsed URL/datetime/UUID/email dump as received for digest round-tripping; the 301 generated-file changes are mechanical tokenizer-based type renames (AwareDatetime→WireDatetime), not hand-edits — codegen churn, not a semantic schema delta. Repo floors hold: TYPE_CHECKING alias keeps the type-checker surface unchanged, import layering respected, no CI gates disabled, no new wire vocabulary.

One non-blocking observation (WireAnyUrl at L165 relies on inherited AnyUrl serializer rather than pinning its own) is test-guarded by test_every_url_spelling_survives, so any regression fails CI rather than shipping silently. Not a finding.

Decision table: rows 1–8 do not fire (no critical/high, gated_paths false, no deletions, no medium findings, no team gate, no prior escalation). Falls through to row 9 → approve.

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 25, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — subsequent pass with no findings; test hygiene only.

This is a follow-up to a prior approve. The delta since then is two conformance test files:

  • _round_trip.py adds a side-effect import adcp.types._eager (correctly noqa: F401) so import-time patchers run before grading, removing test-order non-determinism and grading the adopter-facing models.
  • test_wire_round_trip.py records core/publisher-property-selector.json in the shrink-only REFUSING set; verified not in DIGEST_COVERED/ECHO_COVERED, so no bound-object assertion is suppressed; the refusal originates in a pre-existing _ergonomic mutual-exclusivity validator outside this PR.

Decision-table walk:

  • Rows 1 (critical/high): no findings — skip.
  • Row 2 (gated_paths): false — skip.
  • Row 3 (deleted high-risk): reasons are (modified) only — skip.
  • Row 4 (medium data-loss/schema/infra): no findings — skip.
  • Row 5 (high_risk modified + medium): high_risk is true with (modified) entries, but there are zero medium findings — does not fire.
  • Row 6 (sticky escalate): prior decision was approve, not escalate — skip.
  • Row 7 (no-auto-approve team): no match — skip.
  • Row 8 (≥3 mediums): zero mediums — skip.
  • Row 9: approve.

The high_risk flag on types/__init__.py and _eager.py is a heuristic LOOK signal; both are (modified) with no medium-or-higher concerns, so presumed safe. Clean iteration.

aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 25, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

The reviewer found no critical, high, or medium findings. The sole delta is a defensive fix in src/adcp/types/_wire.py: WireDatetime.wire (L112) and __reduce_ex__ (L120) now read the _wire slot via getattr(self, "_wire", None) instead of a bare attribute access. This correctly handles the Python 3.10/3.11 case where C-level datetime.replace()/astimezone()/arithmetic builds the subclass without calling __new__, leaving the __slots__ member unset — a bare self._wire would AttributeError on .wire access and on pickling a derived value. The getattr fall-through to rfc3339(self) is the documented normalized-form behavior and mirrors the existing WireAnyUrl/WireUuid pattern. Remaining _wire touches are writes, which are slot-safe.

Decision-table walk: no critical/high (row 1 n/a); gated_paths false (row 2 n/a); no (deleted) high-risk reasons — only (modified) on two type files (row 3 n/a); no medium findings at all (rows 4, 5, 8 n/a); prior decision was approve, not escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a). The high_risk flag is a heuristic only — the two modified type files carry no medium-or-higher concern, so the modification is presumed safe. Falls through to row 9.

@bokelley bokelley changed the title fix(types): keep wire bytes on parsed URLs, datetimes, UUIDs and emails so digests round-trip feat(types): keep wire bytes on parsed URLs, datetimes, UUIDs and emails so digests round-trip Oct 26, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve.

Clean pass. The delta tightens wire-text preservation to strict-only forms via a new cycle-free leaf module adcp._formats, shared by schema_loader and the WireAnyUrl/WireDatetime/WireUuid/WireEmail types. The reviewer verified the str()/.host security guarantee (fast is_plain_uri subset grammar + slow-path explicit host comparison with a tested backslash fallback), single-source-of-truth consistency (is_plain_uri ⟹ is_uri_syntax), import safety (unused date/quote removed; _formats is a cycle-free leaf), pickle/deepcopy slot handling, and alignment with docs and new parametrized tests.

No Critical/High/Medium findings. Decision-table walk:

  • Row 1 (critical/high): no such findings.
  • Row 2 (gated_paths unsatisfied): gated_paths is false — not triggered.
  • Row 3/5 (high-risk deleted/modified + medium): both _eager.py and types/__init__.py are (modified), but there are no medium findings, so neither fires.
  • Row 4 (medium data-loss/schema/infra): none.
  • Row 6 (sticky escalate): prior decision was approve, not applicable.
  • Row 7 (no-auto-approve team): no match.
  • Row 8 (≥3 medium): zero medium findings.
  • Row 9: approve.

Prior decision was also approve on an earlier head; this is a net-clean subsequent pass. high_risk is true only via (modified) entries with no accompanying findings, which is presumed safe per the high-risk reading rules.

bokelley and others added 4 commits October 26, 2026 03:28
…ls so digests round-trip

A generated model parsed `baseline.system` into pydantic `AnyUrl` (which
re-serializes a bare origin with a trailing slash) and `last_updated` into
`AwareDatetime` (which rewrites `+00:00` as `Z` and drops `.000`), so a
parsed-then-dumped `AudienceEvidence` no longer matched its own
`content_digest`. The same gap had been patched one field or object at a time
in #1384, #1449, #1587 and #1575.

Invariant: a value the model did not change is dumped as it was received.

- `adcp.types._wire`: `WireDatetime`, `WireAnyUrl`, `WireUuid`, `WireEmail`
  validate as the pydantic types they replace, keep the attribute API and
  JSON Schema, and serialize the text they were parsed from; a value built in
  Python serializes in the documented deterministic form. To a type checker
  the re-exported names are the pydantic types, so adopters' verdicts are
  unchanged.
- `post_generate_fixes.preserve_wire_text_on_normalizing_fields` points every
  generated `format: date-time` / `uri` / `uuid` / `email` field at them
  (token-level rename; descriptions untouched).
- `AdCPBaseModel.model_dump_as_received()`: the explicit "as received" dump
  (`exclude_unset`, nulls kept, `by_alias`) that a digest verifier uses until
  the default dump stops materializing schema defaults (#1613).
- `tests/conformance/test_wire_round_trip.py`: a permanent sweep of every
  bundle schema with a model, under JCS, with a shrink-only allowlist for the
  remaining classes (#1613, #1614, #1615) and none for digest- or
  echo-covered objects on the as-received path.
- `AudienceEvidence` and its pin/selection siblings are exported from
  `adcp.types`.
- `docs/validation-contract.md`: root-cause analysis and the invariant.

Fixes #1610

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-trip sweep

The verdict for a schema depended on whether adcp.types._eager had already
run its import-time patchers: CI's full session refused a
PublisherPropertySelector document that an isolated run accepted. The sweep
now imports the eager surface up front and declares that schema as refusing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On Python 3.10 and 3.11, datetime.replace(), astimezone() and arithmetic
build the subclass instance without calling the overriding __new__, so the
_wire slot was unset on a derived value and .wire raised AttributeError.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The validators saved any str input, so a spelling pydantic's lenient parser
accepts but the schema rejects (a space instead of T, an offset without a
colon, an epoch string, surrounding whitespace, a braced UUID, a display-name
email) was echoed verbatim where main normalized it. The strict date-time and
URI checks now live in adcp._formats, shared with schema_loader; a UUID must
be the hyphenated form; an email is kept only as the trimmed address. str()
and .host of a URL can no longer disagree: text is kept only when its own
authority names the parsed host, so https://good.example\@evil.example falls
back to the normalized form.

Also cheaper: datetime/UUID serializers run in JSON mode only, the datetime is
built from its packed state in one C call, the URL validator is one Python
layer around the core parser, and a fast RFC 3986 subset grammar settles the
common URI before the full validator runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bokelley
bokelley force-pushed the conductor/1610-digest-round-trip-anyurl-awaredatetime branch from 6004a56 to d91e8e3 Compare October 26, 2026 03:29
aao-secretariat[bot]
aao-secretariat Bot previously approved these changes Oct 26, 2026

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean delta pass over three follow-up commits, no blocking or medium findings.

What I checked:

  • No author no-auto-approve team match; gated_paths: false (row 2 not triggered despite review_decision: REVIEW_REQUIRED).
  • high_risk: true but all reasons are (modified) with NO medium-or-higher findings, so row 5 does not fire; no (deleted) entries (row 3 clear).
  • Prior decision was approve (not escalate), so sticky row 6 is irrelevant.
  • Reviewer confirms the _wire.py strict-format gate is sound: _keep_url_text fast path is safe because the subset grammar admits only [A-Za-z0-9.\-]+ authorities (no @, \, %, [), with a full RFC 3986 + host comparison on the normalized arm keeping the good.example\@evil.example spoof contained. Fail-toward-normalize applied consistently for datetime/UUID/email. WireDatetime.__new__ drop verified safe. No new public-surface, import-layering, generated-code, union, or credential concerns.

No Critical/High/Medium findings → falls through rows 1–8 to row 9 (approve).

…newline

is_plain_uri used match with a trailing $, which also accepts a final
newline, and the fast path kept the text without comparing hosts, so a
shorthand IPv4 host (2130706433, 0x7f.1, 127.1) kept its spelling while
.host read 127.0.0.1. Use fullmatch and compare hosts on every path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

Pushed the two fixes from the re-review:

  1. is_plain_uri now uses fullmatch. $ alone accepted a trailing \n, so "https://x.example\n" was echoed verbatim.
  2. Hosts are compared on the fast path too. A shorthand IPv4 host such as 2130706433, 0x7f.1, 127.1 or 0177.0.0.1 kept its spelling while .host read 127.0.0.1, which defeats an ipaddress-based SSRF guard on urlsplit(str(url)).hostname.

New tests for both fail without the fix. The wire tests pass (854). Perf for the record: on a 2,000-product GetProductsResponse, model_dump_json is now at parity with main (2–4%), and model_validate_json is about 9% slower.

Comment thread src/adcp/_formats.py
"""Whether ``text`` is a URI the fast subset grammar accepts.

True implies :func:`is_uri_syntax` is true and that the host the text
names is the host a URL parser reports (lower-cased). The common case,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium: This docstring still claims is_plain_uri True "implies ... the host the text names is the host a URL parser reports (lower-cased)." The delta disproves exactly that: https://2130706433/ passes _PLAIN_URI (host class [A-Za-z0-9.\-]+), yet the WHATWG parser rewrites it to 127.0.0.1 — which is why _keep_url_text just dropped its is_plain_uri fast-echo path in favor of comparing hosts on every path. _wire.py:250-251 carries the same stale claim ("the fast grammar ... guarantees the host agrees"). Load-bearing SSRF-adjacent invariant: a maintainer trusting "guarantees the host agrees" could re-add the removed fast path and reintroduce the host/echo mismatch. Correct the two docstrings to match the code.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean security tightening with one non-blocking medium.

This PR hardens the private URL-wire layer: is_plain_uri switches match→fullmatch (closing a trailing-newline echo bypass), and _keep_url_text drops the fast-echo path so hosts are compared on every path (keeping str() consistent with .host for shorthand-IPv4 like 2130706433 → 127.0.0.1, preserving an SSRF-guard invariant). Both changes are correct and covered by three new test cases. Both modules are private, so there is no public-surface or semver impact.

Medium findings

  • src/adcp/_formats.py:105 — Stale security invariant in is_plain_uri / _keep_url_text docstrings (docstrings still assert the now-false guarantee that the fast grammar implies host agreement).

Decision rationale: No critical/high findings (row 1 N/A). gated_paths is false (row 2 N/A). High-risk reasons are (modified) only, no (deleted) (row 3 N/A). The single medium is a docstring-accuracy issue, not data-loss/schema/infra (row 4 N/A). Row 5 requires a (modified) high-risk file plus a medium finding — but the medium is in src/adcp/_formats.py, which is NOT one of the high-risk-flagged files (types/__init__.py, types/_eager.py); the high-risk-modified files carry no medium finding, so row 5 does not fire. No author team gate (row 7 N/A). Only one medium finding, well under three (row 8 N/A). Prior decision was approve, so sticky escalation (row 6) does not apply. Falls through to row 9 → approve.

Medium findings

  • src/adcp/_formats.py:105 — Stale security invariant in is_plain_uri / _keep_url_text docstrings

@bokelley
bokelley merged commit a7f5acd into main Oct 26, 2026
46 checks passed
@bokelley
bokelley deleted the conductor/1610-digest-round-trip-anyurl-awaredatetime branch October 26, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The generated AudienceEvidence can't round-trip its own content_digest: AnyUrl and AwareDatetime rewrite the digested fields

1 participant