feat(types): keep wire bytes on parsed URLs, datetimes, UUIDs and emails so digests round-trip

docs/validation-contract.md

@@ -43,6 +43,100 @@ directly. One consequence for subclasses: a `field_serializer` that substitutes
 a default for `None` no longer runs for a `None` value on a marked field,
 because `exclude_if` drops the field before the serializer is reached.
 
+## Wire round trip
+
+AdCP binds bytes to several objects. `core/audience-evidence.json` defines
+`content_digest` as SHA-256 over the RFC 8785 (JCS) form of the entry *as
+declared*; `core/canonical-proposal.json` binds `terms_digest` to
+`commercial_terms` the same way; `core/rights-constraint.json`,
+`core/creative-representation-set.json`, `media-buy/acceptance-policy-profile.json`,
+the attestation schemas and the governance check carry their own; a `context`
+is "echoed unchanged". A model that changes a value between parse and dump
+breaks every one of those contracts without failing anything, and until 9.1
+the SDK had no invariant saying it must not.
+
+### Root cause
+
+The generator maps each JSON Schema `format` to the pydantic type that
+*parses* it best — `AnyUrl`, `AwareDatetime`, `UUID`, `EmailStr` — and those
+types re-serialize the value they parsed rather than the text they were given:
+a bare origin gains a trailing slash, `+00:00` becomes `Z`, `.000` is dropped,
+a UUID is lower-cased, an email's domain is lower-cased. The runtime then adds
+rewrites of its own: `model_dump` emits every field, so a schema `default` the
+generator wrote into the field appears in the output though the document
+omitted it; `exclude_none` (the default) drops an explicit `null` the schema
+admits; `by_alias` is off, so `$schema` dumps as `field_schema`; and
+`extra='ignore'` drops a property the pinned model does not declare. Each of
+these is reasonable for an application type and wrong for a document whose
+bytes are an identity.
+
+Four earlier fixes were point patches of this one gap: #1384 typed one field
+(`agent_url`) as a wire string because an option ID hashed it; #1449 reported
+that dumping a validated `ProductFormatDeclaration` "adds every branch default
+to the published format"; #1587 made nested models leave unset fields off from
+any parent; #1575 kept a `ContextObject`'s null keys. #1610 found the next
+instance in `AudienceEvidence`.
+
+### The invariant
+
+**A value the model did not change is dumped as it was received.** Two
+mechanisms carry it, and a conformance suite measures it:
+
+- **Wire-preserving scalar types.** `scripts/post_generate_fixes.py`
+  (`preserve_wire_text_on_normalizing_fields`) points every generated
+  `format: date-time`, `uri`, `uuid` and `email` field at
+  `adcp.types._wire` — `WireDatetime`, `WireAnyUrl`, `WireUuid`, `WireEmail`.
+  Each validates exactly as the pydantic type did, keeps the attribute API
+  (`.year`, `.tzinfo`, `.host`, `.hex`), compares and hashes as the same
+  value, advertises the same JSON Schema, and in JSON mode serializes the text
+  it was parsed from — when that text passes the strict wire check the bundled
+  validator applies (`adcp._formats`: RFC 3339 for a date-time, RFC 3986
+  syntax with an agreeing host for a URI, the hyphenated 8-4-4-4-12 form for a
+  UUID, a trimmed address for an email). A string pydantic's lenient parser
+  accepts but the schema rejects (`"2026-10-01 00:00:00+00:00"`,
+  `"https:x.example"`, `"{...}"`) is normalized, as it always was. A
+  `datetime`, `AnyUrl` or `UUID` *object* passed to a constructor has no wire
+  text and serializes as before: RFC 3339 with `Z` for UTC and `±HH:MM`
+  otherwise, fractional seconds only when non-zero; pydantic's normalized URL
+  string; a lower-case hyphenated UUID. Every such field is covered, not only the
+  digest-covered ones: a shared class (`AttestationReference`, `Provenance`,
+  `Targeting`) sits inside digested and undigested objects alike, so a
+  per-subtree rule would have to split classes. The rule covers the generated
+  AdCP schema types (`adcp.types.domains`); `adcp.types.registry`, the
+  OpenAPI-generated registry client, keeps pydantic's `AnyUrl` and
+  `AwareDatetime` — no registry object is digest-covered.
+- **An explicit "as received" dump.** `AdCPBaseModel.model_dump_as_received()`
+  is `model_dump(mode="json", exclude_unset=True, exclude_none=False,
+  by_alias=True)`: it leaves off every field the document did not set (where
+  a generated default would otherwise appear), keeps an explicit `null` the
+  schema admits, and writes wire names. Recompute a digest over this dump, not
+  the default one. For a model built in Python it emits exactly the fields the
+  caller set.
+
+`tests/conformance/test_wire_round_trip.py` runs every bundle schema that has
+a model — the bundle's `examples`, a synthesized document, the differential
+fixtures, a checked-in corpus for the digest-covered schemas, and variants of
+each exercising every form a normalizing parser rewrites — through
+`model_validate_json(doc).model_dump(mode="json")` and compares the result to
+the input under JCS. A rewritten URL, datetime, UUID, email, value or shape
+fails anywhere. The digest- and echo-covered schemas must round-trip with no
+difference of any class under `model_dump_as_received`. Under the default
+dump the suite names the classes that remain, each with the issue that tracks
+it, and the list only shrinks:
+
+| class | what happens | where it stands |
+| --- | --- | --- |
+| `default_emitted` | the dump materializes a schema default the document omitted | #1613; `model_dump_as_received` recovers the bytes. Two digest-covered schemas (`canonical-proposal`, `commercial-terms`) are affected under the default dump and are listed in the suite. |
+| `field_dropped` | `extra='ignore'` drops an unknown or sibling-arm property | #1614; not recoverable by any dump. No digest-covered schema is affected at pin 3.2.4. |
+| `alias_emitted` | `model_dump()` writes `pass_` for `pass`, `field_schema` for `$schema` | #1615; `model_dump_as_received` writes the wire name. |
+| `null_dropped` | `exclude_none` drops an admitted explicit `null` | #1615; `model_dump_as_received` keeps it. |
+
+Why the default dump was not changed in 9.x: leaving unset defaults off every
+`model_dump()` is wire-equivalent for a schema-applying reader but visible to
+any code that indexes a dumped dict (`payload["sandbox"]`), including a
+counterparty that does not apply schema defaults. That is a 10.0 decision,
+written up in #1613.
+
 ## Wire-boundary behavior
 
 - SDK clients validate requests before sending and responses after receiving,

scripts/post_generate_fixes.py

@@ -25,9 +25,11 @@
 import ast
 import hashlib
 import importlib.util
+import io
 import json
 import re
 import sys
+import tokenize
 from collections.abc import Mapping, Sequence
 from copy import deepcopy
 from pathlib import Path
@@ -1007,6 +1009,97 @@ def preserve_format_reference_agent_url_wire_string() -> None:
     print("  core/format_id.py: agent_url keeps the wire string (WireUrl)")
 
 
+#: The pydantic type the generator renders for each normalizing ``format`` and
+#: the wire-preserving type from ``adcp.types._wire`` that replaces it. Each
+#: replacement validates identically, keeps the attribute API (a ``datetime``,
+#: an ``AnyUrl``, a ``UUID``, a ``str``) and advertises the same JSON Schema;
+#: the only change is that JSON-mode serialization writes the text the wire
+#: delivered instead of the parser's normalized form.
+_WIRE_PRESERVING_TYPES: dict[tuple[str, str], str] = {
+    ("pydantic", "AwareDatetime"): "WireDatetime",
+    ("pydantic", "AnyUrl"): "WireAnyUrl",
+    ("pydantic", "EmailStr"): "WireEmail",
+    ("uuid", "UUID"): "WireUuid",
+}
+
+
+def _rename_code_names(source: str, renames: Mapping[str, str]) -> str:
+    """Rename identifier tokens, leaving strings and comments untouched.
+
+    A description such as "Use a fresh UUID v4 for each request" must keep
+    its wording; only a ``NAME`` token the tokenizer reports is a reference.
+    """
+    lines = source.splitlines(keepends=True)
+    edits: list[tuple[int, int, int, str]] = []
+    previous = ""
+    for token in tokenize.generate_tokens(io.StringIO(source).readline):
+        # ``uuid.UUID`` is an attribute, not a reference to the imported name.
+        if token.type == tokenize.NAME and token.string in renames and previous != ".":
+            (row, col), (_, end_col) = token.start, token.end
+            edits.append((row - 1, col, end_col, renames[token.string]))
+        if token.type not in (tokenize.NL, tokenize.NEWLINE, tokenize.COMMENT):
+            previous = token.string
+    for row, col, end_col, replacement in reversed(edits):
+        line = lines[row]
+        lines[row] = line[:col] + replacement + line[end_col:]
+    return "".join(lines)
+
+
+def preserve_wire_text_on_normalizing_fields() -> None:
+    """Point every normalizing scalar field at its wire-preserving type.
+
+    ``format: date-time``, ``format: uri``, ``format: uuid`` and
+    ``format: email`` fields are rendered ``AwareDatetime``, ``AnyUrl``,
+    ``UUID`` and ``EmailStr``, each of which re-serializes the value it parsed:
+    ``https://measurement.example`` comes back with a trailing slash,
+    ``2026-10-01T00:00:00.000+00:00`` as ``2026-10-01T00:00:00Z``, an
+    upper-case UUID lower-cased, an email with its domain lower-cased. AdCP
+    binds bytes to several of these objects -- ``content_digest`` is SHA-256
+    over the JCS form of an entry's declared fields -- so a parsed-then-dumped
+    entry stopped matching its own digest (#1610). The types in
+    ``adcp.types._wire`` validate the same way and serialize the text they
+    were parsed from; a value built in Python serializes as before.
+
+    Every field is pointed there, not only the digest-covered ones: a shared
+    class (``AttestationReference``, ``Provenance``) sits inside digested and
+    undigested objects alike, so a per-subtree rule would have to split
+    classes, and the invariant worth having is that the SDK never rewrites a
+    wire value it did not change.
+    """
+    renames = {name: wire for (_module, name), wire in _WIRE_PRESERVING_TYPES.items()}
+    modified = 0
+    for path in sorted(OUTPUT_DIR.rglob("*.py")):
+        source = path.read_text()
+        if not any(re.search(rf"\b{name}\b", source) for name in renames):
+            continue
+        tree = ast.parse(source)
+        referenced = {
+            node.id for node in ast.walk(tree) if isinstance(node, ast.Name) and node.id in renames
+        }
+        if not referenced:
+            continue
+        for (module, name), wire in _WIRE_PRESERVING_TYPES.items():
+            if name in referenced:
+                source = drop_from_imports(source, module, name)
+        source = _rename_code_names(source, {name: renames[name] for name in referenced})
+        wires = sorted({renames[name] for name in referenced})
+        if re.search(r"^from adcp\.types\.base import ", source, re.MULTILINE):
+            source = add_to_import(source, "adcp.types.base", *wires)
+        else:
+            anchor = _import_pattern("pydantic").search(source)
+            if anchor is None:
+                anchor = re.search(r"^from |^import ", source, re.MULTILINE)
+            assert anchor is not None, path
+            source = (
+                source[: anchor.start()]
+                + f"from adcp.types.base import {', '.join(wires)}\n"
+                + source[anchor.start() :]
+            )
+        path.write_text(source)
+        modified += 1
+    print(f"  Pointed normalizing fields at wire-preserving types in {modified} file(s)")
+
+
 def point_integer_fields_at_the_schema_integer_type() -> None:
     """Rewrite the generator's ``StrictInt`` marker to ``SchemaInt``.
 
@@ -8360,6 +8453,9 @@ def main(argv: list[str] | None = None) -> int:
         annotate_registry_track_verdict,
         preserve_geo_place_system_map_keys,
         preserve_format_reference_agent_url_wire_string,
+        # After every pass that adds a validator typed ``AnyUrl`` and before
+        # the import cleanups, which would read the names this pass drops.
+        preserve_wire_text_on_normalizing_fields,
         insert_task_message_markers,
         point_integer_fields_at_the_schema_integer_type,
         remove_imports_shadowed_by_a_local_class,

scripts/post_generation_manifest.json

@@ -1,6 +1,7 @@
 {
   "_comment": "Effect manifest for scripts/post_generate_fixes.py. ``true`` records a fix that changes the generated tree; a string records a fix that changes nothing on the pinned schema set, and states why. Regenerate with `python scripts/post_generate_fixes.py --update-manifest`.",
   "fixes": {
+    "point_omittable_fields_at_no_explicit_null": true,
     "add_model_validator_to_product": "No-op since #213 added an explicit discriminator to publisher_properties: Pydantic validates the union, and the body only prints.",
     "fix_preview_render_self_reference": "datamodel-code-generator 0.64.0 emits PreviewRender1..3 unqualified in creative/preview_render.py; the module-qualified form this fix rewrites appears nowhere.",
     "fix_brand_manifest_references": "Its target core/promoted_offerings.py is not generated \u2014 brand.json is in GENERATED_SCHEMA_EXCLUDE_FILES \u2014 and the unaliased brand_manifest.BrandManifest reference appears nowhere in the tree.",
@@ -69,9 +70,9 @@
     "annotate_registry_track_verdict": true,
     "preserve_geo_place_system_map_keys": true,
     "preserve_format_reference_agent_url_wire_string": true,
+    "preserve_wire_text_on_normalizing_fields": true,
     "insert_task_message_markers": true,
     "point_integer_fields_at_the_schema_integer_type": true,
-    "point_omittable_fields_at_no_explicit_null": true,
     "remove_imports_shadowed_by_a_local_class": true,
     "remove_unused_pydantic_field_imports": true,
     "emit_open_asset_unions": true,

src/adcp/_formats.py

@@ -0,0 +1,122 @@
+"""Strict wire-format checks shared by schema validation and the wire types.
+
+The bundled JSON Schema validator and the generated models must agree on
+which strings are a ``format: date-time`` or a ``format: uri``
+(``adcp.validation.schema_loader``), and the wire-preserving scalar types
+(``adcp.types._wire``) keep a string's spelling only when it passes the same
+check -- a string pydantic's lenient parser accepts but the schema rejects
+(``"2026-10-01 00:00:00+00:00"``, ``"https:x.example"``) is normalized
+instead, as it always was. This module is a leaf: it imports nothing from
+``adcp.types`` or ``adcp.validation``, so both can import it.
+"""
+
+from __future__ import annotations
+
+import re
+from datetime import date
+from urllib.parse import quote
+
+__all__ = [
+    "RFC3339_DATE_TIME",
+    "UUID_TEXT",
+    "is_plain_uri",
+    "is_rfc3339_date_time",
+    "is_uri_syntax",
+    "is_uuid_text",
+    "uri_host",
+]
+
+RFC3339_DATE_TIME = re.compile(
+    r"^\d{4}-\d{2}-\d{2}[Tt]"
+    r"(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d"
+    r"(?:\.\d+)?"
+    r"(?:[Zz]|[+-](?:[01]\d|2[0-3]):[0-5]\d)$",
+    re.ASCII,
+)
+
+#: The hyphenated 8-4-4-4-12 form, either case: the only spelling
+#: ``format: uuid`` admits. Braces, ``urn:uuid:`` and the bare 32 hex digits
+#: are accepted by ``uuid.UUID`` and normalized.
+UUID_TEXT = re.compile(
+    r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"
+)
+
+# scheme "://" [userinfo "@"] host [":" port] -- enough to read the host a
+# URI's own text names, for comparison with what the URL parser reports.
+_URI_AUTHORITY = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*://(?:[^/?#@]*@)?(\[[^\]]*\]|[^/?#:]*)")
+
+# A strict SUBSET of RFC 3986: scheme, a plain ASCII reg-name host, an
+# optional port, and path/query/fragment made of unreserved, sub-delimiter
+# and well-formed percent-escape characters only. No userinfo, IP literal,
+# non-ASCII or stray ``%``, so anything it matches is a valid URI whose host
+# every parser reads the same way; everything else takes the full check.
+_PCHAR = r"(?:[A-Za-z0-9._~!$&'()*+,;=:@/-]|%[0-9A-Fa-f]{2})"
+_PLAIN_URI = re.compile(
+    r"^[A-Za-z][A-Za-z0-9+.\-]*://[A-Za-z0-9.\-]+(?::[0-9]*)?"
+    rf"(?:/{_PCHAR}*)?(?:\?(?:{_PCHAR}|\?)*)?(?:#(?:{_PCHAR}|\?)*)?$",
+    re.ASCII,
+)
+
+
+def is_rfc3339_date_time(text: str) -> bool:
+    """Whether ``text`` is an RFC 3339 date-time.
+
+    The grammar checks the time and offset ranges; only the calendar is
+    checked here, because Python 3.10's parser accepts three or six
+    fractional digits where RFC 3339 permits any positive number, and the
+    check must neither coerce nor truncate the value.
+    """
+    if RFC3339_DATE_TIME.fullmatch(text) is None:
+        return False
+    try:
+        date.fromisoformat(text[:10])
+    except ValueError:
+        return False
+    return True
+
+
+def is_uri_syntax(text: str) -> bool:
+    """Whether ``text`` is RFC 3986 URI syntax, with non-ASCII escaped first.
+
+    Whitespace, C1 controls and a backslash fail here: pydantic's ``AnyUrl``
+    trims or rewrites them, which is exactly what must not be hidden. Only
+    ASCII is checked as written, so a malformed percent escape stays visible;
+    ``AnyUrl`` still checks internationalized hosts.
+    """
+    from rfc3986_validator import validate_rfc3986
+
+    if any(char.isspace() or 0x7F <= ord(char) <= 0x9F for char in text):
+        return False
+    try:
+        syntax = (
+            text
+            if text.isascii()
+            else "".join(char if char.isascii() else quote(char, safe="") for char in text)
+        )
+    except UnicodeEncodeError:
+        return False
+    return bool(validate_rfc3986(syntax))
+
+
+def is_plain_uri(text: str) -> bool:
+    """Whether ``text`` is a URI the fast subset grammar accepts.
+
+    True implies :func:`is_uri_syntax` is true and that the host the text
+    names is the host a URL parser reports (lower-cased). The common case,
+    checked in a fraction of the full RFC 3986 validator's time.
+    """
+    # fullmatch: ``$`` alone would also accept a single trailing newline.
+    return _PLAIN_URI.fullmatch(text) is not None
+
+
+def is_uuid_text(text: str) -> bool:
+    """Whether ``text`` is the hyphenated 8-4-4-4-12 UUID spelling."""
+    return UUID_TEXT.fullmatch(text) is not None
+
+
+def uri_host(text: str) -> str | None:
+    """The host ``text`` names in its authority, lower-cased, or ``None``."""
+    match = _URI_AUTHORITY.match(text)
+    if match is None:
+        return None
+    return match.group(1).lower()

src/adcp/reporting/adjustment_evidence.py

@@ -32,6 +32,7 @@
     ReportingObligation,
     ReportingRevision,
 )
+from adcp.types._wire import rfc3339
 from adcp.validation.schema_loader import get_named_validator
 
 __all__ = [
@@ -463,7 +464,10 @@ def build_reporting_adjustment_receipt(
         moment = aware_utc(observed_at)
         if _instant(moment) < _instant(adjustment.created_at, raw["created_at"]):
             _fail("INVALID_RECEIPT")
-        payload["observed_at"] = moment.isoformat()
+        # The receipt's own spelling, written here: a parsed datetime keeps the
+        # text it was given (#1610), so ``isoformat()``'s ``+00:00`` would be
+        # emitted as such; the receipt has always carried ``Z``.
+        payload["observed_at"] = rfc3339(moment)
         if failures:
             payload["rejection_codes"] = sorted(failures)
         if current_receipt is not None: