feat(codegen)!: mark wire-required fields, and leave an unset schema default off the dump

MIGRATION_v9_to_v10.md

@@ -27,3 +27,4 @@ the SDK-relevant inventory is [docs/adcp-3.3-sdk-inventory.md](docs/adcp-3.3-sdk
 | A2A requests must meet the AdCP 3.3 operation-resolution hardening when request signing is configured | Send one JSON-RPC request per body with no duplicate or case-variant member names, one content member per Part, no FileParts, and exactly one DataPart carrying `skill`. Otherwise the request fails with `request_body_malformed`. A `message.taskId` continuation must come from the principal that created the task and name its original operation: a different principal gets task-not-found, and a different operation gets `INVALID_REQUEST`. |
 | `model_dump()` writes wire names | A field the generator aliased because its wire key is not a Python identifier is emitted under the wire key without `by_alias=True`: `$schema` instead of `field_schema`, `pass` instead of `pass_`, `from` instead of `from_`, `media-buy` instead of `media_buy`, `_truncation` instead of `field_truncation`, `2xl` instead of `field_2xl`. Code that reads a dump by the Python name must read the wire name; pass `by_alias=False` to keep the SDK 9 output. Construction accepts both spellings (`validate_by_name` is on). |
 | A received `null` is re-emitted; a Python `None` keyword argument is "not supplied" | A property whose schema admits `null` and that arrived as `null` (`model_validate`, `model_validate_json`, nested validation) stays in `model_fields_set` and is written back as `null` by `model_dump()`. Passing `None` as a keyword argument to a constructor, assigning `None` to a field, or passing `None` in `model_copy(update=...)` now leaves the field *unset*: it is omitted from the dump, and `model_dump(exclude_unset=True)` no longer lists it. A `None` inside a dict you validate is a wire `null`. Properties whose schema does not admit `null` are unchanged (`NoExplicitNull` refuses a JSON `null` and never emits one). |
+| `model_dump()` leaves an unset schema default off | A property the schema lists outside `required` with a `default` (or a `const`) is emitted only when it was set: by the document that was parsed, by a keyword argument, or by assignment. `sandbox: false`, `timezone: "inventory_local"`, `assignment_limit: 50` and the other schema defaults no longer appear on a dump unless you set them; the attribute still reads the default. Required discriminators (`asset_type`, `pricing_model`) are emitted either way. Pass `exclude_unset=False` for the SDK 9 output. Golden-file tests that compare whole dumps need regenerating. If you relied on a default reaching the peer, set it explicitly. `model_dump_as_received()` is deprecated: it returns the same document as `model_dump(mode="json")` and is removed in SDK 11. Generated root models (a union root such as `LocalizedCreativeAsset`) now subclass `AdCPRootModel` and dump with `exclude_none=True` like every other AdCP model. |

docs/validation-contract.md

@@ -105,37 +105,32 @@ mechanisms carry it, and a conformance suite measures it:
   AdCP schema types (`adcp.types.domains`); `adcp.types.registry`, the
   OpenAPI-generated registry client, keeps pydantic's `AnyUrl` and
   `AwareDatetime` — no registry object is digest-covered.
-- **An explicit "as received" dump.** `AdCPBaseModel.model_dump_as_received()`
-  is `model_dump(mode="json", exclude_unset=True, exclude_none=False,
-  by_alias=True)`: it leaves off every field the document did not set (where
-  a generated default would otherwise appear), keeps an explicit `null` the
-  schema admits, and writes wire names. Recompute a digest over this dump, not
-  the default one. For a model built in Python it emits exactly the fields the
-  caller set.
+- **The default dump is the document.** `model_dump(mode="json")` leaves off
+  a schema-optional default the document did not carry (`OmitUnsetDefault`,
+  #1613), re-emits a `null` the document carried on a property whose schema
+  admits one (#1615), writes wire names (`$schema`, `pass`, `from`; #1615),
+  and keeps an unknown property where the schema object is open (#1614).
+  Recompute a digest over the default dump. For a model built in Python it
+  emits the fields the caller set plus the ones the schema requires.
+  `model_dump_as_received()` is a deprecated alias of the same dump and is
+  removed in SDK 11; `model_dump(exclude_unset=False)` gives the SDK 9 output
+  with every default materialized.
 
 `tests/conformance/test_wire_round_trip.py` runs every bundle schema that has
 a model — the bundle's `examples`, a synthesized document, the differential
 fixtures, a checked-in corpus for the digest-covered schemas, and variants of
 each exercising every form a normalizing parser rewrites — through
 `model_validate_json(doc).model_dump(mode="json")` and compares the result to
-the input under JCS. A rewritten URL, datetime, UUID, email, value or shape
-fails anywhere. The digest- and echo-covered schemas must round-trip with no
-difference of any class under `model_dump_as_received`. Under the default
-dump the suite names the classes that remain, each with the issue that tracks
-it, and the list only shrinks:
+the input under JCS. A rewritten URL, datetime, UUID, email, value or shape,
+an emitted default, a dropped null or a Python-named key fails anywhere. The
+digest- and echo-covered schemas must round-trip with no difference of any
+class. Under the default dump the suite names the classes that remain, each
+with the issue that tracks it, and the list only shrinks:
 
 | class | what happens | where it stands |
 | --- | --- | --- |
-| `default_emitted` | the dump materializes a schema default the document omitted | #1613; `model_dump_as_received` recovers the bytes. Two digest-covered schemas (`canonical-proposal`, `commercial-terms`) are affected under the default dump and are listed in the suite. |
-| `field_dropped` | `extra='ignore'` drops an unknown or sibling-arm property | #1614; not recoverable by any dump. No digest-covered schema is affected at pin 3.2.7. |
-| `alias_emitted` | `model_dump()` writes `pass_` for `pass`, `field_schema` for `$schema` | #1615; `model_dump_as_received` writes the wire name. |
-| `null_dropped` | `exclude_none` drops an admitted explicit `null` | #1615; `model_dump_as_received` keeps it. |
-
-Why the default dump was not changed in 9.x: leaving unset defaults off every
-`model_dump()` is wire-equivalent for a schema-applying reader but visible to
-any code that indexes a dumped dict (`payload["sandbox"]`), including a
-counterparty that does not apply schema defaults. That is a 10.0 decision,
-written up in #1613.
+| `field_dropped` | `extra='ignore'` drops an unknown or sibling-arm property | #1614; not recoverable by any dump. No digest-covered schema is affected. |
+| `discriminator_emitted` | a document valid under one arm is routed to a sibling arm, whose required single-value `Literal` it does not carry, and the arm's injected default appears | #1414; closed by #1522's construction dispatch. |
 
 ## Wire-boundary behavior
 

scripts/generate_types.py

@@ -967,6 +967,14 @@ def flatten_root_object_validation_union(schema: dict, schema_path: Path) -> dic
 #: carry it.
 NO_EXPLICIT_NULL_KEY = "x-adcp-no-explicit-null"
 
+#: Pre-generation mark on an optional property whose schema declares a
+#: ``default``. Carried the same way, and turned into the
+#: ``adcp.types.base.OmitUnsetDefault`` annotation by the same post-generation
+#: pass: the default dump then leaves the property off unless it was set
+#: (#1613). A required property with a default is not marked; it is emitted
+#: either way.
+OMIT_UNSET_DEFAULT_KEY = "x-adcp-omit-unset-default"
+
 
 def _file_uri_resource(path: Path):
     """Load *path* as a ``referencing`` resource addressed by its file URI.
@@ -1034,6 +1042,21 @@ def _required_names(node: Any, resolver, seen: frozenset[str] = frozenset()) ->
     return required
 
 
+def _carries_a_default(property_schema: dict[str, Any]) -> bool:
+    """Whether the generated field for this optional property gets a default.
+
+    A schema ``default`` becomes the field's default. A ``const`` or a one-value
+    ``enum`` becomes a single-value ``Literal``, and
+    ``post_generate_fixes.inject_literal_discriminator_defaults`` gives that a
+    default too; on an optional property that default is a value the wire did
+    not send, so it is marked the same way.
+    """
+    if "default" in property_schema or "const" in property_schema:
+        return True
+    enum = property_schema.get("enum")
+    return isinstance(enum, list) and len(enum) == 1
+
+
 def _mark_no_explicit_null(node: Any, validator, resolver, counter: list[int]) -> None:
     if isinstance(node, list):
         for item in node:
@@ -1050,8 +1073,11 @@ def _mark_no_explicit_null(node: Any, validator, resolver, counter: list[int]) -
             if not _property_admits_null(validator, property_schema):
                 property_schema[NO_EXPLICIT_NULL_KEY] = True
                 counter[0] += 1
+            if _carries_a_default(property_schema):
+                property_schema[OMIT_UNSET_DEFAULT_KEY] = True
+                counter[1] += 1
     for key, value in node.items():
-        if key == NO_EXPLICIT_NULL_KEY:
+        if key in (NO_EXPLICIT_NULL_KEY, OMIT_UNSET_DEFAULT_KEY):
             continue
         _mark_no_explicit_null(value, validator, resolver, counter)
 
@@ -1100,6 +1126,7 @@ def retrieve(uri: str):
         for path in sorted(temp_dir.rglob("*.json"))
     )
     marked = 0
+    defaulted = 0
     for schema_file in sorted(temp_dir.rglob("*.json")):
         with open(schema_file) as f:
             schema = json.load(f)
@@ -1117,13 +1144,15 @@ def retrieve(uri: str):
             validator_class = validator_for(document)
         validator = validator_class(document, registry=registry)
         resolver = registry.resolver(base_uri=document["$id"])
-        counter = [0]
+        counter = [0, 0]
         _mark_no_explicit_null(schema, validator, resolver, counter)
-        if counter[0]:
+        if counter[0] or counter[1]:
             with open(schema_file, "w") as f:
                 json.dump(schema, f, indent=2)
             marked += counter[0]
+            defaulted += counter[1]
     print(f"  Marked {marked} optional properties that refuse an explicit null")
+    print(f"  Marked {defaulted} optional properties that carry a schema default")
     return marked
 
 
@@ -1391,6 +1420,7 @@ def _run_datamodel_codegen(input_path: Path, output_path: Path) -> subprocess.Co
         # where the post-generation pass turns it into ``NoExplicitNull``.
         "--field-extra-keys",
         NO_EXPLICIT_NULL_KEY,
+        OMIT_UNSET_DEFAULT_KEY,
         "--allow-remote-refs",
     ]
 

scripts/post_generate_fixes.py

@@ -1135,6 +1135,12 @@ def point_integer_fields_at_the_schema_integer_type() -> None:
 #: The ``Field`` keyword the generator emits for ``generate_types.NO_EXPLICIT_NULL_KEY``.
 _NO_EXPLICIT_NULL_EXTRA = "json_schema_extra={'x-adcp-no-explicit-null': True}"
 _NO_EXPLICIT_NULL_MARKER = "NoExplicitNull"
+_NO_EXPLICIT_NULL_KEY = "x-adcp-no-explicit-null"
+#: The second generator mark the same rewrite carries: a schema-optional
+#: property with a schema ``default`` (``generate_types.OMIT_UNSET_DEFAULT_KEY``).
+_OMIT_UNSET_DEFAULT_KEY = "x-adcp-omit-unset-default"
+_OMIT_UNSET_DEFAULT_MARKER = "OmitUnsetDefault"
+_MARK_KEYS = frozenset({_NO_EXPLICIT_NULL_KEY, _OMIT_UNSET_DEFAULT_KEY})
 
 
 def _generated_line_width() -> int:
@@ -1196,7 +1202,7 @@ def _no_explicit_null_keyword(call: ast.expr) -> ast.keyword | None:
         if keyword.arg != "json_schema_extra" or not isinstance(keyword.value, ast.Dict):
             continue
         keys = [k.value for k in keyword.value.keys if isinstance(k, ast.Constant)]
-        if keys == ["x-adcp-no-explicit-null"]:
+        if keys and len(keys) == len(keyword.value.keys) and set(keys) <= _MARK_KEYS:
             return keyword
     return None
 
@@ -1345,8 +1351,16 @@ def _rewrite_no_explicit_null_marks(
         annotation_names = {n.id for n in ast.walk(elements[0]) if isinstance(n, ast.Name)} | {
             n.value for n in ast.walk(elements[0]) if isinstance(n, ast.Constant)
         }
-        if None in annotation_names and marker is not None:
+        assert isinstance(keyword.value, ast.Dict)
+        keys = {k.value for k in keyword.value.keys if isinstance(k, ast.Constant)}
+        appended = False
+        if _NO_EXPLICIT_NULL_KEY in keys and None in annotation_names and marker is not None:
             metadata.append(marker)
+            appended = True
+        if _OMIT_UNSET_DEFAULT_KEY in keys and marker is not None:
+            metadata.append(_OMIT_UNSET_DEFAULT_MARKER)
+            appended = True
+        if appended:
             marked += 1
         # Measure from the start of the line: the indentation counts toward
         # the width black lays the statement out against.
@@ -1372,13 +1386,13 @@ def _rewrite_no_explicit_null_marks(
     return raw.decode("utf-8"), marked
 
 
-def _ensure_no_explicit_null_import(source: str) -> str:
-    """Import ``NoExplicitNull`` from ``adcp.types.base`` when the module lacks it."""
-    if re.search(r"^from adcp\.types\.base import .*\bNoExplicitNull\b", source, re.MULTILINE):
+def _ensure_no_explicit_null_import(source: str, name: str = _NO_EXPLICIT_NULL_MARKER) -> str:
+    """Import ``name`` (a marker) from ``adcp.types.base`` when the module lacks it."""
+    if re.search(rf"^from adcp\.types\.base import .*\b{name}\b", source, re.MULTILINE):
         return source
     if re.search(r"^from adcp\.types\.base import ", source, re.MULTILINE):
-        return add_to_import(source, "adcp.types.base", _NO_EXPLICIT_NULL_MARKER)
-    statement = f"from adcp.types.base import {_NO_EXPLICIT_NULL_MARKER}\n"
+        return add_to_import(source, "adcp.types.base", name)
+    statement = f"from adcp.types.base import {name}\n"
     anchor = _import_pattern("pydantic").search(source)
     if anchor is not None:
         return source[: anchor.start()] + statement + source[anchor.start() :]
@@ -1407,10 +1421,12 @@ def point_omittable_fields_at_no_explicit_null() -> None:
     marked = 0
     for path in sorted(OUTPUT_DIR.rglob("*.py")):
         source = path.read_text()
-        if _NO_EXPLICIT_NULL_EXTRA not in source:
+        no_null = f"'{_NO_EXPLICIT_NULL_KEY}'"
+        omit_default = f"'{_OMIT_UNSET_DEFAULT_KEY}'"
+        if no_null not in source and omit_default not in source:
             continue
         source, count = _rewrite_no_explicit_null_marks(source)
-        assert _NO_EXPLICIT_NULL_EXTRA not in source, path
+        assert no_null not in source and omit_default not in source, path
         # A module whose only ``Field`` or ``Annotated`` was the mark's is left
         # exactly as the generator writes it without the mark.
         if "Field(" not in source:
@@ -1419,8 +1435,10 @@ def point_omittable_fields_at_no_explicit_null() -> None:
             source = drop_from_imports(source, "typing", "Annotated")
             # Dropping the only ``typing`` import leaves its blank line behind.
             source = re.sub(r"\n{3,}(from |import )", r"\n\n\1", source, count=1)
-        if count:
-            source = _ensure_no_explicit_null_import(source)
+        if re.search(r"\bNoExplicitNull\b", source):
+            source = _ensure_no_explicit_null_import(source, _NO_EXPLICIT_NULL_MARKER)
+        if re.search(r"\bOmitUnsetDefault\b", source):
+            source = _ensure_no_explicit_null_import(source, _OMIT_UNSET_DEFAULT_MARKER)
         path.write_text(source)
         modified += 1
         marked += count
@@ -3837,6 +3855,8 @@ def _ensure_sequence_import(content: str) -> str:
     r"(    canceled: Annotated\[\n        )"
     r"Literal\[True\]"
     r"(,\n        Field\(\n            description=[^\n]*\n        \),\n)"
+    # The optional-const mark (#1613) is dropped with the default it qualified.
+    r"(?:        OmitUnsetDefault,\n)?"
     r"(    \])"
     r" = True"
 )
@@ -3850,6 +3870,7 @@ def _ensure_sequence_import(content: str) -> str:
     r"(    unchanged: Annotated\[\n        )"
     r"Literal\[True\]"
     r"(,\n        Field\(\n            description=.*?\n        \),\n)"
+    r"(?:        OmitUnsetDefault,\n)?"
     r"(    \])"
     r" = True",
     re.DOTALL,
@@ -7440,6 +7461,35 @@ def _rebuild_union(arms: list[ast.expr]) -> ast.expr:
     return rebuilt
 
 
+def point_rootmodels_at_adcp_root() -> None:
+    """Give every generated ``RootModel`` the SDK's dump defaults (#1613).
+
+    A composing root that ``unwrap_rootmodel_unions`` keeps wrapped dumped
+    with pydantic's defaults -- every ``None`` of its arm as ``null`` -- while
+    the arm itself, an ``AdCPBaseModel``, left them off. The wrapper now
+    subclasses ``adcp.types.base.AdCPRootModel``, whose ``model_dump`` and
+    ``model_dump_json`` carry the base model's ``exclude_none`` and
+    ``serialize_as_any`` defaults. Runs last: the earlier passes locate root
+    models by the ``RootModel[`` base this one renames.
+    """
+    pattern = re.compile(r"(class \w+\(\s*)RootModel\[")
+    modified = 0
+    for path in sorted(OUTPUT_DIR.rglob("*.py")):
+        source = path.read_text()
+        if "RootModel[" not in source:
+            continue
+        rewritten, count = pattern.subn(r"\1AdCPRootModel[", source)
+        if not count:
+            continue
+        rewritten = _ensure_no_explicit_null_import(rewritten, "AdCPRootModel")
+        body = re.sub(r"^from pydantic import [^\n]*\n", "", rewritten, flags=re.MULTILINE)
+        if not re.search(r"(?<!AdCP)\bRootModel\b", body):
+            rewritten = drop_from_imports(rewritten, "pydantic", "RootModel")
+        path.write_text(rewritten)
+        modified += 1
+    print(f"  Pointed root models at AdCPRootModel in {modified} file(s)")
+
+
 def preserve_request_signing_operation_strings() -> None:
     """Keep request-signing operation lists source-compatible with beta.5.
 
@@ -8650,6 +8700,7 @@ def main(argv: list[str] | None = None) -> int:
         # ``__all__`` it writes has to name the final set.
         declare_root_schema_module_exports,
         strip_extra_blank_lines_at_eof,
+        point_rootmodels_at_adcp_root,
     ]
     observed: dict[str, list[str]] = {}
     digest = _tree_digest(OUTPUT_DIR)

scripts/post_generation_manifest.json

@@ -78,6 +78,7 @@
     "remove_unused_pydantic_field_imports": true,
     "emit_open_asset_unions": true,
     "declare_root_schema_module_exports": true,
-    "strip_extra_blank_lines_at_eof": true
+    "strip_extra_blank_lines_at_eof": true,
+    "point_rootmodels_at_adcp_root": true
   }
 }

src/adcp/reporting/production/offerings.py

@@ -61,8 +61,14 @@ def __post_init__(self) -> None:
         from adcp.validation.schema_loader import get_named_validator
 
         validator = get_named_validator("core/reporting-delivery-offering.json")
-        if validator is None or next(validator.iter_errors(raw), None) is not None:
+        if validator is None:
             raise ValueError("offering must satisfy the complete public contract")
+        error = next(validator.iter_errors(raw), None)
+        if error is not None:
+            where = "/".join(str(part) for part in error.absolute_path) or "/"
+            raise ValueError(
+                f"offering must satisfy the complete public contract: {error.message} at {where}"
+            )
         profile, definition, key = (
             raw["reporting_profile"],
             self.verification_key.definition,