Skip to content

feat(datasets): report retention, pace polls, serve stored files in any format, and name operations by surface - #68

Merged
jm-rivera merged 22 commits into
devfrom
feat/datasets-client-contract
Oct 23, 2026
Merged

jm-rivera merged 22 commits into
devfrom
feat/datasets-client-contract

Conversation

@jm-rivera

@jm-rivera jm-rivera commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

A Datasets client can pace its polls, fetch any stored version's file by format, list a dataset's versions without a principal, and read how long the server holds each version's frame. Every pending or running operation response carries operation.retry_after_ms and a Retry-After header, a quarter of the time since Operation.submitted_at, held between 1,500 and 8,000 ms. A done poll carries neither.

GET /api/datasets/{id}/{content_version}/file?format=&bundle=&layout= answers 302 to one file of that version (src/datasets_api/doors/files.py). It signs a stored copy first, renews an aged copy whose bytes survive, renders from the stored frame only when no copy survives, and refuses artifact_expired only when the frame's bytes are gone too. The rendering uses the version's recorded minting options with the requested format, bundle and layout applied. GET /api/datasets/{id}/versions serves under datasets.snapshot and pages with limit (1 to 500, default 100), a cursor bound to the path id, next_cursor and complete. An id no dataset has refuses 404 spec_not_found, where the old route answered an empty list. A single served id reads its page through a keyset over version_record_recency (0037_version_record_recency.sql, on subject_id, recorded_at DESC, content_version DESC), so a page reads its own rows however long the history; several served ids keep the grouped query, and both compare the cursor as a row. On 20,000 versions of one subject in Postgres 17, a page at offset 10,000 took 0.73 ms where the grouped query took about 260 ms. The migration's CREATE INDEX takes a SHARE lock and fails after its 5 s lock timeout.

Every Manifest the mint, definition GET, snapshot and done-poll routes return carries retention, either {"pinned": true} or {"pinned": false, "held_until": ...} at the frame's stored_at plus 30 days. A pin promotes the frame retention reads, aged or servable, so the next Manifest reports pinned. LiveArtifactStore.copy reads a 404 as an expired artifact only when the source object is gone. A missing destination bucket is a retryable 503.

The OpenAPI document names every operation {surface}_{route.name}, declares one tag per surface, documents the 202 with Location and Retry-After, and marks no field required that the wire can omit. SlotOption and ObservationPoint list their fields, where a wrap serializer's dict[str, Any] annotation had made them open objects, and test_every_object_schema_names_its_properties guards every component. The Scope schema says that on a spec_id input each supplied member replaces the stored spec's value whole, so entities naming one role unbinds the others. Docstrings describe the website proxy as a per-IP fixed window kept in Postgres, failing open, with Authorization passed upstream unread.

The spec-kit pin moves to 0f1a106 (DevelopmentFinanceObservatory/spec-kit#10), which maps the codes the selector and versions routes serve. The pin moves again to the release commit once that release is cut. The list_versions change in DevelopmentFinanceObservatory/ai#30 merges once this deploys to dev. Until then, the harness backfill preflight refuses, an unknown id raises in mint recovery, and recovery sees only the newest 100 versions. Conformance clauses C182 to C186 state the new contract.

Testing: just check at ec65f9c against spec-kit 0f1a106 passed 37,908 tests, with 132 skipped and 5 xfailed. Its one failure, test_a_surface_built_while_serving_a_release_binds_it_and_the_session_s_after in tests/graph_release/test_views.py, fails the same way on dev and is fixed by #65. With the index, tests/datasets_api, tests/dataset_store and the migration immutability test passed (3,115 passed, 82 skipped).

…ce, and document the 202

Every operation id is `{surface}_{handler}`, set where the app includes a surface's router, and every operation carries its surface's tag, declared with a one-line description. Handlers take verb names, and a test fails on a duplicate id, a missing verb or a missing tag.

POST /api/datasets and GET /api/datasets/{id} document their 202 with the OperationResource body and the Location and Retry-After headers.

ColumnOptionModel.distinct is optional and omitted when null, which matches the wire, and a test fails on any datasets response field that is required and admits None.

The app docstrings state what the website proxy does: a per-IP 429, with Authorization passed upstream unread.
…document the build dedup key

Every pending or running operation response, the 202 submit and every poll, carries operation.retry_after_ms and a Retry-After header in whole seconds rounded up. The value is a quarter of the time since the operation was created, held between 1,500 and 8,000 ms, so a caller that joins a long build polls slowly. A done poll carries neither.

The operation_id description states the dedup key: the subject and its sorted member dcids, the origin catalogue build and the rendering key. A repeat submission joins the running operation, and synchronous builds coalesce within one instance only.

The guide, the quickstart loops and the sync-ceiling note read the pacing from each poll.
…the versions list to every caller

GET /api/datasets/{id}/{content_version}/file?format=&bundle=&layout= answers 302 to that version's file for the requested rendering. The options are the version's recorded minting with the requested format, bundle and layout laid over it, so a derived version renders as itself. A rendering the version does not hold yet is rendered from its stored frame by render_version_from_frame, the one function that file renewal also calls, and an aged frame is renewed first. Frame bytes that are gone refuse artifact_expired.

GET /api/datasets/{id}/versions moves to an open door. It resolves the id through the served-id rules, so spec_not_found and identity_quarantined apply, omits revoked versions and versions whose base is revoked, and pages with next_cursor and complete (limit 1 to 500, default 100).

Conformance clauses C182 to C185 cover both routes.
…sions routes

spec-kit 0f1a106 maps the refusals that the file selector and the versions route serve. The route metadata fixture, the OpenAPI path sets and the platform capability digest follow it.

The datasets README, the operations poll docstring, the presentation app docstring and conformance C11 and C94a now state what the website proxy does: it answers a per-IP 429 only while a Redis is configured, and it passes Authorization upstream unread. The README and the dataset guide describe the file selector and the open, paginated versions route.
…version's base frame

Every Manifest door (mint, definition GET, snapshot and the done operation poll) carries `retention: {pinned, held_until?}`. `version_retention` in `datasets_api/retention.py` reads the version's servable frame, or a derived version's base frame, and reports `pinned: true` for a pinned copy or `held_until` as the frame's `stored_at` plus 30 days. A version with no frame reads its own data artifacts. The value is computed when the response is served and never stored, so `version_document` excludes it.

Pinning a derived version also copies its base version's servable frame, recorded under the base content version. Only the named artifact waits on the daily pin budget.
…renews a frame

A failed operation that is resubmitted keeps its `created_at` but restarts its build, so `poll_interval_ms` now measures age from `submitted_at`: a restarted build starts at the 1,500 ms floor, and a submission that joins a running build keeps that build's pace.

`held_until` and the retention docs state which requests renew a version's frame and which refuse `artifact_expired`. The API docstrings and the datasets README state the website proxy's limiter as it now runs: a fixed-window count per IP in Postgres, shared by every instance, that lets a call through when the count fails.
…e layout's format

A 202 that carries a failed operation advised a wait that grew with its age, though its next poll answers at once. The selector's layout parameter states that it needs format=parquet as well as bundle=false.
…iving file before rendering

A pin promotes the frame that retention reads, the servable frame or else an aged frame whose bytes survive (a derived version's base frame), so the next Manifest reports it pinned. A frame whose bytes are gone stays unpinned and the named file still pins. The file selector renews and serves an aged copy of the requested file before it renders from the frame, and refuses `artifact_expired` only when neither survives.

`LiveArtifactStore.copy` raises `ArtifactNotFoundError` for a missing source object, as its protocol states, so callers can tell gone bytes from a write failure. `held_until` states that only a download of the frame extends it.
…and log a skipped frame

A pin whose named file's bytes are gone refuses `410 artifact_expired` in place of a non-retryable 500, and a frame the pin cannot copy is logged with its digest, subject and content version. `held_until` states that a download of the frame renews it only after `held_until`, while its bytes survive, and the guide names which files re-render from the frame. The versions route declares its `limit` bounds in the OpenAPI document.
…ng bucket retryable

GCS answers 404 both for a missing source object and for a missing destination bucket, so `LiveArtifactStore.copy` read a missing bucket as an expired artifact. It now looks the source up after a `NotFound` and raises `ArtifactNotFoundError` only when the source is gone; otherwise it raises `ArtifactWriteError`, a retryable 503.

The served OpenAPI document described `SlotOption` and `ObservationPoint` as open objects, because pydantic publishes a wrap serializer's `dict[str, Any]` return annotation as the model's serialization schema. Both serializers are now unannotated, so each component lists its properties; `ObservationPoint` requires `date` and `value`, which its serializer always writes. `test_every_object_schema_names_its_properties` fails on any component that is an object with no properties. `SpecObservations` joins the models dumped under `exclude_none`, and the file selector's query parameter schemas come from `MintOptions`. The first-answer scenarios fixture moves only its `capability_version`, which hashes the search wire schema.
…members

The served schema said an omitted scope member means unbound, which holds only on a `statvar_set` input. On a `spec_id` input each supplied member replaces the stored spec's value whole and an omitted member keeps it (C42), so `entities` naming one role unbinds the spec's other roles. The `Scope` docstring now states both inputs, and `window`, `entities` and `facets` each carry a description of what they replace.
…plain statements

Docstrings, comments, field descriptions and the datasets docs this branch added read as present-tense sentences. The `_held_wire` test helper's docstring names what it returns. No behaviour changes.
Migration 0037 adds `version_record_recency` on (subject_id,
recorded_at DESC, content_version DESC). `_servable_page_query` in
src/datasets_api/version_store.py reads a single subject's page with a
keyset over that index, so a page reads its own rows however long the
history. Several served ids keep the grouped query. Both shapes compare
the cursor as a row, `(recorded_at, content_version) < (...)`.
@jm-rivera
jm-rivera marked this pull request as ready for review October 23, 2026 10:28
# Conflicts:
#	tests/fixtures/first_answer_place_scenarios.json
@github-actions github-actions Bot removed the run-eval label Oct 23, 2026
# Conflicts:
#	tests/fixtures/first_answer_place_scenarios.json
@jm-rivera
jm-rivera merged commit b6dc8ab into dev Oct 23, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant