Repository navigation
feat(datasets): report retention, pace polls, serve stored files in any format, and name operations by surface - #68
Merged
Conversation
…ce, and document the 202
Every operation id is `{surface}_{handler}`, set where the app includes a surface's router, and every operation carries its surface's tag, declared with a one-line description. Handlers take verb names, and a test fails on a duplicate id, a missing verb or a missing tag.
POST /api/datasets and GET /api/datasets/{id} document their 202 with the OperationResource body and the Location and Retry-After headers.
ColumnOptionModel.distinct is optional and omitted when null, which matches the wire, and a test fails on any datasets response field that is required and admits None.
The app docstrings state what the website proxy does: a per-IP 429, with Authorization passed upstream unread.
…document the build dedup key Every pending or running operation response, the 202 submit and every poll, carries operation.retry_after_ms and a Retry-After header in whole seconds rounded up. The value is a quarter of the time since the operation was created, held between 1,500 and 8,000 ms, so a caller that joins a long build polls slowly. A done poll carries neither. The operation_id description states the dedup key: the subject and its sorted member dcids, the origin catalogue build and the rendering key. A repeat submission joins the running operation, and synchronous builds coalesce within one instance only. The guide, the quickstart loops and the sync-ceiling note read the pacing from each poll.
…the versions list to every caller
GET /api/datasets/{id}/{content_version}/file?format=&bundle=&layout= answers 302 to that version's file for the requested rendering. The options are the version's recorded minting with the requested format, bundle and layout laid over it, so a derived version renders as itself. A rendering the version does not hold yet is rendered from its stored frame by render_version_from_frame, the one function that file renewal also calls, and an aged frame is renewed first. Frame bytes that are gone refuse artifact_expired.
GET /api/datasets/{id}/versions moves to an open door. It resolves the id through the served-id rules, so spec_not_found and identity_quarantined apply, omits revoked versions and versions whose base is revoked, and pages with next_cursor and complete (limit 1 to 500, default 100).
Conformance clauses C182 to C185 cover both routes.
…sions routes spec-kit 0f1a106 maps the refusals that the file selector and the versions route serve. The route metadata fixture, the OpenAPI path sets and the platform capability digest follow it. The datasets README, the operations poll docstring, the presentation app docstring and conformance C11 and C94a now state what the website proxy does: it answers a per-IP 429 only while a Redis is configured, and it passes Authorization upstream unread. The README and the dataset guide describe the file selector and the open, paginated versions route.
…version's base frame
Every Manifest door (mint, definition GET, snapshot and the done operation poll) carries `retention: {pinned, held_until?}`. `version_retention` in `datasets_api/retention.py` reads the version's servable frame, or a derived version's base frame, and reports `pinned: true` for a pinned copy or `held_until` as the frame's `stored_at` plus 30 days. A version with no frame reads its own data artifacts. The value is computed when the response is served and never stored, so `version_document` excludes it.
Pinning a derived version also copies its base version's servable frame, recorded under the base content version. Only the named artifact waits on the daily pin budget.
…renews a frame A failed operation that is resubmitted keeps its `created_at` but restarts its build, so `poll_interval_ms` now measures age from `submitted_at`: a restarted build starts at the 1,500 ms floor, and a submission that joins a running build keeps that build's pace. `held_until` and the retention docs state which requests renew a version's frame and which refuse `artifact_expired`. The API docstrings and the datasets README state the website proxy's limiter as it now runs: a fixed-window count per IP in Postgres, shared by every instance, that lets a call through when the count fails.
…e layout's format A 202 that carries a failed operation advised a wait that grew with its age, though its next poll answers at once. The selector's layout parameter states that it needs format=parquet as well as bundle=false.
…iving file before rendering A pin promotes the frame that retention reads, the servable frame or else an aged frame whose bytes survive (a derived version's base frame), so the next Manifest reports it pinned. A frame whose bytes are gone stays unpinned and the named file still pins. The file selector renews and serves an aged copy of the requested file before it renders from the frame, and refuses `artifact_expired` only when neither survives. `LiveArtifactStore.copy` raises `ArtifactNotFoundError` for a missing source object, as its protocol states, so callers can tell gone bytes from a write failure. `held_until` states that only a download of the frame extends it.
…and log a skipped frame A pin whose named file's bytes are gone refuses `410 artifact_expired` in place of a non-retryable 500, and a frame the pin cannot copy is logged with its digest, subject and content version. `held_until` states that a download of the frame renews it only after `held_until`, while its bytes survive, and the guide names which files re-render from the frame. The versions route declares its `limit` bounds in the OpenAPI document.
…ng bucket retryable GCS answers 404 both for a missing source object and for a missing destination bucket, so `LiveArtifactStore.copy` read a missing bucket as an expired artifact. It now looks the source up after a `NotFound` and raises `ArtifactNotFoundError` only when the source is gone; otherwise it raises `ArtifactWriteError`, a retryable 503. The served OpenAPI document described `SlotOption` and `ObservationPoint` as open objects, because pydantic publishes a wrap serializer's `dict[str, Any]` return annotation as the model's serialization schema. Both serializers are now unannotated, so each component lists its properties; `ObservationPoint` requires `date` and `value`, which its serializer always writes. `test_every_object_schema_names_its_properties` fails on any component that is an object with no properties. `SpecObservations` joins the models dumped under `exclude_none`, and the file selector's query parameter schemas come from `MintOptions`. The first-answer scenarios fixture moves only its `capability_version`, which hashes the search wire schema.
…members The served schema said an omitted scope member means unbound, which holds only on a `statvar_set` input. On a `spec_id` input each supplied member replaces the stored spec's value whole and an omitted member keeps it (C42), so `entities` naming one role unbinds the spec's other roles. The `Scope` docstring now states both inputs, and `window`, `entities` and `facets` each carry a description of what they replace.
…plain statements Docstrings, comments, field descriptions and the datasets docs this branch added read as present-tense sentences. The `_held_wire` test helper's docstring names what it returns. No behaviour changes.
This was referenced Oct 21, 2026
Merged
Migration 0037 adds `version_record_recency` on (subject_id, recorded_at DESC, content_version DESC). `_servable_page_query` in src/datasets_api/version_store.py reads a single subject's page with a keyset over that index, so a page reads its own rows however long the history. Several served ids keep the grouped query. Both shapes compare the cursor as a row, `(recorded_at, content_version) < (...)`.
jm-rivera
marked this pull request as ready for review
October 23, 2026 10:28
# Conflicts: # tests/fixtures/first_answer_place_scenarios.json
# Conflicts: # tests/fixtures/first_answer_place_scenarios.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Datasets client can pace its polls, fetch any stored version's file by format, list a dataset's versions without a principal, and read how long the server holds each version's frame. Every pending or running operation response carries
operation.retry_after_msand aRetry-Afterheader, a quarter of the time sinceOperation.submitted_at, held between 1,500 and 8,000 ms. A done poll carries neither.GET /api/datasets/{id}/{content_version}/file?format=&bundle=&layout=answers 302 to one file of that version (src/datasets_api/doors/files.py). It signs a stored copy first, renews an aged copy whose bytes survive, renders from the stored frame only when no copy survives, and refusesartifact_expiredonly when the frame's bytes are gone too. The rendering uses the version's recorded minting options with the requested format, bundle and layout applied.GET /api/datasets/{id}/versionsserves underdatasets.snapshotand pages withlimit(1 to 500, default 100), a cursor bound to the path id,next_cursorandcomplete. An id no dataset has refuses 404spec_not_found, where the old route answered an empty list. A single served id reads its page through a keyset overversion_record_recency(0037_version_record_recency.sql, onsubject_id, recorded_at DESC, content_version DESC), so a page reads its own rows however long the history; several served ids keep the grouped query, and both compare the cursor as a row. On 20,000 versions of one subject in Postgres 17, a page at offset 10,000 took 0.73 ms where the grouped query took about 260 ms. The migration'sCREATE INDEXtakes a SHARE lock and fails after its 5 s lock timeout.Every Manifest the mint, definition GET, snapshot and done-poll routes return carries
retention, either{"pinned": true}or{"pinned": false, "held_until": ...}at the frame'sstored_atplus 30 days. A pin promotes the frame retention reads, aged or servable, so the next Manifest reports pinned.LiveArtifactStore.copyreads a 404 as an expired artifact only when the source object is gone. A missing destination bucket is a retryable 503.The OpenAPI document names every operation
{surface}_{route.name}, declares one tag per surface, documents the 202 withLocationandRetry-After, and marks no field required that the wire can omit.SlotOptionandObservationPointlist their fields, where a wrap serializer'sdict[str, Any]annotation had made them open objects, andtest_every_object_schema_names_its_propertiesguards every component. TheScopeschema says that on aspec_idinput each supplied member replaces the stored spec's value whole, soentitiesnaming one role unbinds the others. Docstrings describe the website proxy as a per-IP fixed window kept in Postgres, failing open, withAuthorizationpassed upstream unread.The spec-kit pin moves to 0f1a106 (DevelopmentFinanceObservatory/spec-kit#10), which maps the codes the selector and versions routes serve. The pin moves again to the release commit once that release is cut. The
list_versionschange in DevelopmentFinanceObservatory/ai#30 merges once this deploys to dev. Until then, the harness backfill preflight refuses, an unknown id raises in mint recovery, and recovery sees only the newest 100 versions. Conformance clauses C182 to C186 state the new contract.Testing:
just checkat ec65f9c against spec-kit 0f1a106 passed 37,908 tests, with 132 skipped and 5 xfailed. Its one failure,test_a_surface_built_while_serving_a_release_binds_it_and_the_session_s_afterintests/graph_release/test_views.py, fails the same way on dev and is fixed by #65. With the index,tests/datasets_api,tests/dataset_storeand the migration immutability test passed (3,115 passed, 82 skipped).