Repository navigation
Conversation
…record at apis 99e62038c (#26) served/openapi.json and SOURCE.json come from dfo-apis 99e62038c. NoDataOption and SentenceToken carry entity_choice, the index into no_data.ambiguous_entities of the group an option settles or a slot asks, ParsedEntity carries an optional name, and Part.entity_refs has one entry per restatement row. The 32 recorded scenarios are recorded at that commit, including search-place-ambiguous-twice, two entity_ambiguous groups of one text. The long-build questions move to 1964 and 1965, which dev had not built, and the poll-pacing derivation cycles its waits over any recorded build of three or more running polls. Co-authored-by: Claude <noreply@anthropic.com>
…tatuses (#13) * fix(registry): map Presentation write refusals at the write routes' statuses On presentation.write, binding_invalid, constraint_violation and resource_exhausted map at 400, frame_unavailable at 404 and release_mismatch at 503, the statuses the Presentation write routes serve. presentation.render keeps 422 and 409. frame_unavailable's definition names the causes it covers: a frame never stored, expired, revoked, or whose stored artifact is missing, quarantined, or fails its digest check. A frame store that fails to answer is service_unavailable, which presentation.write and presentation.render already map at 503. * fix(registry): map edit_conflict on presentation.render The render routes read the stored revision they draw, and that read refuses edit_conflict (409) when the component has no current revision. edit_conflict maps to presentation.render at 409, and its definition names that case beside a revise of a revision that is no longer current. * docs(registry): name every case frame_unavailable covers frame_unavailable's definition names each case Presentation refuses with it: a frame never stored or published, expired, or revoked, a spec in quarantine, and a stored artifact that is missing, fails its digest check, does not decode, or holds more rows than Presentation loads.
…son (#21) A no_data part becomes a gap of its reason's gap_kind whatever diagnostics.timed_out says. Search marks every part it left unresolved at its time limit with the reason timed_out, so a part it answered before the limit keeps its reason, its message and its choices. answer-timed-out now carries reason timed_out on the unfinished part, and answer-timed-out-keeps-reasons pins a resolved part that keeps variable_not_resolved under a timed-out answer. The validator's answer_gaps reads the part's reason alone, and a new validator test pins that a diagnostics that is not an object holds no warnings.
…se URL rule (#23) A POST for a build identical to one pending or running returns that operation, so a client that stops waiting can post again to join it. fetch waits with no whole-call deadline by default, and a download that delivers no byte for 30 s is a retryable network_error. The new base-url rule orders the argument, DFO_BASE_URL and the public default, with 14 vectors replayed by a reference resolver in tests/test_rules.py.
…lumns for reading (#24) A fetch now posts "without_party_totals": true unless the caller passes party_totals (Python) or partyTotals (TypeScript). A role the caller binds keeps what it binds. The wire flag is unchanged; only the client default moves. The new column-order rule (rules/column-order.json, nine slots with vectors) puts time period, provider, recipient, subject, purpose, value, unit, flow type and provenance first, then the served order, with stat_var last. A client hashes the downloaded file before it orders the columns, so digests and refs keep the server's file. Every scenario is re-recorded against apis dev at c9a1b905, and served/openapi.json is recaptured. Two new recorded scenarios, fetch-spec-party-totals-left-out and fetch-spec-party-totals-kept, pin the default and the option. Every accepted ref example now carries the rendering, parts, content version and sentence of a member its scenario served, so ref-one-member pins the plain rendering from fetch-spec-party-totals-kept and ref-answer-member comes from answer-definite-fetch. The validator checks both, with a fault row each. Every scenario is re-recorded against the deployed dfo-apis dev app at d80df9f5f, with the served document captured from that commit. The long-build scenarios start at 1967 and 1968, years dev had not built.
…he base URL and role keys (#25) The client-identity rule (rules/client-identity.json, 18 vectors) has every API request carry X-DFO-Client: <client>/<version> (<runtime>), such as dfo-python/0.1.0 (CPython 3.14) or dfo-ts/0.1.0 (browser). A caller's headers or header provider naming it raise header_invalid, and no request to another origin, a signed storage URL included, carries it. Scenario records keep no request headers, so a client's replay checks that each path and origin_path request carries the header and each url request does not. The column-order rule becomes reading-order (rules/reading-order.json, 10 vectors with rows). Columns run: the filled reading slots, then the other columns whose values differ between rows, then the constant columns, with stat_var last. A stack puts part first and judges the groups over the stacked table. Each member's rows are sorted by the filled slots without value and the time period, then by the varying columns (stat_var among them when it varies), then by the time period, so each series' years read together. Keys sort ascending with nulls last, ties keep served order, and members keep member order. The validator reads served Parquet once per file through served_table and checks every scenario's columns and every vector's rows. 28 scenarios change their columns only; no body file changes, so digests stay the server's. A base URL is valid when it is an absolute http or https URL with a host and no userinfo (an empty one included), query, fragment, dot segment, character outside visible ASCII, or IPv4 host in a form other than four decimal numbers. A relative URL is valid only in a TypeScript client inside a browser page, which resolves it against the page's address, and a valid value is used less its trailing slashes. Seven new vectors in rules/base-url.json pin these cases. The User-Agent sentence now says TypeScript sends dfo-ts/<version> outside a browser page or browser worker, in Node, Deno, Bun or a server runtime such as Cloudflare Workers. The role-keys rule (rules/role-keys.json, 13 vectors) names the keys of entities=: place for the subject, posted as observationAbout, and origin and destination for flows, posted as themselves. Spec.roles and specRoles(spec) list a spec's roles under those keys, a flow spec takes both directions, and a key outside the spec's roles raises invalid_argument naming the typed key and the spec's roles, so observationAbout and column names such as financingRecipient are refused. A spec that names no role posts place as observationAbout and leaves the rest to the server.
mint-async-build and operation-resume need a 202, which only an unbuilt spec gives, and each live record builds its two specs for good. The tool named their start years as constants, so every record needed new years by hand. Context.async_spec now walks start years 1960 to 2000 for the DAC and OECD donor questions and takes the first spec that is one definite part, sits just above the sync budget, repeats no earlier pick, and has no version on the versions route. It refuses when the range is used up. The party-totals scenarios narrow a built DAC spec and need no unbuilt one, so they keep their own question, PARTY_TOTALS_FLOW. Co-authored-by: Claude <noreply@anthropic.com>
* feat(registry): register the two scope-choice refusal codes Add requested_option_unavailable and conflicting_requested_options as search refusals on the dataset_resubmit door (problem details, 422), in the registry and its packaged mirror. * fix(registry): define the scope-choice refusals as Search raises them Each definition covers every case dfo-apis refuses with it: an axis the part does not offer, an option the axis does not list or an unavailable one, and entries that conflict or rule out every option of an axis. The client contract gives both a fix_request row.
#33) * fix(client-contract): ask the file selector only for the keys it reads A file-selector rule names format, bundle and layout; the validator, capture tool and selector scenarios derive the query from it. The validator compares values by one rule and reports malformed vectors, and the base-URL and client-identity rules close the cases where Python and WHATWG clients disagreed. * fix(client-contract): read a resumed build's rendering and keep nested columns out of sort keys A resumed operation's result takes the rendering its build posted, and refs are checked against the build that served their file. List and struct columns never sort, the reference same_value is the validator's, selector queries lowercase booleans only, and the capture's OECD prerequisite keeps party totals.
… same_value (#34) The nested-cells vector's flags column holds one item type, the validator reports a vector column Arrow cannot build, and served_table judges flat columns by same_value as it does nested ones.
…' codes (#30) * feat(contract): add derived refs, an output digest and analysis rules DatasetRef format 2 names derived members: an ops_ spec with its recipe hashes, the ocv_ version, an output digest (out_), the root versions with their renderings, the execution identity and resolved periods. Format 1 refs validate unchanged. New rule vector sets pin the output digest (JCS over roles, period, value and unit, leaving out stat_var and provenance), output comparison (== against same_data), operation outputs (share_of in percent, join in long format, compare as data minus other, growth as a fraction), rolling composition (rolling_populations_differ) and share publication (slot URLs survive a live replace; a snapshot cannot be replaced). The registry gains the datasets.analysis and presentation.shares endpoints and the codes allowance_exhausted, rolling_populations_differ, several_members, share_not_replaceable, write_token_required and write_token_invalid. operations.json states each verb's output shape and pairs sum and compare by role. * feat(registry): register the analysis engine's refusal codes The datasets.analysis endpoint gains a row for each of the 34 refusal codes the analysis engine raises, including compare_no_key_matched and sum_of_no_key_matched for paired calls, each 422 with the fix_request action. operations.json lists unmatched_key under sum's and compare's left-out reasons and describes their paired form: `on` pairs data's members with other's by role, and compare returns data minus other. * feat(registry): map the Datasets codes the analysis route serves datasets.analysis also answers 27 existing Datasets codes, for roots it cannot read or build (snapshot_not_found, artifact_expired, recipe_not_found, rendering_not_recorded and others) and for platform faults. Each maps at the status it has on the Datasets endpoints. * feat(registry): map timeout on analysis and the platform codes on shares datasets.analysis answers timeout at 504 when a run passes its deadline. presentation.shares maps not_found, request_invalid, frame_unavailable, edit_conflict, service_unavailable, internal_error and timeout at the statuses the presentation write endpoint uses. * feat(registry): map the share routes' refusals and add the SDKs' left-out warning presentation.shares maps binding_invalid, schema_invalid, constraint_violation and resource_exhausted at 400 and release_mismatch at 503, the statuses the components routes use, and a new payload_too_large at 413 for a share body over the routes' limit. pin_not_permitted is retired: the pin route admits any caller under its allowance. The client contract gains the payload_too_large action row and the series_left_out warning, which a client issues when a derived frame left series out for missing data. * feat(registry): map the card allowance on presentation writes allowance_exhausted answers 429 on presentation.write, where creating or revising a card spends the caller's daily card allowance. * fix(contract): name each value's variable in the output digest The output digest (out2) holds each row's stat_var, so frames that pin the same numbers to different variables differ; every derived identity in refs/ and output-comparison.json is computed by the dfo-apis runner, and the validator checks that equal identities agree across fixtures. allowance_exhausted is never retried and a Retry-After above 60 s raises at once. write_token_invalid takes a new action, stop. The series-left-out rule fires on a derived manifest's left_out, growth's left-out reasons are stated with vectors, a derived member's parts are empty, a growth window starts before it ends, and a scenario fetches dataset refs only. share lists unmatched_key among its left-out reasons. * fix(contract): name a computed variable by what was computed The output digest names a variable that compare, sum_of, share_of or quotient_of writes by its verb and its inputs' variables, so equal computations over equal variables share a digest across recipes, engine releases and rebuilds that move no number, and swapped sides differ. Output-digest vectors carry a variables map; three new vectors pin the naming. * fix: name minted variables by a fixed-size digest and admit grouped roots A minted variable is named mv_ plus the first 32 hex of SHA-256 over the RFC 8785 JSON of {verb, slot: sorted input names}, 35 characters at any depth. rules/README.md states the form, three output-digest vectors are regenerated, and rules.schema.json requires the mv_ form in vectors. A derived ref's root takes its own root_rendering, which admits the groupby (by, op sum, levels) DFO read a summed root with. The artifact rendering selector still refuses groupby. Three ref vectors cover a grouped root, a groupby with another op, and groupby on an artifact. The client contract validator compares a root's groupby with the one its scenario posted, and leaves it to the schema when none was posted. * fix: pin nested minted names and record grouped-root ids from a real run Output-digest vectors carry name_documents, the document each mv_ name hashes, and a new quotient-of-a-sum-of vector names a minted variable whose input is another minted variable. The validator checks that each name hashes from its document and that every nested mv_ name has one. The grouped-root ref vectors record by sorted, as DFO writes it, with ids from a dfo-apis run_analysis. rendering and root_rendering share one selector definition, and root_rendering adds only groupby. The validator's derived-identity check includes each root's rendering.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases
devat455e958tomain: client-contract changes (#21, #23, #24, #25, #26), the capture tool's long-build pick (#28), registry changes (#13, #29), and the licence and public docs (#7).The registry adds two codes,
requested_option_unavailableandconflicting_requested_options, and maps the Presentation write refusals at the write routes' statuses. dfo-apisdevpins455e958, and the apis release movesmain's pin there from0d19172, so merge this one first and with a merge commit, so455e958is reachable frommain. dfo-ai pins708f293on both branches, whichmainalready holds.