Repository navigation
fix(common): read every page of a dataset's versions and send no principal - #30
Merged
Merged
Conversation
…cipal
apis now serves `GET /api/datasets/{spec_id}/versions` with no principal, pages it with `next_cursor` and `complete` (100 versions a page by default), and refuses an id no dataset has with 404 `spec_not_found`. `LiveApisClient.list_versions` read one page with the datasets principal, so mint recovery and recipe replay saw only the newest 100 versions and an unknown id raised. It now follows `next_cursor` until the complete page, returns `()` on `spec_not_found`, refuses a page outside the contract (`response_invalid`), and stops at a repeated cursor or past `VERSIONS_MAX_PAGES`.
The recipe backfill's preflight read versions to prove the URL and the datasets principal, and the versions route no longer reads the principal. It now reads the recipe of a derived pair no frame has, which apis answers `recipe_not_found` only after it accepts the principal. The recorded apis wire pairs are copied from the apis recording of the new route.
A dataset that goes away between two page reads raised nothing and returned no versions, dropping the pages already read. `list_versions` now maps `spec_not_found` to `()` only on its first request and raises it on a later page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LiveApisClient.list_versionsinpackages/common/src/common/live_apis.pyfollowsnext_cursoruntil the complete page, returns()when the first page refusesspec_not_found, raises any later refusal, refuses a page outside the contract asresponse_invalid, and stops at a repeated cursor or pastVERSIONS_MAX_PAGES. It sends no principal. This matches apis, which servesGET /api/datasets/{spec_id}/versionswith no principal, pages it withnext_cursorandcomplete(100 versions a page by default), and refuses an id no dataset has with 404spec_not_found(DevelopmentFinanceObservatory/apis#68). The earlier call read one page with the datasets principal, so against that server mint recovery and recipe replay would see only the newest 100 versions and an unknown id would raise.The recipe backfill's preflight (
services/harness/src/harness/recipe_backfill.py) proves the URL and the datasets principal before it walks frames by reading the recipe of a derived pair no frame has, which apis answersrecipe_not_foundonly after it accepts the principal. The versions route takes no principal, so a versions read cannot prove it.packages/common/tests/fixtures/dfo_apis_live_refresh_wire_pairs.jsonis copied from the apis recording of the new route.Merge this after the apis branch deploys to dev. Against today's dev every call fails, because the route refuses a call with no principal and its answer carries no
complete.Testing:
just checkat 7b9a5b2 passed lint, the type check and 11,186 tests, with 22 skipped and 5 xfailed.