Repository navigation
Conversation
…ame them
A fan_out whose select picks keys can now be another fan_out's over, as
deep as they nest. Each key keeps the fields of the keys above it, so a
third level's path can use {order_id} and {item_id}; key= renames a
level's field. Every level keeps its own paginator, concurrency,
on_status, retries and rate limit. plan checks each level's
placeholders against the levels above it, and an error names the keys
that led to the failing request. One-level jobs are unchanged.
… step with its own level's secrets Also run key= renames end to end and correct the in-flight request count in the docs.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sl.httpfan-outs can now nest: afan_outcan be anotherfan_out'sover, as many levels deep as needed (orders → items → notes)./orders/{order_id}/items/{item_id}/notesis filled with the right pair. A new optionalkey=onpaginateandfan_outrenames the key a level feeds down, for APIs that call every level's idid.concurrency,on_status, and (through thesl.httpit comes from) its own retries and rate limit./tree/orders order_id=5 → /tree/orders/5/items item_id=51 → GET …/notes?page=2: HTTP 500. Each breadcrumb step is redacted with the secrets of the level that made it. Theretryableflag (exit 75) is preserved.key=).Existing jobs are unchanged. The job JSON for one-level fan-outs is pinned byte for byte. One-level error texts get no breadcrumb, and plan errors keep their wording and order (two ordered cases fail on the first version of this branch).
Docs: "Multi-level fan-out" in
sources/python.mdx, marked as coming in 0.37.0.Tests:
packages/sqlload/tests/test_http_jobs.py, against a three-level stub API whose notes endpoint returns 404 for a wrong order/item pair:on_status;key=renames at two levels;crates/sqlload/src/http_source/tests/test_job_problems.rs: multi-level plan checks and the one-level ordering.71 http job tests pass and
make checkis clean. Mutation check: dropping the ancestor fields from forwarded keys fails 4 tests.Follow-ups, not in this PR: inner-level plan messages don't say which level they come from, and listed object keys count every common field as given.