Skip to content

feat(http): multi-level fan-out for sl.http - #196

Open
kvlonge wants to merge 2 commits into
mainfrom
feat/http-multi-level-fan-out
Open

kvlonge wants to merge 2 commits into
mainfrom
feat/http-multi-level-fan-out

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

sl.http fan-outs can now nest: a fan_out can be another fan_out's over, as many levels deep as needed (orders → items → notes).

  • Each key carries the fields of every level above it, so a level-3 path such as /orders/{order_id}/items/{item_id}/notes is filled with the right pair. A new optional key= on paginate and fan_out renames the key a level feeds down, for APIs that call every level's id id.
  • Every level keeps its own paginator, concurrency, on_status, and (through the sl.http it comes from) its own retries and rate limit.
  • Errors in nested jobs name the keys that led to them, such as /tree/orders order_id=5 → /tree/orders/5/items item_id=51 → GET …/notes?page=2: HTTP 500. Each breadcrumb step is redacted with the secrets of the level that made it. The retryable flag (exit 75) is preserved.
  • Plan checks cover every level: every placeholder must be given by a level above, and a key name a level above already gives is an error (rename it with key=).
  • An incremental cursor goes on the top level; lower levels only see the keys it picks, so a rerun requests nothing below it.
  • Only the last level's rows load. A level that should be its own table is its own table function, which repeats the levels above it.

Existing jobs are unchanged. The job JSON for one-level fan-outs is pinned byte for byte. One-level error texts get no breadcrumb, and plan errors keep their wording and order (two ordered cases fail on the first version of this branch).

Docs: "Multi-level fan-out" in sources/python.mdx, marked as coming in 0.37.0.

Tests:

  • packages/sqlload/tests/test_http_jobs.py, against a three-level stub API whose notes endpoint returns 404 for a wrong order/item pair:
    • exact rows and the exact set of requests at concurrency 3 per level;
    • a 404 at level 2, with and without on_status;
    • a 500 at level 3, with the full breadcrumb, exit 75, and per-level retries;
    • a cursor rerun that requests only the top level;
    • key= renames at two levels;
    • byte-identical one-level job JSON.
  • crates/sqlload/src/http_source/tests/test_job_problems.rs: multi-level plan checks and the one-level ordering.

71 http job tests pass and make check is clean. Mutation check: dropping the ancestor fields from forwarded keys fails 4 tests.

Follow-ups, not in this PR: inner-level plan messages don't say which level they come from, and listed object keys count every common field as given.

…ame them

A fan_out whose select picks keys can now be another fan_out's over, as
deep as they nest. Each key keeps the fields of the keys above it, so a
third level's path can use {order_id} and {item_id}; key= renames a
level's field. Every level keeps its own paginator, concurrency,
on_status, retries and rate limit. plan checks each level's
placeholders against the levels above it, and an error names the keys
that led to the failing request. One-level jobs are unchanged.
… step with its own level's secrets

Also run key= renames end to end and correct the in-flight request count in the docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant