Skip to content

Add pagination iterator helpers (iter_* or paginate()) to avoid manual while-loop boilerplate #40

Description

@makegov-mark

Summary

Every paginated endpoint in the SDK requires callers to write a manual while loop to collect all pages. As soon as a second paginated endpoint was added (get_entity_budget_flows, which now returns PaginatedResponse[dict[str, Any]]), the same boilerplate appeared again. We should ship a first-class pagination helper before more callers bake the loop into their code.

Boilerplate this would replace:

all_flows: list[dict] = []
page = 1
while True:
    resp = client.get_entity_budget_flows(uei, page=page, limit=100, fiscal_year=2024)
    all_flows.extend(resp.results)
    if resp.next is None:
        break
    page += 1

Options for discussion

Three shapes are on the table. The issue should land a decision before implementation begins.

Option 1 — Per-endpoint generator methods

Add an iter_* method alongside each list_* / sub-resource method:

for flow in client.iter_entity_budget_flows(uei, fiscal_year=2024):
    process(flow)
  • Discoverable via IDE autocomplete.
  • Fully typed — return type is Iterator[T] where T matches the endpoint's item type.
  • Downside: doubles the public surface area (one get_* + one iter_* per paginated endpoint).

Option 2 — Generic paginate() helper

A single helper (as a method or free function) that accepts any paginated callable:

for flow in client.paginate(client.get_entity_budget_flows, uei, fiscal_year=2024):
    process(flow)
# or as a free function:
from tango import paginate
for flow in paginate(client.get_entity_budget_flows, uei, fiscal_year=2024):
    process(flow)
  • Minimal surface area — one new symbol covers every current and future paginated endpoint.
  • Call-site is slightly awkward (passing the method as a callable).
  • Generic typing is harder to express precisely (ParamSpec / TypeVar gymnastics).
  • Works for both page-based and cursor-based conventions if implemented carefully.

Option 3 — Method on PaginatedResponse

Attach iteration behavior to the response object itself:

resp = client.get_entity_budget_flows(uei, fiscal_year=2024)
for flow in resp.iter_remaining(client):
    process(flow)
  • No new top-level surface; discovery happens on the object you already hold.
  • Mixes data and behavior: PaginatedResponse becomes aware of the client and of how to fetch more pages, which doesn't match the current SDK design.
  • Partial-page scenarios (user already consumed some results from resp) are awkward.

Decision criteria

The decision should cover:

  1. Which shape (1, 2, 3, or a blend — e.g. Option 2 as the implementation primitive + Option 1 as thin wrappers built on top of it).
  2. Scope of the first pass: page-based pagination only, or cover cursor-based endpoints in the same PR?

Out of scope

  • Auto-pagination by default: changing list_* / get_* methods to return an iterator would be a breaking change to existing callers.
  • Async iteration: no async client today; that's a separate concern.
  • Bulk/concurrent page fetching utilities.

Acceptance signal

This issue is resolved when the team has agreed on a shape and scope. Implementation is a follow-on PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

DXDeveloper experienceenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions