[jdatcmd]

Fixes #2046.

What was wrong

rest_get_list (src/columnar_iceberg_rest.c) follows next-page-token and keeps every page's JSON document until the list is built, because each list item points into its page. Each response was capped at 64 MB (REST_MAX_RESPONSE). The only bound across pages was REST_MAX_PAGES (100000) and the repeated-token check. A catalog that kept sending fresh tokens could therefore make one backend hold up to 100000 × 64 MB.

This reaches iceberg_rest_namespaces, iceberg_rest_tables and the namespace walk.

The fix

rest_get_list adds up VARSIZE of each page's document and refuses the list once the total passes REST_MAX_RESPONSE. The error is ERRCODE_PROGRAM_LIMIT_EXCEEDED (54000): iceberg: the REST catalog <what> is larger than 64 MB across its pages. A list may now hold as much as one response could, and no more.

Out of scope: the namespace walk calls rest_get_list once per namespace, so the cap applies to each list, not to the whole walk. The walk is still bounded by depth (64) and by its per-list caps. Say if you want a per-call total as well.

Test

The fixture catalog (test/iceberg_rest_server.py) gains --padded-pages N:BYTES. It serves N pages, each padded with an unread string BYTES long and carrying a fresh token until the last. Two checks join the paging checks in test_iceberg_rest_scan.py (§274), so this PR adds no new TESTS.md section:

Test	Reading
test_paging_pages_within_the_response_cap_together_still_list	15 pages of 4 MiB (60 MiB) still list events
test_paging_pages_over_the_response_cap_together_are_refused	17 pages of 4 MiB (68 MiB) are refused with 54000

Red, on main's code (PG18)

E           AssertionError: paging: 17 pages of 4 MiB (68 MiB in all) are refused with 54000, not held in memory: got 'no error' want '54000 over the cap'
accounting: 3 pass + 1 fail + 0 unrun = 4
1 failed, 3 passed, 126 deselected in 7.67s

The 15-page control passed on main, as it should.

Said plainly: my first red run failed for the wrong reason, a bug in my own test. With SET statement_timeout ...; SELECT ... as one string, psycopg's fetchall() read the SET's result, so both checks were refused as failed queries. I removed the SET; 17 local pages need no timeout. The red above is from the corrected test.

Green

-k paging, PG18:                                   4 passed
test_iceberg_rest_scan.py test_iceberg_rest.py test_iceberg_rest_server.py test_iceberg_rest_vended.py
  PostgreSQL 18.6:   accounting: 203 pass + 0 fail + 0 unrun = 203   203 passed
  PostgreSQL 17.11:  accounting: 203 pass + 0 fail + 0 unrun = 203   203 passed
test_docs_cover_the_corpus.py test_compare_to_bash.py test_harness_deps.py:
  accounting: 962 pass + 0 fail + 0 unrun = 962   106 passed
docs_style.sh: PASSED (54)

Registries

• expected_tests.txt: cluster_tests 5350 → 5352, by collection (6230 collected, less guard_tests 878).
• TESTS.md §274 gains two rows. No ledger row is added, because these are pytest-only checks.
• CHANGELOG has an entry under Fixed.

Found by

The security audit of 2026-11-07. It was a suspected finding until the red run above proved it.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD Ready for review. This one comes from the 2026-11-07 security audit and was written test-first; the red and green runs are in the description.

[OffgridwithJD]

Approved at 1d0a87b.

What I ran

• PR head: test_iceberg_rest_scan.py on PG18.6 (private prefix): 130 passed, 130 checks.
• Mutant: the held > REST_MAX_RESPONSE check disabled. Only test_paging_pages_over_the_response_cap_together_are_refused fails (1 failed, 129 passed). On main the 17-page list was held in full.
• Boundary: 15 × 4 MiB lists and 17 × 4 MiB is refused, so the test bounds the cap from both sides. Doubling the cap would let 17 pages through. Halving it would refuse 15. Counting only the last page would let 17 pages through.

Code

• held sums VARSIZE(doc) across pages. That is the parsed size kept per page, not the bytes on the wire, which is the right thing to bound, since the parsed documents are what stays in memory.
• The check sits right after each page is parsed. Peak memory is at most the cap plus one page.

Non-blocking

1. The message says "64 MB", but the cap is 64 MiB. The per-response refusal it pairs with reports bytes (exceeded 67108864 bytes). Either unit works; matching the other message would make the two easier to tell apart in a log.
2. User docs: my docs audit PR (opening shortly) adds the 64 MiB per-response cap to sql-reference.md's REST catalog section. Once both land, I'll add one sentence there saying a paged list shares that cap, unless you'd rather add it here.
3. Registries: cluster_tests 5350 -> 5352 is on 17d9287. Main has since moved (#2043). The diff merges cleanly, and CI collection is what checks the count.

🤖 Generated with Claude Code