Repository navigation
Conversation
A location now sends its snapshot with Content-Encoding: gzip. The server keeps the 64 MiB wire bound and, after authenticating the location token, streams the body through a 512 MiB decompressed-size bound into the JSON parser. Encodings other than gzip get 415, and a body over either bound gets 413. A server that refuses the encoded body with 400 or 415 gets the snapshot again as plain JSON, so new locations still publish to older servers.
…ge to resend plain A snapshot whose plain JSON exceeds the wire bound can never be accepted uncompressed, so a 400 for the gzip body is returned as is instead of being masked by a second upload that ends in 413.
A parsed snapshot holds about 7.6 times its JSON size on the server, so a body just under 512 MiB peaked at 3.8 GiB RSS. 320 MiB keeps four times the headroom of a 10,000-model project with column lineage.
kvlonge
marked this pull request as ready for review
November 8, 2026 18:36
kvlonge
enabled auto-merge (squash)
November 8, 2026 18:36
Contributor
Author
|
Folded into the single combined sqlbuild PR (D625/D626), which the SQLBuild lane is assembling now and opens once #1377 lands. Branch kept. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A location publish posts its whole snapshot as JSON, and the server caps bodies at 64 MiB. A 10,282-model project already makes a 62.0 MiB snapshot, so larger projects cannot publish. Gzipped it is 2.8 MiB.
Changes
sqb locationsends snapshots withContent-Encoding: gzip.Verification
main: an unauthenticated gzip bomb gets 401 undecompressed; an authenticated bomb gets 413; plain JSON over 64 MiB gets 413;brgets 415 and non-gzip 400; a 10,282-node snapshot over 64 MiB publishes; an invalid one keeps its 400; older servers get the plain resend.sqb serverplussqb location publishpublish a generated 10k-model DuckShop end to end./usr/bin/time -v, freshsqb server, SQLite): 480 MiB for that DuckShop publish (idle 15 MiB). A parsed snapshot holds about 7.6x its JSON: a body just under 512 MiB peaked at 3,879 MiB, so the bound is 320 MiB, 4x the snapshot with column lineage (74 MiB). Just under 320 MiB peaked at 2,420 MiB; 328 MiB got 413 at 1,610 MiB.make check-cipasses; the publication suite and TS client passed on the first commit.DuckShop: no edge case added, because this changes the HTTP transport, not a project shape, data value or lifecycle sequence.