Repository navigation
[3.0.0] Emit finally once, and run it on return, break and continue - #16
OffgridwithJD wants to merge 9 commits into
Conversation
Every plx language is trusted, so any role with USAGE can send arbitrary input to the transpiler through CREATE FUNCTION or DO. - plx_span_text: skip type-annotation tokens (which carry no source span) at the edges of a span. A span beginning on one computed its length from a NULL pointer and crashed the backend (plxjs/ts/php/ruby/python3). - plxtsql, plxgo, plxcobol: guard the expression emitters with check_stack_depth() and the PLX_MAX_DEPTH counter. Deeply nested calls, conversions or subscripts overflowed the C stack (SIGSEGV). - finally/ensure: cap finally-inside-finally nesting at 8 levels; each level doubled the generated body, so a few hundred bytes of source could demand gigabytes. The three copies of the try lowering are now one helper, plx_emit_try, which also runs the normal-path finally body after the protected block, so a finally that raises no longer runs twice. - plxgo fmt.Sprintf: require a letter verb before lowering to a || chain; a '%' followed by a quote consumed half of an escaped '' pair and let Go string text escape into the generated SQL. Regression tests cover each case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…growth Second round of the security audit (shared transpiler core): - emit_string_value: escape every literal backslash and force E''. A string ending in a lone backslash inside an interpolation was emitted bare in an E'' literal, escaping its closing quote so the next literal's text ran as code. The synthetic-string scanners no longer step past the end of their input on a trailing backslash. - Locals: index by lower-cased name in a hash table beside the ordered list; the linear lookup made many distinct variables quadratic. - plxpython3 lexer: resume after the blank/comment lines the indentation scan already covered, instead of rescanning the run from every newline (O(K^2), and line numbers were counted twice). - CHECK_FOR_INTERRUPTS in the lexer loop, the expression rewriter and the local table, so statement_timeout and cancel reach a long transpile. - Ternary rewrite: free the condition and both arms after composing the CASE, so a chain of N ternaries no longer holds N copies of its tail. - plxpython3 call(f"..."): reject an f-string procedure name, which was unquoted with its prefix into a broken identifier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d tests From the adversarial review of this PR: - standard_conforming_strings: literals emitted by plxgo, plxtsql, plxplsql and plxcobol, and Ruby/PHP single-quoted strings, were plain '...' literals. plpgsql reads the body with the caller's setting, so with it off a trailing backslash swallowed the closing quote and the next literal ran as code. A final pass over the generated body (a small SQL lexer that skips comments, quoted identifiers, dollar quotes, E'' and U&'' strings) now rewrites every ordinary literal containing a backslash as E'...' with the backslash doubled. One place covers every dialect and future emitters. - plxpython3 lexer: a #:: annotation line ended the blank-line scan's resume point but not the scan, so runs of annotation lines were still quadratic. The scan now stops at an annotation line. - Locals index: a name of 128+ bytes disabled the hash and brought back the linear scan. Keys are now downcase_truncate_identifier(), as the server reads an unquoted identifier, so every name is indexed and case folding matches pg_strncasecmp/the server for all encodings; the fallback is gone. - Tests: standard_conforming_strings off across six dialects; timeouts around 60,000 locals after an over-long name, 400,000 blank lines and 40,000 annotation lines; and a nesting depth under the 500-level counter with max_stack_depth = 100kB, which exercises check_stack_depth(). - CHANGELOG: the finally cap bounds depth, not output size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jdatcmd
left a comment
There was a problem hiding this comment.
Adversarial review of the last two commits. The rest is #12, which has changes requested.
Tested on PostgreSQL 18.4: builds without warnings, make installcheck passes 13/13, and make differentialcheck (including finally_diff.py 300 1) reports 0 mismatches.
These hold up:
- A
breakinside aswitchinside try/finally inside a loop. - A trigger returning
nullandNEWfrom inside try/finally. - A polymorphic result (
anyelement). commit()in the finally body of a procedure.- Output now grows linearly with nesting. An 8-level body of 209 KB stores 876 KB; under #12 the same shape stored 144 MB.
Must fix
1. The re-raise drops the error's object names
A bare RAISE; (the old lowering) re-raises the whole error. The new re-raise copies only the SQLSTATE, message, detail and hint, so the constraint, table, schema, column and datatype names are lost. Code that catches the error and reads them (for example, an application that maps a constraint name to a form field) silently gets empty strings.
CREATE TABLE t16(id int CONSTRAINT t16_pk PRIMARY KEY);
INSERT INTO t16 VALUES (1);
CREATE FUNCTION ins_fin() RETURNS void LANGUAGE plxjs AS $$
try { execute("INSERT INTO t16 VALUES (1)"); } finally { let z = 1; }
$$;
DO $d$ DECLARE c text; t text; s text; BEGIN
PERFORM ins_fin();
EXCEPTION WHEN unique_violation THEN
GET STACKED DIAGNOSTICS c = CONSTRAINT_NAME, t = TABLE_NAME, s = SCHEMA_NAME;
RAISE NOTICE 'constraint=% table=% schema=%', c, t, s;
END $d$;| build | output |
|---|---|
| master | constraint=t16_pk table=t16 schema=public |
| this PR | constraint= table= schema= |
Suggested fix: also capture COLUMN_NAME, CONSTRAINT_NAME, PG_DATATYPE_NAME, TABLE_NAME and SCHEMA_NAME, and pass them back with RAISE ... USING COLUMN = ..., CONSTRAINT = ..., DATATYPE = ..., TABLE = ..., SCHEMA = .... The diagnostics come back as '' when absent, never NULL, so RAISE accepts them. Unlike DETAIL and HINT, these five fields are not printed in the error text, so they can be passed on every branch of the existing IF without a guard of their own. Add the probe above as a regression test.
If this stays as it is, the CHANGELOG must list the lost fields next to the CONTEXT note.
Should fix
2. A reused loop label makes break skip the finally
exit_crosses looks a label up in cx->labels, which only grows. A loop inside the try that reuses the label of the target loop is found first, so the break is judged not to cross the frame and is emitted as a plain EXIT:
CREATE FUNCTION lbl() RETURNS int LANGUAGE plxjs AS $$
let r = 0;
outer: for (let i = 0; i < 2; i++) {
try {
outer: for (let j = 0; j < 1; j++) { r = r * 10 + 1; }
break outer;
} finally { r = r * 10 + 9; }
}
return r;
$$;
SELECT lbl(); -- 1; expected 19 (the same body with the inner label renamed returns 19)JavaScript rejects a label nested in a same-named label, but plx accepts it (and so does plpgsql). Either remove a label from the list when its loop ends, or reject a label that shadows an enclosing one.
3. finally_diff.py counts any error as a match
Python errors only with ValueError("x"), but the harness maps every plx failure to ERROR. A wrong SQLSTATE, a plpgsql runtime error in generated code, or record is not assigned yet would all pass. Please compare the SQLSTATE (P0001) and message x, not just "some error".
4. Say where finally still does not run
The CHANGELOG says the finally body runs on an error. It does not run on a query cancel or a failed assert, because WHEN OTHERS does not catch query_canceled or assert_failure. Both verified:
statement_timeout = '300ms'aroundtry { execute("SELECT pg_sleep(2)"); } finally { raise("warning", ...); }: no warning.try { assert(false, "boom"); } finally { raise("notice", ...); }: no notice.
This was true before too, but the new text promises more than the code does. One sentence in the CHANGELOG and the dialect chapters is enough.
CI has not run on this PR yet (fork run waiting for approval).
The try/finally lowering emitted the finally body twice (normal and error path), so nesting doubled the output per level, and a return, break or continue inside the try left without running finally at all. A try now pushes a frame while its body and catch arms are parsed. An exit that would leave the frame is emitted as a placeholder, since the parser does not yet know whether a finally follows; plx_emit_try then resolves it: without a finally, as the plain statement (or a placeholder for the enclosing frame); with one, as "save the return value, note the pending exit, EXIT the labeled block". After the single copy of the finally body, a saved error is re-raised with its SQLSTATE, message, detail and hint, and the pending exit is completed in the enclosing frame's context. Labeled break/continue resolve their target through the labels recorded with the frame they were declared in. The return value lives in a hoisted __plx_ret typed from the function's result (RECORD for triggers and record, ALIAS FOR $0 for polymorphic results). The 8-level finally nesting cap is no longer needed and is removed. Source containing U+0001, the placeholder delimiter, is rejected. Checked against real Python with 1,200 randomly generated programs mixing nested try/except/finally, loops, return/break/continue (also inside finally) and raised errors: no mismatches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test/finally_diff.py generates random programs mixing nested try/except/finally, loops, return/break/continue (also inside finally) and raised errors, runs each in Python and as a plxpython3 function, and requires the results to agree. `make differentialcheck` runs 300 cases after the existing differential check, so CI covers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f18e5ab to
0d25407
Compare
"let x /*:: int */ = 0;" drops the "= 0" (commandprompt#20 makes that form an error). The test's result does not depend on it, but it should use the documented form so it keeps passing once commandprompt#20 is merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # test/expected/plxjs.out # test/sql/plxjs.sql
…ys linear A return (or break/continue) at every level of nested try/finally made memory grow with the cube of the depth: each level kept its own copy of the text of every level inside it until the end of the transpile, and the text itself grew with the square of the depth because of indentation. 490 levels in a 15 KB plxjs body took the backend to 3.6 GB in 3.4 s; any role can send that, including in a DO block. - plx_emit_try now frees body, arms and ensure_body (and its resolved copies) once they are appended; the callers free each catch handler's text once it is added to the arms. - plx_indent stops at 32 levels (indentation is cosmetic in plpgsql). The same body now takes 31 MB and 0.37 s, and the generated code shrinks from 5.8 MB to 1.2 MB. A plxjs test checks the output size at 300 levels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Pushed 769f818, which fixes a memory blowup in this PR that came up while reviewing plxcsharp. Problem. Put a
A 15 KB plxjs body with 490 levels took the backend to 3.6 GB in 3.4 s, and 5.8 MB of Fix.
Result. The same body now takes 31 MB and 0.37 s, and the output is 1.2 MB, linear in the source. Ruby begin/ensure and nested catches behave the same way. Tests.
🤖 Generated with Claude Code |
- The error re-raised after a finally body now keeps the schema, table, column, constraint and type names, as well as the SQLSTATE, message, detail and hint. They are passed only when at least one is non-empty: an empty RAISE option is sent to the client as a present field (psql prints it in verbose mode), so passing all five on every re-raise would have added five empty fields to every error, a plain raise included. - A loop label is now in scope only inside its loop. A loop inside the try that reused the label of the enclosing loop was found first, so a break/continue to the enclosing loop was judged not to cross the finally and skipped it. - finally_diff.py compares the SQLSTATE and message of an error, not just that some error happened. - The deep-nesting test checks that twice the depth gives at most 2.5 times the code, instead of a fixed size bound. - Docs and CHANGELOG: the finally body does not run on a query cancel or a failed assert, which WHEN OTHERS does not catch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py). doc/plxcsharp.md gets the same note on what finally re-raises and when it does not run.
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py). doc/plxcsharp.md gets the same note on what finally re-raises and when it does not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. I reproduced all four findings. They're fixed in 39f726e, with one correction to the suggested fix for 1. 1. Object names lost on re-raise. Confirmed with your probe: One correction: an empty value doesn't behave like a missing one. With What the fix does instead:
2. A reused label skips the finally. Confirmed, and
3. 4. Where finally doesn't run. Both confirmed (a failed Two related points I didn't change:
Checked on PostgreSQL 18.4: 🤖 Generated with Claude Code |
Depends on #12. This branch is stacked on #12's
plx_emit_tryhelper, so until #12 merges the diff also shows its two commits. Only the last two commits belong here. Merge #12 first.Summary
The try/finally lowering (plxjs, plxts, plxphp, plxruby, plxpython3) had two flaws:
finallydidn't run onreturn,breakorcontinue. Areturninsidetryleft the function without runningfinally, unlike every source language plx models.The finally body is now emitted once:
How it works:
return/break/continuethat would leave the frame is emitted as a placeholder, because the parser doesn't know yet whether afinallyfollows.plx_emit_tryresolves the placeholders. Without a finally they become the plain statement, so try/catch output is unchanged. With one they are routed through the finally body.break/continuefind their target through the frame their label was declared in.__plx_retis typed from the function's result: RECORD for triggers andrecord,ALIAS FOR $0for polymorphic results.Semantics now match the source languages:
finallyruns.returnor an error insidefinallyreplaces the pending one.finallywith its SQLSTATE, message, detail and hint.Trade-off: a re-raised error's
CONTEXTpoints at the re-raise, not the original statement.Also:
Upgrade note: stored functions keep the plpgsql they were created with. Only functions created or replaced after upgrading get the new behaviour (see CHANGELOG). #13 currently documents that
returninsidetryskipsfinally; that note should be removed when this merges.Testing
returnin finally overriding an errorNEWensure, Pythonfinallytest/finally_diff.py: generates random Python programs mixing nested try/except/finally, loops, return/break/continue (also inside finally) and raised errors, runs each in real Python and as plxpython3, and requires identical results. 1,200 programs across three seeds: 0 mismatches.make differentialchecknow runs 300 cases, so CI covers it.make installcheck: all 13 suites pass on PG 18.4.test/fuzz.pyand the annotation fuzzer: no crashes.🤖 Generated with Claude Code