Skip to content

[3.0.0] Emit finally once, and run it on return, break and continue - #16

Open
OffgridwithJD wants to merge 9 commits into
commandprompt:masterfrom
OffgridwithJD:feat/finally-once
Open

OffgridwithJD wants to merge 9 commits into
commandprompt:masterfrom
OffgridwithJD:feat/finally-once

Conversation

@OffgridwithJD

Copy link
Copy Markdown

Depends on #12. This branch is stacked on #12's plx_emit_try helper, so until #12 merges the diff also shows its two commits. Only the last two commits belong here. Merge #12 first.

Summary

The try/finally lowering (plxjs, plxts, plxphp, plxruby, plxpython3) had two flaws:

  1. finally didn't run on return, break or continue. A return inside try left the function without running finally, unlike every source language plx models.
  2. The finally body was emitted twice (once on the normal path, once in the error handler), so nesting doubled the output per level. [2.1.0] Security audit: fix backend crashes, string-literal breakout, and transpiler DoS #12 capped this at 8 levels as a stopgap.

The finally body is now emitted once:

__plx_pend_N := 0; __plx_es_N := NULL;
<<__plx_try_N>>
BEGIN
  BEGIN
    body                 -- return x  ->  __plx_ret := x; __plx_pend_N := 1; EXIT __plx_try_N;
  EXCEPTION catch arms   -- exits from here are routed the same way
  END;
EXCEPTION WHEN OTHERS THEN
  GET STACKED DIAGNOSTICS __plx_es_N = RETURNED_SQLSTATE, ... MESSAGE_TEXT, ... DETAIL, ... HINT;
END;
finally body
IF __plx_es_N IS NOT NULL THEN RAISE EXCEPTION USING ERRCODE = ..., MESSAGE = ..., [DETAIL, HINT]; END IF;
IF __plx_pend_N = 1 THEN RETURN __plx_ret; ELSIF ... END IF;

How it works:

  • While a try body and its catch arms are parsed, a frame is pushed.
  • A return/break/continue that would leave the frame is emitted as a placeholder, because the parser doesn't know yet whether a finally follows.
  • plx_emit_try resolves the placeholders. Without a finally they become the plain statement, so try/catch output is unchanged. With one they are routed through the finally body.
  • Nested frames propagate outward.
  • Labeled break/continue find their target through the frame their label was declared in.
  • __plx_ret is typed from the function's result: RECORD for triggers and record, ALIAS FOR $0 for polymorphic results.

Semantics now match the source languages:

  • The return value is evaluated before finally runs.
  • A return or an error inside finally replaces the pending one.
  • An uncaught error is re-raised after finally with its SQLSTATE, message, detail and hint.

Trade-off: a re-raised error's CONTEXT points at the re-raise, not the original statement.

Also:

Upgrade note: stored functions keep the plpgsql they were created with. Only functions created or replaced after upgrading get the new behaviour (see CHANGELOG). #13 currently documents that return inside try skips finally; that note should be removed when this merges.

Testing

  • New regression cases:
    • return, break and continue
    • nested try/finally, with inner and outer ordering
    • labeled break across a try
    • error re-raise keeping SQLSTATE/detail/hint
    • return in finally overriding an error
    • return from a catch arm
    • triggers returning NEW
    • SRF bare return
    • Ruby ensure, Python finally
    • 40-level nesting
  • New test/finally_diff.py: generates random Python programs mixing nested try/except/finally, loops, return/break/continue (also inside finally) and raised errors, runs each in real Python and as plxpython3, and requires identical results. 1,200 programs across three seeds: 0 mismatches. make differentialcheck now runs 300 cases, so CI covers it.
  • make installcheck: all 13 suites pass on PG 18.4.
  • test/fuzz.py and the annotation fuzzer: no crashes.

🤖 Generated with Claude Code

jdatcmd and others added 2 commits November 9, 2026 18:46
Every plx language is trusted, so any role with USAGE can send arbitrary
input to the transpiler through CREATE FUNCTION or DO.

- plx_span_text: skip type-annotation tokens (which carry no source span)
  at the edges of a span. A span beginning on one computed its length from a
  NULL pointer and crashed the backend (plxjs/ts/php/ruby/python3).
- plxtsql, plxgo, plxcobol: guard the expression emitters with
  check_stack_depth() and the PLX_MAX_DEPTH counter. Deeply nested calls,
  conversions or subscripts overflowed the C stack (SIGSEGV).
- finally/ensure: cap finally-inside-finally nesting at 8 levels; each level
  doubled the generated body, so a few hundred bytes of source could demand
  gigabytes. The three copies of the try lowering are now one helper,
  plx_emit_try, which also runs the normal-path finally body after the
  protected block, so a finally that raises no longer runs twice.
- plxgo fmt.Sprintf: require a letter verb before lowering to a || chain;
  a '%' followed by a quote consumed half of an escaped '' pair and let
  Go string text escape into the generated SQL.

Regression tests cover each case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…growth

Second round of the security audit (shared transpiler core):

- emit_string_value: escape every literal backslash and force E''. A
  string ending in a lone backslash inside an interpolation was emitted
  bare in an E'' literal, escaping its closing quote so the next literal's
  text ran as code. The synthetic-string scanners no longer step past the
  end of their input on a trailing backslash.
- Locals: index by lower-cased name in a hash table beside the ordered
  list; the linear lookup made many distinct variables quadratic.
- plxpython3 lexer: resume after the blank/comment lines the indentation
  scan already covered, instead of rescanning the run from every newline
  (O(K^2), and line numbers were counted twice).
- CHECK_FOR_INTERRUPTS in the lexer loop, the expression rewriter and the
  local table, so statement_timeout and cancel reach a long transpile.
- Ternary rewrite: free the condition and both arms after composing the
  CASE, so a chain of N ternaries no longer holds N copies of its tail.
- plxpython3 call(f"..."): reject an f-string procedure name, which was
  unquoted with its prefix into a broken identifier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d tests

From the adversarial review of this PR:

- standard_conforming_strings: literals emitted by plxgo, plxtsql,
  plxplsql and plxcobol, and Ruby/PHP single-quoted strings, were plain
  '...' literals. plpgsql reads the body with the caller's setting, so
  with it off a trailing backslash swallowed the closing quote and the
  next literal ran as code. A final pass over the generated body (a small
  SQL lexer that skips comments, quoted identifiers, dollar quotes, E''
  and U&'' strings) now rewrites every ordinary literal containing a
  backslash as E'...' with the backslash doubled. One place covers every
  dialect and future emitters.
- plxpython3 lexer: a #:: annotation line ended the blank-line scan's
  resume point but not the scan, so runs of annotation lines were still
  quadratic. The scan now stops at an annotation line.
- Locals index: a name of 128+ bytes disabled the hash and brought back
  the linear scan. Keys are now downcase_truncate_identifier(), as the
  server reads an unquoted identifier, so every name is indexed and case
  folding matches pg_strncasecmp/the server for all encodings; the
  fallback is gone.
- Tests: standard_conforming_strings off across six dialects; timeouts
  around 60,000 locals after an over-long name, 400,000 blank lines and
  40,000 annotation lines; and a nesting depth under the 500-level counter
  with max_stack_depth = 100kB, which exercises check_stack_depth().
- CHANGELOG: the finally cap bounds depth, not output size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review of the last two commits. The rest is #12, which has changes requested.

Tested on PostgreSQL 18.4: builds without warnings, make installcheck passes 13/13, and make differentialcheck (including finally_diff.py 300 1) reports 0 mismatches.

These hold up:

  • A break inside a switch inside try/finally inside a loop.
  • A trigger returning null and NEW from inside try/finally.
  • A polymorphic result (anyelement).
  • commit() in the finally body of a procedure.
  • Output now grows linearly with nesting. An 8-level body of 209 KB stores 876 KB; under #12 the same shape stored 144 MB.

Must fix

1. The re-raise drops the error's object names

A bare RAISE; (the old lowering) re-raises the whole error. The new re-raise copies only the SQLSTATE, message, detail and hint, so the constraint, table, schema, column and datatype names are lost. Code that catches the error and reads them (for example, an application that maps a constraint name to a form field) silently gets empty strings.

CREATE TABLE t16(id int CONSTRAINT t16_pk PRIMARY KEY);
INSERT INTO t16 VALUES (1);
CREATE FUNCTION ins_fin() RETURNS void LANGUAGE plxjs AS $$
  try { execute("INSERT INTO t16 VALUES (1)"); } finally { let z = 1; }
$$;
DO $d$ DECLARE c text; t text; s text; BEGIN
  PERFORM ins_fin();
EXCEPTION WHEN unique_violation THEN
  GET STACKED DIAGNOSTICS c = CONSTRAINT_NAME, t = TABLE_NAME, s = SCHEMA_NAME;
  RAISE NOTICE 'constraint=% table=% schema=%', c, t, s;
END $d$;
build output
master constraint=t16_pk table=t16 schema=public
this PR constraint= table= schema=

Suggested fix: also capture COLUMN_NAME, CONSTRAINT_NAME, PG_DATATYPE_NAME, TABLE_NAME and SCHEMA_NAME, and pass them back with RAISE ... USING COLUMN = ..., CONSTRAINT = ..., DATATYPE = ..., TABLE = ..., SCHEMA = .... The diagnostics come back as '' when absent, never NULL, so RAISE accepts them. Unlike DETAIL and HINT, these five fields are not printed in the error text, so they can be passed on every branch of the existing IF without a guard of their own. Add the probe above as a regression test.

If this stays as it is, the CHANGELOG must list the lost fields next to the CONTEXT note.

Should fix

2. A reused loop label makes break skip the finally

exit_crosses looks a label up in cx->labels, which only grows. A loop inside the try that reuses the label of the target loop is found first, so the break is judged not to cross the frame and is emitted as a plain EXIT:

CREATE FUNCTION lbl() RETURNS int LANGUAGE plxjs AS $$
  let r = 0;
  outer: for (let i = 0; i < 2; i++) {
    try {
      outer: for (let j = 0; j < 1; j++) { r = r * 10 + 1; }
      break outer;
    } finally { r = r * 10 + 9; }
  }
  return r;
$$;
SELECT lbl();  -- 1; expected 19 (the same body with the inner label renamed returns 19)

JavaScript rejects a label nested in a same-named label, but plx accepts it (and so does plpgsql). Either remove a label from the list when its loop ends, or reject a label that shadows an enclosing one.

3. finally_diff.py counts any error as a match

Python errors only with ValueError("x"), but the harness maps every plx failure to ERROR. A wrong SQLSTATE, a plpgsql runtime error in generated code, or record is not assigned yet would all pass. Please compare the SQLSTATE (P0001) and message x, not just "some error".

4. Say where finally still does not run

The CHANGELOG says the finally body runs on an error. It does not run on a query cancel or a failed assert, because WHEN OTHERS does not catch query_canceled or assert_failure. Both verified:

  • statement_timeout = '300ms' around try { execute("SELECT pg_sleep(2)"); } finally { raise("warning", ...); }: no warning.
  • try { assert(false, "boom"); } finally { raise("notice", ...); }: no notice.

This was true before too, but the new text promises more than the code does. One sentence in the CHANGELOG and the dialect chapters is enough.

CI has not run on this PR yet (fork run waiting for approval).

jdatcmd and others added 2 commits November 9, 2026 20:02
The try/finally lowering emitted the finally body twice (normal and error
path), so nesting doubled the output per level, and a return, break or
continue inside the try left without running finally at all.

A try now pushes a frame while its body and catch arms are parsed. An
exit that would leave the frame is emitted as a placeholder, since the
parser does not yet know whether a finally follows; plx_emit_try then
resolves it: without a finally, as the plain statement (or a placeholder
for the enclosing frame); with one, as "save the return value, note the
pending exit, EXIT the labeled block". After the single copy of the
finally body, a saved error is re-raised with its SQLSTATE, message,
detail and hint, and the pending exit is completed in the enclosing
frame's context. Labeled break/continue resolve their target through
the labels recorded with the frame they were declared in.

The return value lives in a hoisted __plx_ret typed from the function's
result (RECORD for triggers and record, ALIAS FOR $0 for polymorphic
results). The 8-level finally nesting cap is no longer needed and is
removed. Source containing U+0001, the placeholder delimiter, is
rejected.

Checked against real Python with 1,200 randomly generated programs
mixing nested try/except/finally, loops, return/break/continue (also
inside finally) and raised errors: no mismatches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test/finally_diff.py generates random programs mixing nested
try/except/finally, loops, return/break/continue (also inside finally)
and raised errors, runs each in Python and as a plxpython3 function, and
requires the results to agree. `make differentialcheck` runs 300 cases
after the existing differential check, so CI covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD OffgridwithJD changed the title Emit finally once, and run it on return, break and continue [3.0.0] Emit finally once, and run it on return, break and continue Nov 9, 2026
jdatcmd and others added 3 commits November 9, 2026 23:51
"let x /*:: int */ = 0;" drops the "= 0" (commandprompt#20 makes that form an error).
The test's result does not depend on it, but it should use the documented
form so it keeps passing once commandprompt#20 is merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	test/expected/plxjs.out
#	test/sql/plxjs.sql
…ys linear

A return (or break/continue) at every level of nested try/finally made
memory grow with the cube of the depth: each level kept its own copy of
the text of every level inside it until the end of the transpile, and the
text itself grew with the square of the depth because of indentation. 490
levels in a 15 KB plxjs body took the backend to 3.6 GB in 3.4 s; any role
can send that, including in a DO block.

- plx_emit_try now frees body, arms and ensure_body (and its resolved
  copies) once they are appended; the callers free each catch handler's
  text once it is added to the arms.
- plx_indent stops at 32 levels (indentation is cosmetic in plpgsql).

The same body now takes 31 MB and 0.37 s, and the generated code shrinks
from 5.8 MB to 1.2 MB. A plxjs test checks the output size at 300 levels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD

Copy link
Copy Markdown
Author

Pushed 769f818, which fixes a memory blowup in this PR that came up while reviewing plxcsharp.

Problem. Put a return at every level of nested try/finally. Memory then grew with the cube of the depth, for two reasons:

  • Each level kept its own copy of the text of every level inside it until the transpile ended.
  • That text grew with the square of the depth, because of indentation.

A 15 KB plxjs body with 490 levels took the backend to 3.6 GB in 3.4 s, and 5.8 MB of prosrc. Any role can send that, including in a DO block. On master and #12 this can't happen, because finally nesting is capped at 8 levels there. It appeared when this PR removed the cap.

Fix.

  • plx_emit_try now frees body, arms and ensure_body, and its resolved copies, once they are appended. Its comment documents the new ownership.
  • The callers free each catch handler's text once it is added to the arms.
  • plx_indent stops at 32 levels. Indentation is cosmetic in plpgsql.

Result. The same body now takes 31 MB and 0.37 s, and the output is 1.2 MB, linear in the source. Ruby begin/ensure and nested catches behave the same way.

Tests.

  • A new plxjs test checks the output size at 300 levels. It fails without the fix.
  • All suites pass, differentialcheck passes, and finally_diff.py with 1,200 programs has 0 mismatches.

🤖 Generated with Claude Code

- The error re-raised after a finally body now keeps the schema, table,
  column, constraint and type names, as well as the SQLSTATE, message,
  detail and hint. They are passed only when at least one is non-empty: an
  empty RAISE option is sent to the client as a present field (psql prints
  it in verbose mode), so passing all five on every re-raise would have
  added five empty fields to every error, a plain raise included.
- A loop label is now in scope only inside its loop. A loop inside the try
  that reused the label of the enclosing loop was found first, so a
  break/continue to the enclosing loop was judged not to cross the finally
  and skipped it.
- finally_diff.py compares the SQLSTATE and message of an error, not just
  that some error happened.
- The deep-nesting test checks that twice the depth gives at most 2.5 times
  the code, instead of a fixed size bound.
- Docs and CHANGELOG: the finally body does not run on a query cancel or a
  failed assert, which WHEN OTHERS does not catch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OffgridwithJD pushed a commit to OffgridwithJD/plx that referenced this pull request Nov 10, 2026
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py). doc/plxcsharp.md gets the same note on what finally re-raises and when it does not run.
OffgridwithJD pushed a commit to OffgridwithJD/plx that referenced this pull request Nov 10, 2026
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py). doc/plxcsharp.md gets the same note on what finally re-raises and when it does not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OffgridwithJD pushed a commit to OffgridwithJD/plx that referenced this pull request Nov 10, 2026
Brings in the review fixes for commandprompt#16 (object names on re-raise, scoped loop labels, exact error comparison in finally_diff.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD

Copy link
Copy Markdown
Author

Thanks. I reproduced all four findings. They're fixed in 39f726e, with one correction to the suggested fix for 1.

1. Object names lost on re-raise. Confirmed with your probe: constraint= table= schema=.

One correction: an empty value doesn't behave like a missing one. With \set VERBOSITY verbose, RAISE EXCEPTION USING MESSAGE = 'plain', SCHEMA = '', TABLE = '', COLUMN = '', CONSTRAINT = '', DATATYPE = '' prints five empty lines (SCHEMA NAME:, TABLE NAME:, ...) that a plain RAISE does not. Drivers also see the empty values. Passing the five names on every branch would therefore add five empty fields to every re-raised error, a plain raise("...") included.

What the fix does instead:

  • The five names are captured and passed back as a group, only when at least one is non-empty. DETAIL and HINT keep their own guards. That makes eight RAISEs.
  • Result: a re-raised plain error carries no extra fields. Your probe now prints constraint=t16_pk table=t16 schema=public.
  • Remaining cost: an error that names some objects also gets the others as empty fields. A unique violation, for example, shows empty COLUMN NAME and DATATYPE NAME in verbose mode. The CHANGELOG says so.
  • An exact copy of every field, and of the original CONTEXT, would need a small C function that raises with only the non-empty fields. That's a new extension object, so I left it out of this PR. I can add it for 3.0 if you want it.
  • Your probe is now a regression test (js_fin_objs). A domain-check case alongside it covers the type and constraint names.
  • The re-raise block is longer: about 4.4 KB of generated code per level instead of about 2.4 KB, still linear. js_fin_ret_deep now checks linearity as a ratio rather than a fixed size: 300 levels must produce at most 2.5 times the code of 150. It measures 2.06; with the indentation cap removed it is 3.28, so the test still catches that regression.

2. A reused label skips the finally. Confirmed, and continue outer has the same bug: the continue form of your body returned 11 instead of 1919.

  • A label reused by a later sibling loop, which JavaScript allows, already worked.
  • Fix: a label is now in scope only inside its loop. The brace, Ruby and Python front ends restore the label list once the labelled loop is parsed.
  • I didn't reject shadowing, because plpgsql accepts it. I didn't filter by loop depth either: plxcsharp also records a label for switch, which doesn't change the depth.
  • Tests: plxjs covers break and continue; plxruby and plxpython3 cover the same shape.

3. finally_diff.py accepted any error. Confirmed. It now requires ERROR P0001:x, the SQLSTATE and message. 300 + 1,500 cases, 0 mismatches.

4. Where finally doesn't run. Both confirmed (a failed assert and a 300 ms statement_timeout). The CHANGELOG and the plxjs, plxphp, plxpython3 and plxruby chapters now say so. They also say that catch doesn't catch those two errors, for the same reason.

Two related points I didn't change:

  • In real Python, except Exception and finally do see an AssertionError, so plxpython3 differs from Python there. Adding assert_failure to the catch-all is a behaviour change I'd rather make separately.
  • The same gap was a bug in the new plxlua: real Lua's pcall catches a failed assert, and plxlua's didn't. Since plxlua isn't released, I fixed it on its branch (WHEN OTHERS OR assert_failure). A query cancel is still not caught.

Checked on PostgreSQL 18.4: make installcheck 13/13, make differentialcheck clean, and 600 fuzz iterations with no crashes. CI on this PR is waiting for workflow approval.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants