diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c3dc8e5..2acefd33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,22 @@ cargo tests, on 13 interpreters (2.6–3.14). ## Unreleased +### Community fixes (PRs #1–#26) +- merged 26 targeted fixes from @EiNSTeiN- covering 3.11–3.14 bytecode + families: chained assignments and comparison chains; COPY/SWAP and + walrus in the bounded expression simulator; keyword calls and ordered + `**` expansions; comprehension filter polarity, Boolean elements, + nested conditional elements and nested comprehensions inside generator + expressions; `for x in await f()` iteration protocol; padded handler + returns; EXTENDED_ARG loop back edges; guarded `else` suites; split + try/except/else/finally printing; marshalled slice constants vs runtime + `slice()` calls; and PEP 649 class-annotation registration gates. +- the 26 branches and their regression tests were merged and verified + together. Three textual conflicts were resolved by keeping both sides: + PR #6 stacks on #5, all PRs append tests at the same point in + `tests/integration.rs`, and #4/#5/#6/#24 append test-module + declarations at the end of `decompiler.rs`. + ### Fixed - value-position and-chains in a ternary: `try_dup_tail_ternary` now absorbs leading same-target cond jumps into the condition, so @@ -16,9 +32,9 @@ cargo tests, on 13 interpreters (2.6–3.14). merge, so `d[a if c and e else b]` keeps the shared lookup. ### Benchmarks -- real-world corpus: 6408/6411 modules recompile (99.95%); pandas is at - 1419/1420 with only the nested-comprehension family left (sklearn ×2, - pandas to_dict) +- real-world corpus: 6411/6411 modules recompile (100%); the former + nested-comprehension family (pandas `to_dict`, sklearn `_logistic`, + `pairwise`) is closed, and strict AST equality rises to 2493/6411 ### Fixed diff --git a/README.md b/README.md index e3f8f8c1..765316d8 100644 --- a/README.md +++ b/README.md @@ -103,14 +103,14 @@ seven projects **at its latest release tag**, compiles every `.py` to | Project (tag) | modules | serial `-j 1` | parallel (default) | recompiles | AST-identical¹ | |---|---|---|---|---|---| -| yt-dlp (2026.08.19) | 1045 | 0.93 s | 0.12 s | **1045/1045** | 500/1045 | -| matplotlib (v3.11.2) | 253 | 0.47 s | 0.09 s | **253/253** | 59/253 | -| pandas (v3.0.5) | 1420 | 1.35 s | 0.21 s | 1419/1420 | 582/1420 | -| django (6.1.1) | 907 | 0.67 s | 0.16 s | **907/907** | 467/907 | -| sympy (1.14.0) | 1532 | 3.68 s | 1.85 s | **1532/1532** | 479/1532 | -| scikit-learn (1.9.1) | 671 | 0.71 s | 0.10 s | 669/671 | 216/671 | -| ansible (v2.21.4) | 583 | 0.35 s | 0.06 s | **583/583** | 185/583 | -| **total** | **6411** | **8.16 s** | **2.59 s** | **6408/6411 (99.95 %)** | 2488/6411 | +| yt-dlp (2026.08.19) | 1045 | 0.46 s | 0.08 s | **1045/1045** | 501/1045 | +| matplotlib (v3.11.2) | 253 | 0.33 s | 0.06 s | **253/253** | 60/253 | +| pandas (v3.0.5) | 1420 | 0.90 s | 0.14 s | **1420/1420** | 584/1420 | +| django (6.1.1) | 907 | 0.38 s | 0.10 s | **907/907** | 468/907 | +| sympy (1.14.0) | 1532 | 3.03 s | 1.63 s | **1532/1532** | 479/1532 | +| scikit-learn (1.9.1) | 671 | 0.48 s | 0.07 s | **671/671** | 216/671 | +| ansible (v2.21.4) | 583 | 0.27 s | 0.05 s | **583/583** | 185/583 | +| **total** | **6411** | **5.85 s** | **2.13 s** | **6411/6411 (100 %)** | 2493/6411 | ¹ strict: docstring-stripped AST equal to the original source — equivalent normalizations (an if/else instead of a ternary, a redundant @@ -118,10 +118,11 @@ equivalent normalizations (an if/else instead of a ternary, a redundant differences, so treat it as a shape-fidelity floor rather than an error rate. The parse/recompile column is the acceptance bar. -One pycdc process per project: ~1.0 ms per module serial, peak RSS -≈99 MB for the whole 6.4 k-module batch. The three remaining modules are one shape: a dict comprehension -nested inside a generator expression's element (call arguments -included) — marked incomplete, never a crash. Raw data: [`benchmarks/realworld.json`](benchmarks/realworld.json), +One pycdc process per project: ~0.9 ms per module serial, peak RSS +≈81 MB for the whole 6.4 k-module batch. **All 6411 modules recompile**; +the last family (a dict comprehension nested inside a generator +expression's element, call arguments included) was closed by the +community PR batch. Raw data: [`benchmarks/realworld.json`](benchmarks/realworld.json), harness: [`tools/bench_realworld.py`](tools/bench_realworld.py). diff --git a/README.zh-CN.md b/README.zh-CN.md index b7de1870..ab8fab4c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -99,23 +99,23 @@ Apple Silicon 实测，526 个真实 `.pyc`（约 9.5MB，2.6–3.14 标准库 | 项目（tag） | 模块数 | 串行 `-j 1` | 并行（默认） | 可重编译 | AST 完全一致¹ | |---|---|---|---|---|---| -| yt-dlp (2026.08.19) | 1045 | 0.93 s | 0.12 s | **1045/1045** | 500/1045 | -| matplotlib (v3.11.2) | 253 | 0.47 s | 0.09 s | **253/253** | 59/253 | -| pandas (v3.0.5) | 1420 | 1.35 s | 0.21 s | 1419/1420 | 582/1420 | -| django (6.1.1) | 907 | 0.67 s | 0.16 s | **907/907** | 467/907 | -| sympy (1.14.0) | 1532 | 3.68 s | 1.85 s | **1532/1532** | 479/1532 | -| scikit-learn (1.9.1) | 671 | 0.71 s | 0.10 s | 669/671 | 216/671 | -| ansible (v2.21.4) | 583 | 0.35 s | 0.06 s | **583/583** | 185/583 | -| **合计** | **6411** | **8.16 s** | **2.59 s** | **6408/6411（99.95%）** | 2488/6411 | +| yt-dlp (2026.08.19) | 1045 | 0.46 s | 0.08 s | **1045/1045** | 501/1045 | +| matplotlib (v3.11.2) | 253 | 0.33 s | 0.06 s | **253/253** | 60/253 | +| pandas (v3.0.5) | 1420 | 0.90 s | 0.14 s | **1420/1420** | 584/1420 | +| django (6.1.1) | 907 | 0.38 s | 0.10 s | **907/907** | 468/907 | +| sympy (1.14.0) | 1532 | 3.03 s | 1.63 s | **1532/1532** | 479/1532 | +| scikit-learn (1.9.1) | 671 | 0.48 s | 0.07 s | **671/671** | 216/671 | +| ansible (v2.21.4) | 583 | 0.27 s | 0.05 s | **583/583** | 185/583 | +| **合计** | **6411** | **5.85 s** | **2.13 s** | **6411/6411（100%）** | 2493/6411 | ¹ 严格口径：去掉 docstring 后 AST 与原源码完全一致。等价归一化 （if/else 代替三元、解包外多余的 `*()`/`**{}`、关键字参数顺序调整）也 会被计为差异，因此它是「形状保真度下限」而非错误率；验收口径是左侧的 可重编译列。 -每个项目一个 pycdc 进程：串行约 1.0 ms/模块，整批 6.4k 模块峰值 -RSS ≈98 MB。剩余 3 个模块属一种形态：生成器表达式元素（含调用参数）内嵌 -dict 推导式——标记为不完整，不崩溃。 +每个项目一个 pycdc 进程：串行约 0.9 ms/模块，整批 6.4k 模块峰值 +RSS ≈81 MB。**6411 个模块全部可重编译**；最后一族（生成器表达式元素 / +调用参数里的 dict 推导式）已由社区 PR 批次修复。 [`benchmarks/realworld.json`](benchmarks/realworld.json)， 脚本见 [`tools/bench_realworld.py`](tools/bench_realworld.py)。 diff --git a/benchmarks/realworld.json b/benchmarks/realworld.json index 7f5b778e..6ed311bd 100644 --- a/benchmarks/realworld.json +++ b/benchmarks/realworld.json @@ -3,17 +3,17 @@ "pycdc": "target/release/pycdc", "interp": "~/.local/share/uv/python/cpython-3.12.14-macos-aarch64-none/bin/python3.12", "host": "Darwin 25.6.0 arm64", - "generated": "2026-09-15 13:38:23", + "generated": "2026-10-23 14:52:30", "note": "one pycdc process per project (batch directory mode); serial = -j 1, parallel = default worker count; recompiles = decompiled output parses under the corpus interpreter", "projects": [ { "project": "ansible", "files": 583, "source_bytes": 5372361, - "serial_wall_s": 0.35, - "serial_rss_mb": 9.0, - "parallel_wall_s": 0.06, - "parallel_rss_mb": 18.3, + "serial_wall_s": 0.27, + "serial_rss_mb": 8.5, + "parallel_wall_s": 0.05, + "parallel_rss_mb": 18.6, "decompiled": 583, "recompiles": 583, "ast_equal": 185, @@ -23,51 +23,51 @@ "project": "django", "files": 907, "source_bytes": 5914265, - "serial_wall_s": 0.67, - "serial_rss_mb": 8.8, - "parallel_wall_s": 0.16, - "parallel_rss_mb": 23.5, + "serial_wall_s": 0.38, + "serial_rss_mb": 8.3, + "parallel_wall_s": 0.1, + "parallel_rss_mb": 23.9, "decompiled": 907, "recompiles": 907, - "ast_equal": 467, + "ast_equal": 468, "ast_checked": 907 }, { "project": "matplotlib", "files": 253, "source_bytes": 7415404, - "serial_wall_s": 0.47, - "serial_rss_mb": 20.9, - "parallel_wall_s": 0.09, - "parallel_rss_mb": 38.0, + "serial_wall_s": 0.33, + "serial_rss_mb": 18.5, + "parallel_wall_s": 0.06, + "parallel_rss_mb": 35.9, "decompiled": 253, "recompiles": 253, - "ast_equal": 59, + "ast_equal": 60, "ast_checked": 253 }, { "project": "pandas", "files": 1420, "source_bytes": 21287834, - "serial_wall_s": 1.35, - "serial_rss_mb": 13.5, - "parallel_wall_s": 0.21, - "parallel_rss_mb": 39.1, + "serial_wall_s": 0.9, + "serial_rss_mb": 12.8, + "parallel_wall_s": 0.14, + "parallel_rss_mb": 38.9, "decompiled": 1420, - "recompiles": 1419, - "ast_equal": 582, + "recompiles": 1420, + "ast_equal": 584, "ast_checked": 1420 }, { "project": "sklearn", "files": 671, "source_bytes": 13444062, - "serial_wall_s": 0.71, - "serial_rss_mb": 10.8, - "parallel_wall_s": 0.1, - "parallel_rss_mb": 25.0, + "serial_wall_s": 0.48, + "serial_rss_mb": 10.6, + "parallel_wall_s": 0.07, + "parallel_rss_mb": 30.1, "decompiled": 671, - "recompiles": 669, + "recompiles": 671, "ast_equal": 216, "ast_checked": 671 }, @@ -75,10 +75,10 @@ "project": "sympy", "files": 1532, "source_bytes": 26169877, - "serial_wall_s": 3.68, - "serial_rss_mb": 49.6, - "parallel_wall_s": 1.85, - "parallel_rss_mb": 78.8, + "serial_wall_s": 3.03, + "serial_rss_mb": 48.6, + "parallel_wall_s": 1.63, + "parallel_rss_mb": 80.6, "decompiled": 1532, "recompiles": 1532, "ast_equal": 479, @@ -88,13 +88,13 @@ "project": "yt_dlp", "files": 1045, "source_bytes": 9208581, - "serial_wall_s": 0.93, - "serial_rss_mb": 15.5, - "parallel_wall_s": 0.12, - "parallel_rss_mb": 23.2, + "serial_wall_s": 0.46, + "serial_rss_mb": 11.8, + "parallel_wall_s": 0.08, + "parallel_rss_mb": 21.0, "decompiled": 1045, "recompiles": 1045, - "ast_equal": 500, + "ast_equal": 501, "ast_checked": 1045 } ], @@ -102,13 +102,13 @@ "project": "TOTAL", "files": 6411, "source_bytes": 88812384, - "serial_wall_s": 8.16, - "parallel_wall_s": 2.59, - "serial_rss_mb": 49.6, - "parallel_rss_mb": 78.8, + "serial_wall_s": 5.85, + "parallel_wall_s": 2.13, + "serial_rss_mb": 48.6, + "parallel_rss_mb": 80.6, "decompiled": 6411, - "recompiles": 6408, - "ast_equal": 2488, + "recompiles": 6411, + "ast_equal": 2493, "ast_checked": 6411 } } \ No newline at end of file diff --git a/docs/LIMITATIONS.md b/docs/LIMITATIONS.md index 6c90274a..b4caccb8 100644 --- a/docs/LIMITATIONS.md +++ b/docs/LIMITATIONS.md @@ -143,57 +143,22 @@ rendering bugs: (300+ regression-driven special cases), so it is left documented rather than patched blind. -### Nested comprehension inside a genexp (3 modules, deliberately not patched) - -`({c: v for c, v in zip(cols, row) if c} for row in rows)` — a genexp whose -*element* (or a call argument inside it) is an inlined (PEP 709) dict -comprehension. The genexp keeps its own code object and is decompiled by -the comprehension walker, which has no notion of a nested level: the -inner `BUILD_MAP 0` accumulator lands on the walker's stack, `SWAP` is -not modeled, and the following `FOR_ITER` treats it as a second `for` -clause (`... for c, v in {}`), so the output does not parse. - -A frame-stack prototype (one frame per comprehension level, `SWAP` -modeled, the accumulator placeholder replaced by Rc identity when a level -ends, `pending_or_filter` flushed per level) was implemented **twice** and -reverted both times. What it achieved, verified by runtime-equivalence -checks against the original sources: the whole-element shape and the -call-argument shapes *as minimal repros* became semantically correct -(`({c: v for c, v in zip(cols, t) if c} for t in rows)`, -`(f(**d) for x in xs)`, `(f(x, y=1) for x in xs)`; the last two were -fixed for real and are committed). - -It still produced *plausible but semantically wrong* output for the three -real modules (`pandas/core/methods/to_dict.py`, -`sklearn/linear_model/_logistic.py`, `metrics/pairwise.py`), e.g. - - fd(func, ret, s, X, Y[s, ...], **{k: (v[s] if k == "Y" else v) for k, v in kwds.items()}) - -> (fd if k == 'Y' else (func, ret, s, X, Y[s, ...])(*{}, **{**k}) for s in ...) - -The blocker is not the nesting itself: the walker's stack model is a -heuristic that happens to agree with the VM on the shapes the corpus -exercises, and a nested level exposes several concrete divergences at -once (all observed with probes during the third attempt, 2026-09-15): - - * `GET_ITER` is a no-op, so the 3.12 prologue's `SWAP`s reorder the - wrong slots (the VM's is `[saved…, iter]` -> `BUILD_* 0` -> `SWAP 2`; - the walker has no `iter` on the stack there); - * the `LOAD_FAST_AND_CLEAR` save slots are pushed but never popped — - the epilogue's `STORE_FAST`s restore them in the VM, while the walker - treats stores as target assignments, so they linger on the stack; - * `PUSH_NULL` markers then sit between the level's value and the call - slot, and the `CALL_FUNCTION_EX` pops pick up the wrong values - (`fd(...)` came out as `fd if k == 'Y' else (func, ret, X, Y)(*{}, **{**k})`). - -Fixing one of these shifts the divergence to the next (truncating the -level's stack to its base fixed the kwargs slot but stripped the outer -operands, and excluding the save slots then leaked the `PUSH_NULL` -marker into the output), which is exactly why this needs the call model -reworked rather than another local patch. A correct fix -therefore starts with making that call model faithful (or by delegating -the enclosing expression to the main engine's `exec`), and only then -re-adding the nesting. Emitting a visible syntax error stays preferable -to silently wrong code, so the modules remain in the failing set. +### ~~Nested comprehension inside a genexp~~ (fixed 2026-10-23) + +PR #24 recovers the inlined comprehension by decompiling the generator's +code object with the main statement parser and extracting its +For/If/Yield tree; the outer iterable is restored at the call site. This +closes the whole-element and call-argument shapes, including +`({c: v for c, v in zip(cols, row) if c} for row in rows)`. + +The three full-module cases also needed the keyword/dictionary handling +from #10 and #22: a large `MAP_ADD 1` kwargs dictionary must not be +mistaken for the comprehension's own key/value, and `DICT_MERGE` +expansions must keep their boundary. With the consolidated merge all +6411 real-world modules recompile. The earlier frame-stack analysis +(kept in git history) confirmed the right fix was to delegate the +enclosing expression to the statement engine rather than to model every +nested level in the comprehension walker. ### ~~Multi-link ternary on the value path~~ (fixed 2026-09-15) diff --git a/src/codegen.rs b/src/codegen.rs index 89be6894..eef087ce 100644 --- a/src/codegen.rs +++ b/src/codegen.rs @@ -535,6 +535,28 @@ impl Printer { orelse, finalbody, } => { + // Exception-region reconstruction can split try/except from + // its success and finally suites. An else without handlers + // cannot belong to the outer try. Rejoin the single inner + // handler-bearing try, retaining every recovered statement. + if self.version.at_least(2, 5) + && handlers.is_empty() && !orelse.is_empty() && !finalbody.is_empty() + && body.len() == 1 + { + if let Stmt::Try { + body: inner_body, handlers: inner_handlers, + orelse: inner_else, finalbody: inner_finally, + } = &body[0] + { + if !inner_handlers.is_empty() && inner_else.is_empty() && inner_finally.is_empty() { + self.stmt(&Stmt::Try { + body: inner_body.clone(), handlers: inner_handlers.clone(), + orelse: orelse.clone(), finalbody: finalbody.clone(), + }); + return; + } + } + } // a bare `except:` must be last (and unique); drop extras let mut handlers = handlers.clone(); handlers.sort_by_key(|h| if h.type_.is_none() { 1 } else { 0 }); @@ -1134,6 +1156,8 @@ impl Printer { first = false; if let Some(kn) = k { self.write(&format!("{kn}=")); + } else { + self.write("**"); } self.expr(v, prec::TERNARY); } diff --git a/src/decompiler.rs b/src/decompiler.rs index 33a3465d..f22428f2 100644 --- a/src/decompiler.rs +++ b/src/decompiler.rs @@ -259,14 +259,16 @@ struct InlineComp { key: Option, cleared_vars: Vec, target_seen: bool, - /// pending element-ternary folds: (merge offset, condition). The + /// Pending value folds: (merge offset, left/condition, optional bool op). + /// An and/or fold combines its saved left with the right value; a ternary + /// combines its condition with both arm values. The /// linear comp walk evaluates BOTH arms of an element ternary onto /// the value stack (then under else); when the walk reaches the /// ternary's merge offset the pair is folded into a Ternary. This /// works for nested ternaries (inner merge folds first) and for /// dict-comp key AND value positions alike (folding at the append /// op cannot tell which slot a pending cond belongs to). - pending_merges: Vec<(usize, ExprRef)>, + pending_merges: Vec<(usize, ExprRef, Option)>, /// else-arm target shared by the current and-chain's ternary links tern_chain_t: Option, } @@ -7540,6 +7542,12 @@ impl<'a> Ctx<'a> { while k2 < self.instrs.len() { let c = &self.instrs[k2]; match c.op { + Op::NOP | Op::NOT_TAKEN | Op::CACHE => { + // Padding does not end the post-POP_EXCEPT return + // trailer. Keep scanning through it so an unnamed + // handler retains its already loaded return value. + k2 += 1; + } Op::LOAD_CONST | Op::STORE_FAST | Op::STORE_NAME | Op::DELETE_FAST | Op::DELETE_NAME | Op::STORE_DEREF | Op::DELETE_DEREF => { @@ -8108,6 +8116,17 @@ impl<'a> Ctx<'a> { loop { let mut grew = false; for e in &self.raw_exc_entries { + // The generator/coroutine StopIteration handler spans the + // whole frame, not this source-level except/finally chain. + // Following its redirects would swallow later main-flow + // statements into the current handler region. + let frame_handler = self.idx_of.get(&e.target) + .and_then(|&index| self.instrs.get(index)) + .map_or(false, |instruction| instruction.op == Op::CALL_INTRINSIC_1 + && instruction.arg == 3); + if frame_handler { + continue; + } let seed = e.start == from || (e.target == from && e.start < from); let inside = e.start >= from @@ -14797,6 +14816,7 @@ impl<'a> Ctx<'a> { | Op::COPY_DICT_WITHOUT_KEYS => { if self.inline_comp.is_some() && matches!(inst.op, Op::LIST_APPEND | Op::SET_ADD | Op::MAP_ADD) + && (!self.version.at_least(3, 10) || arg >= 2) { self.comp_add_element(inst); } else { @@ -15280,8 +15300,8 @@ impl<'a> Ctx<'a> { let (ex_kws, star_kw) = if let Some(kw) = kwargs_opt { let (k, s) = flatten_ex_kwargs(kw.clone()); // a DICT_MERGE-built kwargs dict with const keys - // plus ONE trailing star is the compiler's own - // layout for `f(kw=..., **x)` — decompose it back + // plus expansion entries is the compiler's own + // layout for `f(kw=..., **x, **y)` — decompose it back // (the whole-dict rendering recompiles to // BUILD_MAP + DICT_UPDATE and breaks sig-exactness; // configparser 3.12 _get_conv). A DICT_UPDATE-built @@ -15295,34 +15315,24 @@ impl<'a> Ctx<'a> { break 'dc None; } if let Expr::Dict(entries) = &*kw { - if entries.len() >= 2 - && matches!( - &*entries[entries.len() - 1].0, - Expr::Starred(_) - ) - && entries[..entries.len() - 1].iter().all( - |(kk, _)| { - matches!(&**kk, Expr::Const(o) - if matches!(&**o, PyObject::Str(_))) - }, - ) - { + if entries.iter().any(|(key, _)| matches!(&**key, Expr::Starred(_))) { let mut kws = Vec::new(); - for (kk, v) in &entries[..entries.len() - 1] { - if let Expr::Const(o) = &**kk { - if let PyObject::Str(name) = &**o { - kws.push(( - Some(name.clone()), - v.clone(), - )); + let mut named = Vec::new(); + for (key, value) in entries { + match &**key { + Expr::Starred(inner) => kws.push((None, inner.clone())), + Expr::Const(object) => { + let PyObject::Str(name) = &**object else { break 'dc None; }; + if !is_kw_name_ok(name) || named.contains(name) { + break 'dc None; + } + named.push(name.clone()); + kws.push((Some(name.clone()), value.clone())); } + _ => break 'dc None, } } - if let Expr::Starred(inner) = - &*entries[entries.len() - 1].0 - { - break 'dc Some((kws, Some(inner.clone()))); - } + break 'dc Some((kws, None)); } } None @@ -19314,20 +19324,10 @@ let reopen = self return true; } let target = inst.target.unwrap_or(inst.end()); - // async for: the iterable expression is an Await, or we are - // inside an async def using GET_AITER - let is_async = matches!( - self.stack.last(), - Some(Sv::E(e)) if matches!(&**e, Expr::Await(_)) - ); - if is_async { - if let Some(Sv::E(e)) = self.stack.last_mut() { - if let Expr::Await(inner) = &**e { - *e = inner.clone(); - } - } - } - self.handle_for_iter(target, is_async); + // FOR_ITER iterates synchronously, even when the collection + // was produced by `await fetch()`. Genuine async iteration + // uses GET_AITER/GET_ANEXT and its separate decoder path. + self.handle_for_iter(target, false); true } Op::FOR_LOOP => { @@ -21174,7 +21174,10 @@ let reopen = self let group = std::mem::take(&mut self.pending_stores); self.last_store_line = None; self.pending_unpack_wrap.clear(); - let all_same = group.windows(2).all(|w| expr_eq(&w[0].1, &w[1].1)); + // COPY/DUP preserves the identity of one evaluated stack value. + // Structurally equal expressions can be independent evaluations: + // `left, right = read(), read()` must still call read twice. + let all_same = group.windows(2).all(|w| Rc::ptr_eq(&w[0].1, &w[1].1)); if all_same { let value = group[0].1.clone(); let targets: Vec = group.iter().map(|(t, _)| t.clone()).collect(); @@ -23192,16 +23195,21 @@ impl<'a> Ctx<'a> { fn is_loop_top_prefix(&self, t: usize, start: usize) -> bool { // 3.11+ folds inline CACHE entries into each instruction's size, // so offsets inside a cached instruction have no idx_of entry - // either - a back edge landing <=8 bytes before the loop top - // would falsely match a CACHE tail (compileall 3.13 main tore - // into an INCOMPLETE stub). The EXTENDED_ARG-prefix shape this - // recognizes is pre-3.11 wordcode only (no CACHE gaps). - if self.version.at_least(3, 11) { - return false; - } + // either. Absence from the instruction map cannot distinguish + // a CACHE tail from an EXTENDED_ARG prefix: modern offsets need + // proof from the original bytes. if t >= start || start - t > 8 { return false; } + if self.version.at_least(3, 11) { + if t % 2 != 0 || start % 2 != 0 { + return false; + } + if !(t..start).step_by(2).all(|offset| self.code.code.get(offset) + .map_or(false, |byte| self.table.op(*byte) == Op::EXTENDED_ARG)) { + return false; + } + } // no instruction STARTS at t (idx_of also maps END offsets: the // preceding instruction's end lands on t and must not count as // an instruction start) @@ -26399,18 +26407,18 @@ impl<'a> Ctx<'a> { )) } - /// 3.14 value-position `A1 and ... An or C`: every operand carries its + /// 3.13+ value-position `A1 and ... An or C1 or ... Cm`: each link carries its /// own COPY/TO_BOOL value-preserving idiom; the and-links PJIF to the /// or-continuation (this jump's target) and the LAST and-operand PJITs /// to the merge M; [target, M) evaluates C (t_pb f1/g_val 3.14: the /// per-jump value_merge blocks mis-nest the chain into /// `a and (b or c)`). Read-only; returns (merged value, merge offset). - fn try_value_and_or_314( + fn try_value_and_or( &self, cond: &ExprRef, target: usize, ) -> Option<(ExprRef, usize)> { - if !self.version.at_least(3, 14) { + if !self.version.at_least(3, 13) { return None; } let is_pjif = |o: Op| { @@ -26491,8 +26499,9 @@ impl<'a> Ctx<'a> { if ands.len() < 2 { return None; } - // [target, m) is the or's second operand: a pure value region - // (its leading POP_TOP discards the arriving operand copy) + // [target, m) contains the remaining OR operands. Each non-final + // operand must use the same COPY/TO_BOOL/PJIT-to-M value protocol; + // arbitrary statement branches or other merge targets do not qualify. let mi = *self.idx_of.get(&m)?; if mi <= ti { return None; @@ -26507,14 +26516,42 @@ impl<'a> Ctx<'a> { if cstart >= mi { return None; } - if !self.instrs[cstart..mi].iter().all(|x| { - x.target.is_none() - && (is_pure_value_op(x.op) - || matches!(x.op, Op::NOP | Op::NOT_TAKEN | Op::CACHE | Op::POP_TOP)) - }) { + let mut or_tail = Vec::new(); + let mut operand_start = cstart; + let mut k = cstart; + while k < mi { + let ins = self.instrs[k]; + if is_pjit(ins.op) && idiom_link(k) { + if ins.target != Some(m) + || self.instrs[k - 2].arg != 1 + || k - 2 <= operand_start + { + return None; + } + or_tail.push(self.sim_value_region(operand_start, k - 2)?); + k += 1; + while k < mi && pad(self.instrs[k].op) { + k += 1; + } + if k >= mi || self.instrs[k].op != Op::POP_TOP { + return None; + } + k += 1; + operand_start = k; + continue; + } + if ins.target.is_some() + || !(is_pure_value_op(ins.op) + || matches!(ins.op, Op::COPY | Op::TO_BOOL | Op::NOP | Op::NOT_TAKEN | Op::CACHE)) + { + return None; + } + k += 1; + } + if operand_start >= mi { return None; } - let c = self.sim_value_region(cstart, mi)?; + or_tail.push(self.sim_value_region(operand_start, mi)?); let mut and_flat = Vec::new(); for v in ands { flatten_boolop(v, BoolOpKind::And, &mut and_flat); @@ -26525,7 +26562,9 @@ impl<'a> Ctx<'a> { }) as ExprRef; let mut or_flat = Vec::new(); flatten_boolop(and, BoolOpKind::Or, &mut or_flat); - flatten_boolop(c, BoolOpKind::Or, &mut or_flat); + for value in or_tail { + flatten_boolop(value, BoolOpKind::Or, &mut or_flat); + } Some(( Rc::new(Expr::BoolOp { op: BoolOpKind::Or, @@ -28791,6 +28830,15 @@ return None; if st_t.len() != st_e.len() || st_t.is_empty() { return None; } + // CALL_KW support must not make an arm-local callable/argument + // prefix look like a preload from before the condition. Merging + // only its final slot would hoist that prefix across the truth + // test. Decline this shape until the whole arm can be preserved. + if st_t.len() != base.len() + 1 + && self.instrs[tstart..=re].iter().any(|ins| ins.op == Op::CALL_KW) + { + return None; + } let slot_t = st_t.last()?.clone(); let slot_e = st_e.last()?.clone(); // compare the UNDER-stack only — the tops are the slot values @@ -28917,11 +28965,19 @@ return None; ( Expr::Call { func: fa, args: aa, keywords: ka, star_args: sa, star_kwargs: ska }, Expr::Call { func: fb, args: ab, keywords: kb, star_args: sb, star_kwargs: skb }, - ) if ka.is_empty() && kb.is_empty() && sa.is_none() && sb.is_none() && ska.is_none() && skb.is_none() && aa.len() == ab.len() => { + ) if sa.is_none() && sb.is_none() && ska.is_none() && skb.is_none() + && aa.len() == ab.len() && ka.len() == kb.len() + && ka.iter().zip(kb).all(|((a, _), (b, _))| a == b) => { let mut slots: Vec<(&ExprRef, &ExprRef)> = vec![(fa, fb)]; for (x, y) in aa.iter().zip(ab.iter()) { slots.push((x, y)); } + // CALL_KW can now reach this merge through the bounded + // walker. Keep keyword values in the same ordered slot + // model as positional values, preserving the call itself. + for ((_, x), (_, y)) in ka.iter().zip(kb) { + slots.push((x, y)); + } let mut merged_idx: Option = None; let mut merged_val: Option = None; for (i, (x, y)) in slots.iter().enumerate() { @@ -28940,8 +28996,9 @@ return None; reps[idx] = val; Some(Rc::new(Expr::Call { func: reps[0].clone(), - args: reps[1..].to_vec(), - keywords: Vec::new(), + args: reps[1..1 + aa.len()].to_vec(), + keywords: ka.iter().zip(&reps[1 + aa.len()..]) + .map(|((name, _), value)| (name.clone(), value.clone())).collect(), star_args: None, star_kwargs: None, })) @@ -29054,6 +29111,33 @@ return None; Op::PUSH_NULL => { st.push(self.name_expr("\u{0}null")); } + Op::COPY => { + if arg == 0 || arg > st.len() { + return None; + } + st.push(st[st.len() - arg].clone()); + } + Op::STORE_FAST => { + // A walrus stores a COPY of the value still on top of the + // expression stack. Standalone assignment is a statement. + let value = st.pop()?; + let surviving = st.last_mut()?; + if !Rc::ptr_eq(surviving, &value) { + return None; + } + *surviving = Rc::new(Expr::Named { + target: self.name_expr(self.local_name(arg)), + value, + }); + } + Op::SWAP => { + if arg == 0 || arg > st.len() { + return None; + } + let top = st.len() - 1; + let other = st.len() - arg; + st.swap(top, other); + } Op::LOAD_ATTR | Op::LOAD_METHOD => { let v = pop1(&mut st); let idx = if ins.op == Op::LOAD_ATTR && self.version.at_least(3, 12) { @@ -29111,6 +29195,28 @@ return None; star_kwargs: None, })); } + Op::CALL_KW => { + let names = st.pop()?; + let Expr::Const(names) = &*names else { return None; }; + let PyObject::Tuple(names) = &**names else { return None; }; + let names: Option> = names.iter().map(|value| match &**value { + PyObject::Str(name) => Some(name.clone()), _ => None, + }).collect(); + let names = names?; + if names.len() > arg || st.len() < arg + 1 { return None; } + let all = st.split_off(st.len() - arg); + let positional = arg - names.len(); + let args = all[..positional].to_vec(); + let keywords = names.into_iter().zip(all[positional..].iter().cloned()) + .map(|(name, value)| (Some(name), value)).collect(); + let is_null = |value: &ExprRef| matches!(&**value, Expr::Name(name) if name == "\u{0}null"); + let mut callable = st.pop()?; + if is_null(&callable) { callable = st.pop()?; } + if st.last().map_or(false, is_null) { st.pop(); } + st.push(Rc::new(Expr::Call { + func: callable, args, keywords, star_args: None, star_kwargs: None, + })); + } Op::BINARY_OP => { let r = pop1(&mut st); let l = pop1(&mut st); @@ -30799,6 +30905,100 @@ return None; return; } } + // A 3.12+ lookup rejection guard has exactly one fall-through back + // edge before its success target. Preserve it as `if: continue` only + // when the success span assigns the local checked by an independent + // None guard and returned after that guard. Other suites retain their + // existing handling; changing protected suites also changes their + // exception-region boundaries. + // Treating the success region as an If suite bounded by the NEXT + // back edge closes that suite at an independent later guard and + // ejects the return that relies on its assignments (a coordinator + // lookup then returns an unbound local for non-dictionary rows). + // Inline comprehension filters retain their own bookkeeping. A + // nested while loop is not this for loop's rejection trampoline. + if self.version.at_least(3, 12) && self.inline_comp.is_none() + && self.code.exception_entries().is_empty() + && raw_target == target && target > self.cur_next + { + let current_loop = self.blocks.iter().rev() + .find(|block| matches!(block.kind, BlockType::For | BlockType::While)) + .filter(|block| block.kind == BlockType::For) + .map(|block| (block.start, block.end)); + let trampoline = self.idx_of.get(&self.cur_next).map(|&index| { + self.instrs[index..].iter().take_while(|item| item.offset < target) + .filter(|item| !matches!(item.op, Op::NOP | Op::NOT_TAKEN | Op::CACHE)) + .collect::>() + }); + if let (Some((top, loop_end)), Some(steps)) = (current_loop, trampoline) { + let dependent_return = self.idx_of.get(&target).map_or(false, |&index| { + let body = self.instrs[index..].iter().take_while(|item| item.offset < loop_end) + .filter(|item| !matches!(item.op, Op::NOP | Op::NOT_TAKEN | Op::CACHE)) + .collect::>(); + let guard = match body.iter().position(|item| item.target.is_some() + || matches!(item.op, Op::RETURN_VALUE | Op::RETURN_CONST | Op::RAISE_VARARGS | Op::RERAISE)) { + Some(index) => index, None => return false, + }; + if guard < 2 || !matches!(body[guard].op, Op::POP_JUMP_IF_NONE | Op::POP_JUMP_IF_NOT_NONE) { + return false; + } + let stored = body[..guard].iter().rev().find_map(|item| match item.op { + Op::STORE_FAST => Some(item.arg), + Op::STORE_FAST_LOAD_FAST => Some(item.arg >> 4), + Op::STORE_FAST_STORE_FAST => Some(item.arg & 15), + _ => None, + }); + let local = match stored { Some(local) => local, None => return false }; + let tested = body[guard - 1]; + if !(matches!(tested.op, Op::LOAD_FAST | Op::LOAD_FAST_BORROW | Op::LOAD_FAST_CHECK) + && tested.arg == local + || tested.op == Op::STORE_FAST_LOAD_FAST && tested.arg & 15 == local) { + return false; + } + let back = match body.get(guard + 1) { Some(back) => back, None => return false }; + if !matches!(back.op, Op::JUMP_BACKWARD | Op::JUMP_BACKWARD_NO_INTERRUPT) + || back.target != Some(top) || body[guard].target != Some(back.end()) { + return false; + } + let returned = match body.get(guard + 2) { Some(value) => value, None => return false }; + if !matches!(returned.op, Op::LOAD_FAST | Op::LOAD_FAST_BORROW | Op::LOAD_FAST_CHECK) + || returned.arg != local { return false; } + let mut tail = guard + 3; + while tail + 1 < body.len() && body[tail].op == Op::SWAP + && body[tail].arg == 2 && body[tail + 1].op == Op::POP_TOP { + tail += 2; + } + body.get(tail).map_or(false, |item| item.op == Op::RETURN_VALUE) + }); + if steps.len() == 1 + && dependent_return + && matches!(steps[0].op, Op::JUMP_BACKWARD | Op::JUMP_BACKWARD_NO_INTERRUPT) + && steps[0].target == Some(top) && steps[0].end() == target + { + self.push_stmt(Stmt::If { + // This layout negates a comparison guard by its jump, not + // its operator. Folding `<` to `>=` changes NaN and + // user comparison hooks. None-identity guards use + // their dedicated jump spelling and can invert IS. + cond: if jump_if_true { + match &*cond { + Expr::Compare { operands, ops } + if ops.len() == 1 + && matches!(ops[0], CmpOp::Is | CmpOp::IsNot) + && operands.len() == 2 + && operands.iter().any(|value| matches!(&**value, + Expr::Const(object) if matches!(&**object, PyObject::None))) => + { negate_cond(cond) } + _ => simplify_not_or_wrap(cond), + } + } else { cond }, + body: vec![Stmt::Continue], orelse: Vec::new(), + }); + self.skip_until = Some(target); + return; + } + } + } if self.version.at_least(3, 12) { if let Some((merged, body_end, tramp_end)) = self.try_or_join(&cond, jump_if_true, target) @@ -31631,6 +31831,26 @@ return None; .unwrap_or(false); if let Some(comp) = &mut self.inline_comp { if self.cur_offset < comp.end { + // A 3.12+ value-preserving COPY guard inside an element + // is an and/or expression, rather than a comprehension + // filter. Its fall-through discards the copied left value. + if let Some(&index) = self.idx_of.get(&self.cur_offset) { + let mut previous = index; + while previous > 0 && matches!(self.instrs[previous - 1].op, + Op::TO_BOOL | Op::NOP | Op::NOT_TAKEN | Op::CACHE) + { previous -= 1; } + let copied = previous > 0 + && self.instrs[previous - 1].op == Op::COPY + && self.instrs[previous - 1].arg == 1; + let discard = self.instrs[index + 1..].iter() + .find(|instruction| !matches!(instruction.op, Op::NOP | Op::NOT_TAKEN | Op::CACHE)) + .map_or(false, |instruction| instruction.op == Op::POP_TOP); + if copied && discard && target > self.cur_offset && target < comp.end { + comp.pending_merges.push((target, cond, + Some(if jump_if_true { BoolOpKind::Or } else { BoolOpKind::And }))); + return; + } + } // element ternary guard: same recognition as the // code-object builder (a forward JF between the jump // and its target skips the else arm to a merge, both @@ -31670,7 +31890,7 @@ return None; }); } } else { - comp.pending_merges.push((m_off, cj)); + comp.pending_merges.push((m_off, cj, None)); comp.tern_chain_t = Some(t_off); } return; @@ -31680,9 +31900,12 @@ return None; comp.tern_chain_t = None; // 3.12+ inline filters come in two layouts: // - INVERTED (same-line 3.12; ALL of 3.13/3.14): the - // jump lands ON the append (target is - // LIST_APPEND/SET_ADD/MAP_ADD/YIELD after pads) and - // the fall-through is the skip back edge. The jump + // jump accepts this filter, either at the next filter + // or the append, while fall-through skips the item. + // Recognize the exact pad/back-edge span first: scanning + // from the target can encounter a later filter's skip + // before reaching the append and invert this filter. + // The jump // polarity IS the filter polarity: PJT->`if C`, // PJF->`if not C`. // - CLASSIC (3.12 line-broken filters): PJF->shared @@ -31699,7 +31922,22 @@ return None; // loop TOP, and scanning from there would wrap around // and find the append after the filter again // (_py_abc 3.12 __new__ line-broken setcomp) - let target_is_append = target > self.cur_offset + let fallthrough_skips_item = self.version.at_least(3, 12) + && target > self.cur_offset + && target < comp.end + && self.idx_of.get(&self.cur_offset).zip(self.idx_of.get(&target)) + .map_or(false, |(&ci, &ti)| { + if ti <= ci { return false; } + let mut significant = self.instrs[ci + 1..ti].iter() + .filter(|x| !matches!(x.op, Op::NOP | Op::NOT_TAKEN | Op::CACHE)); + significant.next().map_or(false, |x| { + x.is_backward + && matches!(x.op, Op::JUMP_BACKWARD | Op::JUMP_BACKWARD_NO_INTERRUPT + | Op::JUMP_ABSOLUTE | Op::JUMP) + && x.target == comp.for_iter_offsets.last().copied() + }) && significant.next().is_none() + }); + let target_accepts_item = fallthrough_skips_item || (target > self.cur_offset && self .idx_of .get(&target) @@ -31734,8 +31972,8 @@ return None; | Op::YIELD_VALUE ) }) - }); - let c = if target_is_append { + })); + let c = if target_accepts_item { if jump_if_true { cond } else { @@ -31755,7 +31993,7 @@ return None; cond }; if let Some(cur) = &mut comp.cur { - if comp_if_break && !target_is_append { + if comp_if_break && !target_accepts_item { cur.if_break = true; } cur.ifs.push(c); @@ -31768,10 +32006,30 @@ return None; // elif test would create a nested If at the same target, so keep // the Else open in that case. if let Some(top) = self.blocks.last() { + // A modern Else opened immediately after the then arm's + // unconditional skip has an authoritative finite span. A guard + // inside that span can share its merge without ending the Else: + // closing it here ejects the guarded query result and may read + // its local on the other arm before assignment. Keep older and + // inferred/try-else boundaries on their existing path. + let guarded_else_merge = self.version.at_least(3, 11) + && top.kind == BlockType::Else + && self.cur_offset > top.start + && self.cur_offset < top.end + && self.idx_of.get(&top.start).map_or(false, |&si| { + si > 0 && { + let skip = &self.instrs[si - 1]; + !skip.is_backward + && skip.end() == top.start + && skip.target == Some(top.end) + && matches!(skip.op, Op::JUMP_FORWARD | Op::JUMP | Op::JUMP_ABSOLUTE) + } + }); if matches!(top.kind, BlockType::Else | BlockType::TryElse) && top.end == target && !top.is_elif && self.cur_offset != top.start + && !guarded_else_merge { self.close_blocks_at(target); } @@ -33228,6 +33486,10 @@ if split_cond { // `if c: continue`-style: fall through to the // arm below that opens an If ending at that back // edge instead of closing the loop here. + // A prefixed loop top has the same identity here + // as when matching the conditional above. Otherwise + // a later edge to EXTENDED_ARG is missed and a + // conditional continue prematurely closes the loop. let loop_start0 = self.blocks[i].start; let has_later_back_edge = self .idx_of @@ -33235,7 +33497,10 @@ if split_cond { .map_or(false, |&ci0| { self.instrs.iter().skip(ci0 + 1).any(|ins| { ins.is_backward - && ins.target == Some(loop_start0) + && ins.target.map_or(false, |t| { + t == loop_start0 + || self.is_loop_top_prefix(t, loop_start0) + }) && matches!( ins.op, Op::JUMP_ABSOLUTE @@ -33415,8 +33680,21 @@ if split_cond { } return; } + // A forward conditional can skip this back + // edge and land on the next one. Keep the + // skipped edge inside its fall-through arm; + // ending before it moves `continue` outside + // the guard and changes the recompiled jumps. + let body_end = if self.version.at_least(3, 11) + && raw_target == inst.end() + && self.is_loop_top_prefix(target, loop_start) + { + inst.end() + } else { + inst.start + }; let mut blk = - Block::new(BlockType::If, self.cur_next, inst.start); + Block::new(BlockType::If, self.cur_next, body_end); let c = if jump_if_true { negate_cond(cond.clone()) } else { @@ -34775,10 +35053,20 @@ if split_cond { // (the per-jump value_merge blocks below would mis-nest it). // !jump_if_true path only, where `c` is `cond` un-negated. if !jump_if_true { - if let Some((merged, m)) = self.try_value_and_or_314(&c, target) { - self.push(merged); - self.skip_until = Some(m); - return; + if let Some((merged, m)) = self.try_value_and_or(&c, target) { + // COPY kept this exact operand below its consumed truth + // test. The folded expression already owns that value; + // leaving it live creates an extra call argument (or steals + // the caller's callee/NULL prefix). Consume only this saved + // operand, preserving every unrelated stack entry. + if self.stack.last().map_or(false, |value| { + matches!(value, Sv::E(saved) if expr_eq(saved, &c)) + }) { + self.stack.pop(); + self.push(merged); + self.skip_until = Some(m); + return; + } } } // 3.12+ chained comparison: the then arm recomputes the next link @@ -41885,7 +42173,8 @@ fn merge_chain_compare(cond: &ExprRef, v: &ExprRef) -> Option { if o2.len() < 2 || ops2.len() + 1 != o2.len() { return None; } - let shared = match (&*operands[1], &*o2[0]) { + let last = operands.last()?; + let shared = match (&**last, &*o2[0]) { (Expr::Name(a), Expr::Name(b)) => a == b, (Expr::Const(c1), Expr::Const(c2)) => Rc::ptr_eq(c1, c2) || const_value_eq(c1, c2), (Expr::Attribute { value: v1, attr: a1 }, Expr::Attribute { value: v2, attr: a2 }) => { @@ -41893,14 +42182,17 @@ fn merge_chain_compare(cond: &ExprRef, v: &ExprRef) -> Option { } // calls/subscripts/etc: deep structural equality (side-effecting // operands must not be duplicated across the merged chain) - _ => expr_eq(&operands[1], &o2[0]), + _ => expr_eq(last, &o2[0]), }; if !shared { return None; } let mut merged_ops = ops.clone(); merged_ops.extend(ops2.iter().copied()); - let mut merged_operands = vec![operands[0].clone()]; + // Extend the complete accumulated chain, sharing only its last operand. + // Using operands[1] or retaining just operands[0] loses earlier links + // once the left side already contains more than two operands. + let mut merged_operands = operands[..operands.len() - 1].to_vec(); merged_operands.extend(o2.iter().cloned()); Some(Rc::new(Expr::Compare { operands: merged_operands, @@ -42238,6 +42530,7 @@ impl<'a> Ctx<'a> { | Some(Op::STORE_DEREF) | Some(Op::STORE_GLOBAL) | Some(Op::STORE_FAST_MAYBE_NULL) + | Some(Op::STORE_FAST_STORE_FAST) ) }; if is_store(self.instrs.get(ci + 1).map(|x| x.op)) { @@ -42442,33 +42735,30 @@ fn simplify_not(e: ExprRef) -> ExprRef { } } -/// 3.14: extended slices arrive as `slice(a, b, c)` calls or marshalled -/// slice constants; normalize both to Expr::Slice. +/// Extended indexes can contain actual marshalled slice constants. Preserve +/// runtime slice calls: their global lookup may be shadowed or have effects. fn normalize_slice_call(idx: ExprRef) -> ExprRef { + normalize_slice_constant(idx) +} + +/// Normalize actual marshalled slice values inside multi-axis indexes. +/// A runtime `slice(...)` call must retain its global lookup and evaluation. +fn normalize_slice_constant(idx: ExprRef) -> ExprRef { match &*idx { - Expr::Call { func, args, keywords, star_args: None, star_kwargs: None } - if keywords.is_empty() - && matches!(&**func, Expr::Name(n) if n == "slice") - && (2..=3).contains(&args.len()) - // a starred operand (`slice(*argvals, None)`) cannot be a - // slice display — keep the call (yt-dlp jsinterp - // `obj[*argvals:]` was not valid source) - && args.iter().all(|a| !matches!(&**a, Expr::Starred(_))) => - { - Rc::new(Expr::Slice(Box::new(SliceExpr { - start: none_if_const_none(args[0].clone()), - stop: none_if_const_none(args[1].clone()), - step: args.get(2).cloned().and_then(none_if_const_none), - }))) - } - Expr::Const(o) => match &**o { - PyObject::Slice(a, b, c) => Rc::new(Expr::Slice(Box::new(SliceExpr { - start: none_if_const_const(a.clone()), - stop: none_if_const_const(b.clone()), - step: none_if_const_const(c.clone()), + Expr::Const(object) => match &**object { + PyObject::Slice(start, stop, step) => Rc::new(Expr::Slice(Box::new(SliceExpr { + start: none_if_const_const(start.clone()), + stop: none_if_const_const(stop.clone()), + step: none_if_const_const(step.clone()), }))), + PyObject::Tuple(values) if values.iter().any(|value| matches!(&**value, PyObject::Slice(_ , _, _))) => + Rc::new(Expr::Tuple(values.iter().cloned() + .map(|value| normalize_slice_constant(Rc::new(Expr::Const(value)))).collect())), _ => idx, }, + Expr::Tuple(values) => Rc::new(Expr::Tuple( + values.iter().cloned().map(normalize_slice_constant).collect(), + )), _ => idx, } } @@ -42539,9 +42829,39 @@ fn cmp_from_index(idx: usize) -> CmpOp { // ===================== stack events, calls, functions, classes ===================== impl<'a> Ctx<'a> { + fn discarded_unpack_target(&self, remaining: usize) -> ExprRef { + // Repeated local targets can have their earlier stores replaced by + // POP_TOP. Their names may then be obfuscated while the surviving + // store keeps its slot. Reuse the next direct local store's spelling: + // the compiler elides the duplicate instead of creating an extra `_` + // local. Do not cross nested unpacking, side effects or control flow. + let name = if self.version.at_least(3, 13) { + self.idx_of.get(&self.cur_offset).and_then(|&index| { + let mut discarded = 1usize; + for next in &self.instrs[index + 1..] { + match next.op { + Op::NOP | Op::CACHE | Op::EXTENDED_ARG | Op::NOT_TAKEN => {}, + Op::POP_TOP => { + discarded += 1; + if discarded >= remaining { return None; } + } + Op::STORE_FAST => return self.code.varnames.get(next.arg as usize).cloned(), + Op::STORE_FAST_STORE_FAST | Op::STORE_FAST_LOAD_FAST => { + return self.code.varnames.get(((next.arg >> 4) & 0xF) as usize).cloned(); + } + Op::STORE_DEREF => return self.code.deref_name(next.arg as usize).map(str::to_string), + _ => return None, + } + } + None + }) + } else { None }; + self.name_expr(name.as_deref().unwrap_or("_")) + } + fn handle_pop_top(&mut self) { - // 3.14 `_` elision: the compiler replaces the store of a - // throwaway `_` name inside a tuple unpack with a POP_TOP + // 3.13+ repeated-target elision: the compiler replaces an + // overwritten local store inside a tuple unpack with a POP_TOP // discarding that slot (UNPACK_SEQUENCE; POP_TOP; // STORE_FAST_STORE_FAST). The open unpack frame must consume // the slot as a placeholder target or the tuple never @@ -42552,8 +42872,9 @@ impl<'a> Ctx<'a> { if *remaining > 0 { let is_slot = matches!(self.stack.last(), Some(Sv::E(e)) if Rc::ptr_eq(e, fv)); if is_slot { + let target = self.discarded_unpack_target(*remaining); if let Some(Sv::E(_)) = self.stack.pop() { - self.collect_unpack_target(self.name_expr("_")); + self.collect_unpack_target(target); } return; } @@ -43236,6 +43557,12 @@ impl<'a> Ctx<'a> { cur.target = Some(target); } comp.target_seen = true; + // UNPACK_SEQUENCE consumed the modeled iterator while + // collecting its tuple target. The VM keeps a distinct + // iterator slot until END_FOR/POP_ITER. Restore that slot, + // as the simple-target path already does, instead of + // suppressing cleanup for generators that still own one. + self.push(value); return; } } @@ -43919,6 +44246,24 @@ impl<'a> Ctx<'a> { .iter() .map(|x| (canon(x.op), x.arg)) .collect(); + // Masked runtime exits can leave padding between POP_EXCEPT and + // a return whose value was computed earlier. Padding is not a + // value-building tail: matching it against the mainline would + // swallow the actual response/call and resume with an empty stack. + // Look through padding, then require a real independent value head. + let head = tail.iter().find(|(op, _)| !matches!( + op, Op::NOP | Op::NOT_TAKEN | Op::CACHE | Op::EXTENDED_ARG | Op::RESUME + )); + let value_head = head.map_or(false, |(op, arg)| { + matches!(op, Op::LOAD_FAST | Op::LOAD_FAST_LOAD_FAST | Op::LOAD_NAME + | Op::LOAD_DEREF | Op::LOAD_CLASSDEREF | Op::LOAD_GLOBAL + | Op::LOAD_CONST | Op::LOAD_SMALL_INT | Op::LOAD_COMMON_CONSTANT) + || (*arg == 0 && matches!(op, + Op::BUILD_TUPLE | Op::BUILD_LIST | Op::BUILD_SET | Op::BUILD_MAP)) + }); + if !value_head { + return None; + } // require a NON-DEGENERATE tail: at least one value op before // the RETURN. A lone RETURN_VALUE (the clause's value was // loaded+SWAPed UNDER the POP_EXCEPT, _collections_abc 3.12 @@ -47810,7 +48155,11 @@ impl<'a> Ctx<'a> { _ => Vec::new(), }; match &*other { - Expr::Dict(d) => entries.extend(d.iter().cloned()), + // A call expansion of a constructed dictionary keeps + // that boundary. Flattening it into outer DICT_MERGE + // entries would change its internal DICT_UPDATE's + // overwrite semantics into duplicate-key errors. + Expr::Dict(d) if inst.op == Op::DICT_UPDATE => entries.extend(d.iter().cloned()), o => entries.push(( Rc::new(Expr::Starred(Rc::new(o.clone()))), self.name_expr(""), @@ -48628,7 +48977,7 @@ fn p649_leaked_gate(e: &ExprRef) -> Option { /// statements, recursively — conditional annotations (`if TYPE_CHECKING: /// x: T`) leave their markers inside the nested block. A marker directly /// after `name = value` merges into `name: T = value`. -fn p649_restore_gates(stmts: &mut Vec, entries: &mut Vec) { +fn p649_restore_gates(stmts: &mut Vec, entries: &mut Vec, class_scope: bool) { let mut i = 0; while i < stmts.len() { let gate = match &stmts[i] { @@ -48654,6 +49003,20 @@ fn p649_restore_gates(stmts: &mut Vec, entries: &mut Vec) { annotation: ent.ann.clone(), value: merged, }; + // A class's annotation registration gate is compiler-visible + // even when its condition was constant-folded. An unconditional + // source block retains that registration without emitting a + // runtime condition. Plain AnnAssign would lose the cell and + // the annotation function's guarded entries on recompilation. + let ann_stmt = if class_scope { + Stmt::If { + cond: Rc::new(Expr::Const(Rc::new(PyObject::True))), + body: vec![ann_stmt], + orelse: vec![], + } + } else { + ann_stmt + }; if has_value { stmts.remove(i - 1); stmts[i - 1] = ann_stmt; @@ -48668,26 +49031,47 @@ fn p649_restore_gates(stmts: &mut Vec, entries: &mut Vec) { Stmt::If { body, orelse, .. } | Stmt::While { body, orelse, .. } | Stmt::For { body, orelse, .. } => { - p649_restore_gates(body, entries); - p649_restore_gates(orelse, entries); + p649_restore_gates(body, entries, class_scope); + p649_restore_gates(orelse, entries, class_scope); } Stmt::Try { body, handlers, orelse, finalbody, .. } => { - p649_restore_gates(body, entries); + p649_restore_gates(body, entries, class_scope); for h in handlers.iter_mut() { - p649_restore_gates(&mut h.body, entries); + p649_restore_gates(&mut h.body, entries, class_scope); } - p649_restore_gates(orelse, entries); - p649_restore_gates(finalbody, entries); + p649_restore_gates(orelse, entries, class_scope); + p649_restore_gates(finalbody, entries, class_scope); } _ => {} } i += 1; } + // Keep one readable registration block for consecutive gated fields. + // Each field still has its own compiler gate index in __annotate__. + if class_scope { + fn annotation_block(stmt: &Stmt) -> bool { + matches!(stmt, Stmt::If { cond, body, orelse } + if matches!(&**cond, Expr::Const(value) if matches!(&**value, PyObject::True)) + && orelse.is_empty() && !body.is_empty() + && body.iter().all(|stmt| matches!(stmt, Stmt::AnnAssign { .. }))) + } + let mut index = 1; + while index < stmts.len() { + if annotation_block(&stmts[index - 1]) && annotation_block(&stmts[index]) { + let Stmt::If { body: current, .. } = stmts.remove(index) else { unreachable!() }; + if let Stmt::If { body, .. } = &mut stmts[index - 1] { + body.extend(current); + } + } else { + index += 1; + } + } + } } /// 3.14 PEP 649: rebuild module/class-scope annotation statements from /// the compiler-generated `__annotate__` machinery (see P649Entry). -fn restore_pep649_annotations(body: &mut Vec) { +fn restore_pep649_annotations(body: &mut Vec, code: &CodeObject) { // 1) locate the pseudo def and extract its entries let mut entries: Vec = Vec::new(); let mut def_pos: Option = None; @@ -48768,7 +49152,7 @@ fn restore_pep649_annotations(body: &mut Vec) { // 3) gated (module-scope) entries: replace their `{IDX}` markers, // recursively through nested blocks - p649_restore_gates(body, &mut entries); + p649_restore_gates(body, &mut entries, code.name != "" && code.flags & 1 == 0); // gated entries whose `{IDX}` marker never appeared were dead in the // executable flow (e.g. folded `if False:` bodies — the compiler drops @@ -48977,7 +49361,7 @@ fn postprocess_body(mut body: Vec, code: &CodeObject, version: PythonVersi // `__annotate_func__`) instead of inline `__annotations__` subscript // stores. Restore them to AnnAssign statements and strip the // protocol remnants (see restore_pep649_annotations). - restore_pep649_annotations(&mut body); + restore_pep649_annotations(&mut body, code); // __module__ / __qualname__ / __doc__ handling for class bodies let mut idx = 0; while idx < body.len() { @@ -50939,6 +51323,49 @@ impl<'a> Ctx<'a> { // ===================== comprehensions ===================== +// The main statement parser already understands PEP 709's nested inline +// comprehensions. Reuse it for generator elements instead of flattening an +// inner dict/list/set comprehension into the generator's own clauses. +fn generator_parts_from_body( + body: &[Stmt], generators: &mut Vec, +) -> Option { + let mut result = None; + for statement in body { + match statement { + Stmt::For { target, iter, body, orelse, is_async } if orelse.is_empty() => { + if result.is_some() { return None; } + generators.push(Comprehension { + target: target.clone(), iter: iter.clone(), ifs: Vec::new(), + is_async: *is_async, if_line_break: false, + }); + result = generator_parts_from_body(body, generators); + if result.is_none() { return None; } + } + Stmt::If { cond, body, orelse } if orelse.is_empty() => { + if result.is_some() { return None; } + generators.last_mut()?.ifs.push(cond.clone()); + result = generator_parts_from_body(body, generators); + if result.is_none() { return None; } + } + Stmt::Expr(value) => { + match &**value { + Expr::Yield(Some(value)) => { + if result.is_some() { return None; } + result = Some(value.clone()); + } + Expr::Const(_) => {}, // Docstring. + _ => return None, + } + } + Stmt::Pass | Stmt::Return(None) => {}, + Stmt::Return(Some(value)) + if matches!(&**value, Expr::Const(object) if matches!(&**object, PyObject::None)) => {}, + _ => return None, + } + } + result +} + impl<'a> Ctx<'a> { /// Try to turn a completed call into a comprehension. The callee is a /// Function whose code object is named `` etc. The iterable is @@ -51110,10 +51537,16 @@ fn genexpr_ternary_merge( | Op::POP_JUMP_IF_TRUE | Op::POP_JUMP_FORWARD_IF_FALSE | Op::POP_JUMP_FORWARD_IF_TRUE + | Op::POP_JUMP_IF_NONE + | Op::POP_JUMP_IF_NOT_NONE + | Op::POP_JUMP_FORWARD_IF_NONE + | Op::POP_JUMP_FORWARD_IF_NOT_NONE ) }; let mut jf_target = None; - for x in instrs.iter().skip(ii + 1) { + let mut j = ii + 1; + while j < instrs.len() { + let x = instrs[j]; if x.offset >= t { break; } @@ -51130,6 +51563,18 @@ fn genexpr_ternary_merge( break; } if is_cj(x.op) && x.target == Some(t) && !x.is_backward { + j += 1; + continue; + } + // A then-arm value may itself contain a conditional dictionary or + // another ternary. Consume its complete, pure value region before + // searching for this arm's final jump, just as the else-arm does. + if is_cj(x.op) { + let nested_merge = Self::genexpr_ternary_merge(instrs, j)?; + if nested_merge > t { + return None; + } + j = instrs.iter().position(|item| item.offset == nested_merge)?; continue; } // 3.12+: TO_BOOL sits between the value run and the cond jump @@ -51137,11 +51582,13 @@ fn genexpr_ternary_merge( // element rendered as a plain `x`, silently dropping the else // arm on EVERY version with the folded ternary shape) if matches!(x.op, Op::TO_BOOL | Op::NOP | Op::NOT_TAKEN | Op::CACHE) { + j += 1; continue; } if !Self::comp_arm_value_op(x.op) { return None; } + j += 1; } let m = jf_target?; let ti = instrs.iter().position(|x| x.offset == t)?; @@ -51157,13 +51604,7 @@ fn genexpr_ternary_merge( j += 1; continue; } - if matches!( - x.op, - Op::POP_JUMP_IF_FALSE - | Op::POP_JUMP_IF_TRUE - | Op::POP_JUMP_FORWARD_IF_FALSE - | Op::POP_JUMP_FORWARD_IF_TRUE - ) { + if is_cj(x.op) { let nm = Self::genexpr_ternary_merge(instrs, j)?; let nmi = instrs.iter().position(|x2| x2.offset == nm)?; if nmi > mi { @@ -51186,6 +51627,20 @@ fn genexpr_ternary_merge( let table = table_for(self.version).ok()?; let instrs = crate::bytecode::decode_instructions(code, table, self.version); + if self.version.at_least(3, 12) && matches!(_kind, CompKind::Generator) + && instrs.iter().any(|instruction| instruction.op == Op::LOAD_FAST_AND_CLEAR) + { + if let Some(statements) = decompile(code, self.version).ok().filter(|result| result.clean) { + let mut generators = Vec::new(); + if let Some(element) = generator_parts_from_body(&statements.body, &mut generators) { + if !generators.is_empty() { + generators[0].iter = outer_iter.clone(); + return Some((element, None, generators)); + } + } + } + } + struct PartialGen { target: Option, iter: ExprRef, @@ -51362,7 +51817,7 @@ fn genexpr_ternary_merge( _ => Vec::new(), }; match &*other { - Expr::Dict(d) => entries.extend(d.iter().cloned()), + Expr::Dict(d) if inst.op == Op::DICT_UPDATE => entries.extend(d.iter().cloned()), o => entries.push(( Rc::new(Expr::Starred(Rc::new(o.clone()))), Rc::new(Expr::Name(String::new())), @@ -51833,8 +52288,19 @@ fn genexpr_ternary_merge( let v = stack.pop().unwrap_or_else(|| Rc::new(Expr::Name("?".to_string()))); (k, v) }; - key.get_or_insert(k); - elt.get_or_insert(v); + if self.version.at_least(3, 10) && inst.arg == 1 { + // Large dict displays use MAP_ADD 1 even inside a + // comprehension. They build its element, rather + // than setting the enclosing comprehension's key. + let dictionary = stack.pop()?; + let Expr::Dict(entries) = &*dictionary else { return None; }; + let mut entries = entries.clone(); + entries.push((k, v)); + stack.push(Rc::new(Expr::Dict(entries))); + } else { + key.get_or_insert(k); + elt.get_or_insert(v); + } } Op::YIELD_VALUE => { if let Some(p) = pending_or_filter.take() { @@ -52150,15 +52616,17 @@ fn genexpr_ternary_merge( } Op::CALL_FUNCTION_EX => { // `f(*t)` / `f(*a, **k)` element: [callable(+marker), - // args-iterable, (kwargs dict when arg&1)] + // args-iterable, kwargs dict (or NULL in 3.14 when absent)]. + // Before 3.14 the kwargs slot exists only when arg&1 is set. let underflow = || Rc::new(Expr::Name("?".to_string())); - let star_kw = if inst.arg & 1 != 0 { - stack.pop() + let star_kw = if self.version.at_least(3, 14) || inst.arg & 1 != 0 { + stack.pop().filter(|value| !is_null_marker(value)) } else { None }; let star = stack.pop().unwrap_or_else(underflow); - let func = if self.version.at_least(3, 14) { + // Since 3.13 the NULL marker follows the callable. + let func = if self.version.at_least(3, 13) { if matches!(stack.last(), Some(m) if is_null_marker(m)) { stack.pop(); } @@ -52863,20 +53331,22 @@ impl<'a> Ctx<'a> { // arm-skip JFs all land on the append): fold them all, // innermost (LAST recorded) first so the stack pairs line up while let Some(idx) = self.inline_comp.as_ref().and_then(|c| { - c.pending_merges.iter().rposition(|(m, _)| *m == offset) + c.pending_merges.iter().rposition(|(m, _, _)| *m == offset) }) { - let (_, cond) = self + let (_, cond, bool_op) = self .inline_comp .as_mut() .map(|c| c.pending_merges.remove(idx)) .unwrap(); let else_v = self.pop_expr(); - let then_v = self.pop_expr(); - self.push(Rc::new(Expr::Ternary { - cond, - then_expr: then_v, - else_expr: else_v, - })); + if let Some(op) = bool_op { + self.push(Rc::new(Expr::BoolOp { op, values: vec![cond, else_v] })); + } else { + let then_v = self.pop_expr(); + self.push(Rc::new(Expr::Ternary { + cond, then_expr: then_v, else_expr: else_v, + })); + } } } @@ -52942,7 +53412,17 @@ impl<'a> Ctx<'a> { continue } Op::BUILD_LIST | Op::BUILD_SET | Op::BUILD_MAP => { - return ins.arg == 0; + if ins.arg != 0 { + return false; + } + // Modern accumulators are lifted under the iterator + // by SWAP 2. An empty default argument consumed by a + // preceding CALL is not a new comprehension level. + // Preserve the distinct py2 prologue handled above. + return !self.version.at_least(3, 12) + || self.instrs.get(back + 1).map_or(false, |next| { + next.op == Op::SWAP && next.arg == 2 + }); } o if is_pure_value_op(o) || matches!( @@ -53096,3 +53576,74 @@ impl<'a> Ctx<'a> { } } } + +#[cfg(test)] +#[path = "decompiler_comparison_tests.rs"] +mod comparison_tests; + +#[cfg(test)] +#[path = "decompiler_stack_tests.rs"] +mod stack_tests; + +#[cfg(test)] +mod generator_parts_tests { + use super::*; + + fn yielded_loop() -> Stmt { + Stmt::For { + target: Rc::new(Expr::Name("row".into())), + iter: Rc::new(Expr::Name("rows".into())), + body: vec![Stmt::Expr(Rc::new(Expr::Yield(Some(Rc::new( + Expr::Name("row".into()), + )))))], + orelse: vec![], + is_async: false, + } + } + + #[test] + fn extraction_preserves_one_yield_path_and_rejects_siblings() { + let body = vec![yielded_loop()]; + let mut generators = Vec::new(); + let element = generator_parts_from_body(&body, &mut generators).unwrap(); + assert!(matches!(&*element, Expr::Name(name) if name == "row")); + assert_eq!(generators.len(), 1); + + let mut generators = Vec::new(); + assert!(generator_parts_from_body( + &[yielded_loop(), yielded_loop()], &mut generators, + ).is_none()); + + let branch = Stmt::If { + cond: Rc::new(Expr::Name("ready".into())), + body: vec![Stmt::Expr(Rc::new(Expr::Yield(Some(Rc::new( + Expr::Name("row".into()), + )))))], + orelse: vec![], + }; + let mut body = yielded_loop(); + if let Stmt::For { body, .. } = &mut body { + *body = vec![branch.clone(), branch]; + } + assert!(generator_parts_from_body(&[body], &mut Vec::new()).is_none()); + } + + #[test] + fn extraction_does_not_discard_assignments_or_loop_else() { + for name in ["saved", "__assert_armored__"] { + let assignment = Stmt::Assign { + targets: vec![Rc::new(Expr::Name(name.into()))], + value: Rc::new(Expr::Const(Rc::new(PyObject::None))), + wrap_before: vec![], + }; + assert!(generator_parts_from_body( + &[assignment, yielded_loop()], &mut Vec::new(), + ).is_none()); + } + let mut body = yielded_loop(); + if let Stmt::For { orelse, .. } = &mut body { + *orelse = vec![Stmt::Pass]; + } + assert!(generator_parts_from_body(&[body], &mut Vec::new()).is_none()); + } +} diff --git a/src/decompiler_comparison_tests.rs b/src/decompiler_comparison_tests.rs new file mode 100644 index 00000000..88a72f68 --- /dev/null +++ b/src/decompiler_comparison_tests.rs @@ -0,0 +1,68 @@ +use super::*; + +fn name(value: &str) -> ExprRef { + Rc::new(Expr::Name(value.into())) +} + +fn call(value: &str) -> ExprRef { + Rc::new(Expr::Call { + func: name(value), args: Vec::new(), keywords: Vec::new(), + star_args: None, star_kwargs: None, + }) +} + +fn compare(operands: Vec, ops: Vec) -> ExprRef { + Rc::new(Expr::Compare { operands, ops }) +} + +#[test] +fn extend_accumulated_chain_keeps_all_operands_and_operators() { + // 0 <= first() <= second(), extended by second() < third() <= 15. + let zero = Rc::new(Expr::Const(Rc::new(PyObject::Int(0)))); + let limit = Rc::new(Expr::Const(Rc::new(PyObject::Int(15)))); + let first = call("first"); + let second = call("second"); + let third = call("third"); + let left = compare(vec![zero.clone(), first.clone(), second.clone()], + vec![CmpOp::LtE, CmpOp::LtE]); + let right = compare(vec![call("second"), third.clone(), limit.clone()], + vec![CmpOp::Lt, CmpOp::LtE]); + let merged = merge_chain_compare(&left, &right).expect("shared last operand"); + let expected = compare(vec![zero, first, second, third, limit], + vec![CmpOp::LtE, CmpOp::LtE, CmpOp::Lt, CmpOp::LtE]); + assert!(expr_eq(&merged, &expected), "{merged:?}"); +} + +#[test] +fn matching_an_earlier_operand_does_not_join_unrelated_chains() { + let left = compare(vec![name("a"), name("b"), name("c")], + vec![CmpOp::Lt, CmpOp::LtE]); + let right = compare(vec![name("b"), name("d")], vec![CmpOp::Lt]); + assert!(merge_chain_compare(&left, &right).is_none()); +} + +#[test] +fn retain_every_link_when_repeatedly_extending_a_chain() { + let mut result = compare(vec![name("a"), name("b")], vec![CmpOp::Lt]); + let names = ["a", "b", "c", "d", "e", "f"]; + for end in 2..names.len() { + let right = compare(vec![name(names[end - 1]), name(names[end])], + vec![CmpOp::Lt]); + result = merge_chain_compare(&result, &right).unwrap(); + let expected = compare(names[..=end].iter().map(|n| name(n)).collect(), + vec![CmpOp::Lt; end]); + assert!(expr_eq(&result, &expected)); + } +} + +#[test] +fn unchanged_two_operand_chain_and_its_terminal_arm_are_preserved() { + let left = compare(vec![name("a"), name("b")], vec![CmpOp::Lt]); + let terminal = merge_chain_compare(&left, &left).unwrap(); + assert!(Rc::ptr_eq(&terminal, &left)); + let right = compare(vec![name("b"), name("c")], vec![CmpOp::LtE]); + let merged = merge_chain_compare(&left, &right).unwrap(); + let expected = compare(vec![name("a"), name("b"), name("c")], + vec![CmpOp::Lt, CmpOp::LtE]); + assert!(expr_eq(&merged, &expected)); +} diff --git a/src/decompiler_stack_tests.rs b/src/decompiler_stack_tests.rs new file mode 100644 index 00000000..236a612e --- /dev/null +++ b/src/decompiler_stack_tests.rs @@ -0,0 +1,251 @@ +use super::*; +use crate::pyc::PycModule; + +fn context(code: &CodeObject, version: PythonVersion, instrs: Vec) -> Ctx<'_> { + Ctx { + code, + table: table_for(version).unwrap(), + version, + instrs, + idx_of: Default::default(), + targets: Default::default(), + stack: Vec::new(), + blocks: vec![Block::new(BlockType::Main, 0, usize::MAX)], + clean: true, + unpack_frames: Vec::new(), + comp_target_store: false, + await_mode: false, + pending_async_with_ctx: None, + skip_end_send: false, + with_exit_await_drop: false, + with_handler_starts: std::collections::HashSet::new(), + async_for_guard: None, + legacy_nest: Vec::new(), + legacy_nest_depth: 0, + py26_temps: HashMap::new(), + completed_comp_regions: Vec::new(), + last_py26_temp: None, + as_cleanup_wrappers: Vec::new(), + used_exc_info: false, + region_collect_only: false, + skipped_chain_landing: None, + pending_gen_code: None, + recent_code_const: None, + pending_match_class: None, + last_kw_names: Vec::new(), + globals: Vec::new(), + nonlocals: Vec::new(), + exc_entries: Vec::new(), + raw_exc_entries: Vec::new(), + try_ctxs: HashMap::new(), + handler_zone: None, + chain_heads: Default::default(), + with_regions: Default::default(), + with_region_handlers: Default::default(), + active_try: None, + active_try_stack: Vec::new(), + last_try_body_end: None, + pending_try_tail: None, + pending_post_chain_stmt: None, + tail_pair_at: None, + pending_try_ctx: None, + emitting_try: std::collections::HashSet::new(), + fin_copy_guard: None, + pending_nested_finally: None, + nested_inner_handlers: Vec::new(), + legacy_try: None, + legacy_handler: None, + legacy_handler_end: None, + legacy_handler_name_store: false, + legacy_handler_cleanup: false, + fused_guard_pad_at: std::collections::HashSet::new(), + swallowed_cleanup_del: None, + in_handler_prelude: false, + pending_then: Vec::new(), + pending_handlers: Vec::new(), + pending_try_body: Vec::new(), + pending_orelse_mark: None, + return_offsets: std::collections::HashSet::new(), + legacy_body_redirect: None, + pending_loop_close_at_chain: None, + chain_absorb_body: None, + sunk_return_fold_at: None, + sunk_loop_return_lift: false, + sunk_pair_drop: None, + sunk_while_lift: None, + nested_tail_defer: false, + pending_try_handlers: Vec::new(), + pending_loop: Vec::new(), + pending_with: Vec::new(), + pending_try_orelse: Vec::new(), + with_exits: 0, + pending_print: Vec::new(), + pending_print_dest: None, + unpack_targets: FrameTargets(Vec::new()), + unpack_target_lines: Vec::new(), + pending_unpack_wrap: Vec::new(), + awaiting_for_target: false, + pending_if_stmts: Vec::new(), + cur_offset: 0, + cur_next: 0, + prev_op: None, + prev_op_at_exec: None, + skip_until: None, + import_star_pop_pending: false, + cond_jump_redirect: std::collections::HashMap::new(), + dup_copy_redirect: HashMap::new(), + dup_copy_skip: Vec::new(), + broken_loop_top: None, + py2_else_pop_at: None, + pending_star_kw_call: false, + threaded_break_at: HashSet::new(), + closed_loop_tops: Vec::new(), + while_true_loops: Vec::new(), + or_rot_revals: Vec::new(), + bare_try_parse: false, + handler_sunk_tail: Vec::new(), + dup_tail_ternary_skip: None, + pending_as_cleanup: None, + star_tail: Vec::new(), + star_tail_end: None, + region_result_expr: None, + held_cleanup_store: None, + class_scope: Vec::new(), + inline_comp: None, + inline_comp_stack: Vec::new(), + pending_restore_vars: Vec::new(), + comp_restore_vars: Vec::new(), + cur_line: None, + pending_stores: Vec::new(), + last_store_line: None, + group_start: 0, + last_flush_offset: 0, + flushing: false, + pending_aug: None, + pending_decorators: Vec::new(), + pending_class_decorators: Vec::new(), + pending_py2_class: None, + import_names: Vec::new(), + dotted_as_just_emitted: false, + import_module: None, + } +} + +fn instruction(op: Op, arg: u32) -> Instruction { + Instruction { + offset: 0, start: 0, size: 2, opcode: 0, op, arg, + has_arg: true, target: None, is_backward: false, line: None, + } +} + +fn fixture() -> PycModule { + PycModule::from_bytes(include_bytes!("../tests/pyc/comprehensions.3.11.pyc")).unwrap() +} + +#[test] +fn copy_keeps_one_evaluation_and_swap_reorders_without_losing_operands() { + let module = fixture(); + // The intermediate stack in `lower <= read() <= upper` is reordered + // and copied so the middle expression is evaluated exactly once. + let lower = Rc::new(Expr::Name("lower".into())); + let middle = Rc::new(Expr::Call { + func: Rc::new(Expr::Name("read".into())), args: Vec::new(), + keywords: Vec::new(), star_args: None, star_kwargs: None, + }); + for minor in 11..=14 { + let version = PythonVersion { minor, ..module.version() }; + let ctx = context(&module.code, version, + vec![instruction(Op::SWAP, 2), instruction(Op::COPY, 2)]); + let result = ctx.sim_stack_region(0, 2, vec![lower.clone(), middle.clone()]) + .expect("COPY and SWAP are available from Python 3.11"); + assert_eq!(result.len(), 3); + assert!(Rc::ptr_eq(&result[0], &middle)); + assert!(Rc::ptr_eq(&result[1], &lower)); + assert!(Rc::ptr_eq(&result[2], &middle)); + } +} + +#[test] +fn copy_and_swap_respect_deeper_stack_entries_and_region_boundaries() { + let module = fixture(); + let values: Vec<_> = ["callee", "argument", "value"].iter() + .map(|name| Rc::new(Expr::Name((*name).into()))).collect(); + let ctx = context(&module.code, module.version(), vec![ + instruction(Op::SWAP, 3), instruction(Op::COPY, 3), + instruction(Op::STORE_FAST, 0), + ]); + let result = ctx.sim_stack_region(0, 2, values.clone()).unwrap(); + assert_eq!(result.len(), 4); + for (index, source) in [2, 1, 0, 2].iter().enumerate() { + assert!(Rc::ptr_eq(&result[index], &values[*source])); + } + // Simulation must still reject a region that includes a statement. + assert!(ctx.sim_stack_region(0, 3, values).is_none()); +} + +#[test] +fn copy_and_swap_reject_zero_depth_and_stack_underflow() { + let module = fixture(); + for op in [Op::COPY, Op::SWAP] { + for (depth, length) in [(0, 0), (0, 1), (1, 0), (2, 1), (u32::MAX, 1)] { + let ctx = context(&module.code, module.version(), vec![instruction(op, depth)]); + let values = vec![Rc::new(Expr::Name("value".into())); length]; + assert!(ctx.sim_stack_region(0, 1, values).is_none(), "{op:?} {depth}"); + } + } +} + +#[test] +fn top_copy_and_identity_swap_are_valid() { + let module = fixture(); + let value = Rc::new(Expr::Name("value".into())); + let ctx = context(&module.code, module.version(), + vec![instruction(Op::COPY, 1), instruction(Op::SWAP, 1)]); + let result = ctx.sim_stack_region(0, 2, vec![value.clone()]).unwrap(); + assert_eq!(result.len(), 2); + assert!(result.iter().all(|entry| Rc::ptr_eq(entry, &value))); +} + +#[test] +fn walrus_keeps_the_local_binding_and_one_evaluation() { + let mut module = fixture(); + Rc::get_mut(&mut module.code).unwrap().varnames = vec!["state".into()]; + let value = Rc::new(Expr::Call { + func: Rc::new(Expr::Name("get".into())), + args: vec![Rc::new(Expr::Name("entity_id".into()))], + keywords: Vec::new(), star_args: None, star_kwargs: None, + }); + let outer = Rc::new(Expr::Name("outer".into())); + // Assignment expressions exist since 3.8; this COPY-backed encoding + // starts in 3.11. The bounded walker must preserve the binding in + // `(state := get(entity_id))` while retaining unrelated stack entries. + for minor in 11..=14 { + let version = PythonVersion { minor, ..module.version() }; + let ctx = context(&module.code, version, vec![ + instruction(Op::COPY, 1), instruction(Op::STORE_FAST, 0), + ]); + let result = ctx.sim_stack_region(0, 2, vec![outer.clone(), value.clone()]) + .expect("COPY followed by STORE_FAST is an assignment expression"); + assert_eq!(result.len(), 2); + assert!(Rc::ptr_eq(&result[0], &outer)); + match &*result[1] { + Expr::Named { target, value: assigned } => { + assert!(matches!(&**target, Expr::Name(name) if name == "state")); + assert!(Rc::ptr_eq(assigned, &value)); + } + other => panic!("expected a named expression, got {other:?}"), + } + } +} + +#[test] +fn store_fast_rejects_separate_evaluations_and_standalone_assignments() { + let module = fixture(); + let ctx = context(&module.code, module.version(), + vec![instruction(Op::STORE_FAST, 0)]); + let name = || Rc::new(Expr::Name("value".into())); + // Equal syntax does not prove that the VM copied a single evaluation. + for stack in [vec![], vec![name()], vec![name(), name()]] { + assert!(ctx.sim_stack_region(0, 1, stack).is_none()); + } +} diff --git a/tests/behavior/cases/b_assignment_chains.py b/tests/behavior/cases/b_assignment_chains.py new file mode 100644 index 00000000..f43b9930 --- /dev/null +++ b/tests/behavior/cases/b_assignment_chains.py @@ -0,0 +1,26 @@ +# MIN_VERSION: 3.13 + + +def three(read): + a = b = c = read() + return a, b, c + + +def six(read): + a = b = c = d = e = f = read() + return a, b, c, d, e, f + + +def three_none(): + a = b = c = None + return a, b, c + + +events = [] +def read(): + events.append("read") + return len(events) + +print(three(read), events) +print(six(read), events) +print(three_none()) diff --git a/tests/behavior/cases/b_assignment_identity.py b/tests/behavior/cases/b_assignment_identity.py new file mode 100644 index 00000000..471da697 --- /dev/null +++ b/tests/behavior/cases/b_assignment_identity.py @@ -0,0 +1,20 @@ +# MIN_VERSION: 3.9 + + +def separate(read): + left, right = read(), read() + return left, right + + +def shared(read): + left = right = read() + return left, right + + +events = [] +def read(): + events.append("read") + return len(events) + +print(separate(read), events) +print(shared(read), events) diff --git a/tests/behavior/cases/b_awaited_synchronous_iteration.py b/tests/behavior/cases/b_awaited_synchronous_iteration.py new file mode 100644 index 00000000..ccf15d1a --- /dev/null +++ b/tests/behavior/cases/b_awaited_synchronous_iteration.py @@ -0,0 +1,14 @@ +# MIN_VERSION: 3.9 +import asyncio + +async def awaited(read): + result = [] + for item in await read(): + result.append(item) + return result + + +async def read(): + print("read") + return [1, 2, 3] +print(asyncio.run(awaited(read))) diff --git a/tests/behavior/cases/b_boolean_call_arguments.py b/tests/behavior/cases/b_boolean_call_arguments.py new file mode 100644 index 00000000..173d1eed --- /dev/null +++ b/tests/behavior/cases/b_boolean_call_arguments.py @@ -0,0 +1,31 @@ +# MIN_VERSION: 3.13 + + +def select(call, before, first, second, last, after): + return call(before(), (first() and second()) or last(), after()) + + +def alternatives(a, b, c, d, e): + return bool((a and b) or c or d or e) + + +def before(): + print("before") + return 1 +def first(): + print("first") + return 0 +def second(): + print("second") + return 2 +def last(): + print("last") + return 3 +def after(): + print("after") + return 4 +def call(*args): + return args +print(select(call, before, first, second, last, after)) +print(alternatives(1, 0, 0, 0, 5)) +print(alternatives(1, 2, 0, 0, 0)) diff --git a/tests/behavior/cases/b_chained_comprehension_filters.py b/tests/behavior/cases/b_chained_comprehension_filters.py new file mode 100644 index 00000000..0320c450 --- /dev/null +++ b/tests/behavior/cases/b_chained_comprehension_filters.py @@ -0,0 +1,25 @@ +# MIN_VERSION: 3.12 + + +def methods_equal(rows, wanted): + return [row.tag for row in rows if row.value == wanted if row.ready() if row.final()] + + +def methods_unequal(rows, wanted): + return [row.tag for row in rows if row.value != wanted if not row.ready() if row.final()] + + +class Row: + def __init__(self, tag, value, ready): + self.tag = tag + self.value = value + self.available = ready + def ready(self): + print("ready", self.tag) + return self.available + def final(self): + print("final", self.tag) + return True +rows = [Row("a", 1, True), Row("b", 2, True), Row("c", 1, False)] +print(methods_equal(rows, 1)) +print(methods_unequal(rows, 2)) diff --git a/tests/behavior/cases/b_class_annotation_registration.py b/tests/behavior/cases/b_class_annotation_registration.py new file mode 100644 index 00000000..b69c745f --- /dev/null +++ b/tests/behavior/cases/b_class_annotation_registration.py @@ -0,0 +1,31 @@ +# MIN_VERSION: 3.14 + + +class Always: + "Documented class." + if True: + amount: int = 3 + label: str = "ready" + + def read(self) -> int: + return self.amount + + +def select(enabled, values, default): + class Result: + if True: + amount: int = 3 + if enabled: + label: str = "ready" + for value in values: + limit: int = default(value) + return Result + + +print(Always.__annotate__.__code__.co_freevars) +print(sorted(Always.__annotations__)) +def default(value): + print("default", value) + return value +print(sorted(select(False, [], default).__annotations__)) +print(sorted(select(True, [3, 4], default).__annotations__)) diff --git a/tests/behavior/cases/b_comprehension_boolean_values.py b/tests/behavior/cases/b_comprehension_boolean_values.py new file mode 100644 index 00000000..bdfc5ead --- /dev/null +++ b/tests/behavior/cases/b_comprehension_boolean_values.py @@ -0,0 +1,27 @@ +# MIN_VERSION: 3.12 + + +def select(rows): + return [max(0.0, float(row.get("x") or 0.0)) for row in rows] + + +def calls(rows): + return [row.first() or row.second() for row in rows] + + +def and_calls(rows): + return [row.first() and row.second() for row in rows] + + +print(select([{}, {"x": None}, {"x": 0}, {"x": -2}, {"x": 7}])) +class Row: + def __init__(self, value): + self.value = value + def first(self): + print("first", self.value) + return self.value + def second(self): + print("second", self.value) + return 9 +print(calls([Row(0), Row(3)])) +print(and_calls([Row(0), Row(3)])) diff --git a/tests/behavior/cases/b_comprehension_default_arguments.py b/tests/behavior/cases/b_comprehension_default_arguments.py new file mode 100644 index 00000000..2ff920a8 --- /dev/null +++ b/tests/behavior/cases/b_comprehension_default_arguments.py @@ -0,0 +1,27 @@ +# MIN_VERSION: 3.12 + + +def list_default(rows): + row = "saved-row" + field = "saved-field" + result = [field for row in rows for field in row.get("values", [])] + return result, row, field + + +def dict_default(rows): + row = "saved-row" + field = "saved-field" + result = {field: len(field) for row in rows for field in row.get("values", {})} + return result, row, field + + +def nested_list(rows): + row = "saved-row" + field = "saved-field" + result = [[field for field in row.get("values", [])] for row in rows] + return result, row, field + + +print(list_default([{}, {"values": ["a", "bb"]}, {}])) +print(dict_default([{}, {"values": {"a": 1, "bb": 2}}, {}])) +print(nested_list([{}, {"values": ["a"]}, {}])) diff --git a/tests/behavior/cases/b_comprehension_dictionary_displays.py b/tests/behavior/cases/b_comprehension_dictionary_displays.py new file mode 100644 index 00000000..68c35a62 --- /dev/null +++ b/tests/behavior/cases/b_comprehension_dictionary_displays.py @@ -0,0 +1,25 @@ +# MIN_VERSION: 3.10 + + +def list17(rows): + return [{ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, + } for row in rows] + + +def generator20(rows): + return ({ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, "k17": row + 17, "k18": row + 18, "k19": row + 19, + } for row in rows) + + +print(list17([0, 10])) +print(tuple(generator20([1, 20]))) diff --git a/tests/behavior/cases/b_conditional_loop_prefix.py b/tests/behavior/cases/b_conditional_loop_prefix.py new file mode 100644 index 00000000..2500953a --- /dev/null +++ b/tests/behavior/cases/b_conditional_loop_prefix.py @@ -0,0 +1,63 @@ +# MIN_VERSION: 3.11 + +# Attribute loads make FOR_ITER require EXTENDED_ARG on Python 3.12+. +# Conditional branches lead to adjacent back edges targeting that prefix. +def process(rows, work): + success = True + for row in rows: + if row.is_file: + if not work( + "file", row.ok, + row.a, row.b, row.c, row.d, row.e, row.f, + row.g, row.h, row.i, row.j, row.k, row.l, + ): + success = False + else: + if not work( + "directory", row.ok, + row.a, row.b, row.c, row.d, row.e, row.f, + row.g, row.h, row.i, row.j, row.k, row.l, + ): + success = False + return success + + +class Row: + a = 1 + b = 2 + c = 3 + d = 4 + e = 5 + f = 6 + g = 7 + h = 8 + i = 9 + j = 10 + k = 11 + l = 12 + + def __init__(self, is_file, ok): + self.is_file = is_file + self.ok = ok + + +def work(kind, ok, *values): + calls.append((kind, ok, values)) + return ok + + +for outcomes in [ + [], + [(True, True)], + [(True, False)], + [(False, True)], + [(False, False)], + [(True, True), (False, True)], + [(False, True), (True, True)], + [(True, False), (False, True), (True, True)], + [(False, False), (True, True), (False, True)], +]: + calls = [] + rows = [Row(is_file, ok) for is_file, ok in outcomes] + print(process(rows, work)) + print(calls) diff --git a/tests/behavior/cases/b_extended_loop_prefix.py b/tests/behavior/cases/b_extended_loop_prefix.py new file mode 100644 index 00000000..f8203427 --- /dev/null +++ b/tests/behavior/cases/b_extended_loop_prefix.py @@ -0,0 +1,50 @@ +# MIN_VERSION: 3.11 + + +def size_30(rows, work): + result = [] + for row in rows: + try: + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + except ValueError: + continue + result.append(row) + return result + + +calls = [] +def work(row): + calls.append(row) + if row == 2: + raise ValueError(row) +print(size_30([1, 2, 3], work)) +print(calls) diff --git a/tests/behavior/cases/b_generator_expanded_calls.py b/tests/behavior/cases/b_generator_expanded_calls.py new file mode 100644 index 00000000..f2164184 --- /dev/null +++ b/tests/behavior/cases/b_generator_expanded_calls.py @@ -0,0 +1,20 @@ +# MIN_VERSION: 3.13 + + +def positional(rows, factory): + return tuple(factory(*row) for row in rows) + + +def keywords(rows, factory): + return tuple(factory(**dict(row)) for row in rows) + + +def both(rows, factory): + return tuple(factory(*row[0], **row[1]) for row in rows) + + +def factory(*args, **kwargs): + return args, sorted(kwargs.items()) +print(positional([[1, 2], [3]], factory)) +print(keywords([{"a": 1}, {"b": 2}], factory)) +print(both([([1], {"a": 2})], factory)) diff --git a/tests/behavior/cases/b_guarded_else_merge.py b/tests/behavior/cases/b_guarded_else_merge.py new file mode 100644 index 00000000..a2692415 --- /dev/null +++ b/tests/behavior/cases/b_guarded_else_merge.py @@ -0,0 +1,29 @@ +# MIN_VERSION: 3.11 +import asyncio + +async def awaited(owner): + value = None + active = owner.active + if active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = float(row[0]) + owner.record(value) + return value + + +class Owner: + def __init__(self, active, row): + self.active = active + self.row = row + async def fetch(self): + print("fetch", self.row) + return self.row + def record(self, value): + print("record", value) +print(asyncio.run(awaited(Owner(True, (9,))))) +print(asyncio.run(awaited(Owner(False, (9,))))) +print(asyncio.run(awaited(Owner(False, (None,))))) +print(asyncio.run(awaited(Owner(False, None)))) diff --git a/tests/behavior/cases/b_keyword_conditional_calls.py b/tests/behavior/cases/b_keyword_conditional_calls.py new file mode 100644 index 00000000..ad3f42fe --- /dev/null +++ b/tests/behavior/cases/b_keyword_conditional_calls.py @@ -0,0 +1,25 @@ +# MIN_VERSION: 3.13 + + +def case_0(value, build, other): + return build("reject" if value else "accept", amount=0) + + +def case_1(value, build, other): + return build("prefix", amount=0 if value else 1) + + +def case_9(value, build, other): + return build("prefix", amount=1 if value else 2, last=other(value)) + + +def build(*args, **kwargs): + print("build", args, sorted(kwargs.items())) + return args +def other(value): + print("other", value) + return 8 +print(case_0(False, build, other)) +print(case_0(True, build, other)) +print(case_1(False, build, other)) +print(case_9(True, build, other)) diff --git a/tests/behavior/cases/b_loop_lookup_guards.py b/tests/behavior/cases/b_loop_lookup_guards.py new file mode 100644 index 00000000..558b76a2 --- /dev/null +++ b/tests/behavior/cases/b_loop_lookup_guards.py @@ -0,0 +1,37 @@ +# MIN_VERSION: 3.12 + + +def rating_lookup(values): + for value in values: + if not value.rating < 10: + continue + coordinator = value.coordinator + if coordinator is None: + continue + return coordinator + + +def dictionary_lookup(rows): + for row in rows: + if not isinstance(row, dict): + continue + coordinator = row.get("coordinator") + if coordinator is None: + continue + return coordinator + + +class Rating: + def __init__(self, value): + self.value = value + def __lt__(self, limit): + print("lt", self.value, limit) + return self.value < limit + def __ge__(self, limit): + raise AssertionError("comparison must not be inverted") +class Row: + def __init__(self, rating, coordinator): + self.rating = Rating(rating) + self.coordinator = coordinator +print(rating_lookup([Row(float("nan"), 8), Row(3, None), Row(4, 0)])) +print(dictionary_lookup([1, {}, {"coordinator": False}])) diff --git a/tests/behavior/cases/b_nested_conditional_elements.py b/tests/behavior/cases/b_nested_conditional_elements.py new file mode 100644 index 00000000..f3578c49 --- /dev/null +++ b/tests/behavior/cases/b_nested_conditional_elements.py @@ -0,0 +1,18 @@ +# MIN_VERSION: 3.12 + + +def select(rows): + return [{ + "excluded": ({"count": row["count"], + "reasons": loads(row["reasons"]) if row["reasons"] else {}} + if row["count"] is not None else {}), + "archived": str(row["archived"]) if row["archived"] else None, + } for row in rows] + + +def loads(value): + print("loads", value) + return {"decoded": value} +print(select([{"count": 1, "reasons": "why", "archived": 3}, + {"count": None, "reasons": "skip", "archived": 0}, + {"count": 2, "reasons": "", "archived": None}])) diff --git a/tests/behavior/cases/b_nested_generator_comprehensions.py b/tests/behavior/cases/b_nested_generator_comprehensions.py new file mode 100644 index 00000000..33da158d --- /dev/null +++ b/tests/behavior/cases/b_nested_generator_comprehensions.py @@ -0,0 +1,26 @@ +# MIN_VERSION: 3.12 + + +def dictionary(rows): + return tuple({key: row[key] for key in row} for row in rows) + + +def call_arguments(rows, collect): + return tuple(collect([value * 2 for value in row]) for row in rows) + + +def filtered(rows): + return tuple({key: row[key] for key in row if key} for row in rows if row) + + +def unpacked(rows, columns): + return tuple({key: value for key, value in zip(columns, row) if key} for row in rows) + + +def collect(values): + print("collect", values) + return sum(values) +print(dictionary([{"a": 1, "b": 2}, {}, {"c": 3}])) +print(call_arguments([[1, 2], [], [3]], collect)) +print(filtered([{}, {"": 0, "a": 1}])) +print(unpacked([[1, 2], [3, 4]], ["a", "b"])) diff --git a/tests/behavior/cases/b_ordered_keyword_expansions.py b/tests/behavior/cases/b_ordered_keyword_expansions.py new file mode 100644 index 00000000..1d45533c --- /dev/null +++ b/tests/behavior/cases/b_ordered_keyword_expansions.py @@ -0,0 +1,32 @@ +# MIN_VERSION: 3.9 + + +def direct(fn, first, second): + return fn(1, fixed=2, **first, **second) + + +def dictionary(fn, first, second): + return fn(1, **{"fixed": 2, **first, **second}) + + +def effectful(fn, get_first, get_second): + return fn(1, fixed=2, **get_first(), **get_second()) + + +def fn(*args, **kwargs): + print("call", args, sorted(kwargs.items())) +def first(): + print("first") + return {"fixed": 7} +def second(): + print("second") + return {"extra": 8} +try: + direct(fn, {"fixed": 7}, {"extra": 8}) +except TypeError: + print("duplicate") +dictionary(fn, {"fixed": 7}, {"extra": 8}) +try: + effectful(fn, first, second) +except TypeError: + print("duplicate before second") diff --git a/tests/behavior/cases/b_repeated_unpack_bindings.py b/tests/behavior/cases/b_repeated_unpack_bindings.py new file mode 100644 index 00000000..40f05f9b --- /dev/null +++ b/tests/behavior/cases/b_repeated_unpack_bindings.py @@ -0,0 +1,31 @@ +# MIN_VERSION: 3.13 + + +def underscore_loop(values): + result = [] + for value, _, discarded, discarded in values: + result.append((value, discarded, _)) + return result + + +def underscore_list(values): + return [(value, discarded, _) for value, _, discarded, discarded in values] + + +def nested_loop(values): + result = [] + for value, (discarded, discarded) in values: + result.append((value, discarded)) + return result + + +def nested_list(values): + return [(value, discarded) for value, (discarded, discarded) in values] + + +rows = [(1, 2, 3, 4), (5, 6, 7, 8)] +print(underscore_loop(rows)) +print(underscore_list(rows)) +nested = [(1, (2, 3)), (4, (5, 6))] +print(nested_loop(nested)) +print(nested_list(nested)) diff --git a/tests/behavior/cases/b_sequential_generator_handlers.py b/tests/behavior/cases/b_sequential_generator_handlers.py new file mode 100644 index 00000000..54ae3d4c --- /dev/null +++ b/tests/behavior/cases/b_sequential_generator_handlers.py @@ -0,0 +1,41 @@ +# MIN_VERSION: 3.12 +import asyncio + +def synchronous(work): + try: + yield work("first") + except ValueError: + yield 2 + yield 3 + try: + yield work("last") + except KeyError: + yield 5 + yield 6 + + +async def asynchronous(work): + try: + yield work("first") + except ValueError: + yield 2 + yield 3 + try: + yield work("last") + except KeyError: + yield 5 + yield 6 + + +def work(stage): + print(stage) + if stage == "first": + raise ValueError(stage) + return 9 +print(tuple(synchronous(work))) +async def consume(): + result = [] + async for value in asynchronous(work): + result.append(value) + return result +print(asyncio.run(consume())) diff --git a/tests/behavior/cases/b_slice_constants_and_calls.py b/tests/behavior/cases/b_slice_constants_and_calls.py new file mode 100644 index 00000000..d8301971 --- /dev/null +++ b/tests/behavior/cases/b_slice_constants_and_calls.py @@ -0,0 +1,35 @@ +# MIN_VERSION: 3.9 + + +def dynamic_axes(data, stop): + return data[:stop, :] + + +def constant_axes(data, stop): + return data[:, :] + + +def mixed_axes(data, stop): + return data[1:stop:2, :, 7] + + +def call(data, stop): + return data[slice(None, stop, None)] + + +def call_axes(data, stop): + return data[slice(None, stop, None), :] + + +class Data: + def __getitem__(self, key): + return repr(key) +def slice(*args): + print("custom slice", args) + return args +data = Data() +print(dynamic_axes(data, 4)) +print(constant_axes(data, 4)) +print(mixed_axes(data, 4)) +print(call(data, 4)) +print(call_axes(data, 4)) diff --git a/tests/behavior/cases/b_unpacked_comprehension_cleanup.py b/tests/behavior/cases/b_unpacked_comprehension_cleanup.py new file mode 100644 index 00000000..2b30f9b5 --- /dev/null +++ b/tests/behavior/cases/b_unpacked_comprehension_cleanup.py @@ -0,0 +1,33 @@ +# MIN_VERSION: 3.12 + + +def unpack4(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [a + b + c + d for rows in groups for a, b, c, d in rows] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack_list(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c) for a, b in groups for c, d in b] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack6(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c, e) for a, b in groups for c, d in b for e, f in d] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +print(unpack4([[], [(1, 2, 3, 4)], [(5, 6, 7, 8)]])) +print(unpack_list([(1, [(3, 4), (5, 6)]), (7, [])])) +print(unpack6([(1, [(3, [(5, 6), (7, 8)])]), (9, [])])) diff --git a/tests/corpus/3.11.14/handler_return_padding_3.py b/tests/corpus/3.11.14/handler_return_padding_3.py new file mode 100644 index 00000000..7e7278b7 --- /dev/null +++ b/tests/corpus/3.11.14/handler_return_padding_3.py @@ -0,0 +1,15 @@ +# The .pyc variants insert 1, 3 or 9 NOPs before each RETURN_VALUE. +# Regenerate with the matching CPython: python tools/compile_padded_returns.py +# Jump targets and exception-table boundaries are relocated with the padding. + +async def handled(read, log): + try: + await read() + log("saved") + except ValueError: + log("locked") + return False + except Exception as error: + log("failed", error) + return False + return True diff --git a/tests/corpus/3.11.14/handler_return_padding_3.pyc b/tests/corpus/3.11.14/handler_return_padding_3.pyc new file mode 100644 index 00000000..ff2d6ead Binary files /dev/null and b/tests/corpus/3.11.14/handler_return_padding_3.pyc differ diff --git a/tests/corpus/3.11.14/out/handler_return_padding_3.py b/tests/corpus/3.11.14/out/handler_return_padding_3.py new file mode 100644 index 00000000..2704aa0c --- /dev/null +++ b/tests/corpus/3.11.14/out/handler_return_padding_3.py @@ -0,0 +1,12 @@ +async def handled(read, log): + try: + await read() + log('saved') + except ValueError: + log('locked') + return False + except Exception as error: + log('failed', error) + return False + return True + diff --git a/tests/corpus/3.11.14/out/padded_value_returns_3.py b/tests/corpus/3.11.14/out/padded_value_returns_3.py new file mode 100644 index 00000000..f79afb3c --- /dev/null +++ b/tests/corpus/3.11.14/out/padded_value_returns_3.py @@ -0,0 +1,14 @@ +def synchronous(read, build): + try: + item = read() + except ValueError: + return build('missing') + return build(item) + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build('missing') + return build(item) + diff --git a/tests/corpus/3.11.14/padded_value_returns_3.py b/tests/corpus/3.11.14/padded_value_returns_3.py new file mode 100644 index 00000000..fb1d66fb --- /dev/null +++ b/tests/corpus/3.11.14/padded_value_returns_3.py @@ -0,0 +1,19 @@ +# Generate .pyc files with the matching CPython 3.11–3.14 interpreter: +# python tools/compile_padded_value_returns.py +# Each RETURN_VALUE receives 1, 3 or 9 NOPs, with jumps and exception +# boundaries relocated. Padding never constructs a new return value. + +def synchronous(read, build): + try: + item = read() + except ValueError: + return build("missing") + return build(item) + + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build("missing") + return build(item) diff --git a/tests/corpus/3.11.14/padded_value_returns_3.pyc b/tests/corpus/3.11.14/padded_value_returns_3.pyc new file mode 100644 index 00000000..8d8162a7 Binary files /dev/null and b/tests/corpus/3.11.14/padded_value_returns_3.pyc differ diff --git a/tests/corpus/3.11.14/report.json b/tests/corpus/3.11.14/report.json new file mode 100644 index 00000000..b587362e --- /dev/null +++ b/tests/corpus/3.11.14/report.json @@ -0,0 +1,17 @@ +{ + "version": "3.11.14", + "interp": "~/.local/share/uv/python/cpython-3.11-macos-aarch64-none/bin/python3.11", + "counts": { + "PASS": 2 + }, + "results": [ + { + "module": "handler_return_padding_3", + "verdict": "PASS" + }, + { + "module": "padded_value_returns_3", + "verdict": "PASS" + } + ] +} \ No newline at end of file diff --git a/tests/corpus/3.12.3/handler_return_padding_3.py b/tests/corpus/3.12.3/handler_return_padding_3.py new file mode 100644 index 00000000..7e7278b7 --- /dev/null +++ b/tests/corpus/3.12.3/handler_return_padding_3.py @@ -0,0 +1,15 @@ +# The .pyc variants insert 1, 3 or 9 NOPs before each RETURN_VALUE. +# Regenerate with the matching CPython: python tools/compile_padded_returns.py +# Jump targets and exception-table boundaries are relocated with the padding. + +async def handled(read, log): + try: + await read() + log("saved") + except ValueError: + log("locked") + return False + except Exception as error: + log("failed", error) + return False + return True diff --git a/tests/corpus/3.12.3/handler_return_padding_3.pyc b/tests/corpus/3.12.3/handler_return_padding_3.pyc new file mode 100644 index 00000000..aae4bf0e Binary files /dev/null and b/tests/corpus/3.12.3/handler_return_padding_3.pyc differ diff --git a/tests/corpus/3.12.3/out/handler_return_padding_3.py b/tests/corpus/3.12.3/out/handler_return_padding_3.py new file mode 100644 index 00000000..2704aa0c --- /dev/null +++ b/tests/corpus/3.12.3/out/handler_return_padding_3.py @@ -0,0 +1,12 @@ +async def handled(read, log): + try: + await read() + log('saved') + except ValueError: + log('locked') + return False + except Exception as error: + log('failed', error) + return False + return True + diff --git a/tests/corpus/3.12.3/out/padded_value_returns_3.py b/tests/corpus/3.12.3/out/padded_value_returns_3.py new file mode 100644 index 00000000..f79afb3c --- /dev/null +++ b/tests/corpus/3.12.3/out/padded_value_returns_3.py @@ -0,0 +1,14 @@ +def synchronous(read, build): + try: + item = read() + except ValueError: + return build('missing') + return build(item) + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build('missing') + return build(item) + diff --git a/tests/corpus/3.12.3/padded_value_returns_3.py b/tests/corpus/3.12.3/padded_value_returns_3.py new file mode 100644 index 00000000..fb1d66fb --- /dev/null +++ b/tests/corpus/3.12.3/padded_value_returns_3.py @@ -0,0 +1,19 @@ +# Generate .pyc files with the matching CPython 3.11–3.14 interpreter: +# python tools/compile_padded_value_returns.py +# Each RETURN_VALUE receives 1, 3 or 9 NOPs, with jumps and exception +# boundaries relocated. Padding never constructs a new return value. + +def synchronous(read, build): + try: + item = read() + except ValueError: + return build("missing") + return build(item) + + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build("missing") + return build(item) diff --git a/tests/corpus/3.12.3/padded_value_returns_3.pyc b/tests/corpus/3.12.3/padded_value_returns_3.pyc new file mode 100644 index 00000000..02223be6 Binary files /dev/null and b/tests/corpus/3.12.3/padded_value_returns_3.pyc differ diff --git a/tests/corpus/3.12.3/report.json b/tests/corpus/3.12.3/report.json new file mode 100644 index 00000000..b467b122 --- /dev/null +++ b/tests/corpus/3.12.3/report.json @@ -0,0 +1,17 @@ +{ + "version": "3.12.3", + "interp": "~/.local/share/uv/python/cpython-3.12-macos-aarch64-none/bin/python3.12", + "counts": { + "PASS": 2 + }, + "results": [ + { + "module": "handler_return_padding_3", + "verdict": "PASS" + }, + { + "module": "padded_value_returns_3", + "verdict": "PASS" + } + ] +} \ No newline at end of file diff --git a/tests/corpus/3.13.15/out/code.py b/tests/corpus/3.13.15/out/code.py index 7fba431c..62d133de 100644 --- a/tests/corpus/3.13.15/out/code.py +++ b/tests/corpus/3.13.15/out/code.py @@ -108,7 +108,7 @@ def showsyntaxerror(self, filename=None, **kwargs): source = kwargs.pop('source', '') self._showtraceback(typ, value, None, source) finally: - value = tb = (typ := None) + typ = value = tb = None def showtraceback(self): '''Display the exception that just occurred. @@ -123,7 +123,7 @@ def showtraceback(self): typ, value, tb = sys.exc_info() self._showtraceback(typ, value, tb.tb_next, '') finally: - value = tb = (typ := None) + typ = value = tb = None def _showtraceback(self, typ, value, tb, source): sys.last_type = typ diff --git a/tests/corpus/3.13.7/handler_return_padding_3.py b/tests/corpus/3.13.7/handler_return_padding_3.py new file mode 100644 index 00000000..7e7278b7 --- /dev/null +++ b/tests/corpus/3.13.7/handler_return_padding_3.py @@ -0,0 +1,15 @@ +# The .pyc variants insert 1, 3 or 9 NOPs before each RETURN_VALUE. +# Regenerate with the matching CPython: python tools/compile_padded_returns.py +# Jump targets and exception-table boundaries are relocated with the padding. + +async def handled(read, log): + try: + await read() + log("saved") + except ValueError: + log("locked") + return False + except Exception as error: + log("failed", error) + return False + return True diff --git a/tests/corpus/3.13.7/handler_return_padding_3.pyc b/tests/corpus/3.13.7/handler_return_padding_3.pyc new file mode 100644 index 00000000..0f68db61 Binary files /dev/null and b/tests/corpus/3.13.7/handler_return_padding_3.pyc differ diff --git a/tests/corpus/3.13.7/out/handler_return_padding_3.py b/tests/corpus/3.13.7/out/handler_return_padding_3.py new file mode 100644 index 00000000..2704aa0c --- /dev/null +++ b/tests/corpus/3.13.7/out/handler_return_padding_3.py @@ -0,0 +1,12 @@ +async def handled(read, log): + try: + await read() + log('saved') + except ValueError: + log('locked') + return False + except Exception as error: + log('failed', error) + return False + return True + diff --git a/tests/corpus/3.13.7/out/padded_value_returns_3.py b/tests/corpus/3.13.7/out/padded_value_returns_3.py new file mode 100644 index 00000000..f79afb3c --- /dev/null +++ b/tests/corpus/3.13.7/out/padded_value_returns_3.py @@ -0,0 +1,14 @@ +def synchronous(read, build): + try: + item = read() + except ValueError: + return build('missing') + return build(item) + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build('missing') + return build(item) + diff --git a/tests/corpus/3.13.7/padded_value_returns_3.py b/tests/corpus/3.13.7/padded_value_returns_3.py new file mode 100644 index 00000000..fb1d66fb --- /dev/null +++ b/tests/corpus/3.13.7/padded_value_returns_3.py @@ -0,0 +1,19 @@ +# Generate .pyc files with the matching CPython 3.11–3.14 interpreter: +# python tools/compile_padded_value_returns.py +# Each RETURN_VALUE receives 1, 3 or 9 NOPs, with jumps and exception +# boundaries relocated. Padding never constructs a new return value. + +def synchronous(read, build): + try: + item = read() + except ValueError: + return build("missing") + return build(item) + + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build("missing") + return build(item) diff --git a/tests/corpus/3.13.7/padded_value_returns_3.pyc b/tests/corpus/3.13.7/padded_value_returns_3.pyc new file mode 100644 index 00000000..66632e32 Binary files /dev/null and b/tests/corpus/3.13.7/padded_value_returns_3.pyc differ diff --git a/tests/corpus/3.13.7/report.json b/tests/corpus/3.13.7/report.json new file mode 100644 index 00000000..d0346da5 --- /dev/null +++ b/tests/corpus/3.13.7/report.json @@ -0,0 +1,17 @@ +{ + "version": "3.13.7", + "interp": "/opt/homebrew/bin/python3.13", + "counts": { + "PASS": 2 + }, + "results": [ + { + "module": "handler_return_padding_3", + "verdict": "PASS" + }, + { + "module": "padded_value_returns_3", + "verdict": "PASS" + } + ] +} \ No newline at end of file diff --git a/tests/corpus/3.14.2/handler_return_padding_3.py b/tests/corpus/3.14.2/handler_return_padding_3.py new file mode 100644 index 00000000..7e7278b7 --- /dev/null +++ b/tests/corpus/3.14.2/handler_return_padding_3.py @@ -0,0 +1,15 @@ +# The .pyc variants insert 1, 3 or 9 NOPs before each RETURN_VALUE. +# Regenerate with the matching CPython: python tools/compile_padded_returns.py +# Jump targets and exception-table boundaries are relocated with the padding. + +async def handled(read, log): + try: + await read() + log("saved") + except ValueError: + log("locked") + return False + except Exception as error: + log("failed", error) + return False + return True diff --git a/tests/corpus/3.14.2/handler_return_padding_3.pyc b/tests/corpus/3.14.2/handler_return_padding_3.pyc new file mode 100644 index 00000000..0f1c0628 Binary files /dev/null and b/tests/corpus/3.14.2/handler_return_padding_3.pyc differ diff --git a/tests/corpus/3.14.2/out/handler_return_padding_3.py b/tests/corpus/3.14.2/out/handler_return_padding_3.py new file mode 100644 index 00000000..2704aa0c --- /dev/null +++ b/tests/corpus/3.14.2/out/handler_return_padding_3.py @@ -0,0 +1,12 @@ +async def handled(read, log): + try: + await read() + log('saved') + except ValueError: + log('locked') + return False + except Exception as error: + log('failed', error) + return False + return True + diff --git a/tests/corpus/3.14.2/out/padded_value_returns_3.py b/tests/corpus/3.14.2/out/padded_value_returns_3.py new file mode 100644 index 00000000..f79afb3c --- /dev/null +++ b/tests/corpus/3.14.2/out/padded_value_returns_3.py @@ -0,0 +1,14 @@ +def synchronous(read, build): + try: + item = read() + except ValueError: + return build('missing') + return build(item) + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build('missing') + return build(item) + diff --git a/tests/corpus/3.14.2/padded_value_returns_3.py b/tests/corpus/3.14.2/padded_value_returns_3.py new file mode 100644 index 00000000..fb1d66fb --- /dev/null +++ b/tests/corpus/3.14.2/padded_value_returns_3.py @@ -0,0 +1,19 @@ +# Generate .pyc files with the matching CPython 3.11–3.14 interpreter: +# python tools/compile_padded_value_returns.py +# Each RETURN_VALUE receives 1, 3 or 9 NOPs, with jumps and exception +# boundaries relocated. Padding never constructs a new return value. + +def synchronous(read, build): + try: + item = read() + except ValueError: + return build("missing") + return build(item) + + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build("missing") + return build(item) diff --git a/tests/corpus/3.14.2/padded_value_returns_3.pyc b/tests/corpus/3.14.2/padded_value_returns_3.pyc new file mode 100644 index 00000000..4d984de4 Binary files /dev/null and b/tests/corpus/3.14.2/padded_value_returns_3.pyc differ diff --git a/tests/corpus/3.14.2/report.json b/tests/corpus/3.14.2/report.json new file mode 100644 index 00000000..7dad997e --- /dev/null +++ b/tests/corpus/3.14.2/report.json @@ -0,0 +1,17 @@ +{ + "version": "3.14.2", + "interp": "/opt/homebrew/bin/python3.14", + "counts": { + "PASS": 2 + }, + "results": [ + { + "module": "handler_return_padding_3", + "verdict": "PASS" + }, + { + "module": "padded_value_returns_3", + "verdict": "PASS" + } + ] +} \ No newline at end of file diff --git a/tests/corpus/3.14.7/out/annotationlib.py b/tests/corpus/3.14.7/out/annotationlib.py index e272c210..4cebcc1b 100644 --- a/tests/corpus/3.14.7/out/annotationlib.py +++ b/tests/corpus/3.14.7/out/annotationlib.py @@ -548,7 +548,8 @@ def call_annotate_function(annotate, format, *, owner=None, _is_evaluate=False): if isinstance(result, ForwardRef): return result.evaluate(format=Format.FORWARDREF) return result - return {val.evaluate(format=Format.FORWARDREF): val for key, val in result.items() if not isinstance(val, ForwardRef)} + return result.items() + val if format == Format.VALUE: raise RuntimeError('annotate function does not support VALUE format') raise ValueError(f'Invalid format: {format!r}') @@ -704,7 +705,7 @@ def get_annotations(obj, *, globals=None, locals=None, eval_str=False, format=Fo obj_locals = None unwrap = obj else: - obj_locals = unwrap = (obj_globals := None) + obj_globals = obj_locals = unwrap = None if unwrap is not None: _seen_ids = {id(unwrap)} while True: diff --git a/tests/corpus/3.14.7/out/code.py b/tests/corpus/3.14.7/out/code.py index 26dba2f7..25ee4e64 100644 --- a/tests/corpus/3.14.7/out/code.py +++ b/tests/corpus/3.14.7/out/code.py @@ -107,7 +107,7 @@ def showsyntaxerror(self, filename=None, **kwargs): source = kwargs.pop('source', '') self._showtraceback(typ, value, None, source) finally: - value = tb = (typ := None) + typ = value = tb = None def showtraceback(self): '''Display the exception that just occurred. @@ -122,7 +122,7 @@ def showtraceback(self): typ, value, tb = sys.exc_info() self._showtraceback(typ, value, tb.tb_next, '') finally: - value = tb = (typ := None) + typ = value = tb = None def _showtraceback(self, typ, value, tb, source): sys.last_type = typ diff --git a/tests/corpus/summary.json b/tests/corpus/summary.json index 43f97016..70dbb5df 100644 --- a/tests/corpus/summary.json +++ b/tests/corpus/summary.json @@ -20,6 +20,12 @@ "AST-PASS": 8 } }, + "3.11.14": { + "interp": "~/.local/share/uv/python/cpython-3.11-macos-aarch64-none/bin/python3.11", + "counts": { + "PASS": 2 + } + }, "3.11.16": { "interp": "~/.local/share/uv/python/cpython-3.11-macos-aarch64-none/bin/python3.11", "counts": { @@ -34,6 +40,12 @@ "AST-PASS": 10 } }, + "3.12.3": { + "interp": "~/.local/share/uv/python/cpython-3.12-macos-aarch64-none/bin/python3.12", + "counts": { + "PASS": 2 + } + }, "3.13.15": { "interp": "/opt/homebrew/bin/python3.13", "counts": { @@ -41,6 +53,18 @@ "AST-PASS": 7 } }, + "3.13.7": { + "interp": "/opt/homebrew/bin/python3.13", + "counts": { + "PASS": 2 + } + }, + "3.14.2": { + "interp": "/opt/homebrew/bin/python3.14", + "counts": { + "PASS": 2 + } + }, "3.14.7": { "interp": "/opt/homebrew/bin/python3.14", "counts": { diff --git a/tests/fixtures/assignment_chains.py b/tests/fixtures/assignment_chains.py new file mode 100644 index 00000000..c2a067e0 --- /dev/null +++ b/tests/fixtures/assignment_chains.py @@ -0,0 +1,40 @@ +# COPY-backed chains; CPython 3.13+ can fuse the final two local stores. + +def three(read): + a = b = c = read() + return a, b, c + + +def four(read): + a = b = c = d = read() + return a, b, c, d + + +def five(read): + a = b = c = d = e = read() + return a, b, c, d, e + + +def six(read): + a = b = c = d = e = f = read() + return a, b, c, d, e, f + + +def three_none(): + a = b = c = None + return a, b, c + + +def four_none(): + a = b = c = d = None + return a, b, c, d + + +def five_none(): + a = b = c = d = e = None + return a, b, c, d, e + + +def six_none(): + a = b = c = d = e = f = None + return a, b, c, d, e, f diff --git a/tests/fixtures/assignment_identity.py b/tests/fixtures/assignment_identity.py new file mode 100644 index 00000000..d121ab02 --- /dev/null +++ b/tests/fixtures/assignment_identity.py @@ -0,0 +1,20 @@ +# Separate evaluations must not become one COPY-backed chained assignment. + +def separate(read): + left, right = read(), read() + return left, right + + +def shared(read): + left = right = read() + return left, right + + +def separate_none(): + left, right = None, None + return left, right + + +def shared_none(): + right = left = None + return left, right diff --git a/tests/fixtures/awaited_synchronous_iteration.py b/tests/fixtures/awaited_synchronous_iteration.py new file mode 100644 index 00000000..6d5438be --- /dev/null +++ b/tests/fixtures/awaited_synchronous_iteration.py @@ -0,0 +1,23 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# async/await syntax starts in Python 3.5. + +async def awaited(read): + result = [] + for item in await read(): + result.append(item) + return result + + +async def asynchronous(read): + result = [] + async for item in read(): + result.append(item) + return result + + +async def sync(read): + result = [] + for item in read(): + result.append(item) + return result diff --git a/tests/fixtures/boolean_call_arguments.py b/tests/fixtures/boolean_call_arguments.py new file mode 100644 index 00000000..de66ce33 --- /dev/null +++ b/tests/fixtures/boolean_call_arguments.py @@ -0,0 +1,35 @@ +def select(call, before, first, second, last, after): + return call(before(), (first() and second()) or last(), after()) + + +def repeated_prefix(call, value, second, last, after): + return call(value, (value and second()) or last(), after()) + + +def alternatives(a, b, c, d, e): + return bool((a and b) or c or d or e) + + +def or_prefix(a, b, c, d, e): + return bool(c or d or (a and b) or e) + + +def method(a, b, c, d, e): + return bool((a and a.get("available")) or b or c or d or e) + + +def assignment(a, b, c, d, e): + value = (a and a.get("available")) or b or c or d or e + return bool(value) + + +def pure_or(a, b, c, d, e): + return bool(a or b or c or d or e) + + +def pure_and(a, b, c, d, e): + return bool(a and b and c and d and e) + + +def following_branch(call, a, b, c, d, e): + return call((a and b) or c, d if e else 0) diff --git a/tests/fixtures/chained_comprehension_filters.py b/tests/fixtures/chained_comprehension_filters.py new file mode 100644 index 00000000..e3d73b5e --- /dev/null +++ b/tests/fixtures/chained_comprehension_filters.py @@ -0,0 +1,61 @@ +def list_positive_separate(values): + return [value for value in values if isinstance(value, str) if value] + + +def list_positive_joined(values): + return [value for value in values if isinstance(value, str) and value] + + +def list_negative_separate(values): + return [value for value in values if not isinstance(value, str) if value] + + +def list_negative_joined(values): + return [value for value in values if not isinstance(value, str) and value] + + +def set_positive_separate(values): + return {value for value in values if isinstance(value, str) if value} + + +def set_positive_joined(values): + return {value for value in values if isinstance(value, str) and value} + + +def set_negative_separate(values): + return {value for value in values if not isinstance(value, str) if value} + + +def set_negative_joined(values): + return {value for value in values if not isinstance(value, str) and value} + + +def dict_positive_separate(values): + return {value: value for value in values if isinstance(value, str) if value} + + +def dict_positive_joined(values): + return {value: value for value in values if isinstance(value, str) and value} + + +def dict_negative_separate(values): + return {value: value for value in values if not isinstance(value, str) if value} + + +def dict_negative_joined(values): + return {value: value for value in values if not isinstance(value, str) and value} + + +def methods_equal(rows, wanted): + return [row.tag for row in rows if row.value == wanted if row.ready() if row.final()] + + +def methods_unequal(rows, wanted): + return [row.tag for row in rows if row.value != wanted if not row.ready() if row.final()] + + +def broken_lines(values): + return [value + for value in values + if isinstance(value, str) + if value] diff --git a/tests/fixtures/class_annotation_registration.py b/tests/fixtures/class_annotation_registration.py new file mode 100644 index 00000000..60ae6d60 --- /dev/null +++ b/tests/fixtures/class_annotation_registration.py @@ -0,0 +1,30 @@ +if True: + module_count: int = 5 + + +class Always: + "Documented class." + if True: + amount: int = 3 + label: str = "ready" + + def read(self) -> int: + return self.amount + + +def select(enabled, values, default): + class Result: + if True: + amount: int = 3 + if enabled: + label: str = "ready" + for value in values: + limit: int = default(value) + return Result + + +def loop_only(values, default): + class Result: + for value in values: + amount: int = default(value) + return Result diff --git a/tests/fixtures/comprehension_boolean_values.py b/tests/fixtures/comprehension_boolean_values.py new file mode 100644 index 00000000..5ed4cdf3 --- /dev/null +++ b/tests/fixtures/comprehension_boolean_values.py @@ -0,0 +1,38 @@ +def select(rows): + return [max(0.0, float(row.get("x") or 0.0)) for row in rows] + + +def element_or(rows): + return [row[1] or 0 for row in rows] + + +def element_and(rows): + return [row[1] and 7 for row in rows] + + +def set_values(rows): + return {row[1] or 0 for row in rows} + + +def dict_values(rows): + return {row[0]: row[1] and 7 for row in rows} + + +def calls(rows): + return [row.first() or row.second() for row in rows] + + +def and_calls(rows): + return [row.first() and row.second() for row in rows] + + +def single_filter(rows): + return [row[1] for row in rows if row[0]] + + +def filtered_value(rows): + return [row[1] or 0 for row in rows if row[0]] + + +def ternary_control(rows): + return [row[1] if row[0] else 0 for row in rows] diff --git a/tests/fixtures/comprehension_default_arguments.py b/tests/fixtures/comprehension_default_arguments.py new file mode 100644 index 00000000..254bc415 --- /dev/null +++ b/tests/fixtures/comprehension_default_arguments.py @@ -0,0 +1,66 @@ +# Compile with each named CPython version; the committed .pyc files are +# exercised by the existing Rust integration suite. + +# Python 3.12+ inlines these comprehensions; 3.11 is the legacy control. + +def list_default(rows): + row = "saved-row" + field = "saved-field" + result = [field for row in rows for field in row.get("values", [])] + return result, row, field + + +def set_default(rows): + row = "saved-row" + field = "saved-field" + result = {field for row in rows for field in row.get("values", {})} + return result, row, field + + +def dict_default(rows): + row = "saved-row" + field = "saved-field" + result = {field: len(field) for row in rows for field in row.get("values", {})} + return result, row, field + + +def nested_list(rows): + row = "saved-row" + field = "saved-field" + result = [[field for field in row.get("values", [])] for row in rows] + return result, row, field + + +def nested_set(rows): + row = "saved-row" + field = "saved-field" + result = [{field for field in row.get("values", {})} for row in rows] + return result, row, field + + +def nested_dict(rows): + row = "saved-row" + field = "saved-field" + result = [{field: len(field) for field in row.get("values", {})} for row in rows] + return result, row, field + + +def subscript(rows): + row = "saved-row" + field = "saved-field" + result = [field for row in rows for field in row["values"]] + return result, row, field + + +def no_default(rows): + row = "saved-row" + field = "saved-field" + result = [field for row in rows for field in row.get("values")] + return result, row, field + + +def tuple_default(rows): + row = "saved-row" + field = "saved-field" + result = [field for row in rows for field in row.get("values", ())] + return result, row, field diff --git a/tests/fixtures/comprehension_dictionary_displays.py b/tests/fixtures/comprehension_dictionary_displays.py new file mode 100644 index 00000000..edf421b6 --- /dev/null +++ b/tests/fixtures/comprehension_dictionary_displays.py @@ -0,0 +1,51 @@ +# Compile with each named CPython version; the committed .pyc files are +# exercised by the existing Rust integration suite. + +# Python 3.10+ uses MAP_ADD 1 for large dictionary displays. + +def list17(rows): + return [{ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, + } for row in rows] + + +def generator17(rows): + return ({ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, + } for row in rows) + + +def list20(rows): + return [{ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, "k17": row + 17, "k18": row + 18, "k19": row + 19, + } for row in rows] + + +def generator20(rows): + return ({ + "k0": row + 0, "k1": row + 1, "k2": row + 2, "k3": row + 3, + "k4": row + 4, "k5": row + 5, "k6": row + 6, "k7": row + 7, + "k8": row + 8, "k9": row + 9, "k10": row + 10, "k11": row + 11, + "k12": row + 12, "k13": row + 13, "k14": row + 14, "k15": row + 15, + "k16": row + 16, "k17": row + 17, "k18": row + 18, "k19": row + 19, + } for row in rows) + + +def dictionary_comprehension(rows): + return {row: row + 1 for row in rows} + + +def small_display(rows): + return [{"value": row} for row in rows] diff --git a/tests/fixtures/conditional_loop_prefix.py b/tests/fixtures/conditional_loop_prefix.py new file mode 100644 index 00000000..1a209679 --- /dev/null +++ b/tests/fixtures/conditional_loop_prefix.py @@ -0,0 +1,20 @@ +# Attribute loads make FOR_ITER require EXTENDED_ARG on Python 3.12+. +# Conditional branches lead to adjacent back edges targeting that prefix. +def process(rows, work): + success = True + for row in rows: + if row.is_file: + if not work( + "file", row.ok, + row.a, row.b, row.c, row.d, row.e, row.f, + row.g, row.h, row.i, row.j, row.k, row.l, + ): + success = False + else: + if not work( + "directory", row.ok, + row.a, row.b, row.c, row.d, row.e, row.f, + row.g, row.h, row.i, row.j, row.k, row.l, + ): + success = False + return success diff --git a/tests/fixtures/extended_loop_prefix.py b/tests/fixtures/extended_loop_prefix.py new file mode 100644 index 00000000..706a8340 --- /dev/null +++ b/tests/fixtures/extended_loop_prefix.py @@ -0,0 +1,123 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# The long functions require EXTENDED_ARG before FOR_ITER. + +def size_1(rows, work): + result = [] + for row in rows: + try: + work(row) + except ValueError: + continue + result.append(row) + return result + + +def size_30(rows, work): + result = [] + for row in rows: + try: + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + except ValueError: + continue + result.append(row) + return result + + +def size_60(rows, work): + result = [] + for row in rows: + try: + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + work(row) + except ValueError: + continue + result.append(row) + return result diff --git a/tests/fixtures/generator_expanded_calls.py b/tests/fixtures/generator_expanded_calls.py new file mode 100644 index 00000000..7f20ebc1 --- /dev/null +++ b/tests/fixtures/generator_expanded_calls.py @@ -0,0 +1,12 @@ +# CALL_FUNCTION_EX changes stack layout in CPython 3.13 and 3.14. + +def positional(rows, factory): + return tuple(factory(*row) for row in rows) + + +def keywords(rows, factory): + return tuple(factory(**dict(row)) for row in rows) + + +def both(rows, factory): + return tuple(factory(*row[0], **row[1]) for row in rows) diff --git a/tests/fixtures/generator_positional_call.py b/tests/fixtures/generator_positional_call.py new file mode 100644 index 00000000..e8f727a6 --- /dev/null +++ b/tests/fixtures/generator_positional_call.py @@ -0,0 +1,2 @@ +def select(rows, factory): + return tuple(factory(*row) for row in rows) diff --git a/tests/fixtures/guarded_else_merge.py b/tests/fixtures/guarded_else_merge.py new file mode 100644 index 00000000..96521e49 --- /dev/null +++ b/tests/fixtures/guarded_else_merge.py @@ -0,0 +1,67 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# The finite else-span guard applies to Python 3.11+. + +async def awaited(owner): + value = None + active = owner.active + if active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = float(row[0]) + owner.record(value) + return value + + +async def synchronous(owner): + value = None + active = owner.active + if active: + value = float(1) + else: + row = owner.fetch() + if row and row[0] is not None: + value = 2 + owner.record(value) + return value + + +async def attribute(owner): + value = None + active = owner.active + if active: + value = float(active['total']) + else: + row = owner.row + if row and row[0] is not None: + value = float(row[0]) + owner.record(value) + return value + + +async def following_else_work(owner): + value = None + active = owner.active + if active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = 2 + owner.record('queried') + owner.record(value) + return value + + +async def independent(owner): + value = None + row = None + if owner.active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = 2 + return value diff --git a/tests/fixtures/handler_return_padding.py b/tests/fixtures/handler_return_padding.py new file mode 100644 index 00000000..7e7278b7 --- /dev/null +++ b/tests/fixtures/handler_return_padding.py @@ -0,0 +1,15 @@ +# The .pyc variants insert 1, 3 or 9 NOPs before each RETURN_VALUE. +# Regenerate with the matching CPython: python tools/compile_padded_returns.py +# Jump targets and exception-table boundaries are relocated with the padding. + +async def handled(read, log): + try: + await read() + log("saved") + except ValueError: + log("locked") + return False + except Exception as error: + log("failed", error) + return False + return True diff --git a/tests/fixtures/keyword_call_arm_boundary.py b/tests/fixtures/keyword_call_arm_boundary.py new file mode 100644 index 00000000..1dab94ec --- /dev/null +++ b/tests/fixtures/keyword_call_arm_boundary.py @@ -0,0 +1,5 @@ +# Resolving owner.build belongs after the condition. Reconstructing this as +# outer(owner.build(amount=1 if value else 2)) changes descriptor evaluation +# order. The bounded merger must decline rather than report a clean result. +def select(value, owner, outer): + return outer(owner.build(amount=1) if value else owner.build(amount=2)) diff --git a/tests/fixtures/keyword_conditional_calls.py b/tests/fixtures/keyword_conditional_calls.py new file mode 100644 index 00000000..044becfc --- /dev/null +++ b/tests/fixtures/keyword_conditional_calls.py @@ -0,0 +1,22 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# CALL_KW starts in Python 3.13. + +def case_0(value, build, other): + return build("reject" if value else "accept", amount=0) + + +def case_1(value, build, other): + return build("prefix", amount=0 if value else 1) + + +def case_2(value, build, other): + return build(0 if value else 1, amount=value) + + +def case_9(value, build, other): + return build("prefix", amount=1 if value else 2, last=other(value)) + + +def case_11(value, build, other): + return other(build(amount=1) if value else build(amount=2, other=3)) diff --git a/tests/fixtures/loop_lookup_guards.py b/tests/fixtures/loop_lookup_guards.py new file mode 100644 index 00000000..ed3f08d6 --- /dev/null +++ b/tests/fixtures/loop_lookup_guards.py @@ -0,0 +1,33 @@ +def rating_lookup(values): + for value in values: + if not value.rating < 10: + continue + coordinator = value.coordinator + if coordinator is None: + continue + return coordinator + + +def dictionary_lookup(rows): + for row in rows: + if not isinstance(row, dict): + continue + coordinator = row.get("coordinator") + if coordinator is None: + continue + return coordinator + + +def none_lookup(rows): + for row in rows: + if row is None: + continue + coordinator = row.coordinator + if coordinator is None: + continue + return coordinator + + +def comprehension(rows): + return [row.get("coordinator") for row in rows if isinstance(row, dict)] + diff --git a/tests/fixtures/nested_conditional_elements.py b/tests/fixtures/nested_conditional_elements.py new file mode 100644 index 00000000..001331f8 --- /dev/null +++ b/tests/fixtures/nested_conditional_elements.py @@ -0,0 +1,20 @@ +def select(rows): + return [{ + "excluded": ({"count": row["count"], + "reasons": loads(row["reasons"]) if row["reasons"] else {}} + if row["count"] is not None else {}), + "archived": str(row["archived"]) if row["archived"] else None, + } for row in rows] + + +def then_arm(rows, build): + return [build(1 if row.inner else 2) if row.outer else 3 for row in rows] + + +def else_arm(rows, build): + return [1 if row.outer else build(2 if row.inner else 3) for row in rows] + + +def both_arms(rows, build): + return [build(1 if row.inner is None else 2) if row.outer + else build(3 if row.inner is not None else 4) for row in rows] diff --git a/tests/fixtures/nested_generator_comprehensions.py b/tests/fixtures/nested_generator_comprehensions.py new file mode 100644 index 00000000..77aefa9a --- /dev/null +++ b/tests/fixtures/nested_generator_comprehensions.py @@ -0,0 +1,22 @@ +def dictionary(rows): + return tuple({key: row[key] for key in row} for row in rows) + + +def list_elements(rows): + return tuple([value * 2 for value in row] for row in rows) + + +def set_elements(rows): + return tuple({value * 2 for value in row} for row in rows) + + +def call_arguments(rows, collect): + return tuple(collect([value * 2 for value in row]) for row in rows) + + +def filtered(rows): + return tuple({key: row[key] for key in row if key} for row in rows if row) + + +def unpacked(rows, columns): + return tuple({key: value for key, value in zip(columns, row) if key} for row in rows) diff --git a/tests/fixtures/ordered_keyword_expansions.py b/tests/fixtures/ordered_keyword_expansions.py new file mode 100644 index 00000000..306e8715 --- /dev/null +++ b/tests/fixtures/ordered_keyword_expansions.py @@ -0,0 +1,30 @@ +def direct(fn, first, second): + return fn(1, fixed=2, **first, **second) + + +def interleaved(fn, first, second): + return fn(1, **first, fixed=2, **second) + + +def mappings(fn, first, second): + return fn(1, **first, **second) + + +def dictionary(fn, first, second): + return fn(1, **{"fixed": 2, **first, **second}) + + +def non_identifier(fn, first, second): + return fn(1, **{"my col": 2, **first, **second}) + + +def reserved_word(fn, first, second): + return fn(1, **{"class": 2, **first, **second}) + + +def repeated_literal(fn, first, second): + return fn(1, **{"fixed": 2, "fixed": 3, **first, **second}) + + +def effectful(fn, get_first, get_second): + return fn(1, fixed=2, **get_first(), **get_second()) diff --git a/tests/fixtures/padded_value_returns.py b/tests/fixtures/padded_value_returns.py new file mode 100644 index 00000000..fb1d66fb --- /dev/null +++ b/tests/fixtures/padded_value_returns.py @@ -0,0 +1,19 @@ +# Generate .pyc files with the matching CPython 3.11–3.14 interpreter: +# python tools/compile_padded_value_returns.py +# Each RETURN_VALUE receives 1, 3 or 9 NOPs, with jumps and exception +# boundaries relocated. Padding never constructs a new return value. + +def synchronous(read, build): + try: + item = read() + except ValueError: + return build("missing") + return build(item) + + +async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build("missing") + return build(item) diff --git a/tests/fixtures/repeated_unpack_bindings.py b/tests/fixtures/repeated_unpack_bindings.py new file mode 100644 index 00000000..2b97c72e --- /dev/null +++ b/tests/fixtures/repeated_unpack_bindings.py @@ -0,0 +1,35 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# CPython 3.13 starts replacing overwritten stores with POP_TOP. + +def repeated_loop(values): + result = [] + for value, discarded, discarded, discarded in values: + result.append(value is not None) + return result + + +def repeated_list(values): + return [value is not None for value, discarded, discarded, discarded in values] + + +def underscore_loop(values): + result = [] + for value, _, discarded, discarded in values: + result.append((value, discarded, _)) + return result + + +def underscore_list(values): + return [(value, discarded, _) for value, _, discarded, discarded in values] + + +def nested_loop(values): + result = [] + for value, (discarded, discarded) in values: + result.append((value, discarded)) + return result + + +def nested_list(values): + return [(value, discarded) for value, (discarded, discarded) in values] diff --git a/tests/fixtures/sequential_generator_handlers.py b/tests/fixtures/sequential_generator_handlers.py new file mode 100644 index 00000000..ffc228ba --- /dev/null +++ b/tests/fixtures/sequential_generator_handlers.py @@ -0,0 +1,29 @@ +# Compile with each named CPython version; the committed .pyc files are +# exercised by the existing Rust integration suite. + +# The frame-wide StopIteration intrinsic is emitted starting in Python 3.12. + +def synchronous(work): + try: + yield work("first") + except ValueError: + yield 2 + yield 3 + try: + yield work("last") + except KeyError: + yield 5 + yield 6 + + +async def asynchronous(work): + try: + yield work("first") + except ValueError: + yield 2 + yield 3 + try: + yield work("last") + except KeyError: + yield 5 + yield 6 diff --git a/tests/fixtures/slice_constants_and_calls.py b/tests/fixtures/slice_constants_and_calls.py new file mode 100644 index 00000000..1ed1a6e9 --- /dev/null +++ b/tests/fixtures/slice_constants_and_calls.py @@ -0,0 +1,23 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +# Marshal format 5 adds slice constants in 3.14; runtime calls must +# retain their global lookup on every supported Python version. + +def dynamic_axes(data, stop): + return data[:stop, :] + + +def constant_axes(data, stop): + return data[:, :] + + +def mixed_axes(data, stop): + return data[1:stop:2, :, 7] + + +def call(data, stop): + return data[slice(None, stop, None)] + + +def call_axes(data, stop): + return data[slice(None, stop, None), :] diff --git a/tests/fixtures/sunk_value_return_control.py b/tests/fixtures/sunk_value_return_control.py new file mode 100644 index 00000000..3b28c204 --- /dev/null +++ b/tests/fixtures/sunk_value_return_control.py @@ -0,0 +1,10 @@ +# Versioned .pyc files are consumed by tests/integration.rs. + +def select(value): + try: + entry = value.entry + except AttributeError: + kind = "missing" + else: + kind = "present" + return kind, value diff --git a/tests/fixtures/unpacked_comprehension_cleanup.py b/tests/fixtures/unpacked_comprehension_cleanup.py new file mode 100644 index 00000000..948c5ae3 --- /dev/null +++ b/tests/fixtures/unpacked_comprehension_cleanup.py @@ -0,0 +1,75 @@ +# Compile with each named CPython version; the committed .pyc files are +# exercised by the existing Rust integration suite. + +# Python 3.12+ inlines these comprehensions; 3.11 is the legacy control. + +def unpack4(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [a + b + c + d for rows in groups for a, b, c, d in rows] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack_list(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c) for a, b in groups for c, d in b] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack_set(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = {(a, c) for a, b in groups for c, d in b} + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack_dict(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = {a: c for a, b in groups for c, d in b} + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def unpack6(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c, e) for a, b in groups for c, d in b for e, f in d] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def simple(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c) for a in groups for c in a] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def mixed(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c) for key, a in groups for c in a] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after + + +def mixed_inner(groups): + a, b, c, d, e, f = 10, 20, 30, 40, 50, 60 + result = [(a, c) for a in groups for key, c in a] + after = 0 + for number in (1, 2, 3): + after += number + return result, (a, b, c, d, e, f), after diff --git a/tests/integration.rs b/tests/integration.rs index f4af5082..9733ee15 100644 --- a/tests/integration.rs +++ b/tests/integration.rs @@ -106,6 +106,26 @@ fn py310_basics() { assert_contains(&src, &["while i < 10:", "v = 1 < x < 100"], "py3.10 basics"); } +#[test] +fn assignment_identity() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("assignment_identity.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "def separate(read):\n left, right = read(), read()\n return left, right", + "def shared(read):\n left = right = read()\n return left, right", + "def separate_none():\n left, right = None, None\n return left, right", + "def shared_none():\n right = left = None\n return left, right", + ], + &fixture, + ); + assert_eq!(src.matches("read()").count(), 3, "{fixture}:\n{src}"); + } +} + #[test] fn py311_comprehensions() { let (src, clean) = decompile_fixture("comprehensions.3.11.pyc"); @@ -151,6 +171,35 @@ fn py312_hello() { ); } +#[test] +fn assignment_chains() { + // 3.13 introduces fused stores; older fixtures check COPY-only chains. + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("assignment_chains.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + for (name, names) in [ + ("three", &['a', 'b', 'c'][..]), + ("four", &['a', 'b', 'c', 'd'][..]), + ("five", &['a', 'b', 'c', 'd', 'e'][..]), + ("six", &['a', 'b', 'c', 'd', 'e', 'f'][..]), + ] { + let names: Vec = names.iter().map(char::to_string).collect(); + let targets = names.join(" = "); + let values = names.join(", "); + assert_contains( + &src, + &[ + &format!("def {name}(read):\n {targets} = read()\n return {values}"), + &format!("def {name}_none():\n {targets} = None\n return {values}"), + ], + &fixture, + ); + } + assert_eq!(src.matches("read()").count(), 4, "{fixture}:\n{src}"); + } +} + #[test] fn py313_hello() { let (src, clean) = decompile_fixture("hello.3.13.pyc"); @@ -158,6 +207,25 @@ fn py313_hello() { assert_contains(&src, &["class Greeter:", "while a < n:", "a, b = b, a + b"], "py3.13 hello"); } +#[test] +fn generator_expanded_calls() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("generator_expanded_calls.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "def positional(rows, factory):\n return tuple((factory(*row) for row in rows))", + "def keywords(rows, factory):\n return tuple((factory(*(), **{**dict(row)}) for row in rows))", + "def both(rows, factory):\n return tuple((factory(*row[0], **{**row[1]}) for row in rows))", + ], + &fixture, + ); + assert_eq!(src.matches("factory(").count(), 3, "{fixture}:\n{src}"); + } +} + #[test] fn py314_hello() { // 3.14 is brand new; accept incomplete-marker output as long as the @@ -169,3 +237,591 @@ fn py314_hello() { "py3.14 hello", ); } + +#[test] +fn sequential_generator_handlers_keep_the_shared_mainline() { + // CALL_INTRINSIC_1 3 starts in 3.12; 3.11 has no frame-wide handler. + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture( + &format!("sequential_generator_handlers.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + let body = " try:\n yield work('first')\n except ValueError:\n yield 2\n yield 3\n try:\n yield work('last')\n except KeyError:\n yield 5\n yield 6"; + for definition in ["def synchronous(work):", "async def asynchronous(work):"] { + let expected = format!("{definition}\n{body}"); + assert_contains(&src, &[&expected], version); + } + assert!(!src.contains("else:"), "Python {version}: spurious handler else:\n{src}"); + } +} + +#[test] +fn empty_defaults_keep_the_comprehension_level() { + // PEP 709 inlining starts in 3.12; retain the 3.11 compatibility case. + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("comprehension_default_arguments.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + for (name, expression) in [ + ("list_default", "[field for row in rows for field in row.get('values', [])]"), + ("set_default", "{field for row in rows for field in row.get('values', {})}"), + ("dict_default", "{field: len(field) for row in rows for field in row.get('values', {})}"), + ("nested_list", "[[field for field in row.get('values', [])] for row in rows]"), + ("nested_set", "[{field for field in row.get('values', {})} for row in rows]"), + ("nested_dict", "[{field: len(field) for field in row.get('values', {})} for row in rows]"), + ("subscript", "[field for row in rows for field in row['values']]"), + ("no_default", "[field for row in rows for field in row.get('values')]"), + ("tuple_default", "[field for row in rows for field in row.get('values', ())]"), + ] { + let expected = format!( + "def {name}(rows):\n row = 'saved-row'\n field = 'saved-field'\n result = {expression}\n return result, row, field"); + assert_contains(&src, &[&expected], version); + } + } +} + +#[test] +fn unpacked_targets_keep_iterator_cleanup_and_saved_locals() { + // PEP 709 inlining starts in 3.12; retain the 3.11 compatibility case. + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("unpacked_comprehension_cleanup.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + for (name, expression) in [ + ("unpack4", "[a + b + c + d for rows in groups for a, b, c, d in rows]"), + ("unpack_list", "[(a, c) for a, b in groups for c, d in b]"), + ("unpack_set", "{(a, c) for a, b in groups for c, d in b}"), + ("unpack_dict", "{a: c for a, b in groups for c, d in b}"), + ("unpack6", "[(a, c, e) for a, b in groups for c, d in b for e, f in d]"), + ("simple", "[(a, c) for a in groups for c in a]"), + ("mixed", "[(a, c) for key, a in groups for c in a]"), + ("mixed_inner", "[(a, c) for a in groups for key, c in a]"), + ] { + let expected = format!( + "def {name}(groups):\n a, b, c, d, e, f = 10, 20, 30, 40, 50, 60\n result = {expression}\n after = 0\n for number in (1, 2, 3):\n after += number\n return result, (a, b, c, d, e, f), after"); + assert_contains(&src, &[&expected], version); + } + } +} + +#[test] +fn padded_handler_returns_keep_false_before_the_success_tail() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + for padding in [1, 3, 9] { + let (src, clean) = decompile_fixture( + &format!("handler_return_padding_{padding}.{version}.pyc")); + assert!(clean, "Python {version}, {padding} NOPs: incomplete:\n{src}"); + assert_contains(&src, &[ + "async def handled(read, log):\n try:\n await read()\n log('saved')\n except ValueError:\n log('locked')\n return False\n except Exception as error:\n log('failed', error)\n return False\n return True", + ], version); + } + } +} + +#[test] +fn large_dictionary_displays_remain_comprehension_elements() { + // CPython 3.10 starts using MAP_ADD 1 for large displays. Before 3.12, + // list comprehensions also use the separate code-object walker. + for version in ["3.10", "3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture( + &format!("comprehension_dictionary_displays.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + for size in [17, 20] { + let fields = (0..size).map(|index| format!("'k{index}': row + {index}")) + .collect::>().join(", "); + for (name, open, close) in [("list", "[", "]"), ("generator", "(", ")")] { + let expected = format!( + "def {name}{size}(rows):\n return {open}{{{fields}}} for row in rows{close}"); + assert_contains(&src, &[&expected], version); + } + } + assert_contains(&src, &[ + "def dictionary_comprehension(rows):\n return {row: row + 1 for row in rows}", + "def small_display(rows):\n return [{'value': row} for row in rows]", + ], version); + } +} + +#[test] +fn extended_loop_prefix_preserves_handler_continue() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("extended_loop_prefix.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + for count in [1, 30, 60] { + let body = " work(row)\n".repeat(count); + let expected = format!( + "def size_{count}(rows, work):\n result = []\n for row in rows:\n try:\n{body} except ValueError:\n continue\n result.append(row)"); + assert_contains(&src, &[&expected], version); + } + } +} + +#[test] +fn conditional_loop_prefix_keeps_both_arms_inside_loop() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("conditional_loop_prefix.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + let args = "row.ok, row.a, row.b, row.c, row.d, row.e, row.f, row.g, row.h, row.i, row.j, row.k, row.l"; + let file_condition = if version == "3.12" { "" } else { "not " }; + let directory_condition = if version == "3.11" { "elif not" } else { "if" }; + assert_contains( + &src, + &[ + &format!(" for row in rows:\n if row.is_file:\n if {file_condition}work('file', {args}):"), + &format!(" {directory_condition} work('directory', {args}):"), + " return success", + ], + version, + ); + assert!(!src.contains("\n success = False"), "Python {version}: failure assignment escaped the loop:\n{src}"); + if version == "3.13" || version == "3.14" { + assert_contains( + &src, + &[" success = False\n continue\n continue"], + version, + ); + } + } +} + +#[test] +fn awaited_collections_use_synchronous_iteration() { + for version in ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("awaited_synchronous_iteration.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"async def awaited(read): + result = [] + for item in await read(): + result.append(item) + return result"#, + r#"async def asynchronous(read): + result = [] + async for item in read(): + result.append(item) + return result"#, + r#"async def sync(read): + result = [] + for item in read(): + result.append(item) + return result"#, + ], version); + } +} + +#[test] +fn real_value_build_still_matches_the_sunk_tail() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("sunk_value_return_control.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"def select(value): + try: + entry = value.entry + except AttributeError: + kind = 'missing' + else: + kind = 'present' + return kind, value"#, + ], version); + } +} + +#[test] +fn padding_does_not_match_a_mainline_value_build() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + for padding in [1, 3, 9] { + let (src, clean) = decompile_fixture( + &format!("padded_value_returns_{padding}.{version}.pyc")); + assert!(clean, "Python {version}, {padding} NOPs: incomplete:\n{src}"); + assert_contains(&src, &[ + r#"def synchronous(read, build): + try: + item = read() + except ValueError: + return build('missing') + return build(item)"#, + r#"async def asynchronous(read, build): + try: + item = await read() + except ValueError: + return build('missing') + return build(item)"#, + ], version); + } + } +} + +#[test] +fn finite_else_span_keeps_its_inner_guard() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("guarded_else_merge.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"async def awaited(owner): + value = None + active = owner.active + if active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = float(row[0]) + owner.record(value) + return value"#, + r#"async def synchronous(owner): + value = None + active = owner.active + if active: + value = float(1) + else: + row = owner.fetch() + if row and row[0] is not None: + value = 2 + owner.record(value) + return value"#, + r#"async def attribute(owner): + value = None + active = owner.active + if active: + value = float(active['total']) + else: + row = owner.row + if row and row[0] is not None: + value = float(row[0]) + owner.record(value) + return value"#, + r#"async def following_else_work(owner): + value = None + active = owner.active + if active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = 2 + owner.record('queried') + owner.record(value) + return value"#, + r#"async def independent(owner): + value = None + row = None + if owner.active: + value = 1 + else: + row = await owner.fetch() + if row and row[0] is not None: + value = 2 + return value"#, + ], version); + } +} + +#[test] +fn split_exception_suites_generate_one_complete_try_statement() { + use pycdc::ast::{ExceptHandler, Expr, Stmt}; + use pycdc::version::PythonVersion; + use std::rc::Rc; + let name = |value: &str| Rc::new(Expr::Name(value.into())); + let inner = Stmt::Try { + body: vec![Stmt::Expr(name("work")), Stmt::Expr(name("more_work"))], + handlers: vec![ExceptHandler { + type_: Some(name("ValueError")), name: Some(name("error")), + body: vec![Stmt::Expr(name("handled"))], is_star: false, + }], + orelse: vec![], finalbody: vec![], + }; + let outer = Stmt::Try { + body: vec![inner.clone()], handlers: vec![], + orelse: vec![Stmt::Expr(name("success"))], + finalbody: vec![Stmt::Expr(name("cleanup")), Stmt::Expr(name("more_cleanup"))], + }; + let module = PycModule::from_bytes(include_bytes!("pyc/hello.3.14.pyc")).unwrap(); + // Unified try/except/finally syntax starts in Python 2.5. The repair + // concerns the recovered AST shape, rather than one opcode version. + for (major, minor) in [(2, 5), (2, 7), (3, 8), (3, 11), (3, 12), (3, 13), (3, 14)] { + let version = PythonVersion { major, minor, ..module.version() }; + let binding = if major == 2 { ", error" } else { " as error" }; + let expected = format!("try:\n work\n more_work\nexcept ValueError{binding}:\n handled\nelse:\n success\nfinally:\n cleanup\n more_cleanup\n"); + assert_eq!(generate(&[outer.clone()], version, true), expected); + + // A legitimate outer try/finally has no success suite to reunite. + let nested = Stmt::Try { + body: vec![inner.clone()], handlers: vec![], orelse: vec![], + finalbody: vec![Stmt::Expr(name("cleanup"))], + }; + let expected = format!("try:\n try:\n work\n more_work\n except ValueError{binding}:\n handled\nfinally:\n cleanup\n"); + assert_eq!(generate(&[nested], version, true), expected); + } +} + +#[test] +fn keyword_calls_survive_conditional_value_reconstruction() { + for version in ["3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("keyword_conditional_calls.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"def case_0(value, build, other): + return build('reject' if value else 'accept', amount=0)"#, + r#"def case_1(value, build, other): + return build('prefix', amount=0 if value else 1)"#, + r#"def case_2(value, build, other): + return build(0 if value else 1, amount=value)"#, + r#"def case_9(value, build, other): + return build('prefix', amount=1 if value else 2, last=other(value))"#, + r#"def case_11(value, build, other): + return other(build(amount=1) if value else build(amount=2, other=3))"#, + ], version); + } +} + +#[test] +fn keyword_call_merge_does_not_hoist_an_arm_local_callable() { + for version in ["3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("keyword_call_arm_boundary.{version}.pyc")); + assert!(!clean, "Python {version}: an unproved call merge must remain incomplete:\n{src}"); + assert!(!src.contains("owner.build(amount=1 if value else 2)"), + "Python {version}: callable resolution moved before the condition:\n{src}"); + } +} + +#[test] +fn overwritten_unpack_slots_reuse_the_surviving_binding() { + for version in ["3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("repeated_unpack_bindings.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"def repeated_loop(values): + result = [] + for value, discarded, discarded, discarded in values: + result.append(value is not None) + return result"#, + r#"def repeated_list(values): + return [value is not None for value, discarded, discarded, discarded in values]"#, + r#"def underscore_loop(values): + result = [] + for value, _, discarded, discarded in values: + result.append((value, discarded, _)) + return result"#, + r#"def underscore_list(values): + return [(value, discarded, _) for value, _, discarded, discarded in values]"#, + r#"def nested_loop(values): + result = [] + for value, (discarded, discarded) in values: + result.append((value, discarded)) + return result"#, + r#"def nested_list(values): + return [(value, discarded) for value, (discarded, discarded) in values]"#, + ], version); + } +} + +#[test] +fn slice_constants_preserve_axes_and_runtime_calls() { + for version in ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] { + let (src, clean) = decompile_fixture(&format!("slice_constants_and_calls.{version}.pyc")); + assert!(clean, "Python {version}: output marked incomplete:\n{src}"); + assert_contains(&src, &[ + r#"def dynamic_axes(data, stop): + return data[:stop, :]"#, + r#"def constant_axes(data, stop): + return data[:, :]"#, + r#"def mixed_axes(data, stop): + return data[1:stop:2, :, 7]"#, + r#"def call(data, stop): + return data[slice(None, stop, None)]"#, + r#"def call_axes(data, stop): + return data[slice(None, stop, None), :]"#, + ], version); + } +} + +#[test] +fn nested_conditional_comprehension_arms_keep_their_values() { + for version in ["3.12", "3.13", "3.14"] { + let fixture = format!("nested_conditional_elements.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "return [{'excluded': {'count': row['count'], 'reasons': loads(row['reasons']) if row['reasons'] else {}} if row['count'] is not None else {}, 'archived': str(row['archived']) if row['archived'] else None} for row in rows]", + "return [build(1 if row.inner else 2) if row.outer else 3 for row in rows]", + "return [1 if row.outer else build(2 if row.inner else 3) for row in rows]", + "return [build(1 if row.inner is None else 2) if row.outer else build(3 if row.inner is not None else 4) for row in rows]", + ], + &fixture, + ); + } +} + +#[test] +fn chained_comprehension_filters_keep_each_polarity() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("chained_comprehension_filters.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + for (kind, open, close, element) in [ + ("list", "[", "]", "value"), + ("set", "{", "}", "value"), + ("dict", "{", "}", "value: value"), + ] { + for (polarity, negation) in [("positive", ""), ("negative", "not ")] { + for join in ["separate", "joined"] { + let expected = format!( + "def {kind}_{polarity}_{join}(values):\n return {open}{element} for value in values if {negation}isinstance(value, str) if value{close}" + ); + assert_contains(&src, &[&expected], &fixture); + } + } + } + assert_contains( + &src, + &[ + "return [row.tag for row in rows if row.value == wanted if row.ready() if row.final()]", + "return [row.tag for row in rows if row.value != wanted if not row.ready() if row.final()]", + ], + &fixture, + ); + let normalized = src.split_whitespace().collect::>().join(" "); + assert_contains( + &normalized, + &["def broken_lines(values): return [value for value in values if isinstance(value, str) if value]"], + &fixture, + ); + } +} + +#[test] +fn comprehension_boolean_values_remain_elements_not_filters() { + for version in ["3.12", "3.13", "3.14"] { + let fixture = format!("comprehension_boolean_values.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "return [max(0.0, float(row.get('x') or 0.0)) for row in rows]", + "return [row[1] or 0 for row in rows]", + "return [row[1] and 7 for row in rows]", + "return {row[1] or 0 for row in rows}", + "return {row[0]: row[1] and 7 for row in rows}", + "return [row.first() or row.second() for row in rows]", + "return [row.first() and row.second() for row in rows]", + "return [row[1] for row in rows if row[0]]", + "return [row[1] or 0 for row in rows if row[0]]", + "return [row[1] if row[0] else 0 for row in rows]", + ], + &fixture, + ); + } +} + +#[test] +fn keyword_expansions_keep_merge_checks_and_dictionary_boundaries() { + for version in ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("ordered_keyword_expansions.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "return fn(1, fixed=2, **first, **second)", + "return fn(1, **first, **{'fixed': 2}, **second)", + "return fn(1, **first, **second)", + "return fn(1, **{'fixed': 2, **first, **second})", + "return fn(1, **{'my col': 2, **first, **second})", + "return fn(1, **{'class': 2, **first, **second})", + "return fn(1, **{'fixed': 2, 'fixed': 3, **first, **second})", + "return fn(1, fixed=2, **get_first(), **get_second())", + ], + &fixture, + ); + } +} + +#[test] +fn deferred_class_annotations_keep_registration_gates() { + let fixture = "class_annotation_registration.3.14.pyc"; + let (src, clean) = decompile_fixture(fixture); + assert!(clean, "output marked incomplete:\n{src}"); + assert!(src.starts_with("module_count: int = 5\n"), "{src}"); + assert_contains( + &src, + &[ + "class Always:", + "'''Documented class.'''", + " if True:\n amount: int = 3\n label: str = 'ready'", + " if enabled:\n if True:\n label: str = 'ready'", + " for value in values:\n if True:\n limit: int = default(value)", + " for value in values:\n if True:\n amount: int = default(value)", + "def read(self) -> int:", + ], + fixture, + ); + assert!(!src.contains("__conditional_annotations__"), "{src}"); +} + +#[test] +fn nested_comprehensions_remain_generator_elements() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("nested_generator_comprehensions.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "return tuple(({key: row[key] for key in row} for row in rows))", + "return tuple(([value * 2 for value in row] for row in rows))", + "return tuple(({value * 2 for value in row} for row in rows))", + "return tuple((collect([value * 2 for value in row]) for row in rows))", + "return tuple(({key: row[key] for key in row if key} for row in rows if row))", + "return tuple(({key: value for key, value in zip(columns, row) if key} for row in rows))", + ], + &fixture, + ); + } +} + +#[test] +fn boolean_value_chains_preserve_call_arguments_and_prefixes() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("boolean_call_arguments.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + assert_contains( + &src, + &[ + "return call(before(), first() and second() or last(), after())", + "return call(value, value and second() or last(), after())", + "return bool(a and b or c or d or e)", + "return bool(c or d or a and b or e)", + "return bool(a and a.get('available') or b or c or d or e)", + "value = a and a.get('available') or b or c or d or e\n return bool(value)", + "return bool(a or b or c or d or e)", + "return bool(a and b and c and d and e)", + "return call(a and b or c, d if e else 0)", + ], + &fixture, + ); + } +} + +#[test] +fn independent_lookup_guards_keep_assignment_and_return_inside_the_loop() { + for version in ["3.11", "3.12", "3.13", "3.14"] { + let fixture = format!("loop_lookup_guards.{version}.pyc"); + let (src, clean) = decompile_fixture(&fixture); + assert!(clean, "{fixture}: output marked incomplete:\n{src}"); + // The existing None-jump printer uses either equivalent spelling. + let src = src.replace("not coordinator is not None", "coordinator is None"); + assert_contains( + &src, + &[ + "def rating_lookup(values):\n for value in values:\n if not value.rating < 10:\n continue\n coordinator = value.coordinator\n if coordinator is None:\n continue\n return coordinator", + "def dictionary_lookup(rows):\n for row in rows:\n if not isinstance(row, dict):\n continue\n coordinator = row.get('coordinator')\n if coordinator is None:\n continue\n return coordinator", + "def none_lookup(rows):\n for row in rows:\n if row is None:\n continue\n coordinator = row.coordinator\n if coordinator is None:\n continue\n return coordinator", + "return [row.get('coordinator') for row in rows if isinstance(row, dict)]", + ], + &fixture, + ); + assert!(!src.contains("value.rating >="), "{fixture}: comparison inverted"); + } +} diff --git a/tests/pyc/assignment_chains.3.11.pyc b/tests/pyc/assignment_chains.3.11.pyc new file mode 100644 index 00000000..65e13709 Binary files /dev/null and b/tests/pyc/assignment_chains.3.11.pyc differ diff --git a/tests/pyc/assignment_chains.3.12.pyc b/tests/pyc/assignment_chains.3.12.pyc new file mode 100644 index 00000000..d6d6bf68 Binary files /dev/null and b/tests/pyc/assignment_chains.3.12.pyc differ diff --git a/tests/pyc/assignment_chains.3.13.pyc b/tests/pyc/assignment_chains.3.13.pyc new file mode 100644 index 00000000..e90e0a0b Binary files /dev/null and b/tests/pyc/assignment_chains.3.13.pyc differ diff --git a/tests/pyc/assignment_chains.3.14.pyc b/tests/pyc/assignment_chains.3.14.pyc new file mode 100644 index 00000000..76a7ae4b Binary files /dev/null and b/tests/pyc/assignment_chains.3.14.pyc differ diff --git a/tests/pyc/assignment_identity.3.11.pyc b/tests/pyc/assignment_identity.3.11.pyc new file mode 100644 index 00000000..d3f73a5a Binary files /dev/null and b/tests/pyc/assignment_identity.3.11.pyc differ diff --git a/tests/pyc/assignment_identity.3.12.pyc b/tests/pyc/assignment_identity.3.12.pyc new file mode 100644 index 00000000..2b9520b5 Binary files /dev/null and b/tests/pyc/assignment_identity.3.12.pyc differ diff --git a/tests/pyc/assignment_identity.3.13.pyc b/tests/pyc/assignment_identity.3.13.pyc new file mode 100644 index 00000000..1d19476a Binary files /dev/null and b/tests/pyc/assignment_identity.3.13.pyc differ diff --git a/tests/pyc/assignment_identity.3.14.pyc b/tests/pyc/assignment_identity.3.14.pyc new file mode 100644 index 00000000..b3cd8796 Binary files /dev/null and b/tests/pyc/assignment_identity.3.14.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.10.pyc b/tests/pyc/awaited_synchronous_iteration.3.10.pyc new file mode 100644 index 00000000..9cc60d84 Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.10.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.11.pyc b/tests/pyc/awaited_synchronous_iteration.3.11.pyc new file mode 100644 index 00000000..866f96b3 Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.11.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.12.pyc b/tests/pyc/awaited_synchronous_iteration.3.12.pyc new file mode 100644 index 00000000..2d87c32b Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.12.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.13.pyc b/tests/pyc/awaited_synchronous_iteration.3.13.pyc new file mode 100644 index 00000000..882a80e5 Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.13.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.14.pyc b/tests/pyc/awaited_synchronous_iteration.3.14.pyc new file mode 100644 index 00000000..fb7bb10b Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.14.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.8.pyc b/tests/pyc/awaited_synchronous_iteration.3.8.pyc new file mode 100644 index 00000000..f88c010c Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.8.pyc differ diff --git a/tests/pyc/awaited_synchronous_iteration.3.9.pyc b/tests/pyc/awaited_synchronous_iteration.3.9.pyc new file mode 100644 index 00000000..1adf702d Binary files /dev/null and b/tests/pyc/awaited_synchronous_iteration.3.9.pyc differ diff --git a/tests/pyc/boolean_call_arguments.3.11.pyc b/tests/pyc/boolean_call_arguments.3.11.pyc new file mode 100644 index 00000000..82e30327 Binary files /dev/null and b/tests/pyc/boolean_call_arguments.3.11.pyc differ diff --git a/tests/pyc/boolean_call_arguments.3.12.pyc b/tests/pyc/boolean_call_arguments.3.12.pyc new file mode 100644 index 00000000..4e97c86e Binary files /dev/null and b/tests/pyc/boolean_call_arguments.3.12.pyc differ diff --git a/tests/pyc/boolean_call_arguments.3.13.pyc b/tests/pyc/boolean_call_arguments.3.13.pyc new file mode 100644 index 00000000..1f0cf92c Binary files /dev/null and b/tests/pyc/boolean_call_arguments.3.13.pyc differ diff --git a/tests/pyc/boolean_call_arguments.3.14.pyc b/tests/pyc/boolean_call_arguments.3.14.pyc new file mode 100644 index 00000000..98fbb201 Binary files /dev/null and b/tests/pyc/boolean_call_arguments.3.14.pyc differ diff --git a/tests/pyc/chained_comprehension_filters.3.11.pyc b/tests/pyc/chained_comprehension_filters.3.11.pyc new file mode 100644 index 00000000..c1ed1162 Binary files /dev/null and b/tests/pyc/chained_comprehension_filters.3.11.pyc differ diff --git a/tests/pyc/chained_comprehension_filters.3.12.pyc b/tests/pyc/chained_comprehension_filters.3.12.pyc new file mode 100644 index 00000000..4f795f2c Binary files /dev/null and b/tests/pyc/chained_comprehension_filters.3.12.pyc differ diff --git a/tests/pyc/chained_comprehension_filters.3.13.pyc b/tests/pyc/chained_comprehension_filters.3.13.pyc new file mode 100644 index 00000000..278ea6af Binary files /dev/null and b/tests/pyc/chained_comprehension_filters.3.13.pyc differ diff --git a/tests/pyc/chained_comprehension_filters.3.14.pyc b/tests/pyc/chained_comprehension_filters.3.14.pyc new file mode 100644 index 00000000..d1b07c59 Binary files /dev/null and b/tests/pyc/chained_comprehension_filters.3.14.pyc differ diff --git a/tests/pyc/class_annotation_registration.3.14.pyc b/tests/pyc/class_annotation_registration.3.14.pyc new file mode 100644 index 00000000..c17f6348 Binary files /dev/null and b/tests/pyc/class_annotation_registration.3.14.pyc differ diff --git a/tests/pyc/comprehension_boolean_values.3.12.pyc b/tests/pyc/comprehension_boolean_values.3.12.pyc new file mode 100644 index 00000000..44b3d7f9 Binary files /dev/null and b/tests/pyc/comprehension_boolean_values.3.12.pyc differ diff --git a/tests/pyc/comprehension_boolean_values.3.13.pyc b/tests/pyc/comprehension_boolean_values.3.13.pyc new file mode 100644 index 00000000..b59a5800 Binary files /dev/null and b/tests/pyc/comprehension_boolean_values.3.13.pyc differ diff --git a/tests/pyc/comprehension_boolean_values.3.14.pyc b/tests/pyc/comprehension_boolean_values.3.14.pyc new file mode 100644 index 00000000..bf57c917 Binary files /dev/null and b/tests/pyc/comprehension_boolean_values.3.14.pyc differ diff --git a/tests/pyc/comprehension_default_arguments.3.11.pyc b/tests/pyc/comprehension_default_arguments.3.11.pyc new file mode 100644 index 00000000..59434d06 Binary files /dev/null and b/tests/pyc/comprehension_default_arguments.3.11.pyc differ diff --git a/tests/pyc/comprehension_default_arguments.3.12.pyc b/tests/pyc/comprehension_default_arguments.3.12.pyc new file mode 100644 index 00000000..6c90b1c9 Binary files /dev/null and b/tests/pyc/comprehension_default_arguments.3.12.pyc differ diff --git a/tests/pyc/comprehension_default_arguments.3.13.pyc b/tests/pyc/comprehension_default_arguments.3.13.pyc new file mode 100644 index 00000000..efab9741 Binary files /dev/null and b/tests/pyc/comprehension_default_arguments.3.13.pyc differ diff --git a/tests/pyc/comprehension_default_arguments.3.14.pyc b/tests/pyc/comprehension_default_arguments.3.14.pyc new file mode 100644 index 00000000..869ec159 Binary files /dev/null and b/tests/pyc/comprehension_default_arguments.3.14.pyc differ diff --git a/tests/pyc/comprehension_dictionary_displays.3.10.pyc b/tests/pyc/comprehension_dictionary_displays.3.10.pyc new file mode 100644 index 00000000..f7587b2e Binary files /dev/null and b/tests/pyc/comprehension_dictionary_displays.3.10.pyc differ diff --git a/tests/pyc/comprehension_dictionary_displays.3.11.pyc b/tests/pyc/comprehension_dictionary_displays.3.11.pyc new file mode 100644 index 00000000..e0866250 Binary files /dev/null and b/tests/pyc/comprehension_dictionary_displays.3.11.pyc differ diff --git a/tests/pyc/comprehension_dictionary_displays.3.12.pyc b/tests/pyc/comprehension_dictionary_displays.3.12.pyc new file mode 100644 index 00000000..dbdc5f36 Binary files /dev/null and b/tests/pyc/comprehension_dictionary_displays.3.12.pyc differ diff --git a/tests/pyc/comprehension_dictionary_displays.3.13.pyc b/tests/pyc/comprehension_dictionary_displays.3.13.pyc new file mode 100644 index 00000000..c1494d26 Binary files /dev/null and b/tests/pyc/comprehension_dictionary_displays.3.13.pyc differ diff --git a/tests/pyc/comprehension_dictionary_displays.3.14.pyc b/tests/pyc/comprehension_dictionary_displays.3.14.pyc new file mode 100644 index 00000000..340d8546 Binary files /dev/null and b/tests/pyc/comprehension_dictionary_displays.3.14.pyc differ diff --git a/tests/pyc/conditional_loop_prefix.3.11.pyc b/tests/pyc/conditional_loop_prefix.3.11.pyc new file mode 100644 index 00000000..a205329d Binary files /dev/null and b/tests/pyc/conditional_loop_prefix.3.11.pyc differ diff --git a/tests/pyc/conditional_loop_prefix.3.12.pyc b/tests/pyc/conditional_loop_prefix.3.12.pyc new file mode 100644 index 00000000..afeffcd3 Binary files /dev/null and b/tests/pyc/conditional_loop_prefix.3.12.pyc differ diff --git a/tests/pyc/conditional_loop_prefix.3.13.pyc b/tests/pyc/conditional_loop_prefix.3.13.pyc new file mode 100644 index 00000000..c4ca9f1e Binary files /dev/null and b/tests/pyc/conditional_loop_prefix.3.13.pyc differ diff --git a/tests/pyc/conditional_loop_prefix.3.14.pyc b/tests/pyc/conditional_loop_prefix.3.14.pyc new file mode 100644 index 00000000..f3f270d6 Binary files /dev/null and b/tests/pyc/conditional_loop_prefix.3.14.pyc differ diff --git a/tests/pyc/extended_loop_prefix.3.11.pyc b/tests/pyc/extended_loop_prefix.3.11.pyc new file mode 100644 index 00000000..c805c5c2 Binary files /dev/null and b/tests/pyc/extended_loop_prefix.3.11.pyc differ diff --git a/tests/pyc/extended_loop_prefix.3.12.pyc b/tests/pyc/extended_loop_prefix.3.12.pyc new file mode 100644 index 00000000..65423109 Binary files /dev/null and b/tests/pyc/extended_loop_prefix.3.12.pyc differ diff --git a/tests/pyc/extended_loop_prefix.3.13.pyc b/tests/pyc/extended_loop_prefix.3.13.pyc new file mode 100644 index 00000000..f1f773a3 Binary files /dev/null and b/tests/pyc/extended_loop_prefix.3.13.pyc differ diff --git a/tests/pyc/extended_loop_prefix.3.14.pyc b/tests/pyc/extended_loop_prefix.3.14.pyc new file mode 100644 index 00000000..5d4255f4 Binary files /dev/null and b/tests/pyc/extended_loop_prefix.3.14.pyc differ diff --git a/tests/pyc/generator_expanded_calls.3.11.pyc b/tests/pyc/generator_expanded_calls.3.11.pyc new file mode 100644 index 00000000..fb86771d Binary files /dev/null and b/tests/pyc/generator_expanded_calls.3.11.pyc differ diff --git a/tests/pyc/generator_expanded_calls.3.12.pyc b/tests/pyc/generator_expanded_calls.3.12.pyc new file mode 100644 index 00000000..52f56319 Binary files /dev/null and b/tests/pyc/generator_expanded_calls.3.12.pyc differ diff --git a/tests/pyc/generator_expanded_calls.3.13.pyc b/tests/pyc/generator_expanded_calls.3.13.pyc new file mode 100644 index 00000000..2a963867 Binary files /dev/null and b/tests/pyc/generator_expanded_calls.3.13.pyc differ diff --git a/tests/pyc/generator_expanded_calls.3.14.pyc b/tests/pyc/generator_expanded_calls.3.14.pyc new file mode 100644 index 00000000..ac01da07 Binary files /dev/null and b/tests/pyc/generator_expanded_calls.3.14.pyc differ diff --git a/tests/pyc/guarded_else_merge.3.11.pyc b/tests/pyc/guarded_else_merge.3.11.pyc new file mode 100644 index 00000000..226ca408 Binary files /dev/null and b/tests/pyc/guarded_else_merge.3.11.pyc differ diff --git a/tests/pyc/guarded_else_merge.3.12.pyc b/tests/pyc/guarded_else_merge.3.12.pyc new file mode 100644 index 00000000..9d8e7958 Binary files /dev/null and b/tests/pyc/guarded_else_merge.3.12.pyc differ diff --git a/tests/pyc/guarded_else_merge.3.13.pyc b/tests/pyc/guarded_else_merge.3.13.pyc new file mode 100644 index 00000000..1c15b4af Binary files /dev/null and b/tests/pyc/guarded_else_merge.3.13.pyc differ diff --git a/tests/pyc/guarded_else_merge.3.14.pyc b/tests/pyc/guarded_else_merge.3.14.pyc new file mode 100644 index 00000000..6ea01b6c Binary files /dev/null and b/tests/pyc/guarded_else_merge.3.14.pyc differ diff --git a/tests/pyc/handler_return_padding_1.3.11.pyc b/tests/pyc/handler_return_padding_1.3.11.pyc new file mode 100644 index 00000000..0f1660ad Binary files /dev/null and b/tests/pyc/handler_return_padding_1.3.11.pyc differ diff --git a/tests/pyc/handler_return_padding_1.3.12.pyc b/tests/pyc/handler_return_padding_1.3.12.pyc new file mode 100644 index 00000000..aae4bf0e Binary files /dev/null and b/tests/pyc/handler_return_padding_1.3.12.pyc differ diff --git a/tests/pyc/handler_return_padding_1.3.13.pyc b/tests/pyc/handler_return_padding_1.3.13.pyc new file mode 100644 index 00000000..0f68db61 Binary files /dev/null and b/tests/pyc/handler_return_padding_1.3.13.pyc differ diff --git a/tests/pyc/handler_return_padding_1.3.14.pyc b/tests/pyc/handler_return_padding_1.3.14.pyc new file mode 100644 index 00000000..be6567e2 Binary files /dev/null and b/tests/pyc/handler_return_padding_1.3.14.pyc differ diff --git a/tests/pyc/handler_return_padding_3.3.11.pyc b/tests/pyc/handler_return_padding_3.3.11.pyc new file mode 100644 index 00000000..ff2d6ead Binary files /dev/null and b/tests/pyc/handler_return_padding_3.3.11.pyc differ diff --git a/tests/pyc/handler_return_padding_3.3.12.pyc b/tests/pyc/handler_return_padding_3.3.12.pyc new file mode 100644 index 00000000..aae4bf0e Binary files /dev/null and b/tests/pyc/handler_return_padding_3.3.12.pyc differ diff --git a/tests/pyc/handler_return_padding_3.3.13.pyc b/tests/pyc/handler_return_padding_3.3.13.pyc new file mode 100644 index 00000000..0f68db61 Binary files /dev/null and b/tests/pyc/handler_return_padding_3.3.13.pyc differ diff --git a/tests/pyc/handler_return_padding_3.3.14.pyc b/tests/pyc/handler_return_padding_3.3.14.pyc new file mode 100644 index 00000000..0f1c0628 Binary files /dev/null and b/tests/pyc/handler_return_padding_3.3.14.pyc differ diff --git a/tests/pyc/handler_return_padding_9.3.11.pyc b/tests/pyc/handler_return_padding_9.3.11.pyc new file mode 100644 index 00000000..9a90c4f1 Binary files /dev/null and b/tests/pyc/handler_return_padding_9.3.11.pyc differ diff --git a/tests/pyc/handler_return_padding_9.3.12.pyc b/tests/pyc/handler_return_padding_9.3.12.pyc new file mode 100644 index 00000000..aae4bf0e Binary files /dev/null and b/tests/pyc/handler_return_padding_9.3.12.pyc differ diff --git a/tests/pyc/handler_return_padding_9.3.13.pyc b/tests/pyc/handler_return_padding_9.3.13.pyc new file mode 100644 index 00000000..0f68db61 Binary files /dev/null and b/tests/pyc/handler_return_padding_9.3.13.pyc differ diff --git a/tests/pyc/handler_return_padding_9.3.14.pyc b/tests/pyc/handler_return_padding_9.3.14.pyc new file mode 100644 index 00000000..57241865 Binary files /dev/null and b/tests/pyc/handler_return_padding_9.3.14.pyc differ diff --git a/tests/pyc/keyword_call_arm_boundary.3.13.pyc b/tests/pyc/keyword_call_arm_boundary.3.13.pyc new file mode 100644 index 00000000..02c090f0 Binary files /dev/null and b/tests/pyc/keyword_call_arm_boundary.3.13.pyc differ diff --git a/tests/pyc/keyword_call_arm_boundary.3.14.pyc b/tests/pyc/keyword_call_arm_boundary.3.14.pyc new file mode 100644 index 00000000..6506f42c Binary files /dev/null and b/tests/pyc/keyword_call_arm_boundary.3.14.pyc differ diff --git a/tests/pyc/keyword_conditional_calls.3.13.pyc b/tests/pyc/keyword_conditional_calls.3.13.pyc new file mode 100644 index 00000000..d453af65 Binary files /dev/null and b/tests/pyc/keyword_conditional_calls.3.13.pyc differ diff --git a/tests/pyc/keyword_conditional_calls.3.14.pyc b/tests/pyc/keyword_conditional_calls.3.14.pyc new file mode 100644 index 00000000..891c547c Binary files /dev/null and b/tests/pyc/keyword_conditional_calls.3.14.pyc differ diff --git a/tests/pyc/loop_lookup_guards.3.11.pyc b/tests/pyc/loop_lookup_guards.3.11.pyc new file mode 100644 index 00000000..3c1bfc22 Binary files /dev/null and b/tests/pyc/loop_lookup_guards.3.11.pyc differ diff --git a/tests/pyc/loop_lookup_guards.3.12.pyc b/tests/pyc/loop_lookup_guards.3.12.pyc new file mode 100644 index 00000000..8230f1de Binary files /dev/null and b/tests/pyc/loop_lookup_guards.3.12.pyc differ diff --git a/tests/pyc/loop_lookup_guards.3.13.pyc b/tests/pyc/loop_lookup_guards.3.13.pyc new file mode 100644 index 00000000..8124b444 Binary files /dev/null and b/tests/pyc/loop_lookup_guards.3.13.pyc differ diff --git a/tests/pyc/loop_lookup_guards.3.14.pyc b/tests/pyc/loop_lookup_guards.3.14.pyc new file mode 100644 index 00000000..0b46ed6a Binary files /dev/null and b/tests/pyc/loop_lookup_guards.3.14.pyc differ diff --git a/tests/pyc/nested_conditional_elements.3.12.pyc b/tests/pyc/nested_conditional_elements.3.12.pyc new file mode 100644 index 00000000..71b44852 Binary files /dev/null and b/tests/pyc/nested_conditional_elements.3.12.pyc differ diff --git a/tests/pyc/nested_conditional_elements.3.13.pyc b/tests/pyc/nested_conditional_elements.3.13.pyc new file mode 100644 index 00000000..a2b79a36 Binary files /dev/null and b/tests/pyc/nested_conditional_elements.3.13.pyc differ diff --git a/tests/pyc/nested_conditional_elements.3.14.pyc b/tests/pyc/nested_conditional_elements.3.14.pyc new file mode 100644 index 00000000..5a5fc236 Binary files /dev/null and b/tests/pyc/nested_conditional_elements.3.14.pyc differ diff --git a/tests/pyc/nested_generator_comprehensions.3.11.pyc b/tests/pyc/nested_generator_comprehensions.3.11.pyc new file mode 100644 index 00000000..2a803036 Binary files /dev/null and b/tests/pyc/nested_generator_comprehensions.3.11.pyc differ diff --git a/tests/pyc/nested_generator_comprehensions.3.12.pyc b/tests/pyc/nested_generator_comprehensions.3.12.pyc new file mode 100644 index 00000000..6c39d374 Binary files /dev/null and b/tests/pyc/nested_generator_comprehensions.3.12.pyc differ diff --git a/tests/pyc/nested_generator_comprehensions.3.13.pyc b/tests/pyc/nested_generator_comprehensions.3.13.pyc new file mode 100644 index 00000000..5eb814e4 Binary files /dev/null and b/tests/pyc/nested_generator_comprehensions.3.13.pyc differ diff --git a/tests/pyc/nested_generator_comprehensions.3.14.pyc b/tests/pyc/nested_generator_comprehensions.3.14.pyc new file mode 100644 index 00000000..a00b18f4 Binary files /dev/null and b/tests/pyc/nested_generator_comprehensions.3.14.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.10.pyc b/tests/pyc/ordered_keyword_expansions.3.10.pyc new file mode 100644 index 00000000..54ae9dcc Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.10.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.11.pyc b/tests/pyc/ordered_keyword_expansions.3.11.pyc new file mode 100644 index 00000000..0a33d062 Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.11.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.12.pyc b/tests/pyc/ordered_keyword_expansions.3.12.pyc new file mode 100644 index 00000000..aaa035b7 Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.12.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.13.pyc b/tests/pyc/ordered_keyword_expansions.3.13.pyc new file mode 100644 index 00000000..444fa357 Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.13.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.14.pyc b/tests/pyc/ordered_keyword_expansions.3.14.pyc new file mode 100644 index 00000000..02f4bed8 Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.14.pyc differ diff --git a/tests/pyc/ordered_keyword_expansions.3.9.pyc b/tests/pyc/ordered_keyword_expansions.3.9.pyc new file mode 100644 index 00000000..b3905ae4 Binary files /dev/null and b/tests/pyc/ordered_keyword_expansions.3.9.pyc differ diff --git a/tests/pyc/padded_value_returns_1.3.11.pyc b/tests/pyc/padded_value_returns_1.3.11.pyc new file mode 100644 index 00000000..2ef2c6bf Binary files /dev/null and b/tests/pyc/padded_value_returns_1.3.11.pyc differ diff --git a/tests/pyc/padded_value_returns_1.3.12.pyc b/tests/pyc/padded_value_returns_1.3.12.pyc new file mode 100644 index 00000000..d55d035a Binary files /dev/null and b/tests/pyc/padded_value_returns_1.3.12.pyc differ diff --git a/tests/pyc/padded_value_returns_1.3.13.pyc b/tests/pyc/padded_value_returns_1.3.13.pyc new file mode 100644 index 00000000..7ed911c6 Binary files /dev/null and b/tests/pyc/padded_value_returns_1.3.13.pyc differ diff --git a/tests/pyc/padded_value_returns_1.3.14.pyc b/tests/pyc/padded_value_returns_1.3.14.pyc new file mode 100644 index 00000000..f6ccbc43 Binary files /dev/null and b/tests/pyc/padded_value_returns_1.3.14.pyc differ diff --git a/tests/pyc/padded_value_returns_3.3.11.pyc b/tests/pyc/padded_value_returns_3.3.11.pyc new file mode 100644 index 00000000..8d8162a7 Binary files /dev/null and b/tests/pyc/padded_value_returns_3.3.11.pyc differ diff --git a/tests/pyc/padded_value_returns_3.3.12.pyc b/tests/pyc/padded_value_returns_3.3.12.pyc new file mode 100644 index 00000000..02223be6 Binary files /dev/null and b/tests/pyc/padded_value_returns_3.3.12.pyc differ diff --git a/tests/pyc/padded_value_returns_3.3.13.pyc b/tests/pyc/padded_value_returns_3.3.13.pyc new file mode 100644 index 00000000..66632e32 Binary files /dev/null and b/tests/pyc/padded_value_returns_3.3.13.pyc differ diff --git a/tests/pyc/padded_value_returns_3.3.14.pyc b/tests/pyc/padded_value_returns_3.3.14.pyc new file mode 100644 index 00000000..4d984de4 Binary files /dev/null and b/tests/pyc/padded_value_returns_3.3.14.pyc differ diff --git a/tests/pyc/padded_value_returns_9.3.11.pyc b/tests/pyc/padded_value_returns_9.3.11.pyc new file mode 100644 index 00000000..adb60127 Binary files /dev/null and b/tests/pyc/padded_value_returns_9.3.11.pyc differ diff --git a/tests/pyc/padded_value_returns_9.3.12.pyc b/tests/pyc/padded_value_returns_9.3.12.pyc new file mode 100644 index 00000000..cc0d5730 Binary files /dev/null and b/tests/pyc/padded_value_returns_9.3.12.pyc differ diff --git a/tests/pyc/padded_value_returns_9.3.13.pyc b/tests/pyc/padded_value_returns_9.3.13.pyc new file mode 100644 index 00000000..7812335a Binary files /dev/null and b/tests/pyc/padded_value_returns_9.3.13.pyc differ diff --git a/tests/pyc/padded_value_returns_9.3.14.pyc b/tests/pyc/padded_value_returns_9.3.14.pyc new file mode 100644 index 00000000..64d17f01 Binary files /dev/null and b/tests/pyc/padded_value_returns_9.3.14.pyc differ diff --git a/tests/pyc/repeated_unpack_bindings.3.12.pyc b/tests/pyc/repeated_unpack_bindings.3.12.pyc new file mode 100644 index 00000000..3b14a1b6 Binary files /dev/null and b/tests/pyc/repeated_unpack_bindings.3.12.pyc differ diff --git a/tests/pyc/repeated_unpack_bindings.3.13.pyc b/tests/pyc/repeated_unpack_bindings.3.13.pyc new file mode 100644 index 00000000..de827138 Binary files /dev/null and b/tests/pyc/repeated_unpack_bindings.3.13.pyc differ diff --git a/tests/pyc/repeated_unpack_bindings.3.14.pyc b/tests/pyc/repeated_unpack_bindings.3.14.pyc new file mode 100644 index 00000000..b94832cd Binary files /dev/null and b/tests/pyc/repeated_unpack_bindings.3.14.pyc differ diff --git a/tests/pyc/sequential_generator_handlers.3.11.pyc b/tests/pyc/sequential_generator_handlers.3.11.pyc new file mode 100644 index 00000000..008ddc55 Binary files /dev/null and b/tests/pyc/sequential_generator_handlers.3.11.pyc differ diff --git a/tests/pyc/sequential_generator_handlers.3.12.pyc b/tests/pyc/sequential_generator_handlers.3.12.pyc new file mode 100644 index 00000000..ccfe238b Binary files /dev/null and b/tests/pyc/sequential_generator_handlers.3.12.pyc differ diff --git a/tests/pyc/sequential_generator_handlers.3.13.pyc b/tests/pyc/sequential_generator_handlers.3.13.pyc new file mode 100644 index 00000000..3c083b3a Binary files /dev/null and b/tests/pyc/sequential_generator_handlers.3.13.pyc differ diff --git a/tests/pyc/sequential_generator_handlers.3.14.pyc b/tests/pyc/sequential_generator_handlers.3.14.pyc new file mode 100644 index 00000000..fa151051 Binary files /dev/null and b/tests/pyc/sequential_generator_handlers.3.14.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.10.pyc b/tests/pyc/slice_constants_and_calls.3.10.pyc new file mode 100644 index 00000000..6d170997 Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.10.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.11.pyc b/tests/pyc/slice_constants_and_calls.3.11.pyc new file mode 100644 index 00000000..ef6510a8 Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.11.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.12.pyc b/tests/pyc/slice_constants_and_calls.3.12.pyc new file mode 100644 index 00000000..63c9d7a8 Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.12.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.13.pyc b/tests/pyc/slice_constants_and_calls.3.13.pyc new file mode 100644 index 00000000..8383befd Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.13.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.14.pyc b/tests/pyc/slice_constants_and_calls.3.14.pyc new file mode 100644 index 00000000..376bceb9 Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.14.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.8.pyc b/tests/pyc/slice_constants_and_calls.3.8.pyc new file mode 100644 index 00000000..258bdc8c Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.8.pyc differ diff --git a/tests/pyc/slice_constants_and_calls.3.9.pyc b/tests/pyc/slice_constants_and_calls.3.9.pyc new file mode 100644 index 00000000..60518fac Binary files /dev/null and b/tests/pyc/slice_constants_and_calls.3.9.pyc differ diff --git a/tests/pyc/sunk_value_return_control.3.11.pyc b/tests/pyc/sunk_value_return_control.3.11.pyc new file mode 100644 index 00000000..96c6d1a8 Binary files /dev/null and b/tests/pyc/sunk_value_return_control.3.11.pyc differ diff --git a/tests/pyc/sunk_value_return_control.3.12.pyc b/tests/pyc/sunk_value_return_control.3.12.pyc new file mode 100644 index 00000000..58db421e Binary files /dev/null and b/tests/pyc/sunk_value_return_control.3.12.pyc differ diff --git a/tests/pyc/sunk_value_return_control.3.13.pyc b/tests/pyc/sunk_value_return_control.3.13.pyc new file mode 100644 index 00000000..68224696 Binary files /dev/null and b/tests/pyc/sunk_value_return_control.3.13.pyc differ diff --git a/tests/pyc/sunk_value_return_control.3.14.pyc b/tests/pyc/sunk_value_return_control.3.14.pyc new file mode 100644 index 00000000..7f8c9e4d Binary files /dev/null and b/tests/pyc/sunk_value_return_control.3.14.pyc differ diff --git a/tests/pyc/unpacked_comprehension_cleanup.3.11.pyc b/tests/pyc/unpacked_comprehension_cleanup.3.11.pyc new file mode 100644 index 00000000..e5f5cc50 Binary files /dev/null and b/tests/pyc/unpacked_comprehension_cleanup.3.11.pyc differ diff --git a/tests/pyc/unpacked_comprehension_cleanup.3.12.pyc b/tests/pyc/unpacked_comprehension_cleanup.3.12.pyc new file mode 100644 index 00000000..bb9e343f Binary files /dev/null and b/tests/pyc/unpacked_comprehension_cleanup.3.12.pyc differ diff --git a/tests/pyc/unpacked_comprehension_cleanup.3.13.pyc b/tests/pyc/unpacked_comprehension_cleanup.3.13.pyc new file mode 100644 index 00000000..21d84ac7 Binary files /dev/null and b/tests/pyc/unpacked_comprehension_cleanup.3.13.pyc differ diff --git a/tests/pyc/unpacked_comprehension_cleanup.3.14.pyc b/tests/pyc/unpacked_comprehension_cleanup.3.14.pyc new file mode 100644 index 00000000..08964896 Binary files /dev/null and b/tests/pyc/unpacked_comprehension_cleanup.3.14.pyc differ diff --git a/tools/compile_padded_returns.py b/tools/compile_padded_returns.py new file mode 100644 index 00000000..bdb472d9 --- /dev/null +++ b/tools/compile_padded_returns.py @@ -0,0 +1,81 @@ +"""Regenerate the padded handler-return fixtures with CPython 3.11–3.14. + +Run once per interpreter from any directory. This prepares bytecode inputs +for tests/integration.rs; it is not a separate test runner. Only RETURN_VALUE +is padded, leaving versions that emit RETURN_CONST as compatibility controls. +""" +import importlib.util +import marshal +from pathlib import Path +import sys +import types + +def pad_return_values(code, count=3): + """Add NOP padding while retaining jumps and exact handler boundaries.""" + import dis + instructions = list(dis.get_instructions(code)) + # This compact fixture needs no extended operands. NOPs + # reproduce the padded return layout, preserving the return's value. + assert not any(item.opname == 'EXTENDED_ARG' for item in instructions) + starts, sizes, cursor = {}, {}, 0 + for index, item in enumerate(instructions): + starts[item.offset] = cursor + end = instructions[index + 1].offset if index + 1 < len(instructions) else len(code.co_code) + sizes[item.offset] = end - item.offset + cursor += sizes[item.offset] + (2 * count if item.opname == 'RETURN_VALUE' else 0) + starts[len(code.co_code)] = cursor + output = bytearray() + for item in instructions: + if item.opname == 'RETURN_VALUE': + output.extend(bytes((dis.opmap['NOP'], 0)) * count) + actual = len(output) + raw = bytearray(code.co_code[item.offset:item.offset + sizes[item.offset]]) + if item.opcode in getattr(dis, "hasjump", dis.hasjrel + dis.hasjabs): + delta = (starts[item.argval] - (actual + sizes[item.offset])) // 2 + operand = -delta if 'BACKWARD' in item.opname else delta + assert 0 <= operand <= 255 + raw[1] = operand + output.extend(raw) + def varint(value, marker=False): + digits = [value & 63] + while value >> 6: + value >>= 6 + digits.append(value & 63) + digits.reverse() + for index in range(len(digits) - 1): + digits[index] |= 64 + if marker: + digits[0] |= 128 + return bytes(digits) + table = b''.join(varint(starts[row.start] // 2, True) + + varint((starts[row.end] - starts[row.start]) // 2) + + varint(starts[row.target] // 2) + + varint((row.depth << 1) | row.lasti) + for row in dis.Bytecode(code).exception_entries) + padded = code.replace(co_code=bytes(output), co_exceptiontable=table, co_linetable=b'') + return padded + + +if __name__ == "__main__": + if not (3, 11) <= sys.version_info[:2] <= (3, 14): + raise SystemExit("Use CPython 3.11, 3.12, 3.13 or 3.14") + root = Path(__file__).resolve().parents[1] + source_path = Path("tests/fixtures/handler_return_padding.py") + source = (root / source_path).read_bytes() + module = compile(source, source_path.as_posix(), "exec", dont_inherit=True) + version = f"{sys.version_info.major}.{sys.version_info.minor}" + header = (importlib.util.MAGIC_NUMBER + (3).to_bytes(4, "little") + + importlib.util.source_hash(source)) + for count in (1, 3, 9): + padded = module.replace(co_consts=tuple( + pad_return_values(value, count) if isinstance(value, types.CodeType) else value + for value in module.co_consts)) + path = root / f"tests/pyc/handler_return_padding_{count}.{version}.pyc" + path.write_bytes(header + marshal.dumps(padded)) + if count == 3: + # Also exercise a representative padded input in verify_corpus.py. + # Keep the directory named after the interpreter producing the pyc. + corpus = root / "tests/corpus" / ".".join(map(str, sys.version_info[:3])) + corpus.mkdir(parents=True, exist_ok=True) + (corpus / "handler_return_padding_3.py").write_bytes(source) + (corpus / "handler_return_padding_3.pyc").write_bytes(header + marshal.dumps(padded)) diff --git a/tools/compile_padded_value_returns.py b/tools/compile_padded_value_returns.py new file mode 100644 index 00000000..52e1f539 --- /dev/null +++ b/tools/compile_padded_value_returns.py @@ -0,0 +1,81 @@ +"""Regenerate the padded value-return fixtures with CPython 3.11–3.14. + +Run once per interpreter from any directory. This prepares bytecode inputs +for tests/integration.rs; it is not a separate test runner. Only RETURN_VALUE +is padded, leaving versions that emit RETURN_CONST as compatibility controls. +""" +import importlib.util +import marshal +from pathlib import Path +import sys +import types + +def pad_return_values(code, count=3): + """Add NOP padding while retaining jumps and exact handler boundaries.""" + import dis + instructions = list(dis.get_instructions(code)) + # This compact fixture needs no extended operands. NOPs + # reproduce the padded return layout, preserving the return's value. + assert not any(item.opname == 'EXTENDED_ARG' for item in instructions) + starts, sizes, cursor = {}, {}, 0 + for index, item in enumerate(instructions): + starts[item.offset] = cursor + end = instructions[index + 1].offset if index + 1 < len(instructions) else len(code.co_code) + sizes[item.offset] = end - item.offset + cursor += sizes[item.offset] + (2 * count if item.opname == 'RETURN_VALUE' else 0) + starts[len(code.co_code)] = cursor + output = bytearray() + for item in instructions: + if item.opname == 'RETURN_VALUE': + output.extend(bytes((dis.opmap['NOP'], 0)) * count) + actual = len(output) + raw = bytearray(code.co_code[item.offset:item.offset + sizes[item.offset]]) + if item.opcode in getattr(dis, "hasjump", dis.hasjrel + dis.hasjabs): + delta = (starts[item.argval] - (actual + sizes[item.offset])) // 2 + operand = -delta if 'BACKWARD' in item.opname else delta + assert 0 <= operand <= 255 + raw[1] = operand + output.extend(raw) + def varint(value, marker=False): + digits = [value & 63] + while value >> 6: + value >>= 6 + digits.append(value & 63) + digits.reverse() + for index in range(len(digits) - 1): + digits[index] |= 64 + if marker: + digits[0] |= 128 + return bytes(digits) + table = b''.join(varint(starts[row.start] // 2, True) + + varint((starts[row.end] - starts[row.start]) // 2) + + varint(starts[row.target] // 2) + + varint((row.depth << 1) | row.lasti) + for row in dis.Bytecode(code).exception_entries) + padded = code.replace(co_code=bytes(output), co_exceptiontable=table, co_linetable=b'') + return padded + + +if __name__ == "__main__": + if not (3, 11) <= sys.version_info[:2] <= (3, 14): + raise SystemExit("Use CPython 3.11, 3.12, 3.13 or 3.14") + root = Path(__file__).resolve().parents[1] + source_path = Path("tests/fixtures/padded_value_returns.py") + source = (root / source_path).read_bytes() + module = compile(source, source_path.as_posix(), "exec", dont_inherit=True) + version = f"{sys.version_info.major}.{sys.version_info.minor}" + header = (importlib.util.MAGIC_NUMBER + (3).to_bytes(4, "little") + + importlib.util.source_hash(source)) + for count in (1, 3, 9): + padded = module.replace(co_consts=tuple( + pad_return_values(value, count) if isinstance(value, types.CodeType) else value + for value in module.co_consts)) + path = root / f"tests/pyc/padded_value_returns_{count}.{version}.pyc" + path.write_bytes(header + marshal.dumps(padded)) + if count == 3: + # Also exercise a representative padded input in verify_corpus.py. + # Keep the directory named after the interpreter producing the pyc. + corpus = root / "tests/corpus" / ".".join(map(str, sys.version_info[:3])) + corpus.mkdir(parents=True, exist_ok=True) + (corpus / "padded_value_returns_3.py").write_bytes(source) + (corpus / "padded_value_returns_3.pyc").write_bytes(header + marshal.dumps(padded))