fix: derive notebook cell boundaries from the AST, not a line prefix - #246
Merged
Conversation
`add_notebook_quotes` decided what was a narrative docstring with a line-prefix test, so a triple-quoted string literal *assigned in code* fooled it: the opener is indented behind `s = ` and invisible to the test, but the closing delimiter sits at column 0 and matched. That flipped the docstring state and inverted every cell boundary after it — the enclosing code cell became an unterminated-string SyntaxError and the code that followed was emitted as narrative prose. Segmentation now comes from `_narrative_docstring_ranges`, which parses the script and takes the module-level bare string expression statements. `ast` tells a docstring from a string bound to a name by node type, so the confusion cannot recur. `strip_env_declarations` moves to the same helper — it carried the same prefix assumption, absorbed rather than acted on, and there is now one segmentation source instead of two tests that can drift. `navigator.py` needs no change; it already reads its blocks back out of `add_notebook_quotes`. The parsed span also closes an *indented* closing delimiter, which the prefix test missed — the mirror of the opener defect in #211. Two shapes now raise instead of silently emitting a broken cell, matching the stray-`# %%` guard in #214: a script that does not parse, and a column-0 single-line docstring whose delimiters share a line. Latent, not shipping: the only file workspace-wide with the trigger shape, `autolens_workspace_test/gallery/gallery_build.py`, sits outside `scripts/` and is never converted. Closes #244
The gallery-build shape is described generically instead. Organ code must not carry instance facts; the firewall's rule is to reword, not to allowlist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
add_notebook_quotesdecided what was a narrative docstring with a line-prefixtest (
line.startswith('"""')). A triple-quoted string literal assigned incode defeats it: the opener is indented behind
s =and so is invisible tothe test, but the closing delimiter sits at column 0 and does match. That flipped
the docstring state and inverted every cell boundary after it.
Splitting the reproducer's converted output the way
ipynb-py-convert'spy2nbdoes (on the literal
"\n\n# %%\n"):SyntaxError: unterminated triple-quoted string literalprint(s)— real code rendered as narrative proseSegmentation now comes from
_narrative_docstring_ranges, which parses thescript and takes the module-level bare string expression statements written at
column 0 with a triple-quote delimiter.
astdistinguishes a docstring from astring bound to a name by node type, so the confusion cannot recur.
Latent, not shipping. Scanning every
.pyin the four workspaces, their_test/_developersiblings and the three HowTo repos found exactly one filewith the trigger shape —
autolens_workspace_test/gallery/gallery_build.py:42(a module-level
CSS = """block).gallery/sits outsidescripts/anditer_script_pathsonly walksscripts/, so it is never converted. That is whyit was left out of #211 rather than fixed there.
Two further findings while fixing it:
prefix test missed entirely — the block never closed and everything after was
swallowed into the markdown cell. This is the mirror of the opener defect in
fix: notebook generation mangles a docstring that follows code; drop the Finish. hack #211. Three files carry the shape (
autocti_workspace_test/legacy/tvac/),all outside
scripts/.strip_env_declarationswas not a live defect. It carried the sameprefix assumption at the same line, and the false "opener" match does fire —
but every non-
__Env__path emits the block verbatim, so the misparse wasabsorbed. It is migrated to the shared helper as hardening, so there is one
segmentation source rather than two tests that can drift apart, not because it
was broken.
Closes #244.
API Changes
autohands.add_notebook_quotesgains two loud failures where it previouslyemitted a silently broken cell — an unparseable source script, and a column-0
single-line docstring. Both are
ValueError, matching the stray-# %%guardadded in #214. Cell segmentation for every currently-converted script is
byte-for-byte unchanged.
navigator.pyis untouched and inherits the fix bydelegation. See full details below.
Test Plan
pytest PyAutoHands/tests/— 349 passed (was 340; +9 new).pre-fix module and all 8 fail, so they pin the regression rather than
passing vacuously. The 9th (
test_strip_leaves_a_code_string_literal_untouched)passes before and after — correctly, since
strip_env_declarationswasnever a live defect; it is a pin, not a regression.
add_notebook_quoteswere run over all693 converted scripts in the seven artifact-bearing repos
(
auto{lens,galaxy,fit,cti}_workspace,HowTo{Lens,Galaxy,Fit}) andcompared byte-for-byte — 0 output differences, 0 new raises. No live
script has the shape, so a correct fix must change nothing, and this checks
that directly rather than via a regeneration that could mask a diff.
.pyfiles: 18files where old and new disagree, all outside
scripts/(i.e. none inthe converted set). Each was inspected and the new behaviour is correct in
every case — 15 are code string literals the old test wrongly flagged, 3
are indented closers the old test wrongly missed.
navigatorcatalogues unchanged —_docstring_blocksis a pure functionof
add_notebook_quotesoutput, which is byte-identical for everycatalogued script.
test_check_navigator.pygreen.Full API Changes (for automation & release notes)
Added
add_notebook_quotes._narrative_docstring_ranges(lines)— private. Returns(start, end)0-based line-index pairs for each module-level narrativedocstring block, derived from
ast.parse.Changed Behaviour
add_notebook_quotes.add_notebook_quotes(lines)— cell boundaries are now thedelimiter lines of AST-derived docstring blocks instead of any line beginning
with
"""/'''. A triple-quoted string literal bound to a name is no longera cell boundary; an indented closing delimiter now correctly closes its block.
Output is unchanged for every currently-converted script.
add_notebook_quotes.strip_env_declarations(lines)— locates docstring blocksvia the same helper. No behavioural change on any real input; the removed
forward-scan also removes the now-unreachable unterminated-block branch.
Raises (new)
ValueError— source script does not parse as Python. Previously the prefixtest would proceed and emit a mangled artefact.
ValueError— column-0 single-line docstring, whose delimiters share a lineand so cannot bracket a cell. Zero occurrences across all 1517 workspace files.
Migration
None required. No workspace script is affected; both new
ValueErrors areraised only by shapes that do not occur in any current script.
Generated by the PyAutoLabs agent workflow.