Skip to content

fix: derive notebook cell boundaries from the AST, not a line prefix - #246

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/notebook-quotes-string-literal
Aug 20, 2026
Merged

fix: derive notebook cell boundaries from the AST, not a line prefix#246
Jammy2211 merged 2 commits into
mainfrom
feature/notebook-quotes-string-literal

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

add_notebook_quotes decided what was a narrative docstring with a line-prefix
test (line.startswith('"""')). A triple-quoted string literal assigned in
code
defeats it: the opener is indented behind s = and so is invisible to
the test, but the closing delimiter sits at column 0 and does match. That flipped
the docstring state and inverted every cell boundary after it.

Splitting the reproducer's converted output the way ipynb-py-convert's py2nb
does (on the literal "\n\n# %%\n"):

cell kind result
1 code SyntaxError: unterminated triple-quoted string literal
2 markdown print(s) — real code rendered as narrative prose

Segmentation now comes from _narrative_docstring_ranges, which parses the
script and takes the module-level bare string expression statements written at
column 0 with a triple-quote delimiter. ast distinguishes a docstring from a
string bound to a name by node type, so the confusion cannot recur.

Latent, not shipping. Scanning every .py in the four workspaces, their
_test / _developer siblings and the three HowTo repos found exactly one file
with the trigger shape — autolens_workspace_test/gallery/gallery_build.py:42
(a module-level CSS = """ block). gallery/ sits outside scripts/ and
iter_script_paths only walks scripts/, so it is never converted. That is why
it was left out of #211 rather than fixed there.

Two further findings while fixing it:

  • The parsed span also closes an indented closing delimiter, which the
    prefix test missed entirely — the block never closed and everything after was
    swallowed into the markdown cell. This is the mirror of the opener defect in
    fix: notebook generation mangles a docstring that follows code; drop the Finish. hack #211. Three files carry the shape (autocti_workspace_test/legacy/tvac/),
    all outside scripts/.
  • strip_env_declarations was not a live defect. It carried the same
    prefix assumption at the same line, and the false "opener" match does fire —
    but every non-__Env__ path emits the block verbatim, so the misparse was
    absorbed. It is migrated to the shared helper as hardening, so there is one
    segmentation source rather than two tests that can drift apart, not because it
    was broken.

Closes #244.

API Changes

autohands.add_notebook_quotes gains two loud failures where it previously
emitted a silently broken cell — an unparseable source script, and a column-0
single-line docstring. Both are ValueError, matching the stray-# %% guard
added in #214. Cell segmentation for every currently-converted script is
byte-for-byte unchanged. navigator.py is untouched and inherits the fix by
delegation. See full details below.

Test Plan

  • pytest PyAutoHands/tests/349 passed (was 340; +9 new).
  • Control test: the 8 new behavioural tests were run against the
    pre-fix module and all 8 fail, so they pin the regression rather than
    passing vacuously. The 9th (test_strip_leaves_a_code_string_literal_untouched)
    passes before and after — correctly, since strip_env_declarations was
    never a live defect; it is a pin, not a regression.
  • Zero-diff proof: old and new add_notebook_quotes were run over all
    693 converted scripts in the seven artifact-bearing repos
    (auto{lens,galaxy,fit,cti}_workspace, HowTo{Lens,Galaxy,Fit}) and
    compared byte-for-byte — 0 output differences, 0 new raises. No live
    script has the shape, so a correct fix must change nothing, and this checks
    that directly rather than via a regeneration that could mask a diff.
  • Boundary-divergence audit across all 1517 workspace .py files: 18
    files where old and new disagree, all outside scripts/ (i.e. none in
    the converted set). Each was inspected and the new behaviour is correct in
    every case — 15 are code string literals the old test wrongly flagged, 3
    are indented closers the old test wrongly missed.
  • navigator catalogues unchanged — _docstring_blocks is a pure function
    of add_notebook_quotes output, which is byte-identical for every
    catalogued script. test_check_navigator.py green.
Full API Changes (for automation & release notes)

Added

  • add_notebook_quotes._narrative_docstring_ranges(lines) — private. Returns
    (start, end) 0-based line-index pairs for each module-level narrative
    docstring block, derived from ast.parse.

Changed Behaviour

  • add_notebook_quotes.add_notebook_quotes(lines) — cell boundaries are now the
    delimiter lines of AST-derived docstring blocks instead of any line beginning
    with """ / '''. A triple-quoted string literal bound to a name is no longer
    a cell boundary; an indented closing delimiter now correctly closes its block.
    Output is unchanged for every currently-converted script.
  • add_notebook_quotes.strip_env_declarations(lines) — locates docstring blocks
    via the same helper. No behavioural change on any real input; the removed
    forward-scan also removes the now-unreachable unterminated-block branch.

Raises (new)

  • ValueError — source script does not parse as Python. Previously the prefix
    test would proceed and emit a mangled artefact.
  • ValueError — column-0 single-line docstring, whose delimiters share a line
    and so cannot bracket a cell. Zero occurrences across all 1517 workspace files.

Migration

None required. No workspace script is affected; both new ValueErrors are
raised only by shapes that do not occur in any current script.

Generated by the PyAutoLabs agent workflow.

`add_notebook_quotes` decided what was a narrative docstring with a line-prefix
test, so a triple-quoted string literal *assigned in code* fooled it: the opener
is indented behind `s = ` and invisible to the test, but the closing delimiter
sits at column 0 and matched. That flipped the docstring state and inverted every
cell boundary after it — the enclosing code cell became an unterminated-string
SyntaxError and the code that followed was emitted as narrative prose.

Segmentation now comes from `_narrative_docstring_ranges`, which parses the
script and takes the module-level bare string expression statements. `ast` tells
a docstring from a string bound to a name by node type, so the confusion cannot
recur. `strip_env_declarations` moves to the same helper — it carried the same
prefix assumption, absorbed rather than acted on, and there is now one
segmentation source instead of two tests that can drift. `navigator.py` needs no
change; it already reads its blocks back out of `add_notebook_quotes`.

The parsed span also closes an *indented* closing delimiter, which the prefix
test missed — the mirror of the opener defect in #211.

Two shapes now raise instead of silently emitting a broken cell, matching the
stray-`# %%` guard in #214: a script that does not parse, and a column-0
single-line docstring whose delimiters share a line.

Latent, not shipping: the only file workspace-wide with the trigger shape,
`autolens_workspace_test/gallery/gallery_build.py`, sits outside `scripts/` and
is never converted.

Closes #244
@Jammy2211 Jammy2211 added the pending-release Merged PR awaiting inclusion in the next release build label Aug 20, 2026
The gallery-build shape is described generically instead. Organ code must not
carry instance facts; the firewall's rule is to reword, not to allowlist.
@Jammy2211
Jammy2211 merged commit 10a3178 into main Aug 20, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/notebook-quotes-string-literal branch August 20, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release Merged PR awaiting inclusion in the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: derive notebook cell boundaries from the AST, not a line prefix

1 participant