Let a spec reach the items nested inside a block (t51)

Let a spec reach the items nested inside a block

Author: peterbjohnson

CHANGELOG.md

@@ -20,6 +20,8 @@
 - An export now names its images by the file names `media/` holds them under. Every markdown image reference a question holds — in the question's text, a part's text, a worked solution, a final answer or an answer box's wording — is rewritten to the file name the image was carried under, so a document writing `![](figures/train.png)` exports as `![](train.png)` beside `media/train.png`, and Lambda Feedback finds the figure where it looks for one. A file two questions use is carried once. Where two different files have the same name, in2lambda names the second as Lambda Feedback's own exports name an image, `question_001_<Title>_0001.png`. Reading an export back is unchanged, because an export already names its images this way.
 - A draft can freeze more than one document, which is how a sheet written as a question file and a separate solutions file is drafted. `in2lambda source add questions.docx solutions.docx` freezes them as source 1 and source 2 of the one `questions.draft.json`, and `in2lambda source add solutions.docx --draft questions.docx` adds a file to an existing draft as its next source. Every block id and line range of a source after the first carries that source's number — `2/b3`, `2/s10:14` — and `1/b3` names the block `b3` names. A field quoted from a source records which source it came from, so that the same line number in two documents is two places. `in2lambda source show` prints each source under its number and its name. `in2lambda spec run` runs the spec over every source: the first source is laid out as the spec's `layout` says, and in any source after it the `question` selector picks out the marker written above each question's solutions while every other match is a solution, paired onto the questions and parts of the first source as `in2lambda convert -a` pairs an answers file. A draft now holds `sources`, a list of `{source, hash, blocks}` in the order they were frozen, in place of those three keys at the top level, so a draft written before this release is refused as a draft in2lambda did not write; `in2lambda source add --start-over` freezes the document again. Four changes to the Python API break existing scripts: `in2lambda.source.add` takes a list of files and the draft to freeze them into; `in2lambda.source.frozen` returns the markdown of every source and `in2lambda.draft.apply` takes the markdown of every source, in place of one; `in2lambda.spec.fields` takes one `(blocks, markdown)` pair per source in place of its `elements` and `markdown` arguments; and `in2lambda.spec.Field` carries the number of the source its ranges are lines of, which every caller constructing a `Field` must pass.
 - Every command that works on a draft takes `--draft`, naming either the draft or the source it was frozen from: `in2lambda source show`, each `in2lambda draft` command, `in2lambda spec run`, `in2lambda validate`, `in2lambda build` and `in2lambda render`. Left off, each command uses the one draft in the current directory, and where the directory holds more than one draft, the command is refused, naming them. `in2lambda spec run` resolves its SPEC from the draft's directory. The Python functions behind those commands take the draft's path in place of a directory: `in2lambda.source.frozen`, `in2lambda.source.show`, `in2lambda.draft.execute`, `in2lambda.draft.replay`, `in2lambda.draft.spec_command`, `in2lambda.draft.report.validate`, `in2lambda.draft.export.build` and `in2lambda.draft.export.render`. `in2lambda.source.draft_of` returns the path of a document's draft, and `in2lambda.source.find` resolves `--draft` for the command line.
+- `in2lambda source add` records the blocks nested inside a block as well as the top-level blocks, so that a sheet written as one list — each question an item, each part an item of a list inside that item — holds a block per part for a spec to select. A list item and a fenced div, which is what pandoc writes a `\begin{solution}` environment as, are the two blocks that hold blocks of their own. A list item or div holding a single element other than a list is one block. A nested block's id is the id of the block holding it and a number, such as `b3.1` and `b3.2.1`, and the block carries a `depth`: 1 for a top-level element, 2 for a block inside one. A block spans the blocks nested inside it. A fenced div spans the lines its content is written on, and not the `:::` lines pandoc wrote around it. `in2lambda source show` prints the ids against the line each block starts on, indented two spaces for each level below the top, so that a line starting a block and the first block inside it carries both ids. A spec selects a nested block by `depth`, as in `part: ListItem depth=2`. A block whose children hold a role holds no role itself, so one selector may match a block and its children. `in2lambda validate` reports the children of a block with children, in place of the block itself, as being in no field. A draft frozen from a document holding nothing nested is written as it was before this release and replays. A draft frozen before this release from a document holding a list does not replay, because freezing that document now records blocks the draft does not hold; `in2lambda source add --start-over` freezes the document again.
+- A spec written before this release can match blocks it did not match before, because a selector that names no `depth` now matches a block nested inside a list item or a `solution` environment as well as a top-level block. The `PartsOneSol` example ships a `solution` environment holding two paragraphs and a display maths, which pandoc writes as a `Para` as well, so a spec reading `question: Para` now matches all three of them; the div holding them then holds no role, and the run writes three questions the document does not write in place of the first question's worked solution. The second question's worked solution survives, because that `solution` environment holds one element and stays one block. Write `depth=1` on each selector — `question: Para depth=1` — to keep the meaning the selector had before this release. One change to the Python API breaks existing scripts: `in2lambda.spec.Selector.matches` takes a list of `(block, element)` pairs in place of a list of panflute elements, so a caller matching a selector itself passes each `in2lambda.source.Block` beside its element.
 - `in2lambda convert FILE PartsOneSol` now exports the worked solution a document writes in a `solution` environment. Pandoc writes that environment as a Div whose classes hold `solution`, and the filter recognised only a Div whose first block reads `Solution`, so a document using the environment exported every question with an empty worked solution. A Div whose first block reads `Solution` is still recognised.
 - Importing `in2lambda.katex_convert` no longer writes a file called `log` into the working directory. That module reports what it changed in an expression to the `in2lambda.katex_convert` logger, which is silent unless the application configures logging.
 - `in2lambda convert` now reads a .docx that holds an image. in2lambda looks in the document for the directories a `\graphicspath` names, and read the document as UTF-8 text to find them. A .docx is a zip file, so converting a Word document holding a figure raised `UnicodeDecodeError`. in2lambda now reads a document that is not UTF-8 text as naming no directory, which is what a .docx names.

docs/source/drafts.md

@@ -105,8 +105,10 @@ $ cat sheet.draft.json
 }
 ```
 
-A block is one top-level element pandoc found - a heading, a paragraph, a list item - with the
-lines it spans and an id to quote it by. `fields` holds the questions, parts and solutions
+A block is one element pandoc found - a heading, a paragraph, a list item - with the lines it
+spans and an id to quote it by. A list item, or a `\begin{solution}` environment, holds elements
+of its own, and each of those is a block too: `b3` holds `b3.1` and `b3.2`, and `b3` spans them.
+[Specs](spec.md) says more about the nested ones. `fields` holds the questions, parts and solutions
 written from the sheet, and `log` holds the commands that wrote them. Both are empty until a spec
 or a command fills them in.
 
@@ -143,8 +145,9 @@ b8  15  The flow rate is $Q = \pi d^2 v / 4$.
 ```
 
 `in2lambda source show` prints the frozen markdown numbered, with the id of each block against
-the line it starts on. A command names a block by its id, `b3`, a line of the source, `s5`, or a
-range of lines, `s15:16`.
+the line it starts on. A block and the first block nested inside it start on the same line, so a
+line may carry several ids, and the margin is indented two spaces for each level of nesting. A
+command names a block by its id, `b3`, a line of the source, `s5`, or a range of lines, `s15:16`.
 
 ## Fill the draft in from a spec
 

docs/source/spec.md

@@ -71,13 +71,14 @@ A selector is a block type followed by any number of constraints:
 ```
 
 The type is a pandoc element: `Header`, `Para`, `ListItem`. A selector that omits the type matches
-any block. A constraint names one of three attributes:
+any block. A constraint names one of four attributes:
 
 | Attribute | Meaning |
 |-----------|---------|
 | `level`   | A heading's level. `level=2` matches `##`. |
 | `text`    | The whole block as text, with the markup removed. |
 | `label`   | The first word of that text, which usually numbers a question. |
+| `depth`   | How deep the block sits. `depth=1` matches a top-level element of the document, `depth=2` a block nested inside one. See [Nested blocks](#nested-blocks). |
 
 `=` matches the whole value. `~` matches a regular expression anywhere in the value. `after
 SELECTOR,` requires the block to follow the first block that the named selector matches, which is
@@ -92,6 +93,45 @@ reads `(a)` as a list marker. Its value is dedented as pandoc reads the item: th
 the first line, and the same width of indentation off every line below it. Use `strip` for the
 labels pandoc does not read as a marker, such as `Q1. ` and `Solution: `.
 
+(nested-blocks)=
+## Nested blocks
+
+Many sheets are written as one list: each question is an item, and the parts of a question are a
+list nested inside that item. `in2lambda source add` records the blocks inside a block as well as
+the top-level blocks, so that a selector reaches a part.
+
+A list item and a fenced div, which is what pandoc writes a `\begin{solution}` environment as, are
+the two blocks that hold blocks of their own. A list item or div holding a single element other
+than a list is that element, and stays one block. A nested block's id is the id of the block
+holding it and a number: `b3` holds `b3.1` and `b3.2`, and `b3.2` holds `b3.2.1`. `in2lambda source
+show` prints the ids against the line each block starts on, indented two spaces for each level
+below the top.
+
+A block spans the blocks nested inside it, so `depth` distinguishes a question from its parts:
+
+```yaml
+question: Para depth=2
+part:     Para depth=3
+solution: Div
+layout:   PartSolPartSol
+```
+
+That spec reads a sheet whose questions are top-level list items. The question is the item's own
+paragraph `b3.1`, and not the whole item `b3`. The `PartSolPartSol` layout pairs each solution
+with the part written above it.
+
+A block whose children hold a role holds no role itself. Writing `b3` into `q1.text` and `b3.2`
+into `q1.p1.text` would be two fields quoted from the same lines, which no command writes. A spec
+quotes the item's own paragraphs into the question and the nested items into the parts. So one
+selector may match a block and its children, and in2lambda assigns the role to the children.
+
+A fenced div spans the lines its content is written on. The `:::` lines pandoc wrote around the
+content are pandoc's, as a list marker is, and no field quotes them.
+
+`in2lambda validate` reports the children of a block with children, in place of the block itself,
+as being in no field. A block with children spans the blank lines and the fences between those
+children, and no field can quote those lines.
+
 (predicates)=
 ## Predicates
 

in2lambda/draft/__init__.py

@@ -18,14 +18,15 @@
 import in2lambda.spec
 from in2lambda.draft.report import _order, _where, checks, overlapping, uncovered
 from in2lambda.source import (
+    Block,
     SourceError,
     _digest,
     _elements,
     _numbered,
     _require_conversion_tools,
     blocks,
-    dedented,
     frozen,
+    quoted,
     save,
     serialise,
 )
@@ -548,24 +549,38 @@ def _quoted(
 ) -> str:
     """Lines of one frozen source as a field holds them.
 
-    Lines quoted out of a list item are dedented by the item's own indentation, which the
-    markdown requires and the author did not write. The ranges still name the source
-    lines. The block the lines fall in decides whether they are dedented, and the text
-    does not, so that a paragraph reading like a list item is quoted as it is written.
+    Lines quoted out of a list item, or out of a block nested inside a list item, are
+    dedented by the indentation the markdown requires and the author did not write. The
+    ranges still name the source lines. The block the lines fall in decides whether they
+    are dedented, and the text does not, so that a paragraph reading like a list item is
+    quoted as it is written.
     """
     text = "\n".join(markdown.splitlines()[start - 1 : end])
-    # Blocks do not overlap, so the block holding the first line is the block the lines
-    # belong to. A nested item falls in that block as well, because only a top-level item
-    # is a block of its own.
+    # The innermost block holding the first line, which is the last block in the list to
+    # hold it, because in2lambda writes a block after the block holding it. A block and
+    # the block holding it stand at the same indentation, so either dedents by the same
+    # width; the innermost block states the depth where the block holding it is a fenced
+    # div and not a list item.
     block = next(
         (
             held
-            for held in draft["sources"][source - 1]["blocks"]
+            for held in reversed(draft["sources"][source - 1]["blocks"])
             if held["start"] <= start <= held["end"]
         ),
         None,
     )
-    return dedented(text) if block and block["type"] == "list item" else text
+    if block is None:
+        return text
+    return quoted(
+        text,
+        Block(
+            block["id"],
+            block["type"],
+            block["start"],
+            block["end"],
+            block.get("depth", 1),
+        ),
+    )
 
 
 def _next(draft: dict[str, Any], prefix: str) -> str:

in2lambda/draft/report.py

@@ -110,6 +110,11 @@ def uncovered(draft: dict[str, Any]) -> list[Finding]:
     block by the lines the fields were taken from and not by the block's name, so that an
     ignore of a whole block covers both halves of a block `split block` has cut in two.
 
+    A block with blocks nested inside it is not reported; its children are. Such a block
+    spans its children and the blank lines and fences pandoc wrote between them, which no
+    field can quote, so reporting it would name lines nobody can account for and say
+    again what each child already says.
+
     Args:
         draft: A draft, as `in2lambda.source.frozen` reads one.
 
@@ -128,7 +133,16 @@ def uncovered(draft: dict[str, Any]) -> list[Finding]:
     }
     found = []
     for number, source in enumerate(draft["sources"], start=1):
+        # A nested block's id is its parent's and a number, so a block whose id is the
+        # stem of another's is one holding blocks of its own.
+        parents = {
+            block["id"].rsplit(".", 1)[0]
+            for block in source["blocks"]
+            if "." in block["id"]
+        }
         for block in source["blocks"]:
+            if block["id"] in parents:
+                continue
             free = _runs(
                 [
                     line