Skip to content

Strict mode: ZDOCS_STRICT reports every document's warnings in the doc-check gate - #36

Open
tobiaskaestner wants to merge 1 commit into
mainfrom
tkaestner/29-strict-mode
Open

tobiaskaestner wants to merge 1 commit into
mainfrom
tkaestner/29-strict-mode

Conversation

@tobiaskaestner

Copy link
Copy Markdown
Contributor

Closes #29

Based on main, independent of #35 (sphinx-needs 9.2): unit suites pass with 8.3 (561) as well. ADR number: this adds ADR 0015. The PRs for #25 and #26 add 0016 and 0017. Each of them adds one line after 0014 in decisions/index.rst, so the second and third to merge get a trivial conflict there. I can rebase on request.

What changed

A cache option ZDOCS_STRICT (default OFF). When it is on:

  • Sphinx runs as before, without -W. A warning does not fail its target, so every document is built, whatever the build tool and its options (no -k 0 needed).
  • add_doc_check runs doccheck.py --strict in the gate (the POST_BUILD step of all-docs and the doc-check target). New check D reads the stage-2 warning file that sphinx-build -w writes for every Sphinx build of the set (<build>/<doc>/build/<builder>.log). It reports each warning and error as <doc> (<builder>): <message>, so every document with a warning is named. The build directory is cut from the messages, so the text is the same on every host and works with doc_check_accepted:.
  • Stage-1 xref.log is not read. Stage 2 parses every document again, and stage 1 has the warnings that xref_builder explains. This matches the documentation guidelines: "doc-check plus a clean stage-two build log, not sphinx-build -W". It also means that needflow's "only supported for HTML" from the stage-1 builder (needflow warns in the stage-1 xref builder and fails strict builds #30) does not fail a strict build.
  • A strict check that finds no warning file at all fails, because a check that read nothing must not pass. ZDOCS_STRICT=ON without an add_doc_check() call is a configure error (cmake_language(DEFER)).
  • Docs: ADR 0015, the consumer contract (ZDOCS_STRICT, plus a hint at ZDOCS_SPHINXOPTS_EXTRA), the CLI reference (check D), and the add_doc_check reference.

Why

With -W, ninja stops at the first failed document. With a dangling link imported from a peer, that is often an innocent document, and the culprit never reports. -k 0 cannot be set from CMake and differs per build tool.

How tested

  • Unit: new test_doccheck_strict.py (5 tests) uses real sphinx-build -w runs (a dangling need link, an undefined label, a clean document and an external one). It covers pass without --strict, every affected document named, no host path, acceptance by text, and that an empty check fails. Full suites: 561 passed (sphinx-needs 8.3, CI pins) and 560 + the known 8.3 control (9.2).
  • Acceptance (zdocs-tests): same result as main (238 passed, the same 11 environment-caused failures, 20 skipped; details in sphinx-needs 9.2; unique prefixes; no group with a document's target name #35). The suite does not turn strict mode on.
  • zdocs' own docs with -DZDOCS_STRICT=ON: doccheck: OK (deploy tree, strict: 2 Sphinx warning file(s) clean).
  • Real consumer (3-repo west workspace, sphinx-needs 9.2 with per-type link schemas). I copied it, replaced its -DZDOCS_SPHINXOPTS_EXTRA="-W;--keep-going" with ZDOCS_STRICT ON, and dropped -- -k 0 from its build command:
    • clean: all three checks pass;
    • a wrong-direction link (needs.undeclared_for_type) plus a dangling link (needs.link_outgoing) in the upstream requirements document: the build fails once at the gate, with 6 findings that name all 5 documents. The culprit is listed with both of its own warnings, and the 4 importers with needs.external_link_outgoing;
    • ZDOCS_STRICT=ON without add_doc_check(): configure fails with the message.

Open questions

  1. Gate instead of -W. The issue proposed -W --keep-going plus -k 0. I put the failure in the gate instead, because CMake cannot make the build tool keep going. Consequence: a single document target (cmake --build . --target se-req-html) does not fail on its warnings in strict mode. Only all-docs/doc-check fail, and Sphinx still prints the warnings. Is that acceptable, or should strict mode also add -W (and accept the stop-at-first-failure behaviour for direct target builds)?
  2. Like -W, the gate sees only the warnings of the documents the last build read again. An incremental build that reads nothing again reports nothing, so CI should build from scratch. This is documented.
  3. Not done: reading sphinx-needs' schema_violations.json (the violations are already in the warning file), and turning strict mode on for zdocs' own doc/ set. It builds clean in strict mode, so this could be a one-line follow-up.
  4. A dangling link is still reported once per importing document. Suppressing needs.external_link_outgoing in strict mode would leave only the culprit, but it would hide real problems in a set where the defining document is not built. Per-document needs imports instead of all-to-all #26 (imports:) reduces the repetition instead.

🤖 Generated with Claude Code

By default a dangling need link, a sphinx-needs schema violation or an
unresolved reference does not fail the build. With -W in
ZDOCS_SPHINXOPTS_EXTRA it does, but the build tool stops at the first
failed document, and with a dangling link imported from a peer that is
often an innocent one: the document with the broken link never reports.

The cache option ZDOCS_STRICT (default OFF) makes the doc-check gate
(add_doc_check: the POST_BUILD step of all-docs and the doc-check
target) run doccheck.py --strict. That reads the stage-2 warning file of
every Sphinx build of the set (<build>/<doc>/build/<builder>.log, written
by sphinx-build -w) and reports each warning and error as
"<doc> (<builder>): <message>", with the build directory cut from the
paths. Sphinx runs without -W, so every document is built whatever the
build tool, and the gate names every document with a warning. Stage-1
files are not read: stage 2 parses every document again.

The findings go through doc_check_accepted: like the others. A strict
check that finds no warning file fails, and ZDOCS_STRICT without an
add_doc_check() call is a configure error.

ADR 0015 records the decision; the consumer contract and the CLI
reference describe the option and the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tobias Kaestner <tobias.kaestner@inovex.de>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A documented strict mode

1 participant