From a79c79e7060cb5bac639ab6eb313e394db202525 Mon Sep 17 00:00:00 2001 From: Christophe Dervieux Date: Tue, 10 Nov 2026 16:51:50 +0100 Subject: [PATCH 1/5] Add triage-ci-failures skill for red PR checks A dependency bump like Pandoc 3.13 turns a PR red across dozens of bucket x OS jobs, each log several hundred KB. Check-run annotations only name the bucket, and `gh run view --log` silently truncates near 750 KB, so finding which tests failed meant pulling raw logs into context. The script takes a PR number (every failing run on the head commit) or a run id, caches failed-job logs per job id via the jobs/ID/logs endpoint, and prints one row per unique failing test with the OSes it failed on and the verify message. When verify hits a missing output file, the row shows the render error from the OUTPUT tail instead. `##[error]` lines are reported only for jobs with no failing test, since smoke jobs echo each failed bucket there. Jobs whose log shape is not parsed (Playwright, ff-matrix are unverified) are listed with their log path. The skill then has the agent group rows by cause and classify each as dependency output drift or a regression before touching assertions or snapshots. On PR #15320 it reports the 18 failing tests plus the runtime-boundary error in about 6 KB. --- .claude/skills/triage-ci-failures/SKILL.md | 40 ++++++ .../triage-ci-failures/scripts/ci-failures.sh | 121 ++++++++++++++++++ 2 files changed, 161 insertions(+) create mode 100644 .claude/skills/triage-ci-failures/SKILL.md create mode 100644 .claude/skills/triage-ci-failures/scripts/ci-failures.sh diff --git a/.claude/skills/triage-ci-failures/SKILL.md b/.claude/skills/triage-ci-failures/SKILL.md new file mode 100644 index 0000000000..c04b02fece --- /dev/null +++ b/.claude/skills/triage-ci-failures/SKILL.md @@ -0,0 +1,40 @@ +--- +name: triage-ci-failures +description: Use when a quarto-cli PR has red CI checks or a workflow run failed, to find which tests failed on which OSes and why, without loading raw job logs into context. Covers dependency-bump PRs (Pandoc, Typst, Deno) where many assertions break at once. +--- + +# Triage CI Failures + +A red PR on quarto-cli fans out into dozens of failed jobs (test buckets × OSes), each log several hundred KB. Check-run annotations only name the bucket, and `gh run view --log` silently truncates near 750 KB. The script below downloads the failed-job logs to a cache directory and prints a compact digest: one row per unique failing test, the OSes it failed on, and the assertion that broke. + +## Step 1: Digest the failures + +```bash +bash .claude/skills/triage-ci-failures/scripts/ci-failures.sh +bash .claude/skills/triage-ci-failures/scripts/ci-failures.sh --run # one run, e.g. a scheduled run on main +``` + +The PR form takes every failing run on the PR's head commit. Logs are cached per job id under `$CI_FAILURES_DIR` (default `${TMPDIR:-/tmp}/quarto-ci-failures`), so re-running after a new push only fetches the new jobs. + +The output has up to three sections: + +- **unique failing tests**: `[OSes] `, then the `[verify]` check and its message. When verify hit a missing output file, the render itself failed, and the row shows the render error from the test's `OUTPUT:` tail instead. +- **step errors**: `##[error]` lines from failed jobs that had no failing test (runtime-boundary, lint, setup). +- **failed jobs with no recognised failure**: a log shape the script does not parse (Playwright and feature-format-matrix jobs are unverified). Grep the printed log path for the cause (`grep -n -E 'Error|FAIL' | head`) rather than reading the log whole. + +Step 1 is done when every failed job is accounted for by a test row, a step error, or a cause found by grep in its log. + +## Step 2: Group rows by cause + +Several rows usually share one cause (one Pandoc change breaks every docx crossref test). Group them, and for each cause note whether it is OS-specific: a test failing on one OS only is a platform bug or a flake, see `dev-docs/debugging-flaky-tests.md`. + +## Step 3: Classify each cause before changing any test + +For each cause, decide **drift** or **regression** before touching an assertion or snapshot: + +- **Drift**: the dependency changed its output in a way that is correct or neutral for users (reordered attributes, hashed bookmark names, new whitespace). Update the assertion or snapshot, and say in the commit which upstream change caused it. +- **Regression**: the rendered result is now wrong for users (missing content, broken cross-reference, a document that no longer renders). Fix Quarto or report upstream; the test stays as it is. + +Decide from the rendered output, not the assertion text: render the test document with the dev build and inspect it (`.claude/rules/testing/smoke-all-tests.md` covers how smoke-all tests declare their checks). For a dependency bump, the dependency's changelog usually names the change. + +Step 3 is done when every cause carries a drift-or-regression verdict with the evidence behind it. Track one work item per cause, so each verdict and fix can be reviewed on its own. diff --git a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh new file mode 100644 index 0000000000..6f3d3e400c --- /dev/null +++ b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# Summarise the failing CI jobs of a PR (or of one workflow run) without +# loading raw logs: one row per unique failing test, deduplicated across OSes. +# +# Usage: ci-failures.sh [repo] +# ci-failures.sh --run [repo] +# +# Logs are cached per job id under $CI_FAILURES_DIR +# (default: ${TMPDIR:-/tmp}/quarto-ci-failures), so re-runs are cheap. +set -euo pipefail + +if [ "${1:-}" = "--run" ]; then + RUNS="${2:?run id required}"; REPO="${3:-quarto-dev/quarto-cli}" + echo "Run $RUNS" +else + PR="${1:?usage: ci-failures.sh | --run [repo]}" + REPO="${2:-quarto-dev/quarto-cli}" + SHA="$(gh pr view "$PR" --repo "$REPO" --json headRefOid --jq .headRefOid)" + RUNS="$(gh run list --repo "$REPO" --commit "$SHA" --limit 100 \ + --json databaseId,conclusion --jq '.[] | select(.conclusion=="failure") | .databaseId')" + echo "PR #$PR head ${SHA:0:9}: failing runs: $(echo $RUNS)" + [ -n "$RUNS" ] || { echo "No failing runs."; exit 0; } +fi + +D="${CI_FAILURES_DIR:-${TMPDIR:-/tmp}/quarto-ci-failures}"; mkdir -p "$D" +JOBS="$D/jobs-$$.tsv"; : > "$JOBS" +trap 'rm -f "$JOBS"' EXIT + +for RUN in $RUNS; do + gh api "repos/$REPO/actions/runs/$RUN/jobs?per_page=100&filter=latest" --paginate \ + --jq '.jobs[] | select(.conclusion=="failure") | [.id, .name] | @tsv' >> "$JOBS" +done +# Linux jobs first, so excerpts show forward-slash paths. +awk -F'\t' '{print ($2 ~ /ubuntu|linux/ ? 0 : 1) "\t" $0}' "$JOBS" \ + | sort -s -k1,1 | cut -f2- > "$JOBS.sorted" && mv "$JOBS.sorted" "$JOBS" +echo "Failed jobs: $(wc -l < "$JOBS" | tr -d ' ') Logs: $D" + +# gh run view --log silently truncates large logs; the jobs/ID/logs endpoint does not. +while IFS=$'\t' read -r id name; do + f="$D/$id.log" + [ -s "$f" ] || gh api --allow-escape-sequences "repos/$REPO/actions/jobs/$id/logs" \ + | sed -e 's/\x1b\[[0-9;]*m//g' -e 's/^[0-9T:.-]*Z //' > "$f" +done < "$JOBS" + +# Pass 1: one record per (failure, job): kind \t key \t os \t job-id \t excerpt (lines joined by \037) +while IFS=$'\t' read -r id name; do + os="$(printf '%s' "$name" | sed -nE 's/.*\(([^)]*)\)$/\1/p')"; os="${os:-$name}" + group="$(printf '%s' "$name" | sed -E 's/ *\([^)]*\)$//')" + awk -v os="$os" -v id="$id" -v group="$group" ' + function norm(s) { sub(/^\[smoke\] > /, "", s); gsub(/\\/, "/", s); return s } + function flush( i, n, out, seen, start) { + if (title == "") return + out = verify (msg != "" ? "\037" msg : "") + if (notfound && nout > 0) { + # Render failed before verify ran: show the render error from OUTPUT instead. + out = out "\037render error (OUTPUT tail):" + n = 0; start = nout + for (i = nout; i >= 1 && n < 6; i--) if (!(outl[i] in seen)) { seen[outl[i]] = 1; n++; start = i } + delete seen + for (i = start; i <= nout; i++) if (!(outl[i] in seen)) { seen[outl[i]] = 1; out = out "\037 " outl[i] } + } + print "T\t" norm(title) "\t" os "\t" id "\t" out + title = ""; verify = ""; msg = ""; notfound = 0; nout = 0; inout = 0; hdr = 0 + } + / \.\.\. FAILED \(/ { t = $0; sub(/ \.\.\. FAILED.*/, "", t); print "T\t" norm(t) "\t" os "\t" id "\t"; next } + /━━━ TEST FAILURE: / { flush(); title = substr($0, index($0, "TEST FAILURE: ") + 14); next } + title != "" { + if ($0 ~ /^\[verify\] > /) { verify = substr($0, 1, 200); next } + if (verify != "" && msg == "" && $0 !~ /^[ \t]*$/) { + msg = substr($0, 1, 240); if ($0 ~ /NotFound|os error 2|cannot find the file/) notfound = 1; next + } + if ($0 ~ /^OUTPUT:/) { inout = 1; next } + if ($0 ~ /Stack trace:|throw new AssertionError/) { inout = 0; next } + if (inout) { + l = $0; sub(/^[ \t]+/, "", l); sub(/[ \t]+$/, "", l) + # Skip the echoed "pandoc" options and "metadata" blocks; each ends at a blank line. + if (l == "pandoc" || l == "metadata") { hdr = 1; next } + if (l == "") { hdr = 0; next } + if (!hdr && l !~ /^[0-9]* *\|/ && l != "^") outl[++nout] = substr(l, 1, 240) + } + } + /^##\[error\]/ && !/Process completed with exit code/ { + print "E\t" substr($0, 10, 300) "\t" os "\t" id "\t" group + } + END { flush() } + ' "$D/$id.log" +done < "$JOBS" > "$D/records-$$.tsv" + +# Pass 2: aggregate into one row per test (and per step error), listing the OSes that failed it. +awk -F'\t' -v jobs="$JOBS" -v dir="$D" ' + function addos(list, os) { return index(", " list ", ", ", " os ", ") ? list : (list == "" ? os : list ", " os) } + BEGIN { while ((getline l < jobs) > 0) { split(l, a, "\t"); jname[a[1]] = a[2]; jorder[++nj] = a[1] } } + { seenjob[$4] = 1 } + $1 == "T" { + testjob[$4] = 1 + if (!($2 in tos)) order[++nt] = $2 + tos[$2] = addos(tos[$2], $3) + if (ex[$2] == "" && $5 != "") ex[$2] = $5 + } + $1 == "E" { ejob[++nr] = $4; ekey[nr] = $5 "\t" $2; eosr[nr] = $3 } + END { + printf "\n== %d unique failing tests\n", nt + for (i = 1; i <= nt; i++) { + t = order[i]; printf "\n[%s] %s\n", tos[t], t + n = split(ex[t], lines, "\037"); for (j = 1; j <= n; j++) if (lines[j] != "") print " " lines[j] + } + # A step error in a job that has failing tests only restates those tests. + for (r = 1; r <= nr; r++) if (!(ejob[r] in testjob)) { + k = ekey[r]; if (!(k in eos)) eorder[++ne] = k + eos[k] = addos(eos[k], eosr[r]) + } + if (ne) printf "\n== %d step errors\n", ne + for (i = 1; i <= ne; i++) { split(eorder[i], p, "\t"); printf "\n[%s] %s\n %s\n", eos[eorder[i]], p[1], p[2] } + m = 0 + for (i = 1; i <= nj; i++) if (!(jorder[i] in seenjob)) { + if (!m++) print "\n== Failed jobs with no recognised failure (grep their log for the cause)" + printf "%s\n %s/%s.log\n", jname[jorder[i]], dir, jorder[i] + } + } +' "$D/records-$$.tsv" +rm -f "$D/records-$$.tsv" From 2101d2d33d90c7e06fc9eee0c763a7f0b6cfbc62 Mon Sep 17 00:00:00 2001 From: Christophe Dervieux Date: Tue, 10 Nov 2026 17:03:22 +0100 Subject: [PATCH 2/5] Keep diagnostics for more test failure shapes in ci-failures.sh - Capture the assertion message up to the stack trace (max 6 lines). noErrors and noErrorsOrWarnings put the actual errors after a generic first line, which the digest dropped. - Treat "No output created message" and "Path ... doesn't exist" as a failed render, like an OS NotFound, and show the OUTPUT tail. - A FAILED test with no TEST FAILURE block (prereq or teardown-only failure, which test.ts rethrows undecorated) now names the log to grep instead of printing an empty row. - Download logs to a .part file and rename on success, so a failed fetch is retried next run instead of being cached truncated. The PR #15320 digest is unchanged. The new shapes were checked on a log written in the test.ts failure format. --- .claude/skills/triage-ci-failures/SKILL.md | 2 +- .../triage-ci-failures/scripts/ci-failures.sh | 26 ++++++++++++++----- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/.claude/skills/triage-ci-failures/SKILL.md b/.claude/skills/triage-ci-failures/SKILL.md index c04b02fece..6b197fe95e 100644 --- a/.claude/skills/triage-ci-failures/SKILL.md +++ b/.claude/skills/triage-ci-failures/SKILL.md @@ -18,7 +18,7 @@ The PR form takes every failing run on the PR's head commit. Logs are cached per The output has up to three sections: -- **unique failing tests**: `[OSes] `, then the `[verify]` check and its message. When verify hit a missing output file, the render itself failed, and the row shows the render error from the test's `OUTPUT:` tail instead. +- **unique failing tests**: `[OSes] `, then the `[verify]` check and its message (up to 6 lines, so `noErrors` diagnostics come through). When verify found no output (`No output created message`, a missing path), the render itself failed, and the row adds the render error from the test's `OUTPUT:` tail. A row with no excerpt failed outside the verify phase (prereq, teardown) and names the log to grep. - **step errors**: `##[error]` lines from failed jobs that had no failing test (runtime-boundary, lint, setup). - **failed jobs with no recognised failure**: a log shape the script does not parse (Playwright and feature-format-matrix jobs are unverified). Grep the printed log path for the cause (`grep -n -E 'Error|FAIL' | head`) rather than reading the log whole. diff --git a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh index 6f3d3e400c..0429b5e437 100644 --- a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh +++ b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh @@ -38,8 +38,10 @@ echo "Failed jobs: $(wc -l < "$JOBS" | tr -d ' ') Logs: $D" # gh run view --log silently truncates large logs; the jobs/ID/logs endpoint does not. while IFS=$'\t' read -r id name; do f="$D/$id.log" - [ -s "$f" ] || gh api --allow-escape-sequences "repos/$REPO/actions/jobs/$id/logs" \ - | sed -e 's/\x1b\[[0-9;]*m//g' -e 's/^[0-9T:.-]*Z //' > "$f" + [ -s "$f" ] && continue + # Publish only a complete download, so a failed fetch is retried next run. + gh api --allow-escape-sequences "repos/$REPO/actions/jobs/$id/logs" \ + | sed -e 's/\x1b\[[0-9;]*m//g' -e 's/^[0-9T:.-]*Z //' > "$f.part" && mv "$f.part" "$f" done < "$JOBS" # Pass 1: one record per (failure, job): kind \t key \t os \t job-id \t excerpt (lines joined by \037) @@ -50,7 +52,7 @@ while IFS=$'\t' read -r id name; do function norm(s) { sub(/^\[smoke\] > /, "", s); gsub(/\\/, "/", s); return s } function flush( i, n, out, seen, start) { if (title == "") return - out = verify (msg != "" ? "\037" msg : "") + out = verify msg if (notfound && nout > 0) { # Render failed before verify ran: show the render error from OUTPUT instead. out = out "\037render error (OUTPUT tail):" @@ -60,14 +62,21 @@ while IFS=$'\t' read -r id name; do for (i = start; i <= nout; i++) if (!(outl[i] in seen)) { seen[outl[i]] = 1; out = out "\037 " outl[i] } } print "T\t" norm(title) "\t" os "\t" id "\t" out - title = ""; verify = ""; msg = ""; notfound = 0; nout = 0; inout = 0; hdr = 0 + title = ""; verify = ""; msg = ""; nmsg = 0; inmsg = 0; notfound = 0; nout = 0; inout = 0; hdr = 0 } / \.\.\. FAILED \(/ { t = $0; sub(/ \.\.\. FAILED.*/, "", t); print "T\t" norm(t) "\t" os "\t" id "\t"; next } /━━━ TEST FAILURE: / { flush(); title = substr($0, index($0, "TEST FAILURE: ") + 14); next } title != "" { - if ($0 ~ /^\[verify\] > /) { verify = substr($0, 1, 200); next } - if (verify != "" && msg == "" && $0 !~ /^[ \t]*$/) { - msg = substr($0, 1, 240); if ($0 ~ /NotFound|os error 2|cannot find the file/) notfound = 1; next + if ($0 ~ /^\[verify\] > /) { verify = substr($0, 1, 200); inmsg = 1; next } + # The assertion message runs from the first non-blank line after [verify] to the + # stack trace; noErrors-style messages carry the real diagnostics on later lines. + if (inmsg) { + if ($0 ~ /^[ \t]*$/) { if (nmsg) inmsg = 0; next } + if ($0 ~ /^(AssertionError|NotFound|Error|TypeError|Path): |^[ \t]+at /) { inmsg = 0; next } + if (nmsg < 6) { msg = msg "\037" substr($0, 1, 240); nmsg++ } + # Render failed, so no output file: the render error is in OUTPUT. + if ($0 ~ /NotFound|os error 2|cannot find the file|No output created message|^Path .* doesn.t exist/) notfound = 1 + next } if ($0 ~ /^OUTPUT:/) { inout = 1; next } if ($0 ~ /Stack trace:|throw new AssertionError/) { inout = 0; next } @@ -96,6 +105,7 @@ awk -F'\t' -v jobs="$JOBS" -v dir="$D" ' if (!($2 in tos)) order[++nt] = $2 tos[$2] = addos(tos[$2], $3) if (ex[$2] == "" && $5 != "") ex[$2] = $5 + if (!($2 in tjob)) tjob[$2] = $4 } $1 == "E" { ejob[++nr] = $4; ekey[nr] = $5 "\t" $2; eosr[nr] = $3 } END { @@ -103,6 +113,8 @@ awk -F'\t' -v jobs="$JOBS" -v dir="$D" ' for (i = 1; i <= nt; i++) { t = order[i]; printf "\n[%s] %s\n", tos[t], t n = split(ex[t], lines, "\037"); for (j = 1; j <= n; j++) if (lines[j] != "") print " " lines[j] + # prereq and teardown-only failures print no TEST FAILURE block. + if (ex[t] == "") printf " no TEST FAILURE block (prereq or teardown failure?); grep %s/%s.log\n", dir, tjob[t] } # A step error in a job that has failing tests only restates those tests. for (r = 1; r <= nr; r++) if (!(ejob[r] in testjob)) { From ed30185746ae0f926987cb0d3b65804987788a72 Mon Sep 17 00:00:00 2001 From: Christophe Dervieux Date: Tue, 10 Nov 2026 17:12:12 +0100 Subject: [PATCH 3/5] Keep assertion messages across blank lines in ci-failures.sh A Sass failure reaches noErrorsOrWarnings as "Theme file compilation failed:\n\n" (src/core/dart-sass.ts), and the diagnostics start with "Error:". Stopping the message at a blank line, or at any "Error:" line, dropped the compiler error. The message now ends at the stack trace, recognised as "at" frames or as ": ", the line JS stacks begin with. Blank lines are skipped. XPath and layout verifiers append the whole document after "... follows:", so capture ends at that line; the blank line had been doing that job before. The PR #15320 digest only gains the "Path:" line of the NotFound message. --- .../triage-ci-failures/scripts/ci-failures.sh | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh index 0429b5e437..1996f1e7bc 100644 --- a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh +++ b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh @@ -62,18 +62,25 @@ while IFS=$'\t' read -r id name; do for (i = start; i <= nout; i++) if (!(outl[i] in seen)) { seen[outl[i]] = 1; out = out "\037 " outl[i] } } print "T\t" norm(title) "\t" os "\t" id "\t" out - title = ""; verify = ""; msg = ""; nmsg = 0; inmsg = 0; notfound = 0; nout = 0; inout = 0; hdr = 0 + title = ""; verify = ""; msg = ""; m1 = ""; nmsg = 0; inmsg = 0; notfound = 0; nout = 0; inout = 0; hdr = 0 } / \.\.\. FAILED \(/ { t = $0; sub(/ \.\.\. FAILED.*/, "", t); print "T\t" norm(t) "\t" os "\t" id "\t"; next } /━━━ TEST FAILURE: / { flush(); title = substr($0, index($0, "TEST FAILURE: ") + 14); next } title != "" { if ($0 ~ /^\[verify\] > /) { verify = substr($0, 1, 200); inmsg = 1; next } - # The assertion message runs from the first non-blank line after [verify] to the - # stack trace; noErrors-style messages carry the real diagnostics on later lines. + # The assertion message runs from [verify] to the stack trace, whose first line is + # ": ". noErrors-style messages carry the real + # diagnostics on later lines, possibly after blank lines. if (inmsg) { - if ($0 ~ /^[ \t]*$/) { if (nmsg) inmsg = 0; next } - if ($0 ~ /^(AssertionError|NotFound|Error|TypeError|Path): |^[ \t]+at /) { inmsg = 0; next } + if ($0 ~ /^[ \t]*$/) next + if ($0 ~ /^OUTPUT:/) inmsg = 0 + else if ($0 ~ /^[ \t]+at / || (m1 != "" && $0 ~ /^[A-Za-z]+: / && substr($0, index($0, ": ") + 2) == m1)) { inmsg = 0; next } + } + if (inmsg) { + if (m1 == "") m1 = $0 if (nmsg < 6) { msg = msg "\037" substr($0, 1, 240); nmsg++ } + # XPath and layout verifiers end with "... follows:" and then dump the whole document. + if ($0 ~ /follows:[ \t]*$/) inmsg = 0 # Render failed, so no output file: the render error is in OUTPUT. if ($0 ~ /NotFound|os error 2|cannot find the file|No output created message|^Path .* doesn.t exist/) notfound = 1 next From e976cd4880405c71cd8019bfed87ef75e31b76db Mon Sep 17 00:00:00 2001 From: Christophe Dervieux Date: Tue, 10 Nov 2026 17:19:13 +0100 Subject: [PATCH 4/5] Read the OS label from suffixed smoke job names in ci-failures.sh test-smokes.yml names jobs "Run smoke ()" plus an optional " with timed file" or " - with some extra R packages". Requiring the parenthesis at the end of the name made those rows show the full job name instead of the OS. --- .claude/skills/triage-ci-failures/scripts/ci-failures.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh index 1996f1e7bc..adbdd9c44b 100644 --- a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh +++ b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh @@ -46,8 +46,9 @@ done < "$JOBS" # Pass 1: one record per (failure, job): kind \t key \t os \t job-id \t excerpt (lines joined by \037) while IFS=$'\t' read -r id name; do - os="$(printf '%s' "$name" | sed -nE 's/.*\(([^)]*)\)$/\1/p')"; os="${os:-$name}" - group="$(printf '%s' "$name" | sed -E 's/ *\([^)]*\)$//')" + # "Run smoke ()" may carry a suffix such as " with timed file" (test-smokes.yml). + os="$(printf '%s' "$name" | sed -nE 's/.*\(([^)]*)\).*/\1/p')"; os="${os:-$name}" + group="$(printf '%s' "$name" | sed -E 's/ *\([^)]*\)//')" awk -v os="$os" -v id="$id" -v group="$group" ' function norm(s) { sub(/^\[smoke\] > /, "", s); gsub(/\\/, "/", s); return s } function flush( i, n, out, seen, start) { From dfcb68e82befde549edc7f28ceb6a058145ee947 Mon Sep 17 00:00:00 2001 From: Christophe Dervieux Date: Tue, 10 Nov 2026 17:24:39 +0100 Subject: [PATCH 5/5] Keep CI digest excerpts traceable across OSes A merged test row can contain failures with different causes. Identify the representative excerpt's OS and job and list all contributing logs so those differences can be checked before grouping failures by cause. Clarify that missing output and failures limited to one OS require investigation before assigning a cause. Verified on PR #15320: 18 unique failing tests and the runtime-boundary error, with the render diagnostics preserved in an 8.5 KB digest. Shell syntax and diff checks pass. --- .claude/skills/triage-ci-failures/SKILL.md | 4 ++-- .../triage-ci-failures/scripts/ci-failures.sh | 20 ++++++++++++------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/.claude/skills/triage-ci-failures/SKILL.md b/.claude/skills/triage-ci-failures/SKILL.md index 6b197fe95e..6bb6eb3149 100644 --- a/.claude/skills/triage-ci-failures/SKILL.md +++ b/.claude/skills/triage-ci-failures/SKILL.md @@ -18,7 +18,7 @@ The PR form takes every failing run on the PR's head commit. Logs are cached per The output has up to three sections: -- **unique failing tests**: `[OSes] `, then the `[verify]` check and its message (up to 6 lines, so `noErrors` diagnostics come through). When verify found no output (`No output created message`, a missing path), the render itself failed, and the row adds the render error from the test's `OUTPUT:` tail. A row with no excerpt failed outside the verify phase (prereq, teardown) and names the log to grep. +- **unique failing tests**: `[OSes] `, then one representative `[verify]` check and its message (up to 6 lines, including `noErrors` diagnostics). The excerpt identifies its OS and job; the row lists every contributing log in the cache directory. Missing output (`No output created message`, a missing path) may indicate a render failure, so the row includes the captured render output from the test's `OUTPUT:` tail. A row with no excerpt may have failed during prereq or teardown and names the log to grep. - **step errors**: `##[error]` lines from failed jobs that had no failing test (runtime-boundary, lint, setup). - **failed jobs with no recognised failure**: a log shape the script does not parse (Playwright and feature-format-matrix jobs are unverified). Grep the printed log path for the cause (`grep -n -E 'Error|FAIL' | head`) rather than reading the log whole. @@ -26,7 +26,7 @@ Step 1 is done when every failed job is accounted for by a test row, a step erro ## Step 2: Group rows by cause -Several rows usually share one cause (one Pandoc change breaks every docx crossref test). Group them, and for each cause note whether it is OS-specific: a test failing on one OS only is a platform bug or a flake, see `dev-docs/debugging-flaky-tests.md`. +Several rows may share one cause, such as a Pandoc change affecting several docx crossref tests. Compare diagnostics in the listed logs before grouping failures across OSes; the representative excerpt may not describe every failure. Investigate failures limited to one OS for platform differences or intermittent behavior; see `dev-docs/debugging-flaky-tests.md`. ## Step 3: Classify each cause before changing any test diff --git a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh index adbdd9c44b..1c1079fcac 100644 --- a/.claude/skills/triage-ci-failures/scripts/ci-failures.sh +++ b/.claude/skills/triage-ci-failures/scripts/ci-failures.sh @@ -6,7 +6,7 @@ # ci-failures.sh --run [repo] # # Logs are cached per job id under $CI_FAILURES_DIR -# (default: ${TMPDIR:-/tmp}/quarto-ci-failures), so re-runs are cheap. +# (default: ${TMPDIR:-/tmp}/quarto-ci-failures). set -euo pipefail if [ "${1:-}" = "--run" ]; then @@ -35,7 +35,7 @@ awk -F'\t' '{print ($2 ~ /ubuntu|linux/ ? 0 : 1) "\t" $0}' "$JOBS" \ | sort -s -k1,1 | cut -f2- > "$JOBS.sorted" && mv "$JOBS.sorted" "$JOBS" echo "Failed jobs: $(wc -l < "$JOBS" | tr -d ' ') Logs: $D" -# gh run view --log silently truncates large logs; the jobs/ID/logs endpoint does not. +# Download job logs directly to avoid truncation. while IFS=$'\t' read -r id name; do f="$D/$id.log" [ -s "$f" ] && continue @@ -55,8 +55,8 @@ while IFS=$'\t' read -r id name; do if (title == "") return out = verify msg if (notfound && nout > 0) { - # Render failed before verify ran: show the render error from OUTPUT instead. - out = out "\037render error (OUTPUT tail):" + # Missing output: include the captured render output. + out = out "\037render output (OUTPUT tail):" n = 0; start = nout for (i = nout; i >= 1 && n < 6; i--) if (!(outl[i] in seen)) { seen[outl[i]] = 1; n++; start = i } delete seen @@ -82,7 +82,7 @@ while IFS=$'\t' read -r id name; do if (nmsg < 6) { msg = msg "\037" substr($0, 1, 240); nmsg++ } # XPath and layout verifiers end with "... follows:" and then dump the whole document. if ($0 ~ /follows:[ \t]*$/) inmsg = 0 - # Render failed, so no output file: the render error is in OUTPUT. + # Missing output: include the captured render output. if ($0 ~ /NotFound|os error 2|cannot find the file|No output created message|^Path .* doesn.t exist/) notfound = 1 next } @@ -112,19 +112,25 @@ awk -F'\t' -v jobs="$JOBS" -v dir="$D" ' testjob[$4] = 1 if (!($2 in tos)) order[++nt] = $2 tos[$2] = addos(tos[$2], $3) - if (ex[$2] == "" && $5 != "") ex[$2] = $5 + if (ex[$2] == "" && $5 != "") { ex[$2] = $5; exos[$2] = $3; exjob[$2] = $4 } if (!($2 in tjob)) tjob[$2] = $4 + if (!logseen[$2, $4]) { + logseen[$2, $4] = 1 + tlogs[$2] = tlogs[$2] (tlogs[$2] == "" ? "" : "; ") "[" $3 "] " $4 ".log" + } } $1 == "E" { ejob[++nr] = $4; ekey[nr] = $5 "\t" $2; eosr[nr] = $3 } END { printf "\n== %d unique failing tests\n", nt for (i = 1; i <= nt; i++) { t = order[i]; printf "\n[%s] %s\n", tos[t], t + if (ex[t] != "") printf " representative excerpt (%s, job %s):\n", exos[t], exjob[t] n = split(ex[t], lines, "\037"); for (j = 1; j <= n; j++) if (lines[j] != "") print " " lines[j] + printf " logs: %s\n", tlogs[t] # prereq and teardown-only failures print no TEST FAILURE block. if (ex[t] == "") printf " no TEST FAILURE block (prereq or teardown failure?); grep %s/%s.log\n", dir, tjob[t] } - # A step error in a job that has failing tests only restates those tests. + # Hide step annotations when this job already has test rows. for (r = 1; r <= nr; r++) if (!(ejob[r] in testjob)) { k = ekey[r]; if (!(k in eos)) eorder[++ne] = k eos[k] = addos(eos[k], eosr[r])