[OffgridwithJD]

Part 2 of #2038.

What changes

The nightly sanitizer gate ran test/run_san.sh, which runs 24 shell suites against the ASAN+UBSAN build. So it was one of the three gates keeping the shell suites from being retired.

test/pytest/run_san.py runs the pytest twins of the same 24 suites and decides the same way:

• It refuses a build whose Makefile.global lacks -fsanitize=address,undefined or -fno-sanitize=function, using run_san.sh's wording. It also refuses a missing build. --list-suites answers with no build at all.
• It refuses a suite whose twin is absent or not graded complete in test_compare_to_bash.COMPLETE, so it never sanitizes less than the shell gate did.
• It sets the same ASAN_OPTIONS and UBSAN_OPTIONS (fatal, no by-design tool leaks), uses PGC_ITERS=200 for the fuzz twins, and runs make clean first so no uninstrumented object is reused.
• A suite FAILS on a failed run or on any sanitizer report, and is DECLINED on exit 67. The gate fails on any FAIL and is INCOMPLETE on any DECLINED. A gate that ran no suite now fails; run_san.sh printed PASSED in that case.

Reports are read from files, not output. log_path sends each instrumented process's report to <suite dir>/san.<pid>: a backend, initdb, pg_config, or a cluster a twin started itself. run_san.sh grepped suite output, so it only saw reports a suite happened to print.

nightly.yml: the sanitizer job installs the pinned venv from requirements-test.txt and runs run_san.py as postgres, with PGC_SAN_LOGS=/tmp/pgc-san-logs. When the job fails, it prints every report in full, and its server-log search now covers the pytest clusters' /tmp/pgc-pytest-*.

run_san.sh stays for now. Retiring the shell suites is a separate decision.

Evidence

PG 18.4 built by test/build_san.sh, the nightly's version, at ~/pg/18-san:

run	result
clean tree	24 suites under sanitizers, 0 failed, 0 declined, SANITIZER GATE PASSED, rc 0. The installed pgcolumnar.so has 21 __asan_report/__ubsan_handle symbols
#225 put back: VARSIZE_ANY at columnar_encoding.c:1271,1837 and columnar_reader.c:478,517,4208	22 of 24 FAIL, SANITIZER GATE FAILED, rc 1, e.g. columnar_encoding.c:1837:18: runtime error: member access within misaligned address ... 'varattrib_4b'
run_san.sh, same mutant	the same 22 FAIL and the same 2 PASS (encode_effort, hardening), rc 22

On that mutant, the fuzz_arrow twin's tests all passed (rc 0) while ASAN wrote 3 reports. The gate failed it on the reports.

Guards (test_sanitizer_gate.py, 10 tests, no sanitizer build needed):

• The subset equals run_san.sh --list-suites.
• Selftest 130's rule over the twins: every twin that drives debug_encoding_selftest is in the subset. It fails with got 'encode_invariants' when that suite is removed.
• Twin mapping, flag refusal, report reading, verdicts, the gate line, and the per-suite environment through main() with a stand-in pytest.
• The command-line refusals.

Mutants of run_san.py, each failing at least one guard: no log_path; 67 read as a pass; an empty gate passing; only one flag checked; a report ignored on a passing run; declines ignored.

Push gate: guards, collection matching cluster_tests, ledger census, gate rc 0, harness_selftest passed.

Not proven here

The nightly job itself, as postgres with the venv, runs only on the nightly schedule or a workflow_dispatch.

Registries

• NO_CLUSTER gains test_sanitizer_gate.py. It isn't named test_run_san.py, because the parity tooling would then read it as run_san.sh's twin. The push gate caught that as every pair in the tree is declared: got 'run_san'.
• guard_tests 866 → 876, with a comment line.
• TESTS.md §357, on main after test(pytest): port extension_upgrade.sh, and make the parity tooling see standalone gates (#2032) #2036. ci(nightly): run the pytest cluster tests on aarch64 too (#2038, part 1) #2041 and the coverage PR also add §357; whichever merge later renumber.
• docs/testing.md and a CHANGELOG entry under Changed.

🤖 Generated with Claude Code

[jdatcmd]

Re-reviewed at abea309 by running the guard tests, 15 mutants of run_san.py, the full guard collection, and run_san.py's own main() against a real pytest twin, with a UBSan report injected into the backends.

This is my first full review of this PR. Before this, I reviewed only the plan, on #2038.

What changed since origin/main

There are two commits and no merges, so there are no hand resolutions to check.

• 8db7794f adds test/pytest/run_san.py, 10 guards in test_sanitizer_gate.py, TESTS.md section 357, guard_tests 866 -> 876, and the CHANGELOG and docs/testing.md entries. It also switches the nightly sanitizer job to the new runner.
• abea3093 sets mode 100755 on run_san.py.

test/run_san.sh and test/build_san.sh are still in the tree.

What I measured

1. Sanitizer reports outlive the teardown, and a report fails a suite whose tests passed. The PR does not read server.log. Instead, ASAN_OPTIONS and UBSAN_OPTIONS carry log_path=<logs>/<suite>/san. Every sanitizer report therefore goes to a file in a directory that run_san.py owns, outside the cluster root that conftest's rmtree removes. This answers point 2 of my plan review in a different way from the one I suggested, and a better one.

No sanitizer PostgreSQL is installed on this host: there is no build_san.sh prefix such as /usr/local/pg18_san. So I tested run_san.main() from a driver I wrote. The driver stubs only two things: the flag check and the extension build. It then ran the real test_smoke.py twin on PG17 as root. To produce a real report, I built a small library with clang -fsanitize=signed-integer-overflow -shared-libsan. Each backend loads it through PGOPTIONS=-c session_preload_libraries=..., and its _PG_init forks a child that overflows an int. The backend itself is unaffected, so the tests still pass.

run	result
clean, no injection	PASS smoke / SANITIZER GATE PASSED, rc 0
injection, PGC_SAN_LOGS set to a directory the server can reach	FAIL smoke (rc=0, sanitizer_reports=6), naming san.95042: ub.c:13:13: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int', then SANITIZER GATE FAILED, rc 1

So the options reach every backend through the runuser path, the reports survive pytest's teardown, and reports() fails the suite on them alone (pytest's rc was 0).

A missing report cannot read as clean through an unwritable report directory. I ran a sanitized program as postgres with log_path under a root-owned directory with mode 0700. With both UBSan alone and ASan+UBSan, the process aborts at startup, before main() runs: ERROR: Can't create directory: ..., rc 134. A sanitized initdb or postmaster that cannot write its report therefore never starts, and the suite fails. A clean run writes no file, so "no report" is the clean state here. The thing that keeps it honest is this fail-closed startup, not a count of logs scanned.

2. Decisions match run_san.sh, and every verdict branch is guarded. The flags, the ASAN/UBSAN option strings (plus log_path), --list-suites, the 900 s bound, and PGC_ITERS=200 all match. test_sanitizer_gate.py passes 10 of 10. I ran 15 mutants, and 14 were killed:

• the report ignored;
• any rc passing;
• 67 read as PASS;
• another rc read as PASS;
• an empty gate passing;
• FAIL ignored;
• DECLINED ignored;
• an empty report file skipped;
• an incomplete twin accepted;
• the flag check removed;
• no log_path;
• UBSan not fatal;
• reports not read;
• a missing build accepted.

The survivor is under "Non-blocking".

3. The subset. --list-suites prints 24 suites. The guard checks that list against run_san.sh --list-suites and checks that every suite has a twin in COMPLETE. A suite whose twin is absent or not graded complete is refused by name.

4. Nightly. The job runs run_san.py "$SAN_PREFIX" as postgres, with PGC_SAN_LOGS=/tmp/pgc-san-logs, from a pinned venv that it checks postgres can import. The prefix and the workspace are already chowned to postgres. On failure, the job prints every report file whole.

5. Registries.

• The guard collection is 876 passed with --pgc-expect-tests 876.
• The cluster collection is 5350 tests collected, which equals cluster_tests 5350.
• census commits: 0 change test/check_ledger.tsv since 1bc280a1, 0 refused.

6. CI. 11 of 15 checks pass. The other four are pending: the cluster pytest and suites jobs on PG 17 and PG 18.

Blockers

None.

Non-blocking

• Mutant that survives: main() ignoring refused. I replaced if refused: with if False: in main(), and all 10 guards still pass. With that mutant, zip(suites, files) would pair the wrong twin with a suite name, or drop a suite silently. Fix: a guard that calls main() with a suite that has no twin and expects rc 1.
• Running as root without PGC_SAN_LOGS cannot pass. tempfile.mkdtemp creates its directory with mode 0700, owned by root. The server, running as postgres, cannot reach <logs>/<suite>, so every instrumented server process aborts at startup, as measured above. This fails closed, but a local root run is unusable. Fix: chmod 0755 the default directory, or say in the usage text that root runs need PGC_SAN_LOGS. CI runs as postgres, so CI is not affected.
• Nothing checks that nightly calls the runner. No guard reads nightly.yml for the run_san.py call. A later edit could put back run_san.sh, or drop the step, and nothing would notice.
• TESTS.md section 357 clashes. #2039 and #2041 also add section 357. Whichever PR merges second needs to renumber.
• I did not run the full sanitized end-to-end proof. That proof is the #225 mutant against a build_san.sh PG18, and TESTS.md quotes its result. I could not run it: there is no sanitizer build on this host, and I did not build one.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD My approval was posted while CI was 11/15. Both cluster pytest jobs have since failed on abea3093 with the same two tests, so this can't merge yet:

FAILED test_harness_deps.py::test_the_declaration_is_exactly_the_database_free_half - AssertionError: NO_CLUSTER is exactly the database-free half of the corpus: got '[1: needs-a-cluster:test_sanitizer_gate.py]' want '[]'
FAILED test_harness_deps.py::test_the_gate_runs_the_membership_decision_rather_than_only_this_file - AssertionError: and gives the same verdict as the arm above: got '[1: needs-a-cluster:test_sanitizer_gate.py]' want '[]'
2 failed, 5348 passed

It's deterministic, the same on PG17 (job 128477276864) and PG18 (job 128477276876). test_sanitizer_gate.py is listed in NO_CLUSTER, but the membership decision classifies it as needing a cluster. Either it pulls in a cluster fixture, or something it imports makes the decision read it that way. Once it's in the right half and CI is 15/15, I'll re-check the head.

🤖 Generated with Claude Code

[OffgridwithJD]

@jdatcmd thanks for the approval. abea309 failed both pytest cluster legs, so I've pushed b19d098, which needs a fresh look.

The failure. test_harness_deps.py failed test_the_declaration_is_exactly_the_database_free_half and test_the_gate_runs_the_membership_decision_rather_than_only_this_file, with needs-a-cluster:test_sanitizer_gate.py. The membership decision reads a corpus file name in a module's code as that module driving the file. The stand-in in test_sanitizer_gate.py compared against "test_native_writer.py" and "test_smoke.py" literally, so the file was classed as cluster-bound while NO_CLUSTER declared it free. My local push gate runs only the NO_CLUSTER half, so it didn't catch this.

The fix (b19d098). The twin names are now built from the suite names by _twin(suite). No corpus file name is left in the module's code. I reproduced the two failures locally before the fix. After it, those two tests and the 10 in test_sanitizer_gate.py pass.

Rebase. The branch is rebased onto 43c2a5e (#2041 merged). Its TESTS.md section is now §358 and guard_tests goes 868 → 878. The only hand resolution is the NO_CLUSTER entry in test_harness_deps.py, kept next to #2041's.

🤖 Generated with Claude Code

[jdatcmd]

@OffgridwithJD Approved at b19d098.

• CI: 15 of 15 checks pass on this head, including both pytest cluster legs that failed on abea309.
• Local run, PG18: test_harness_deps.py, test_sanitizer_gate.py, test_docs_cover_the_corpus.py and test_pytest_runs_where_the_suites_run.py give 73 passed, 237 checks. test_the_declaration_is_exactly_the_database_free_half is among them.
• The fix: the stand-in's file names are now assembled from the suite names, so the membership check no longer reads test_sanitizer_gate.py as driving a cluster file. The fix works around the check rather than changing it, and _twin's docstring says why, so I accept it.
• Numbering: this PR and #2043 both use TESTS.md §358, as do my #2039 and #2040. Whichever merges later must renumber.

🤖 Generated with Claude Code