From 6ad3f362cb4796332033f8f804c723e947d084f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 02:53:31 +0000 Subject: [PATCH 1/5] docs(cladistics): issue 303 experiment 8, the application's placement step, and a correction Experiment 8 repeats the placement-evidence test with the command the application's place step runs (RAxML EPA, -f v -m GTRCATI -G 0.2) on a maximum-likelihood reference tree of 200 SILVA sequences. 2,375 of 2,388 entailed verdicts were right; the 13 errors concentrate in the smallest clades (7 of 62 at two or three references, 0 of 1,515 at fifty or more). The scripts are in docs/cladistics/issue-303/epa/. Correction: the page said an entailed verdict survives zooming out to any enclosing clade and a refuted verdict survives zooming in. The proved laws each need a premise on the bounds; the page now states them. The reading of the results no longer says a verdict becomes undecided, not wrong, as the clade shrinks: that held for experiment 7 and not for experiment 8. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01J4QvDiW8qD4f2JDQJ4buGo --- docs/cladistics/issue-303/README.md | 134 ++++++++++++++++-- docs/cladistics/issue-303/epa/01_run.sh | 10 ++ docs/cladistics/issue-303/epa/01_select.jl | 44 ++++++ docs/cladistics/issue-303/epa/02_tree.sh | 11 ++ docs/cladistics/issue-303/epa/03_prep.jl | 70 +++++++++ docs/cladistics/issue-303/epa/04_score.jl | 79 +++++++++++ docs/cladistics/issue-303/epa/05_run_epa.sh | 11 ++ docs/cladistics/issue-303/epa/06_tables.jl | 60 ++++++++ .../issue-303/epa/07_selfcheck_truth.jl | 44 ++++++ docs/cladistics/issue-303/epa/08_by_depth.jl | 49 +++++++ docs/cladistics/issue-303/epa/lib_nwk.jl | 73 ++++++++++ 11 files changed, 576 insertions(+), 9 deletions(-) create mode 100755 docs/cladistics/issue-303/epa/01_run.sh create mode 100644 docs/cladistics/issue-303/epa/01_select.jl create mode 100755 docs/cladistics/issue-303/epa/02_tree.sh create mode 100644 docs/cladistics/issue-303/epa/03_prep.jl create mode 100644 docs/cladistics/issue-303/epa/04_score.jl create mode 100755 docs/cladistics/issue-303/epa/05_run_epa.sh create mode 100644 docs/cladistics/issue-303/epa/06_tables.jl create mode 100644 docs/cladistics/issue-303/epa/07_selfcheck_truth.jl create mode 100644 docs/cladistics/issue-303/epa/08_by_depth.jl create mode 100644 docs/cladistics/issue-303/epa/lib_nwk.jl diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 2722bf63..9a1189a1 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -30,7 +30,10 @@ tested reached 60%. An earlier reading of experiment 4, that zoomed-in views kee of well-supported branches, did not survive at 1,024 sequences (66%). Placing an amplicon-length fragment on a reference tree built from full-length sequences does work: clade verdicts from bootstrapped placements were right for 1,105 of 1,106 entailed and 57,116 of 57,128 refuted -verdicts, and pin a read to a clade of median size 18 among 399 references. +verdicts, and pin a read to a clade of median size 18 among 399 references. With the +application's own placement step (RAxML EPA) the picture is sharper and less safe: 2,375 of +2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more +references and 7 errors among 62 on clades of two or three. ## Data @@ -203,15 +206,90 @@ control shows the score can fail. What it does not show is in the limits below. The counting rule that turns per-read verdicts into a per-sample verdict for a clade, with separate bounds for wrongly counted and missed reads, is proved in `metadatastician/MetaManifold-Evidence` (`agda/src/MetaManifold/Evidence/TwoSided.agda`, -not yet merged there). It also proves that an entailed verdict survives zooming out to any -enclosing clade and a refuted verdict survives zooming in. +not yet merged there). It also proves two zoom laws, each under a premise on the bounds. An +entailed verdict survives zooming out to an enclosing clade if the same bound on wrongly +counted reads holds for the larger clade. A refuted verdict survives zooming in to a +contained clade if that clade's bound on missed reads is no larger. An earlier version of +this page stated both without their premises; containment alone is not enough for either. + +### 8. The application's own placement step (`epa/`) + +Experiment 7 placed by least squares. This repeats it with the command the application runs +in its `place` step: `raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2` (RAxML 8.2.12). 200 +full-length sequences from the same SILVA alignment give a maximum-likelihood reference tree +(IQ-TREE 2.0.7, GTR+G4). Each of the 200 in turn is pruned from the tree and its 240-base V4 +fragment is placed back. For every clade of the pruned tree, the mass is the sum of the +likelihood weight ratios of the placements inside it, as RAxML reports them. The verdict is +entailed at mass 0.99 or more, refuted at 0.01 or less, unresolved between. + +| | Amplicon fragment | Whole sequence (positive control) | A different sequence placed (negative control) | +| --- | --- | --- | --- | +| Placed on exactly the true edge | 91 of 200 (45.5%) | 188 of 200 (94.0%) | 0 of 200 | +| Within two edges of it | 76.5% | 99.5% | 1.0% | +| Entailed verdicts that are right | 2,375 of 2,388 (99.46%) | 3,065 of 3,065 | 1,507 of 3,176 (47.45%) | +| Refuted verdicts that are right | 35,988 of 36,016 (99.92%) | 36,208 of 36,211 (99.99%) | 34,392 of 35,952 (95.66%) | +| Clades that truly hold the sequence: entailed / unresolved / wrongly refuted | 75.2% / 23.9% / 0.9% | 97.0% / 2.9% / 0.1% | 47.7% / 2.9% / 49.4% | +| Smallest entailed clade, median size of 199 | 4 | 4 | 3 | +| Smallest entailed clade holds the truth | 139 of 146 queries (95.2%) | 193 of 193 | 9 of 189 (4.8%) | +| Queries with no entailed clade | 54 | 7 | 11 | + +In this negative control the other sequence stays in the tree, so its fragment is placed +beside it; experiment 7 left it out. The two negative columns are not comparable. + +The errors of the amplicon fragment depend on how small the clade is: + +| Clade size (of 199 references) | Entailed verdicts | Wrong | +| --- | --- | --- | +| 2 to 3 | 62 | 7 | +| 4 to 7 | 132 | 1 | +| 8 to 15 | 228 | 1 | +| 16 to 49 | 451 | 4 | +| 50 to 199 | 1,515 | 0 | + +Verdicts on nested clades of one query are not independent, so the per-query count is the +honest one. Reporting the smallest entailed clade, 7 of 146 queries get a wrong clade. +Reporting the smallest entailed clade of at least 4, 8 or 16 references, 4, 3 or 2 of the +146 do, at median sizes 6.5, 13 and 23. + +RAxML reports few placements per query: a median of 3, at most 7, and a single one for 60 of +the 200. Four of the seven wrong queries had a single placement with weight 0.99 or more. The +mass is therefore lumpy. Clades given mass in (0, 0.05] hold the truth for 9 of 52, +(0.05, 0.25] 22 of 96, (0.25, 0.5] 24 of 46, (0.5, 0.75] 53 of 98, (0.75, 0.95] 280 of 324 +and (0.95, 1] 2,745 of 2,773. + +With the looser rule (entailed at 0.95 or more, refuted at 0.05 or less) 2,745 of 2,773 +entailed verdicts are right (98.99%), and the smallest entailed clade is wrong for 17 of 172 +queries. + +Three self-checks passed. An independent calculation of the truth from path distances on the +unpruned tree matched in all 10 tests (9 distinct queries). In all 600 runs the 395 edge +numbers of the placement file map to 395 distinct branches of the pruned tree. Clade mass +never decreased +from a clade to the clade containing it (39,200 pairs per column). `08_by_depth.jl` +recomputes the tables from the per-clade output separately from `06_tables.jl`; the two +agree. + +What this shows: with the application's placement step, "present in this clade" was right +in every case seen for clades of a quarter of the tree or more, and wrong about one time in +nine for clades of two or three references. At that depth the verdict is confidently wrong, +not unresolved. What it does not show is in the limits below. ## Reading of the results -- **Evidence on placements.** Experiment 7 is the one positive real-data result. Verdicts by - clade are sound at the rates above, and they give the "thumbnail then zoom" idea a form - that holds: the verdict is decided at a coarse clade and becomes undecided, not wrong, as - the clade shrinks. +- **Evidence on placements.** Experiments 7 and 8 are the positive real-data results. Verdicts + by clade are sound at the rates above for coarse clades, and they give the "thumbnail then + zoom" idea a form that holds at the coarse end: the verdict is decided at a coarse clade. + How it fails as the clade shrinks depends on where the mass comes from. With bootstrapped + least squares (experiment 7) it became undecided. With the likelihood weights of the + application's placement step (experiment 8) it became wrong for 7 of 62 verdicts on the + smallest clades. A rule for the application therefore needs a minimum clade size or a + better-calibrated mass, and neither has been chosen or tested here. +- **Counting reads does not average this error away.** Each query in experiments 7 and 8 is + one sequence. Reads from the same organism share that sequence, so if it is misplaced they + are all misplaced together. The bound on wrongly counted reads that the counting rule + takes as input cannot be read off the number of reads; it has to come from the number of + distinct sequences and a per-depth error rate. That rate has not been measured beyond the + table above. - **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic number is enough. On this dataset exact arithmetic gives the same decisions as double precision. What it adds is order-independence and a provable comparison, not a different @@ -241,8 +319,19 @@ enclosing clade and a refuted verdict survives zooming in. verdict in 1,106 means "entailed" is a measured rate, not a proof. Verdicts for nested clades of one query are not independent: counted per query, the smallest entailed clade was wrong for 1 of 130 queries, and 20 further queries had no entailed clade at all. -- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with the application's - own placement step or with real reads, and any change to the application. +- Experiment 8: the truth is where the full-length sequence sits in a maximum-likelihood tree + that was not bootstrapped, so some errors on the smallest clades may be the reference tree's + own uncertainty; the two were not separated. The reference tree used a fixed model (GTR+G4) + with IQ-TREE 2.0.7, where the application selects a model, bootstraps and pins IQ-TREE 3.1.4. + The application's alignment and trimming steps (MAFFT, trimAl) were skipped: queries sit in + the columns of the SILVA alignment. Queries are error-free fragments of reference sequences. + 200 references and 200 queries, one draw. Experiments 7 and 8 differ in the number of + references, the tree method, the source of the mass and the thresholds, so the difference + between them is not the effect of the placement method alone. A cheaper agent ran the + experiment; the tables were recomputed separately from its per-clade output. +- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with real reads, the + application's placement step with its speed heuristic turned off, and any change to the + application. ## Running @@ -278,6 +367,33 @@ julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP julia -O2 -t 4 docs/cladistics/issue-303/placement_evidence.jl /path/to/silva.bacteria.fasta ``` +Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. Copy `epa/` to +`WORK/scripts`, then run its steps in order: + +```bash +sh WORK/scripts/01_run.sh WORK /path/to/silva.bacteria.fasta +``` + +```bash +sh WORK/scripts/02_tree.sh WORK +``` + +```bash +julia WORK/scripts/03_prep.jl WORK 200 7 +``` + +```bash +for k in amp pos neg; do sh WORK/scripts/05_run_epa.sh WORK $k && julia WORK/scripts/04_score.jl WORK $k; done +``` + +```bash +julia WORK/scripts/06_tables.jl WORK amp pos neg +``` + +```bash +julia WORK/scripts/08_by_depth.jl WORK +``` + The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its own copy of the shared code because it is the script as it was run; the later scripts include `lib_tree.jl`. diff --git a/docs/cladistics/issue-303/epa/01_run.sh b/docs/cladistics/issue-303/epa/01_run.sh new file mode 100755 index 00000000..3444470f --- /dev/null +++ b/docs/cladistics/issue-303/epa/01_run.sh @@ -0,0 +1,10 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 01_run.sh epa_dir silva.bacteria.fasta +# epa_dir holds scripts/ (a copy of this directory), and receives work/ and raw/. +E=$1; SILVA=$2 +mkdir -p "$E/work/tree" "$E/raw" +julia -O2 "$E/scripts/01_select.jl" "$SILVA" "$E/work" 200 20261117 diff --git a/docs/cladistics/issue-303/epa/01_select.jl b/docs/cladistics/issue-303/epa/01_select.jl new file mode 100644 index 00000000..0bfcb7ac --- /dev/null +++ b/docs/cladistics/issue-303/epa/01_select.jl @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 01_select.jl silva.fasta outdir N seed +# Same eligibility / column rule as placement_evidence.jl load_alignment. +using Random +const V4S = 11894; const V4E = 25319; const FRAG = 240 +code(c::UInt8) = c == UInt8('A') ? 0x01 : c == UInt8('C') ? 0x02 : c == UInt8('G') ? 0x03 : (c == UInt8('T') || c == UInt8('U')) ? 0x04 : 0x00 +path, out, N, seed = ARGS[1], ARGS[2], parse(Int, ARGS[3]), parse(Int, ARGS[4]) +rng = MersenneTwister(seed) +eligible = Int[]; idx = 0 +open(path) do io + while !eof(io) + readline(io); s = codeunits(readline(io)); global idx += 1 + tot = 0; v4 = 0 + @inbounds for (k, c) in enumerate(s) + if code(c) != 0x00; tot += 1; (V4S <= k <= V4E) && (v4 += 1); end + end + (tot >= 1200 && v4 >= FRAG) && push!(eligible, idx) + end +end +chosen = sort(shuffle(rng, eligible)[1:N]); cs = Set(chosen) +names = String[]; rows = Vector{Vector{UInt8}}(); idx = 0; orig = String[] +open(path) do io + while !eof(io) + h = readline(io); s = readline(io); global idx += 1 + if idx in cs; push!(orig, h[2:end]); push!(rows, UInt8[code(c) for c in codeunits(s)]); end + end +end +# shuffle order so names s001.. are not in file order +perm = shuffle(rng, 1:N); orig = orig[perm]; rows = rows[perm] +names = ["s" * lpad(i, 3, '0') for i in 1:N] +L = length(rows[1]); keep = [k for k in 1:L if count(r -> r[k] != 0x00, rows) >= 2] +v4cols = [j for (j, k) in enumerate(keep) if V4S <= k <= V4E] +A = zeros(UInt8, N, length(keep)); for i in 1:N, (j, k) in enumerate(keep); A[i, j] = rows[i][k]; end +ch = ['-', 'A', 'C', 'G', 'T'] +mkpath(out) +open(joinpath(out, "ref_full.fasta"), "w") do io + for i in 1:N; println(io, ">", names[i]); println(io, String([ch[A[i, j] + 1] for j in 1:size(A, 2)])); end +end +open(joinpath(out, "names.tsv"), "w") do io; for i in 1:N; println(io, names[i], "\t", orig[i]); end; end +open(joinpath(out, "v4cols.txt"), "w") do io; println(io, join(v4cols, ",")); end +println("eligible=$(length(eligible)) N=$N kept_columns=$(size(A,2)) v4_columns=$(length(v4cols)) seed=$seed") diff --git a/docs/cladistics/issue-303/epa/02_tree.sh b/docs/cladistics/issue-303/epa/02_tree.sh new file mode 100755 index 00000000..b975ba92 --- /dev/null +++ b/docs/cladistics/issue-303/epa/02_tree.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 02_tree.sh epa_dir +# Application: iqtree -s ALN -m MFP -b 100 -nt 4 -seed S -pre PFX -redo +# Used here: iqtree2 -s ALN -m GTR+G4 -nt 4 -seed 1 -pre PFX -redo +# (-m MFP aborted after 12 min at 90/286 models; -b 100 omitted: it does not change the ML tree) +E=$1 +cd $E/work/tree && iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -nt 4 -seed 1 -pre $E/work/tree/reference -redo > $E/raw/02_iqtree.stdout 2>&1 diff --git a/docs/cladistics/issue-303/epa/03_prep.jl b/docs/cladistics/issue-303/epa/03_prep.jl new file mode 100644 index 00000000..ffef9614 --- /dev/null +++ b/docs/cladistics/issue-303/epa/03_prep.jl @@ -0,0 +1,70 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 03_prep.jl epa_dir Q seed +# Leave-one-out preparation: for each query q, prune, write pruned tree, truth, and three alignments (amp/pos/neg). +using Random, Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, Q, seed = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]) +W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +seqs = Dict{String,String}(); cur = "" +for l in eachline(joinpath(W, "ref_full.fasta")); startswith(l, ">") ? (global cur = l[2:end]) : (seqs[cur] = l); end +v4cols = parse.(Int, split(strip(read(joinpath(W, "v4cols.txt"), String)), ",")) +g0 = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g0.names[u] => u for u in eachindex(g0.names) if g0.names[u] != "") +@assert length(nid) == N +rng = MersenneTwister(seed) +queries = sort(shuffle(rng, 1:N)[1:Q]); negsrc = [rand(rng, setdiff(1:N, q)) for q in queries] +function fragment(s::String, cols) + v = collect(s); out = fill('-', length(v)); k = 0 + for j in cols; if v[j] != '-'; out[j] = v[j]; k += 1; k == 240 && break; end; end + @assert k == 240; String(out) +end +function newick(g::NG, root) + io = IOBuffer() + function w(u, p) + if isleaf(g, u); print(io, g.names[u]); return; end + print(io, "("); first = true + for (v, l) in g.adj[u]; v == p && continue; first || print(io, ","); first = false; w(v, u); print(io, ":", l); end + print(io, ")") + end + # root must be an internal node + print(io, "("); first = true + for (v, l) in g.adj[root]; first || print(io, ","); first = false; w(v, root); print(io, ":", l); end + print(io, ");"); String(take!(io)) +end +open(joinpath(W, "queries.tsv"), "w") do qio + println(qio, "qi\tquery\tnegsrc") + for (qi, q) in enumerate(queries) + qn = names[q]; u = nid[qn] + adj = [copy(d) for d in g0.adj]; g = NG(copy(g0.names), adj, [Dict{Int,Int}() for _ in adj]) + @assert length(adj[u]) == 1 + p = first(keys(adj[u])); @assert length(adj[p]) == 3 "parent of query has degree $(length(adj[p]))" + delete!(adj[p], u); empty!(adj[u]) + (a, la), (b, lb) = collect(adj[p]) + delete!(adj[a], p); delete!(adj[b], p); empty!(adj[p]); adj[a][b] = la + lb; adj[b][a] = la + lb + refs = [n for n in names if n != qn]; rl = leafidx[minimum(refs)] + # truth: root at the alphabetically first leaf; the merged edge a-b, child side + par, kids, below, order = root_at(g, nid[names[rl]], leafidx) + child = par[a] == b ? a : (par[b] == a ? b : error("a,b not adjacent after merge")) + truth = sort([names[i] for i in below[child]]) + dir = joinpath(W, "q", @sprintf("q%03d", qi)); mkpath(dir) + root = first(u2 for u2 in eachindex(g.names) if !isleaf(g, u2) && length(g.adj[u2]) == 3) + write(joinpath(dir, "pruned.nwk"), newick(g, root) * "\n") + write(joinpath(dir, "truth.txt"), join(truth, "\n") * "\n") + write(joinpath(dir, "query.txt"), qn * "\n") + for (kind, row) in (("amp", fragment(seqs[qn], v4cols)), ("pos", seqs[qn]), ("neg", fragment(seqs[names[negsrc[qi]]], v4cols))) + kd = joinpath(dir, kind); mkpath(kd) + open(joinpath(kd, "aln.fasta"), "w") do io + for n in refs; println(io, ">", n); println(io, seqs[n]); end + println(io, ">qry"); println(io, row) + end + cp(joinpath(dir, "pruned.nwk"), joinpath(kd, "tree.nwk"); force=true) + end + println(qio, qi, "\t", qn, "\t", names[negsrc[qi]]) + end +end +println("prepared $Q queries") diff --git a/docs/cladistics/issue-303/epa/04_score.jl b/docs/cladistics/issue-303/epa/04_score.jl new file mode 100644 index 00000000..982f7703 --- /dev/null +++ b/docs/cladistics/issue-303/epa/04_score.jl @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 04_score.jl epa_dir kind -> raw/perquery_.tsv, raw/perclade_.tsv, raw/checks_.txt +using Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, kind = ARGS[1], ARGS[2]; W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +pq = open(joinpath(E, "raw", "perquery_$kind.tsv"), "w"); pc = open(joinpath(E, "raw", "perclade_$kind.tsv"), "w") +ck = open(joinpath(E, "raw", "checks_$kind.txt"), "w") +println(pq, "qi\tquery\tstatus\tnplacements\ttotal_lwr\tbest_lwr\texact\tedges_apart") +println(pc, "qi\tclade_size\tmass\ttruth_inside") +nedges_ok = true; distinct_ok = true; mono_viol = 0; mono_checked = 0; nscored = 0 +for q in qs + qi = parse(Int, q[1]); dir = joinpath(W, "q", @sprintf("q%03d", qi)); kd = joinpath(dir, kind) + jp = joinpath(kd, "RAxML_portableTree.epa.jplace") + if !isfile(jp); println(pq, qi, "\t", q[2], "\tMISSING\t0\t0\t0\t0\t0"); continue; end + g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String))) + truthleaves = split(strip(read(joinpath(dir, "truth.txt"), String)), "\n") + refs = sort([n for n in names if n != q[2]]); rl = leafidx[refs[1]] + nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") + par, kids, below, order = root_at(g, nid[refs[1]], leafidx) + # edge above node u (u != root leaf) <-> below[u]; root leaf's pendant edge == edge above its only child + root_child = first(kids[nid[refs[1]]]) + edgekey = Dict{BitSet,Int}() # bipartition (side without refs[1]) -> node + for u in order; u == nid[refs[1]] && continue; edgekey[below[u]] = u; end + ne = length(edgekey) + 0 # includes root_child's edge == pendant of root leaf + expected = 2 * (N - 1) - 3 + # jplace tree + tree, fields, rows = parse_jplace(jp) + jg = parse_nwk(tree) + jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "") + @assert !haskey(jn, "qry") "query leaf present in jplace tree" + jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx) + enum2node = Dict{Int,Int}(); jkey = Dict{Int,BitSet}() + for u in jorder; u == jn[refs[1]] && continue; e = jg.lab[u][jpar[u]]; enum2node[e] = u; jkey[e] = jbelow[u]; end + # the edge above the root leaf's only child and the pendant edge of root leaf are the same unrooted edge: jplace labels it once + nj = length(jkey) + if nj != expected; global nedges_ok = false; end + # edge-number -> bipartition uniqueness + if length(Set(values(jkey))) != length(jkey); global distinct_ok = false; end + println(ck, "q$qi: jplace edges=$(length(jkey)) distinct_bipartitions=$(length(Set(values(jkey)))) expected=$expected pruned_tree_edges=$(length(edgekey)) all_in_pruned=$(all(k -> haskey(edgekey, k), values(jkey)))") + ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields) + pls = [(Int(r[ei]), r[li]) for r in rows] + total = sum(p[2] for p in pls) + # best placement + bi = argmax([p[2] for p in pls]); bkey = jkey[pls[bi][1]] + tkey = BitSet(leafidx[n] for n in truthleaves) + # truth is the below-set of its child in the pruned tree rooted at refs[1] + @assert haskey(edgekey, tkey) "truth bipartition not an edge of pruned tree" + function apart(a, b) + a == b && return 0 + da = Dict{Int,Int}(); u = a; k = 0; while u != 0 && u != nid[refs[1]]; da[u] = k; u = par[u]; k += 1; end + u = b; k = 0; while u != 0 && u != nid[refs[1]]; haskey(da, u) && return da[u] + k; u = par[u]; k += 1; end + typemax(Int) + end + tnode = edgekey[tkey]; bnode = edgekey[bkey] + # root-leaf pendant edge: represent by root_child's node (same edge) + println(pq, qi, "\t", q[2], "\tOK\t", length(pls), "\t", total, "\t", pls[bi][2], "\t", Int(bkey == tkey), "\t", apart(bnode, tnode)) + # clades: internal nodes other than the root leaf, whose parent... internal non-leaf nodes of the rooted tree (root leaf excluded) + plnode = [(edgekey[jkey[e]], w) for (e, w) in pls] + clades = [u for u in order if u != nid[refs[1]] && !isleaf(g, u)] + masses = Dict{Int,Float64}() + for c in clades + S = below[c]; m = 0.0 + for (nd, w) in plnode; below[nd] != S && issubset(below[nd], S) && (m += w); end + masses[c] = m + println(pc, qi, "\t", length(S), "\t", m, "\t", Int(tkey != S && issubset(tkey, S))) + end + # monotonicity under inclusion: mass(child) <= mass(parent) for parent clade strictly containing + for c in clades; p = par[c]; (p == nid[refs[1]] || isleaf(g, p)) && continue; global mono_checked += 1; masses[c] <= masses[p] + 1e-12 || (global mono_viol += 1); end + global nscored += 1 +end +println(ck, "SUMMARY scored=$nscored edge_count_ok=$nedges_ok distinct_bipartitions_ok=$distinct_ok monotone_pairs_checked=$mono_checked monotone_violations=$mono_viol") +close(pq); close(pc); close(ck) +println("scored $nscored; ", read(joinpath(E, "raw", "checks_$kind.txt"), String)[end-200:end]) diff --git a/docs/cladistics/issue-303/epa/05_run_epa.sh b/docs/cladistics/issue-303/epa/05_run_epa.sh new file mode 100755 index 00000000..32bd356a --- /dev/null +++ b/docs/cladistics/issue-303/epa/05_run_epa.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 05_run_epa.sh epa_dir kind (kind = amp | pos | neg) +# Application command (src/pipeline/phylogeny.jl, step "place"), threads=4 -> -T max(2,threads) = -T 4 in the app; +# here -T 2 with two runs in parallel on 4 cores (thread count does not change the result). +# raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s -t -T N -w / +E=$1; K=$2 +ls -d "$E"/work/q/q*/"$K" | xargs -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ diff --git a/docs/cladistics/issue-303/epa/06_tables.jl b/docs/cladistics/issue-303/epa/06_tables.jl new file mode 100644 index 00000000..22c21dd5 --- /dev/null +++ b/docs/cladistics/issue-303/epa/06_tables.jl @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 06_tables.jl epa_dir kind [kind...] -> prints tables from raw/perquery_.tsv and raw/perclade_.tsv +using Printf, Statistics +E = ARGS[1] +function table(kind) + pq = [split(l, "\t") for l in eachline(joinpath(E, "raw", "perquery_$kind.tsv"))][2:end] + ok = filter(r -> r[3] == "OK", pq) + missing_n = length(pq) - length(ok) + pc = [split(l, "\t") for l in eachline(joinpath(E, "raw", "perclade_$kind.tsv"))][2:end] + byq = Dict{Int,Vector{Tuple{Int,Float64,Bool}}}() + for r in pc; push!(get!(byq, parse(Int, r[1]), Tuple{Int,Float64,Bool}[]), (parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")); end + ex = [r[7] == "1" for r in ok]; ap = [parse(Int, r[8]) for r in ok]; tot = [parse(Float64, r[5]) for r in ok]; npl = [parse(Int, r[4]) for r in ok] + println("\n===== ", kind, " =====") + @printf("queries scored: %d (missing/failed: %d); placed on exactly the true edge: %.1f%% (%d); within 2 edges: %.1f%%; median edges apart: %g\n", + length(ok), missing_n, 100mean(ex), sum(ex), 100mean(ap .<= 2), median(ap)) + @printf("placements reported per query: median %g, max %d; queries with a single placement: %d; total LWR reported per query: min %.4f, median %.4f, max %.4f\n", + median(npl), maximum(npl), count(==(1), npl), minimum(tot), median(tot), maximum(tot)) + for alpha in (0.01, 0.05) + ent_ok = ent_bad = ref_ok = ref_bad = unr_in = unr_out = 0; pin = Int[]; pin_ok = 0; nopin = 0; q_wrong = 0 + for (qi, cl) in byq + best = 0; bsz = typemax(Int); bt = false + for (sz, m, t) in cl + if m >= 1 - alpha - 1e-12 + t ? (ent_ok += 1) : (ent_bad += 1) + sz < bsz && ((bsz, bt, best) = (sz, t, 1)) + elseif m <= alpha + 1e-12 + t ? (ref_bad += 1) : (ref_ok += 1) + else + t ? (unr_in += 1) : (unr_out += 1) + end + end + if best == 0; nopin += 1 else push!(pin, bsz); bt ? (pin_ok += 1) : (q_wrong += 1) end + end + @printf("\n alpha = %.2f: entailed if mass >= %.2f, refuted if mass <= %.2f\n", alpha, 1 - alpha, alpha) + @printf(" entailed %7d verdicts, wrong %5d (truth not inside) -> sound %.2f%%\n", ent_ok + ent_bad, ent_bad, 100ent_ok / max(ent_ok + ent_bad, 1)) + @printf(" refuted %7d verdicts, wrong %5d (truth inside) -> sound %.3f%%\n", ref_ok + ref_bad, ref_bad, 100ref_ok / max(ref_ok + ref_bad, 1)) + @printf(" unresolved %7d verdicts (%d truth inside, %d truth not inside)\n", unr_in + unr_out, unr_in, unr_out) + d = ent_ok + unr_in + ref_bad + @printf(" of the %d clades that truly contain the query: entailed %.1f%%, unresolved %.1f%%, wrongly refuted %.1f%%\n", d, 100ent_ok / d, 100unr_in / d, 100ref_bad / d) + if !isempty(pin) + @printf(" smallest entailed clade per query: median size %g (quartiles %g, %g) of %d references; contains truth for %.1f%% of %d queries; no entailed clade for %d queries\n", + median(pin), quantile(pin, 0.25), quantile(pin, 0.75), 199, 100pin_ok / length(pin), length(pin), nopin) + else + @printf(" no entailed clade for any query (%d)\n", nopin) + end + @printf(" queries whose smallest entailed clade is wrong: %d of %d (with an entailed clade)\n", q_wrong, length(pin)) + end + println("\n calibration of clade mass (share of clades that contain the truth)") + edges = [0.05, 0.25, 0.5, 0.75, 0.95]; hit = zeros(Int, 7); cnt = zeros(Int, 7) + for cl in values(byq), (sz, m, t) in cl + b = m <= 0 ? 1 : m > 0.95 + 1e-12 ? 7 : 1 + findfirst(x -> m <= x + 1e-12, edges) + cnt[b] += 1; hit[b] += t + end + labels = ["0", "(0, .05]", "(.05, .25]", "(.25, .5]", "(.5, .75]", "(.75, .95]", "(.95, 1]"] + for b in 1:7; cnt[b] > 0 && @printf(" mass %-11s clades %7d truth inside %6.2f%%\n", labels[b], cnt[b], 100hit[b] / cnt[b]); end +end +for k in ARGS[2:end]; table(k); end diff --git a/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl b/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl new file mode 100644 index 00000000..88c70513 --- /dev/null +++ b/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 07_selfcheck_truth.jl epa_dir -- independent truth: from leaf-to-leaf path distances in the UNPRUNED tree +# Rule: q hangs by pendant length l from p. For leaves x,y != q: g(x,y) = d(q,x)+d(q,y)-d(x,y). +# x,y on different sides of p (sides A/B = the two other neighbours' subtrees): g = 2l (the minimum); same side: g = 2(l + d(p, meet)) >= 2l. +# Leaves are clustered by linking pairs with g > min + tol; the two largest components must be the sides. +using Printf, Random +include(joinpath(@__DIR__, "lib_nwk.jl")) +E = ARGS[1]; W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))] +g = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") +function dists(src) + d = fill(NaN, length(g.names)); d[src] = 0.0; st = [src] + while !isempty(st); u = pop!(st); for (v, l) in g.adj[u]; isnan(d[v]) && (d[v] = d[u] + l; push!(st, v)); end; end + d +end +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +rng = MersenneTwister(99); pick = shuffle(rng, 1:length(qs))[1:5] +big = [i for i in shuffle(rng, 1:length(qs)) if length(readlines(joinpath(W, "q", @sprintf("q%03d", parse(Int, qs[i][1])), "truth.txt"))) >= 5][1:5]; append!(pick, big) +allok = true +for i in pick + qi = parse(Int, qs[i][1]); qn = qs[i][2] + truth = Set(split(strip(read(joinpath(W, "q", @sprintf("q%03d", qi), "truth.txt"), String)), "\n")) + dq = dists(nid[qn]); others = [n for n in names if n != qn]; m = length(others) + D = Dict(n => dists(nid[n]) for n in others) + G = [dq[nid[x]] + dq[nid[y]] - D[x][nid[y]] for x in others, y in others] + mn = minimum(G[i2, j2] for i2 in 1:m, j2 in 1:m if i2 != j2) + tol = 1e-7 + comp = collect(1:m); find(x) = (while comp[x] != x; comp[x] = comp[comp[x]]; x = comp[x]; end; x) + for i2 in 1:m, j2 in i2+1:m; G[i2, j2] > mn + tol && (comp[find(i2)] = find(j2)); end + groups = Dict{Int,Vector{String}}(); for i2 in 1:m; push!(get!(groups, find(i2), String[]), others[i2]); end + gs = sort(collect(values(groups)); by = length, rev = true) + sides = [Set(x) for x in gs[1:min(2, length(gs))]] + # the "truth" side in the prep is the side not containing the alphabetically first reference leaf + first_leaf = minimum(others) + cand = [s for s in sides if !(first_leaf in s)] + ok = length(gs) == 2 && length(cand) == 1 && cand[1] == truth + global allok &= ok + @printf(" %s (q%03d): components=%d sizes=%s; independent side without %s has %d leaves; prepared truth has %d leaves; match=%s\n", qn, qi, length(gs), string(length.(gs)), first_leaf, isempty(cand) ? -1 : length(cand[1]), length(truth), ok) +end +println("SELF-CHECK (a): all ten match (5 random + 5 with truth side >= 5 leaves) = ", allok) diff --git a/docs/cladistics/issue-303/epa/08_by_depth.jl b/docs/cladistics/issue-303/epa/08_by_depth.jl new file mode 100644 index 00000000..f4c1b6fd --- /dev/null +++ b/docs/cladistics/issue-303/epa/08_by_depth.jl @@ -0,0 +1,49 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# Independent recomputation of the tables from raw/perclade_*.tsv and raw/perquery_*.tsv, and the +# breakdown of wrong entailed verdicts by clade size. Written separately from 06_tables.jl as a +# cross-check. usage: julia 08_by_depth.jl epa_dir +using Printf, Statistics +function load(kind) + pc = [split(l, '\t') for l in readlines(joinpath(ARGS[1], "raw", "perclade_$kind.tsv"))[2:end]] + C = [(parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1") for r in pc] + pq = [split(l, '\t') for l in readlines(joinpath(ARGS[1], "raw", "perquery_$kind.tsv"))[2:end]] + Q = Dict(parse(Int, r[1]) => (n = parse(Int, r[4]), tot = parse(Float64, r[5]), best = parse(Float64, r[6]), exact = r[7] == "1", apart = parse(Int, r[8])) for r in pq) + C, Q +end +for kind in ("amp", "pos", "neg") + C, Q = load(kind) + println("== $kind: $(length(C)) clade rows, $(length(Q)) queries") + for a in (0.01, 0.05) + ent = [c for c in C if c[3] >= 1 - a]; ref = [c for c in C if c[3] <= a] + ew = count(c -> !c[4], ent); rw = count(c -> c[4], ref) + # smallest entailed clade per query + sm = Dict{Int,Tuple{Int,Bool}}() + for c in ent; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + wrongq = sort([q for (q, v) in sm if !v[2]]) + @printf(" alpha %.2f: entailed %d wrong %d | refuted %d wrong %d | smallest entailed: %d queries, median size %.1f, wrong %d\n", + a, length(ent), ew, length(ref), rw, length(sm), median([v[1] for v in values(sm)]), length(wrongq)) + if kind == "amp" + println(" wrong-smallest queries (qi: clade size, edges apart, placements, best LWR): ", + join(["$q: $(sm[q][1]), $(Q[q].apart), $(Q[q].n), $(round(Q[q].best, digits=3))" for q in wrongq], " | ")) + # entailed error by clade size + for (lo, hi) in ((2, 3), (4, 7), (8, 15), (16, 49), (50, 199)) + e = [c for c in ent if lo <= c[2] <= hi] + @printf(" entailed, clade size %3d..%3d: %5d verdicts, wrong %2d\n", lo, hi, length(e), count(c -> !c[4], e)) + end + # rule: smallest entailed clade of size >= k + for k in (2, 4, 8, 16) + s2 = Dict{Int,Tuple{Int,Bool}}() + for c in ent; c[2] >= k || continue; (!haskey(s2, c[1]) || c[2] < s2[c[1]][1]) && (s2[c[1]] = (c[2], c[4])); end + @printf(" smallest entailed clade of size >= %2d: %3d queries, median size %.1f, wrong %d\n", k, length(s2), median([v[1] for v in values(s2)]), count(v -> !v[2], values(s2))) + end + # by number of placements reported + for (lab, f) in (("1 placement", n -> n == 1), ("2+ placements", n -> n >= 2)) + qs = [q for q in keys(sm) if f(Q[q].n)] + @printf(" %s: %d queries with an entailed clade, wrong smallest %d\n", lab, length(qs), count(q -> !sm[q][2], qs)) + end + end + end +end diff --git a/docs/cladistics/issue-303/epa/lib_nwk.jl b/docs/cladistics/issue-303/epa/lib_nwk.jl new file mode 100644 index 00000000..46a3a7cc --- /dev/null +++ b/docs/cladistics/issue-303/epa/lib_nwk.jl @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# Minimal Newick + jplace reading; standard library only. Data is read as data only. +# Graph = Vector{Dict{Int,Float64}} adjacency; names[i] = leaf name or "". +mutable struct NG; names::Vector{String}; adj::Vector{Dict{Int,Float64}}; lab::Vector{Dict{Int,Int}}; end +# lab[u][v] = jplace edge number of edge u-v (only filled by parse with {n}) +function parse_nwk(s::AbstractString) + names = String[]; adj = Dict{Int,Float64}[]; lab = Dict{Int,Int}[] + newnode!(n) = (push!(names, n); push!(adj, Dict{Int,Float64}()); push!(lab, Dict{Int,Int}()); length(names)) + s = strip(s); pos = 1; n = lastindex(s) + # returns (node, branchlength, edgenum) + function rd() + if s[pos] == '(' + pos += 1; kids = Tuple{Int,Float64,Int}[] + while true + push!(kids, rd()); c = s[pos]; pos += 1 + c == ',' && continue; c == ')' && break; error("bad newick at $pos") + end + lbl = "" + while s[pos] ∉ (':', ',', ')', ';', '{'); lbl *= s[pos]; pos += 1; end + u = newnode!("") + for (k, l, e) in kids; adj[u][k] = l; adj[k][u] = l; e >= 0 && (lab[u][k] = e; lab[k][u] = e); end + else + st = pos; while s[pos] ∉ (':', ',', ')', ';', '{'); pos += 1; end + u = newnode!(String(s[st:pos-1])) + end + bl = 0.0; en = -1 + if s[pos] == ':'; pos += 1; st = pos; while s[pos] ∉ (',', ')', ';', '{'); pos += 1; end; bl = parse(Float64, s[st:pos-1]); end + if s[pos] == '{'; pos += 1; st = pos; while s[pos] != '}'; pos += 1; end; en = parse(Int, s[st:pos-1]); pos += 1; end + (u, bl, en) + end + (root, _, _) = rd() + g = NG(names, adj, lab) + # suppress a degree-2 root + if length(adj[root]) == 2 && names[root] == "" + (a, la), (b, lb) = collect(adj[root]) + delete!(adj[a], root); delete!(adj[b], root); delete!(adj[root], a); delete!(adj[root], b) + adj[a][b] = la + lb; adj[b][a] = la + lb + if haskey(lab[root], a) && haskey(lab[root], b); lab[a][b] = lab[root][a]; lab[b][a] = lab[root][a]; end + end + g +end +isleaf(g::NG, u) = g.names[u] != "" +alive(g::NG, u) = !isempty(g.adj[u]) +# Root at leaf `rl`; returns parent, children, below (BitSet of leaf *indices in leafidx*), order (preorder) +function root_at(g::NG, rl::Int, leafidx::Dict{String,Int}) + n = length(g.names); parent = zeros(Int, n); order = Int[]; st = [rl]; seen = falses(n); seen[rl] = true + kids = [Int[] for _ in 1:n] + while !isempty(st) + u = pop!(st); push!(order, u) + for v in keys(g.adj[u]); if !seen[v]; seen[v] = true; parent[v] = u; push!(kids[u], v); push!(st, v); end; end + end + below = Vector{BitSet}(undef, n) + for u in reverse(order) + if isleaf(g, u); below[u] = BitSet([leafidx[g.names[u]]]) + else b = BitSet(); for c in kids[u]; union!(b, below[c]); end; below[u] = b; end + end + parent, kids, below, order +end +function parse_jplace(path) + txt = read(path, String) + m = match(r"\"tree\"\s*:\s*\"([^\"]*)\"", txt); tree = m.captures[1] + fm = match(r"\"fields\"\s*:\s*\[([^\]]*)\]", txt) + fields = [strip(replace(x, "\"" => "")) for x in split(fm.captures[1], ",")] + pm = match(r"\"p\"\s*:\s*\[(.*?)\]\s*,\s*\"n\""s, txt) + rows = Vector{Vector{Float64}}() + for rm in eachmatch(r"\[([^\[\]]+)\]", pm.captures[1] * "]") + push!(rows, parse.(Float64, split(rm.captures[1], ","))) + end + tree, fields, rows +end From 41631b3f97e5e281176cd87d95647ba410ed9110 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 03:56:12 +0000 Subject: [PATCH 2/5] docs(cladistics): issue 303 experiment 9, error of clade verdicts by clade depth Two draws of 400 references and 600 amplicon queries with the application's placement step. The smallest entailed clade was wrong for 32 of 448 queries (7.1%, exact 95% interval 4.9 to 9.9); requiring at least 64 references in the clade brought it to 4 of 448 (0.9%, 0.2 to 2.3). Errors also occur on large clades whose branch cuts off a small group, which experiment 8's smaller sample did not show. The scripts are in docs/cladistics/issue-303/epa-depth/. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01J4QvDiW8qD4f2JDQJ4buGo --- docs/cladistics/issue-303/README.md | 105 ++++++++++- .../cladistics/issue-303/epa-depth/02_tree.sh | 21 +++ .../cladistics/issue-303/epa-depth/03_prep.jl | 71 ++++++++ .../issue-303/epa-depth/04_score.jl | 81 +++++++++ .../issue-303/epa-depth/05_run_epa.sh | 11 ++ .../issue-303/epa-depth/06_tables.jl | 168 ++++++++++++++++++ .../issue-303/epa-depth/08_crosscheck.jl | 64 +++++++ .../issue-303/epa-depth/run_rest.sh | 16 ++ .../issue-303/epa-depth/run_trees.sh | 17 ++ 9 files changed, 547 insertions(+), 7 deletions(-) create mode 100755 docs/cladistics/issue-303/epa-depth/02_tree.sh create mode 100644 docs/cladistics/issue-303/epa-depth/03_prep.jl create mode 100644 docs/cladistics/issue-303/epa-depth/04_score.jl create mode 100755 docs/cladistics/issue-303/epa-depth/05_run_epa.sh create mode 100644 docs/cladistics/issue-303/epa-depth/06_tables.jl create mode 100644 docs/cladistics/issue-303/epa-depth/08_crosscheck.jl create mode 100755 docs/cladistics/issue-303/epa-depth/run_rest.sh create mode 100755 docs/cladistics/issue-303/epa-depth/run_trees.sh diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 9a1189a1..304ea597 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -33,7 +33,9 @@ bootstrapped placements were right for 1,105 of 1,106 entailed and 57,116 of 57, verdicts, and pin a read to a clade of median size 18 among 399 references. With the application's own placement step (RAxML EPA) the picture is sharper and less safe: 2,375 of 2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more -references and 7 errors among 62 on clades of two or three. +references and 7 errors among 62 on clades of two or three. On a larger sample the smallest +confident clade was wrong for 32 of 448 queries (7.1%), falling to 4 of 448 (0.9%) when the +clade had to hold at least 64 of 399 references. ## Data @@ -272,7 +274,69 @@ agree. What this shows: with the application's placement step, "present in this clade" was right in every case seen for clades of a quarter of the tree or more, and wrong about one time in nine for clades of two or three references. At that depth the verdict is confidently wrong, -not unresolved. What it does not show is in the limits below. +not unresolved. Experiment 9 repeats this on a larger sample and finds errors on large clades +too. What it does not show is in the limits below. + +### 9. Error by clade depth (`epa-depth/`) + +The same test as experiment 8 on a larger sample, to put a number on how the error depends +on clade size. Two independent draws of 400 references (13 sequences are in both), a +maximum-likelihood tree for each, and 300 amplicon queries per draw: 600 queries. As a +control, the whole sequence of 60 queries was placed: 60 of 60 landed on the exact edge and +none of their 1,134 entailed verdicts was wrong. + +For the amplicon fragment, 9,555 of 9,613 entailed verdicts were right. The 58 wrong ones by +clade size: + +| Clade size (of 399 references) | Entailed verdicts | Wrong | Queries with a wrong verdict | +| --- | --- | --- | --- | +| 2 to 3 | 215 | 15 | 13 | +| 4 to 7 | 417 | 12 | 11 | +| 8 to 15 | 507 | 11 | 9 | +| 16 to 31 | 748 | 7 | 6 | +| 32 to 63 | 999 | 6 | 2 | +| 64 to 127 | 431 | 2 | 1 | +| 128 to 399 | 6,296 | 5 | 3 | + +The five errors in the last row are on clades holding 357 to 372 of the 399 references. The +branch there cuts off a small group, and the sequence truly sits in that small group. Binned +by the smaller side of the branch instead, the wrong count falls from 15 of 444 at two or +three references to 2 of 1,271 at 64 to 127, and 0 of 1,119 at 128 or more. + +One result per query is the honest unit. The rule "report the smallest entailed clade with +at least k references" gave: + +| k | Queries given a clade (of 600) | Median clade size | Wrong | Rate (exact 95% interval) | +| --- | --- | --- | --- | --- | +| 2 | 448 | 4 | 32 | 7.1% (4.9 to 9.9) | +| 4 | 448 | 7 | 25 | 5.6% (3.6 to 8.1) | +| 8 | 448 | 14 | 17 | 3.8% (2.2 to 6.0) | +| 16 | 448 | 26 | 11 | 2.5% (1.2 to 4.4) | +| 32 | 448 | 46.5 | 6 | 1.3% (0.5 to 2.9) | +| 64 | 448 | 173 | 4 | 0.9% (0.2 to 2.3) | + +The other 152 queries had no entailed clade. The two draws agree within their intervals: +19 of 230 and 13 of 218 at k = 2. Of the 32 queries wrong at k = 2, 18 had a single reported +placement and 18 had a best weight of 0.99 or more. With the looser rule (0.95) the rates +are higher: 61 of 540 at k = 2 and 10 of 540 at k = 64. + +The intervals are Clopper-Pearson and treat queries as independent. The queries of one draw +share one reference tree, so the intervals cover the sampling of queries on these two trees, +not the variation between trees. + +Self-checks passed: the control above; an independent truth calculation matched for 10 +queries in each draw; all 793 edge numbers of every placement file map to distinct branches +of the pruned tree; clade mass never decreased from a clade to the clade containing it; and +the new table code reproduces experiment 8's counts from experiment 8's output. +`epa-depth/08_crosscheck.jl` recomputes the tables separately from `06_tables.jl`; the two +agree. + +What this shows: the error of a "present in this clade" verdict from an amplicon fragment +falls as the clade grows and does not reach zero. Requiring 64 references in the clade +brought it under 1% of queries, at the price of a median clade of 173 of 399. The whole +sequence, placed the same way, was never wrong in 60 queries. The truth was itself built +from the whole sequence, so that control shows the machinery is sound; it does not show the +reference tree is right at the finest depth. ## Reading of the results @@ -282,14 +346,16 @@ not unresolved. What it does not show is in the limits below. How it fails as the clade shrinks depends on where the mass comes from. With bootstrapped least squares (experiment 7) it became undecided. With the likelihood weights of the application's placement step (experiment 8) it became wrong for 7 of 62 verdicts on the - smallest clades. A rule for the application therefore needs a minimum clade size or a - better-calibrated mass, and neither has been chosen or tested here. + smallest clades. Experiment 9 puts numbers on it: a minimum clade size trades error for + resolution (7.1% of queries wrong with no minimum, 0.9% with a minimum of 64 references). + A rule for the application needs that minimum or a better-calibrated mass. Neither has + been chosen, and no rule has been tested inside the application. - **Counting reads does not average this error away.** Each query in experiments 7 and 8 is one sequence. Reads from the same organism share that sequence, so if it is misplaced they are all misplaced together. The bound on wrongly counted reads that the counting rule takes as input cannot be read off the number of reads; it has to come from the number of - distinct sequences and a per-depth error rate. That rate has not been measured beyond the - table above. + distinct sequences and a per-depth error rate. Experiment 9 gives that rate for error-free + fragments on two trees; it has not been measured on real reads. - **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic number is enough. On this dataset exact arithmetic gives the same decisions as double precision. What it adds is order-independence and a provable comparison, not a different @@ -325,7 +391,10 @@ not unresolved. What it does not show is in the limits below. with IQ-TREE 2.0.7, where the application selects a model, bootstraps and pins IQ-TREE 3.1.4. The application's alignment and trimming steps (MAFFT, trimAl) were skipped: queries sit in the columns of the SILVA alignment. Queries are error-free fragments of reference sequences. - 200 references and 200 queries, one draw. Experiments 7 and 8 differ in the number of + 200 references and 200 queries, one draw. Experiment 9 shares every limit of experiment 8 + except size: 400 references, two draws, 600 queries. The second draw's tree was built + without bootstrap after the bootstrapped run exceeded 25 minutes; the bootstrap supports of + the first tree were saved and not analysed. Experiments 7 and 8 differ in the number of references, the tree method, the source of the mass and the thresholds, so the difference between them is not the effect of the placement method alone. A cheaper agent ran the experiment; the tables were recomputed separately from its per-clade output. @@ -394,6 +463,28 @@ julia WORK/scripts/06_tables.jl WORK amp pos neg julia WORK/scripts/08_by_depth.jl WORK ``` +Experiment 9: copy `epa/` and then `epa-depth/` over it to `WORK/scripts`, then: + +```bash +sh WORK/scripts/run_trees.sh WORK /path/to/silva.bacteria.fasta +``` + +```bash +sh WORK/scripts/run_rest.sh WORK +``` + +```bash +mkdir -p WORK/raw && for t in "A amp" "A pos" "B amp"; do set -- $t; julia WORK/scripts/04_score.jl WORK/$1 $1 $2; done +``` + +```bash +julia WORK/scripts/06_tables.jl new WORK/raw +``` + +```bash +julia WORK/scripts/08_crosscheck.jl WORK/raw +``` + The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its own copy of the shared code because it is the script as it was run; the later scripts include `lib_tree.jl`. diff --git a/docs/cladistics/issue-303/epa-depth/02_tree.sh b/docs/cladistics/issue-303/epa-depth/02_tree.sh new file mode 100755 index 00000000..891cfb27 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/02_tree.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: 02_tree.sh epa_dir draw -- IQ-TREE GTR+G4 with UFBoot 1000 (4 threads); fallback without -B after 25 min wall clock +E=$1; DR=$2; T=$E/work/tree; mkdir -p $T; cd $T +LOG=$E/../raw/tree_$DR.log +s=$(date +%s) +timeout 1500 iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -B 1000 -nt 4 -seed 1 -pre $T/reference -redo > $E/../raw/02_iqtree_$DR.stdout 2>&1 +rc=$? +if [ $rc -ne 0 ] || [ ! -s $T/reference.treefile ]; then + echo "draw $DR: -B 1000 run rc=$rc after $(( $(date +%s) - s )) s; RERUN WITHOUT -B" > $LOG + s=$(date +%s) + iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -nt 4 -seed 1 -pre $T/reference -redo > $E/../raw/02_iqtree_$DR.nob.stdout 2>&1 + echo "draw $DR: rerun without -B took $(( $(date +%s) - s )) s" >> $LOG +else + echo "draw $DR: -B 1000 completed, $(( $(date +%s) - s )) s wall clock, rc=0" > $LOG +fi +cp $T/reference.treefile $T/reference_with_support.treefile diff --git a/docs/cladistics/issue-303/epa-depth/03_prep.jl b/docs/cladistics/issue-303/epa-depth/03_prep.jl new file mode 100644 index 00000000..f212b608 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/03_prep.jl @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 03_prep.jl epa_dir Q seed npos (amp for all Q queries; pos for the first npos queries; no neg) +# Leave-one-out preparation: for each query q, prune, write pruned tree, truth, and three alignments (amp/pos/neg). +using Random, Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, Q, seed, NPOS = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]), parse(Int, ARGS[4]) +W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +seqs = Dict{String,String}(); cur = "" +for l in eachline(joinpath(W, "ref_full.fasta")); startswith(l, ">") ? (global cur = l[2:end]) : (seqs[cur] = l); end +v4cols = parse.(Int, split(strip(read(joinpath(W, "v4cols.txt"), String)), ",")) +g0 = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g0.names[u] => u for u in eachindex(g0.names) if g0.names[u] != "") +@assert length(nid) == N +rng = MersenneTwister(seed) +queries = sort(shuffle(rng, 1:N)[1:Q]) +function fragment(s::String, cols) + v = collect(s); out = fill('-', length(v)); k = 0 + for j in cols; if v[j] != '-'; out[j] = v[j]; k += 1; k == 240 && break; end; end + @assert k == 240; String(out) +end +function newick(g::NG, root) + io = IOBuffer() + function w(u, p) + if isleaf(g, u); print(io, g.names[u]); return; end + print(io, "("); first = true + for (v, l) in g.adj[u]; v == p && continue; first || print(io, ","); first = false; w(v, u); print(io, ":", l); end + print(io, ")") + end + # root must be an internal node + print(io, "("); first = true + for (v, l) in g.adj[root]; first || print(io, ","); first = false; w(v, root); print(io, ":", l); end + print(io, ");"); String(take!(io)) +end +open(joinpath(W, "queries.tsv"), "w") do qio + println(qio, "qi\tquery") + for (qi, q) in enumerate(queries) + qn = names[q]; u = nid[qn] + adj = [copy(d) for d in g0.adj]; g = NG(copy(g0.names), adj, [Dict{Int,Int}() for _ in adj]) + @assert length(adj[u]) == 1 + p = first(keys(adj[u])); @assert length(adj[p]) == 3 "parent of query has degree $(length(adj[p]))" + delete!(adj[p], u); empty!(adj[u]) + (a, la), (b, lb) = collect(adj[p]) + delete!(adj[a], p); delete!(adj[b], p); empty!(adj[p]); adj[a][b] = la + lb; adj[b][a] = la + lb + refs = [n for n in names if n != qn]; rl = leafidx[minimum(refs)] + # truth: root at the alphabetically first leaf; the merged edge a-b, child side + par, kids, below, order = root_at(g, nid[names[rl]], leafidx) + child = par[a] == b ? a : (par[b] == a ? b : error("a,b not adjacent after merge")) + truth = sort([names[i] for i in below[child]]) + dir = joinpath(W, "q", @sprintf("q%03d", qi)); mkpath(dir) + root = first(u2 for u2 in eachindex(g.names) if !isleaf(g, u2) && length(g.adj[u2]) == 3) + write(joinpath(dir, "pruned.nwk"), newick(g, root) * "\n") + write(joinpath(dir, "truth.txt"), join(truth, "\n") * "\n") + write(joinpath(dir, "query.txt"), qn * "\n") + for (kind, row) in (qi <= NPOS ? [("amp", fragment(seqs[qn], v4cols)), ("pos", seqs[qn])] : [("amp", fragment(seqs[qn], v4cols))]) + kd = joinpath(dir, kind); mkpath(kd) + open(joinpath(kd, "aln.fasta"), "w") do io + for n in refs; println(io, ">", n); println(io, seqs[n]); end + println(io, ">qry"); println(io, row) + end + cp(joinpath(dir, "pruned.nwk"), joinpath(kd, "tree.nwk"); force=true) + end + println(qio, qi, "\t", qn) + end +end +println("prepared $Q queries") diff --git a/docs/cladistics/issue-303/epa-depth/04_score.jl b/docs/cladistics/issue-303/epa-depth/04_score.jl new file mode 100644 index 00000000..19c34a66 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/04_score.jl @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 04_score.jl epa_dir draw kind -> ../raw/perquery__.tsv, perclade__.tsv, checks__.txt +using Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, draw, kind = ARGS[1], ARGS[2], ARGS[3]; W = joinpath(E, "work"); R = joinpath(E, "..", "raw") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +pq = open(joinpath(R, "perquery_$(draw)_$kind.tsv"), "w"); pc = open(joinpath(R, "perclade_$(draw)_$kind.tsv"), "w") +ck = open(joinpath(R, "checks_$(draw)_$kind.txt"), "w") +println(pq, "qi\tquery\tstatus\tnplacements\ttotal_lwr\tbest_lwr\texact\tedges_apart") +println(pc, "qi\tclade_size\tmass\ttruth_inside\tn_refs") +nedges_ok = true; distinct_ok = true; mono_viol = 0; mono_checked = 0; nscored = 0 +for q in qs + qi = parse(Int, q[1]); dir = joinpath(W, "q", @sprintf("q%03d", qi)); kd = joinpath(dir, kind) + isdir(kd) || continue + jp = joinpath(kd, "RAxML_portableTree.epa.jplace") + if !isfile(jp); println(pq, qi, "\t", q[2], "\tMISSING\t0\t0\t0\t0\t0"); continue; end + g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String))) + truthleaves = split(strip(read(joinpath(dir, "truth.txt"), String)), "\n") + refs = sort([n for n in names if n != q[2]]); rl = leafidx[refs[1]] + nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") + par, kids, below, order = root_at(g, nid[refs[1]], leafidx) + # edge above node u (u != root leaf) <-> below[u]; root leaf's pendant edge == edge above its only child + root_child = first(kids[nid[refs[1]]]) + edgekey = Dict{BitSet,Int}() # bipartition (side without refs[1]) -> node + for u in order; u == nid[refs[1]] && continue; edgekey[below[u]] = u; end + ne = length(edgekey) + 0 # includes root_child's edge == pendant of root leaf + expected = 2 * (N - 1) - 3 + # jplace tree + tree, fields, rows = parse_jplace(jp) + jg = parse_nwk(tree) + jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "") + @assert !haskey(jn, "qry") "query leaf present in jplace tree" + jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx) + enum2node = Dict{Int,Int}(); jkey = Dict{Int,BitSet}() + for u in jorder; u == jn[refs[1]] && continue; e = jg.lab[u][jpar[u]]; enum2node[e] = u; jkey[e] = jbelow[u]; end + # the edge above the root leaf's only child and the pendant edge of root leaf are the same unrooted edge: jplace labels it once + nj = length(jkey) + if nj != expected; global nedges_ok = false; end + # edge-number -> bipartition uniqueness + if length(Set(values(jkey))) != length(jkey); global distinct_ok = false; end + println(ck, "q$qi: jplace edges=$(length(jkey)) distinct_bipartitions=$(length(Set(values(jkey)))) expected=$expected pruned_tree_edges=$(length(edgekey)) all_in_pruned=$(all(k -> haskey(edgekey, k), values(jkey)))") + ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields) + pls = [(Int(r[ei]), r[li]) for r in rows] + total = sum(p[2] for p in pls) + # best placement + bi = argmax([p[2] for p in pls]); bkey = jkey[pls[bi][1]] + tkey = BitSet(leafidx[n] for n in truthleaves) + # truth is the below-set of its child in the pruned tree rooted at refs[1] + @assert haskey(edgekey, tkey) "truth bipartition not an edge of pruned tree" + function apart(a, b) + a == b && return 0 + da = Dict{Int,Int}(); u = a; k = 0; while u != 0 && u != nid[refs[1]]; da[u] = k; u = par[u]; k += 1; end + u = b; k = 0; while u != 0 && u != nid[refs[1]]; haskey(da, u) && return da[u] + k; u = par[u]; k += 1; end + typemax(Int) + end + tnode = edgekey[tkey]; bnode = edgekey[bkey] + # root-leaf pendant edge: represent by root_child's node (same edge) + println(pq, qi, "\t", q[2], "\tOK\t", length(pls), "\t", total, "\t", pls[bi][2], "\t", Int(bkey == tkey), "\t", apart(bnode, tnode)) + # clades: internal nodes other than the root leaf, whose parent... internal non-leaf nodes of the rooted tree (root leaf excluded) + plnode = [(edgekey[jkey[e]], w) for (e, w) in pls] + clades = [u for u in order if u != nid[refs[1]] && !isleaf(g, u)] + masses = Dict{Int,Float64}() + for c in clades + S = below[c]; m = 0.0 + for (nd, w) in plnode; below[nd] != S && issubset(below[nd], S) && (m += w); end + masses[c] = m + println(pc, qi, "\t", length(S), "\t", m, "\t", Int(tkey != S && issubset(tkey, S)), "\t", N - 1) + end + # monotonicity under inclusion: mass(child) <= mass(parent) for parent clade strictly containing + for c in clades; p = par[c]; (p == nid[refs[1]] || isleaf(g, p)) && continue; global mono_checked += 1; masses[c] <= masses[p] + 1e-12 || (global mono_viol += 1); end + global nscored += 1 +end +println(ck, "SUMMARY scored=$nscored edge_count_ok=$nedges_ok distinct_bipartitions_ok=$distinct_ok monotone_pairs_checked=$mono_checked monotone_violations=$mono_viol") +close(pq); close(pc); close(ck) +println("scored $nscored; ", read(joinpath(R, "checks_$(draw)_$kind.txt"), String)[end-200:end]) diff --git a/docs/cladistics/issue-303/epa-depth/05_run_epa.sh b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh new file mode 100755 index 00000000..387bd635 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh @@ -0,0 +1,11 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: 05_run_epa.sh epa_dir kind (kind = amp | pos); same command as the application's place step, -T 2, two jobs in parallel +E=$1; K=$2 +s=$(date +%s) +ls -d "$E"/work/q/q*/"$K" | xargs -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ +echo "EPA $E $K: $(ls -d $E/work/q/q*/$K | wc -l) runs, wall clock $(( $(date +%s) - s )) s" >> $E/../raw/epa_timing.txt diff --git a/docs/cladistics/issue-303/epa-depth/06_tables.jl b/docs/cladistics/issue-303/epa-depth/06_tables.jl new file mode 100644 index 00000000..88918fd6 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/06_tables.jl @@ -0,0 +1,168 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 06_tables.jl new RAW_DIR -> T1-T6 for draws A, B and pooled (amp), plus pos for A +# julia 06_tables.jl old OLD_RAW_DIR -> reproduces the N=200 headline numbers from ../epa/raw/perclade_amp.tsv (no n_refs column there) +using Printf, Statistics +# ---------- Clopper-Pearson (exact, two-sided) by bisection on the BigFloat binomial CDF ---------- +setprecision(BigFloat, 256) +function bcdf(x::Int, n::Int, p::BigFloat) # P(X <= x) + x < 0 && return big"0.0"; x >= n && return big"1.0" + s = big"0.0"; q = 1 - p + for i in 0:x; s += binomial(big(n), big(i)) * p^i * q^(n - i); end + s +end +function cp_interval(x::Int, n::Int; level = 0.05) + a = BigFloat(level) / 2 + lo = big"0.0"; hi = big"1.0" + if x > 0 # lower: P(X >= x | p) = a (increasing in p) + l, h = big"0.0", big"1.0" + for _ in 1:200; m = (l + h) / 2; (1 - bcdf(x - 1, n, m) < a) ? (l = m) : (h = m); end + lo = (l + h) / 2 + end + if x < n # upper: P(X <= x | p) = a (decreasing in p) + l, h = big"0.0", big"1.0" + for _ in 1:200; m = (l + h) / 2; (bcdf(x, n, m) > a) ? (l = m) : (h = m); end + hi = (l + h) / 2 + else hi = big"1.0" end + Float64(lo), Float64(hi) +end +# ---------- loading ---------- +struct DS; label::String; pq::Dict{Int,Vector{String}}; cl::Dict{Int,Vector{Tuple{Int,Float64,Bool}}}; nref::Int; end +function load(label, pqpath, pcpath; keyoff = 0, nref_default = 199) + pq = Dict{Int,Vector{String}}(); cl = Dict{Int,Vector{Tuple{Int,Float64,Bool}}}() + rows = [split(l, "\t") for l in eachline(pqpath)][2:end] + for r in rows; r[3] == "OK" && (pq[parse(Int, r[1]) + keyoff] = String.(r)); end + hdr = split(readline(pcpath), "\t"); nref = nref_default + for l in Iterators.drop(eachline(pcpath), 1) + r = split(l, "\t"); qi = parse(Int, r[1]) + keyoff + push!(get!(cl, qi, Tuple{Int,Float64,Bool}[]), (parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")) + length(r) >= 5 && (nref = parse(Int, r[5])) + end + DS(label, pq, cl, nref) +end +merge_ds(label, a::DS, b::DS) = DS(label, merge(a.pq, b.pq), merge(a.cl, b.cl), a.nref) +isent(m, alpha) = m >= 1 - alpha - 1e-12 +isref(m, alpha) = m <= alpha + 1e-12 +# ---------- T1 ---------- +function t1(d::DS) + ok = [d.pq[q] for q in sort(collect(keys(d.pq)))] + ex = [r[7] == "1" for r in ok]; ap = [parse(Int, r[8]) for r in ok]; tot = [parse(Float64, r[5]) for r in ok]; npl = [parse(Int, r[4]) for r in ok] + @printf(" queries scored %d; placed on exactly the true edge: %d of %d (%.1f%%); within 2 edges: %d of %d; median edges apart %g\n", length(ok), sum(ex), length(ok), 100mean(ex), count(<=(2), ap), length(ok), median(ap)) + @printf(" placements per query: median %g, max %d, queries with a single placement: %d of %d; total LWR per query: min %.4f, median %.4f\n", median(npl), maximum(npl), count(==(1), npl), length(ok), minimum(tot), median(tot)) + for alpha in (0.01, 0.05) + eo = eb = ro = rb = ui = uo = 0 + for cl in values(d.cl), (sz, m, t) in cl + if isent(m, alpha); t ? (eo += 1) : (eb += 1) + elseif isref(m, alpha); t ? (rb += 1) : (ro += 1) + else t ? (ui += 1) : (uo += 1) end + end + @printf(" alpha %.2f: entailed %d, wrong %d | refuted %d, wrong %d | unresolved %d (truth inside %d, truth not inside %d)\n", alpha, eo + eb, eb, ro + rb, rb, ui + uo, ui, uo) + end +end +# ---------- T2 ---------- +const BINS = [(2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 10^6)] +binlabel(b) = b[2] >= 10^6 ? @sprintf("%d-%d", b[1], 399) : @sprintf("%d-%d", b[1], b[2]) +function t2(d::DS, alpha) + println(" bin entailed verdicts wrong distinct queries with a wrong verdict") + for b in BINS + ne = nw = 0; wq = Set{Int}() + for (q, cl) in d.cl, (sz, m, t) in cl + (b[1] <= sz <= b[2] && isent(m, alpha)) || continue + ne += 1; if !t; nw += 1; push!(wq, q); end + end + @printf(" %-9s %9d %12d %12d\n", binlabel(b), ne, nw, length(wq)) + end +end +# ---------- T3 ---------- +function smallest_entailed(d::DS, alpha, k) # -> Dict qi => (size, truth_inside) + out = Dict{Int,Tuple{Int,Bool}}() + for (q, cl) in d.cl + best = nothing + for (sz, m, t) in cl + (sz >= k && isent(m, alpha)) || continue + (best === nothing || sz < best[1]) && (best = (sz, t)) + end + best !== nothing && (out[q] = best) + end + out +end +function t3(d::DS, alpha; ks = (2, 4, 8, 16, 32, 64)) + nq = length(d.pq) + println(" k queries with a clade (of $nq) median size wrong wrong rate exact 95% CI (Clopper-Pearson)") + for k in ks + s = smallest_entailed(d, alpha, k); n = length(s) + if n == 0; @printf(" %-4d %6d\n", k, 0); continue; end + w = count(v -> !v[2], values(s)); lo, hi = cp_interval(w, n) + @printf(" %-4d %6d %18g %14d %9.4f (%.4f, %.4f) [%d of %d]\n", k, n, median([v[1] for v in values(s)]), w, w / n, lo, hi, w, n) + end +end +# ---------- T4 ---------- +function t4(ds::Vector{DS}, pooled::DS, alpha) + for d in vcat(ds, [pooled]) + s = smallest_entailed(d, alpha, 2); wr = sort([q for (q, v) in s if !v[2]]) + npl1 = count(q -> d.pq[q][4] == "1", wr); b99 = count(q -> parse(Float64, d.pq[q][6]) >= 0.99, wr) + @printf(" %s: %d queries with an entailed clade, %d wrong; of the wrong: single reported placement %d of %d; best LWR >= 0.99: %d of %d\n", d.label, length(s), length(wr), npl1, length(wr), b99, length(wr)) + end + println(" list (key = draw+query index; pooled keys: A = qi, B = 1000+qi): key clade_size edges_best_to_truth placements best_lwr") + for q in sort([q for (q, v) in smallest_entailed(pooled, alpha, 2) if !v[2]]) + r = pooled.pq[q]; sz = smallest_entailed(pooled, alpha, 2)[q][1] + @printf(" %5d %6d %6s %6s %10.4f\n", q, sz, r[8], r[4], parse(Float64, r[6])) + end +end +# ---------- T5 ---------- +function t5(d::DS) + edges = [0.05, 0.25, 0.5, 0.75, 0.95]; hit = zeros(Int, 7); cnt = zeros(Int, 7) + for cl in values(d.cl), (sz, m, t) in cl + b = m <= 0 ? 1 : m > 0.95 + 1e-12 ? 7 : 1 + findfirst(x -> m <= x + 1e-12, edges) + cnt[b] += 1; hit[b] += t + end + labels = ["0", "(0, .05]", "(.05, .25]", "(.25, .5]", "(.5, .75]", "(.75, .95]", "(.95, 1]"] + for b in 1:7; cnt[b] > 0 && @printf(" mass %-11s clades %8d truth inside %8d (%.2f%%)\n", labels[b], cnt[b], hit[b], 100hit[b] / cnt[b]); end +end +# ---------- T6 ---------- +function t6(A::DS, B::DS) + for alpha in (0.01, 0.05) + @printf(" alpha %.2f k draw A: wrong/n rate CI | draw B: wrong/n rate CI\n", alpha) + for k in (2, 8, 32) + f(d) = begin s = smallest_entailed(d, alpha, k); n = length(s); w = count(v -> !v[2], values(s)); lo, hi = cp_interval(w, n); @sprintf("%d/%d %.4f (%.4f, %.4f)", w, n, w / n, lo, hi) end + @printf(" %-3d %s | %s\n", k, f(A), f(B)) + end + end +end +# ---------- main ---------- +mode, R = ARGS[1], ARGS[2] +if mode == "cptest" + for (x, n) in ((0, 100), (7, 146), (13, 2388), (5, 5), (0, 1)) + lo, hi = cp_interval(x, n); @printf("CP %d of %d: (%.4f, %.4f)\n", x, n, lo, hi) + end +elseif mode == "old" + d = load("old", joinpath(R, "perquery_amp.tsv"), joinpath(R, "perclade_amp.tsv")) + for alpha in (0.01, 0.05) + eo = eb = 0 + for cl in values(d.cl), (sz, m, t) in cl; isent(m, alpha) && (t ? (eo += 1) : (eb += 1)); end + @printf("OLD N=200 reproduction, alpha %.2f: entailed %d wrong %d\n", alpha, eo + eb, eb) + end + s = smallest_entailed(d, 0.01, 2); @printf("OLD smallest entailed clade alpha 0.01: queries with clade %d, wrong %d\n", length(s), count(v -> !v[2], values(s))) +else + A = load("A", joinpath(R, "perquery_A_amp.tsv"), joinpath(R, "perclade_A_amp.tsv")) + B = load("B", joinpath(R, "perquery_B_amp.tsv"), joinpath(R, "perclade_B_amp.tsv"), keyoff = 1000) + P = merge_ds("pooled", A, B) + Apos = load("A-pos", joinpath(R, "perquery_A_pos.tsv"), joinpath(R, "perclade_A_pos.tsv")) + println("Per-query unit note: verdicts on nested clades of one query are NOT independent; T3 (one result per query) is the statistically honest unit. T2 counts verdicts and is descriptive only.") + println("Clade size = number of references below the node; n_refs = $(A.nref) (pruned tree). Pooled = draw A + draw B queries (a reference present in both draws may be a query in both).\n") + for d in (A, B, P) + println("\n################ ", d.label, " (amp) ################"); println("T1"); t1(d) + for alpha in (0.01, 0.05) + @printf("\nT2 alpha %.2f (%s)\n", alpha, d.label); t2(d, alpha) + @printf("\nT3 alpha %.2f (%s)\n", alpha, d.label); t3(d, alpha) + end + println("\nT5 calibration (", d.label, ")"); t5(d) + end + println("\n################ A-pos (positive control: full-length sequence, first 60 queries of draw A) ################"); t1(Apos) + for alpha in (0.01, 0.05); @printf("\nT3 alpha %.2f (A-pos)\n", alpha); t3(Apos, alpha); end + for alpha in (0.01, 0.05); @printf("\nT4 alpha %.2f (smallest entailed clade, k=2, wrong)\n", alpha); t4([A, B], P, alpha); end + println("\nT6 agreement between draws"); t6(A, B) +end diff --git a/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl new file mode 100644 index 00000000..691e8534 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# Cross-check of 06_tables.jl, written separately: the size-bin table, the per-query rule, the +# errors on the largest clades, and the same errors binned by the smaller side of the branch. +# usage: julia 08_crosscheck.jl RAW_DIR +using Printf, Statistics +function load(tag) + C = [(r = split(l, '\t'); (parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")) for l in readlines(joinpath(ARGS[1], "perclade_$tag.tsv"))[2:end]] + Q = Dict((r = split(l, '\t'); parse(Int, r[1]) => (n = parse(Int, r[4]), best = parse(Float64, r[6]), exact = r[7] == "1", apart = parse(Int, r[8]))) for l in readlines(joinpath(ARGS[1], "perquery_$tag.tsv"))[2:end]) + C, Q +end +A, QA = load("A_amp"); B, QB = load("B_amp"); P, QP = load("A_pos") +println("rows A $(length(A)) B $(length(B)); queries A $(length(QA)) B $(length(QB)); pos exact $(count(q -> q.exact, values(QP))) of $(length(QP))") +bins = ((2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 399)) +for a in (0.01, 0.05) + tot = 0; wr = 0 + for (lo, hi) in bins + s = ""; n = 0; w = 0; qs = Set{Tuple{Char,Int}}() + for (tag, C) in (('A', A), ('B', B)), c in C + (c[3] >= 1 - a && lo <= c[2] <= hi) || continue + n += 1; c[4] || (w += 1; push!(qs, (tag, c[1]))) + end + tot += n; wr += w + @printf(" alpha %.2f size %3d..%3d: entailed %5d wrong %2d (queries %d)\n", a, lo, hi, n, w, length(qs)) + end + println(" alpha $a total entailed $tot wrong $wr") + for k in (2, 4, 8, 16, 32, 64) + n = 0; w = 0; sizes = Int[] + for C in (A, B) + sm = Dict{Int,Tuple{Int,Bool}}() + for c in C; (c[3] >= 1 - a && c[2] >= k) || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + n += length(sm); w += count(v -> !v[2], values(sm)); append!(sizes, [v[1] for v in values(sm)]) + end + @printf(" alpha %.2f k>=%2d: queries %d median size %.1f wrong %d\n", a, k, n, median(sizes), w) + end +end +println("wrong entailed verdicts on clades of 128+ at alpha 0.01 (draw, query, clade size, edges apart, placements, best LWR):") +for (tag, C, Q) in (('A', A, QA), ('B', B, QB)), c in C + (c[3] >= 0.99 && c[2] >= 128 && !c[4]) && println(" $tag q$(c[1]) size $(c[2]) apart $(Q[c[1]].apart) n $(Q[c[1]].n) best $(round(Q[c[1]].best, digits=3)) mass $(round(c[3], digits=4))") +end +# How far is the best placement from the truth for wrong vs right smallest-entailed queries (k=2) +for (tag, C, Q) in (('A', A, QA), ('B', B, QB)) + sm = Dict{Int,Tuple{Int,Bool}}() + for c in C; c[3] >= 0.99 || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + wq = [q for (q, v) in sm if !v[2]] + println(" draw $tag: wrong smallest-entailed queries $(length(wq)); edges apart: ", sort([Q[q].apart for q in wq])) +end + +# ---- by the smaller side of the branch: min(clade size, references outside it) ---- +rows2(tag) = [(r = split(l, '\t'); (tag, parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1", parse(Int, r[5]))) for l in readlines(joinpath(ARGS[1], "perclade_$tag.tsv"))[2:end]] +C2 = vcat(rows2("A_amp"), rows2("B_amp")) +for a in (0.01, 0.05) + for (lo, hi) in ((1, 1), (2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 200)) + n = 0; w = 0; qs = Set{Tuple{String,Int}}() + for c in C2 + m = min(c[3], c[6] - c[3]); (c[4] >= 1 - a && lo <= m <= hi) || continue + n += 1; c[5] || (w += 1; push!(qs, (c[1], c[2]))) + end + @printf(" alpha %.2f smaller side %3d..%3d: entailed %5d wrong %2d (queries %d)\n", a, lo, hi, n, w, length(qs)) + end +end diff --git a/docs/cladistics/issue-303/epa-depth/run_rest.sh b/docs/cladistics/issue-303/epa-depth/run_rest.sh new file mode 100755 index 00000000..1cc63721 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/run_rest.sh @@ -0,0 +1,16 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: run_rest.sh WORK (after run_trees.sh has finished) +# 300 amplicon queries per draw (selection seeds 7 and 8); whole-sequence control for the +# first 60 queries of draw A. +D=$1 +cd "$D" || exit 1 +julia -O2 scripts/03_prep.jl "$D/A" 300 7 60 > raw/03_prep_A.log 2>&1 +julia -O2 scripts/03_prep.jl "$D/B" 300 8 0 > raw/03_prep_B.log 2>&1 +sh scripts/05_run_epa.sh "$D/A" amp +sh scripts/05_run_epa.sh "$D/A" pos +sh scripts/05_run_epa.sh "$D/B" amp diff --git a/docs/cladistics/issue-303/epa-depth/run_trees.sh b/docs/cladistics/issue-303/epa-depth/run_trees.sh new file mode 100755 index 00000000..71358f7f --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/run_trees.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: run_trees.sh WORK /path/to/silva.bacteria.fasta +# WORK holds scripts/ (../epa/ with this directory copied over it) and receives A/, B/ and raw/. +D=$1; SILVA=$2 +mkdir -p "$D/raw" +for X in A B; do + seed=20261117; [ "$X" = B ] && seed=20261118 + mkdir -p "$D/$X/work" + julia -O2 "$D/scripts/01_select.jl" "$SILVA" "$D/$X/work" 400 "$seed" > "$D/raw/01_select_$X.log" 2>&1 +done +sh "$D/scripts/02_tree.sh" "$D/A" A +sh "$D/scripts/02_tree.sh" "$D/B" B From e506a9182e795abe8bda3dbe468eead0bde310d2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 04:17:11 +0000 Subject: [PATCH 3/5] docs(cladistics): make the issue 303 experiment 8 and 9 scripts runnable from any path The experiment 8 and 9 scripts were changed after the published runs, in response to review of PR 348. The published runs used absolute paths from a scratch directory; the committed copies had only been lightly edited and had not been run end to end. They have now been run end to end at a small size (40 references, 6 queries), once with an absolute WORK containing a space and once with a relative WORK, through the tables and the cross-check, with all self-checks reporting ok. Changed: - Shell drivers resolve directory arguments to absolute paths before any cd, quote every path, use set -eu, and stop on a failed stage. 05_run_epa.sh dispatches NUL-delimited paths to xargs -0 -P 2 and fails if any RAxML worker fails; the timing line is written only on success. run_trees.sh and run_rest.sh stop at the first failed stage. - Optional trailing size arguments (references, queries, controls, bootstrap timeout, threads); defaults are the published sizes. - epa-depth/03_prep.jl and epa/03_prep.jl remove the managed work/q directory before writing, so a smaller control count leaves no stale pos directories. - epa/01_select.jl checks V4 bases after column filtering and fails with the row name instead of failing later in 03_prep.jl. The selection itself is unchanged. - epa/08_by_depth.jl and epa-depth/08_crosscheck.jl use the 1e-12 threshold tolerance of the 06_tables.jl scripts (aggregation code stays separate) and print n/a for the median of an empty set. - README "Running": states that WORK may be relative or contain spaces, lists the size arguments, and wraps the scoring loops so a failure is not masked. Parity: 06_tables.jl and the cross-check scripts were run on the original raw outputs before and after; the output is identical. No reported number, table or conclusion changed. 01_select.jl selects byte-identical references for N=200 seed 20261117 and N=400 seeds 20261117 and 20261118. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01J4QvDiW8qD4f2JDQJ4buGo --- docs/cladistics/issue-303/README.md | 17 ++++++++--- .../cladistics/issue-303/epa-depth/02_tree.sh | 29 ++++++++++++------- .../cladistics/issue-303/epa-depth/03_prep.jl | 3 ++ .../issue-303/epa-depth/05_run_epa.sh | 14 +++++++-- .../issue-303/epa-depth/08_crosscheck.jl | 15 ++++++---- .../issue-303/epa-depth/run_rest.sh | 19 +++++++----- .../issue-303/epa-depth/run_trees.sh | 18 +++++++----- docs/cladistics/issue-303/epa/01_run.sh | 13 +++++---- docs/cladistics/issue-303/epa/01_select.jl | 6 ++++ docs/cladistics/issue-303/epa/02_tree.sh | 9 ++++-- docs/cladistics/issue-303/epa/03_prep.jl | 3 ++ docs/cladistics/issue-303/epa/05_run_epa.sh | 11 +++++-- docs/cladistics/issue-303/epa/08_by_depth.jl | 10 ++++--- 13 files changed, 114 insertions(+), 53 deletions(-) diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 304ea597..a1e7bf98 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -436,8 +436,10 @@ julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP julia -O2 -t 4 docs/cladistics/issue-303/placement_evidence.jl /path/to/silva.bacteria.fasta ``` -Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. Copy `epa/` to -`WORK/scripts`, then run its steps in order: +Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. `WORK` is a directory +of your choice, absolute or relative, and may contain spaces; the drivers resolve it to an +absolute path and stop with a non-zero status if a stage fails. Copy `epa/` to `WORK/scripts`, +then run its steps in order: ```bash sh WORK/scripts/01_run.sh WORK /path/to/silva.bacteria.fasta @@ -452,7 +454,7 @@ julia WORK/scripts/03_prep.jl WORK 200 7 ``` ```bash -for k in amp pos neg; do sh WORK/scripts/05_run_epa.sh WORK $k && julia WORK/scripts/04_score.jl WORK $k; done +(set -e; for k in amp pos neg; do sh WORK/scripts/05_run_epa.sh WORK $k; julia WORK/scripts/04_score.jl WORK $k; done) ``` ```bash @@ -463,6 +465,13 @@ julia WORK/scripts/06_tables.jl WORK amp pos neg julia WORK/scripts/08_by_depth.jl WORK ``` +The drivers take optional trailing size arguments for a quick trial; without them they run the +published sizes. `01_run.sh` takes the number of references (default 200), `02_tree.sh` the +thread count (default 4), and `03_prep.jl` the number of queries. For experiment 9, +`run_trees.sh` takes the references per draw (default 400), the bootstrap timeout in seconds +(default 1500) and the threads (default 4), and `run_rest.sh` takes the queries per draw +(default 300) and the whole-sequence controls (default 60). + Experiment 9: copy `epa/` and then `epa-depth/` over it to `WORK/scripts`, then: ```bash @@ -474,7 +483,7 @@ sh WORK/scripts/run_rest.sh WORK ``` ```bash -mkdir -p WORK/raw && for t in "A amp" "A pos" "B amp"; do set -- $t; julia WORK/scripts/04_score.jl WORK/$1 $1 $2; done +mkdir -p WORK/raw && (set -e; for t in "A amp" "A pos" "B amp"; do set -- $t; julia WORK/scripts/04_score.jl WORK/$1 $1 $2; done) ``` ```bash diff --git a/docs/cladistics/issue-303/epa-depth/02_tree.sh b/docs/cladistics/issue-303/epa-depth/02_tree.sh index 891cfb27..2274d30c 100755 --- a/docs/cladistics/issue-303/epa-depth/02_tree.sh +++ b/docs/cladistics/issue-303/epa-depth/02_tree.sh @@ -4,18 +4,25 @@ # Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the # application's placement step. See ../README.md, section 9. These files replace their # namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. -# usage: 02_tree.sh epa_dir draw -- IQ-TREE GTR+G4 with UFBoot 1000 (4 threads); fallback without -B after 25 min wall clock -E=$1; DR=$2; T=$E/work/tree; mkdir -p $T; cd $T -LOG=$E/../raw/tree_$DR.log +# usage: 02_tree.sh epa_dir draw [timeout_s] [threads] +# IQ-TREE GTR+G4 with UFBoot 1000 (default 4 threads); fallback without -B after timeout_s +# (default 1500 s = 25 min wall clock). Logs go to ../raw relative to epa_dir. +set -eu +mkdir -p "$1/work" +E=$(cd "$1" && pwd) || exit 1; DR=$2; TMO=${3:-1500}; NT=${4:-4} +T=$E/work/tree; mkdir -p "$T"; cd "$T" +RAW=$(cd "$E/.." && pwd)/raw +LOG=$RAW/tree_$DR.log s=$(date +%s) -timeout 1500 iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -B 1000 -nt 4 -seed 1 -pre $T/reference -redo > $E/../raw/02_iqtree_$DR.stdout 2>&1 -rc=$? -if [ $rc -ne 0 ] || [ ! -s $T/reference.treefile ]; then - echo "draw $DR: -B 1000 run rc=$rc after $(( $(date +%s) - s )) s; RERUN WITHOUT -B" > $LOG +rc=0 +timeout "$TMO" iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -B 1000 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.stdout" 2>&1 || rc=$? +if [ "$rc" -ne 0 ] || [ ! -s "$T/reference.treefile" ]; then + echo "draw $DR: -B 1000 run rc=$rc after $(( $(date +%s) - s )) s; RERUN WITHOUT -B" > "$LOG" s=$(date +%s) - iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -nt 4 -seed 1 -pre $T/reference -redo > $E/../raw/02_iqtree_$DR.nob.stdout 2>&1 - echo "draw $DR: rerun without -B took $(( $(date +%s) - s )) s" >> $LOG + iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.nob.stdout" 2>&1 + echo "draw $DR: rerun without -B took $(( $(date +%s) - s )) s" >> "$LOG" else - echo "draw $DR: -B 1000 completed, $(( $(date +%s) - s )) s wall clock, rc=0" > $LOG + echo "draw $DR: -B 1000 completed, $(( $(date +%s) - s )) s wall clock, rc=0" > "$LOG" fi -cp $T/reference.treefile $T/reference_with_support.treefile +[ -s "$T/reference.treefile" ] || { echo "02_tree.sh: no reference.treefile for draw $DR" >&2; exit 1; } +cp "$T/reference.treefile" "$T/reference_with_support.treefile" diff --git a/docs/cladistics/issue-303/epa-depth/03_prep.jl b/docs/cladistics/issue-303/epa-depth/03_prep.jl index f212b608..75afd6a7 100644 --- a/docs/cladistics/issue-303/epa-depth/03_prep.jl +++ b/docs/cladistics/issue-303/epa-depth/03_prep.jl @@ -9,6 +9,9 @@ using Random, Printf include(joinpath(@__DIR__, "lib_nwk.jl")) E, Q, seed, NPOS = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]), parse(Int, ARGS[4]) W = joinpath(E, "work") +# start from a clean query directory: stale q*/ variants (e.g. pos from an earlier run with more +# controls) must not survive into the scorer +rm(joinpath(W, "q"); recursive = true, force = true) names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) leafidx = Dict(n => i for (i, n) in enumerate(names)) seqs = Dict{String,String}(); cur = "" diff --git a/docs/cladistics/issue-303/epa-depth/05_run_epa.sh b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh index 387bd635..8aabec67 100755 --- a/docs/cladistics/issue-303/epa-depth/05_run_epa.sh +++ b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh @@ -5,7 +5,15 @@ # application's placement step. See ../README.md, section 9. These files replace their # namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. # usage: 05_run_epa.sh epa_dir kind (kind = amp | pos); same command as the application's place step, -T 2, two jobs in parallel -E=$1; K=$2 +set -eu +E=$(cd "$1" && pwd) || exit 1; K=$2 +RAW=$(cd "$E/.." && pwd)/raw s=$(date +%s) -ls -d "$E"/work/q/q*/"$K" | xargs -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ -echo "EPA $E $K: $(ls -d $E/work/q/q*/$K | wc -l) runs, wall clock $(( $(date +%s) - s )) s" >> $E/../raw/epa_timing.txt +# one NUL-terminated path per query directory, so paths with spaces stay whole; xargs exits 123 if any worker failed +n=0 +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && n=$((n + 1)); done +[ "$n" -gt 0 ] || { echo "05_run_epa.sh: no query directories for kind $K under $E/work/q" >&2; exit 1; } +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && printf "%s\0" "$D"; done | + xargs -0 -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ || + { echo "05_run_epa.sh: at least one EPA run failed (kind $K); see raxml.stdout in the query directories" >&2; exit 1; } +echo "EPA $E $K: $n runs, wall clock $(( $(date +%s) - s )) s" >> "$RAW/epa_timing.txt" diff --git a/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl index 691e8534..3af94564 100644 --- a/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl +++ b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl @@ -7,6 +7,9 @@ # errors on the largest clades, and the same errors binned by the smaller side of the branch. # usage: julia 08_crosscheck.jl RAW_DIR using Printf, Statistics +# Same boundary tolerance as 06_tables.jl (isent), so a mass of 0.9899999999999999 counts at alpha 0.01. +const TOL = 1e-12 +medstr(x) = isempty(x) ? "n/a" : @sprintf("%.1f", median(x)) function load(tag) C = [(r = split(l, '\t'); (parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")) for l in readlines(joinpath(ARGS[1], "perclade_$tag.tsv"))[2:end]] Q = Dict((r = split(l, '\t'); parse(Int, r[1]) => (n = parse(Int, r[4]), best = parse(Float64, r[6]), exact = r[7] == "1", apart = parse(Int, r[8]))) for l in readlines(joinpath(ARGS[1], "perquery_$tag.tsv"))[2:end]) @@ -20,7 +23,7 @@ for a in (0.01, 0.05) for (lo, hi) in bins s = ""; n = 0; w = 0; qs = Set{Tuple{Char,Int}}() for (tag, C) in (('A', A), ('B', B)), c in C - (c[3] >= 1 - a && lo <= c[2] <= hi) || continue + (c[3] >= 1 - a - TOL && lo <= c[2] <= hi) || continue n += 1; c[4] || (w += 1; push!(qs, (tag, c[1]))) end tot += n; wr += w @@ -31,20 +34,20 @@ for a in (0.01, 0.05) n = 0; w = 0; sizes = Int[] for C in (A, B) sm = Dict{Int,Tuple{Int,Bool}}() - for c in C; (c[3] >= 1 - a && c[2] >= k) || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + for c in C; (c[3] >= 1 - a - TOL && c[2] >= k) || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end n += length(sm); w += count(v -> !v[2], values(sm)); append!(sizes, [v[1] for v in values(sm)]) end - @printf(" alpha %.2f k>=%2d: queries %d median size %.1f wrong %d\n", a, k, n, median(sizes), w) + @printf(" alpha %.2f k>=%2d: queries %d median size %s wrong %d\n", a, k, n, medstr(sizes), w) end end println("wrong entailed verdicts on clades of 128+ at alpha 0.01 (draw, query, clade size, edges apart, placements, best LWR):") for (tag, C, Q) in (('A', A, QA), ('B', B, QB)), c in C - (c[3] >= 0.99 && c[2] >= 128 && !c[4]) && println(" $tag q$(c[1]) size $(c[2]) apart $(Q[c[1]].apart) n $(Q[c[1]].n) best $(round(Q[c[1]].best, digits=3)) mass $(round(c[3], digits=4))") + (c[3] >= 0.99 - TOL && c[2] >= 128 && !c[4]) && println(" $tag q$(c[1]) size $(c[2]) apart $(Q[c[1]].apart) n $(Q[c[1]].n) best $(round(Q[c[1]].best, digits=3)) mass $(round(c[3], digits=4))") end # How far is the best placement from the truth for wrong vs right smallest-entailed queries (k=2) for (tag, C, Q) in (('A', A, QA), ('B', B, QB)) sm = Dict{Int,Tuple{Int,Bool}}() - for c in C; c[3] >= 0.99 || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + for c in C; c[3] >= 0.99 - TOL || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end wq = [q for (q, v) in sm if !v[2]] println(" draw $tag: wrong smallest-entailed queries $(length(wq)); edges apart: ", sort([Q[q].apart for q in wq])) end @@ -56,7 +59,7 @@ for a in (0.01, 0.05) for (lo, hi) in ((1, 1), (2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 200)) n = 0; w = 0; qs = Set{Tuple{String,Int}}() for c in C2 - m = min(c[3], c[6] - c[3]); (c[4] >= 1 - a && lo <= m <= hi) || continue + m = min(c[3], c[6] - c[3]); (c[4] >= 1 - a - TOL && lo <= m <= hi) || continue n += 1; c[5] || (w += 1; push!(qs, (c[1], c[2]))) end @printf(" alpha %.2f smaller side %3d..%3d: entailed %5d wrong %2d (queries %d)\n", a, lo, hi, n, w, length(qs)) diff --git a/docs/cladistics/issue-303/epa-depth/run_rest.sh b/docs/cladistics/issue-303/epa-depth/run_rest.sh index 1cc63721..e2b29dfc 100755 --- a/docs/cladistics/issue-303/epa-depth/run_rest.sh +++ b/docs/cladistics/issue-303/epa-depth/run_rest.sh @@ -4,13 +4,16 @@ # Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the # application's placement step. See ../README.md, section 9. These files replace their # namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. -# usage: run_rest.sh WORK (after run_trees.sh has finished) -# 300 amplicon queries per draw (selection seeds 7 and 8); whole-sequence control for the -# first 60 queries of draw A. -D=$1 -cd "$D" || exit 1 -julia -O2 scripts/03_prep.jl "$D/A" 300 7 60 > raw/03_prep_A.log 2>&1 -julia -O2 scripts/03_prep.jl "$D/B" 300 8 0 > raw/03_prep_B.log 2>&1 +# usage: run_rest.sh WORK [Q [NPOS]] (after run_trees.sh has finished) +# Q amplicon queries per draw (selection seeds 7 and 8), default 300; whole-sequence control for +# the first NPOS queries of draw A, default 60. WORK may be relative. Any failed stage stops +# the script with a non-zero status. +set -eu +D=$(cd "$1" && pwd) || exit 1 +Q=${2:-300}; NPOS=${3:-60} +cd "$D" +julia -O2 scripts/03_prep.jl "$D/A" "$Q" 7 "$NPOS" > raw/03_prep_A.log 2>&1 +julia -O2 scripts/03_prep.jl "$D/B" "$Q" 8 0 > raw/03_prep_B.log 2>&1 sh scripts/05_run_epa.sh "$D/A" amp -sh scripts/05_run_epa.sh "$D/A" pos +if [ "$NPOS" -gt 0 ]; then sh scripts/05_run_epa.sh "$D/A" pos; fi sh scripts/05_run_epa.sh "$D/B" amp diff --git a/docs/cladistics/issue-303/epa-depth/run_trees.sh b/docs/cladistics/issue-303/epa-depth/run_trees.sh index 71358f7f..c286aab9 100755 --- a/docs/cladistics/issue-303/epa-depth/run_trees.sh +++ b/docs/cladistics/issue-303/epa-depth/run_trees.sh @@ -4,14 +4,18 @@ # Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the # application's placement step. See ../README.md, section 9. These files replace their # namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. -# usage: run_trees.sh WORK /path/to/silva.bacteria.fasta -# WORK holds scripts/ (../epa/ with this directory copied over it) and receives A/, B/ and raw/. -D=$1; SILVA=$2 -mkdir -p "$D/raw" +# usage: run_trees.sh WORK /path/to/silva.bacteria.fasta [N [timeout_s [threads]]] +# WORK (absolute or relative) holds scripts/ (../epa/ with this directory copied over it) and +# receives A/, B/ and raw/. Defaults are the published sizes: N = 400 references per draw, +# UFBoot timeout 1500 s, 4 threads. Any failed stage stops the script with a non-zero status. +set -eu +mkdir -p "$1/raw" +D=$(cd "$1" && pwd) || exit 1 +SILVA=$2; N=${3:-400}; TMO=${4:-1500}; NT=${5:-4} for X in A B; do seed=20261117; [ "$X" = B ] && seed=20261118 mkdir -p "$D/$X/work" - julia -O2 "$D/scripts/01_select.jl" "$SILVA" "$D/$X/work" 400 "$seed" > "$D/raw/01_select_$X.log" 2>&1 + julia -O2 "$D/scripts/01_select.jl" "$SILVA" "$D/$X/work" "$N" "$seed" > "$D/raw/01_select_$X.log" 2>&1 done -sh "$D/scripts/02_tree.sh" "$D/A" A -sh "$D/scripts/02_tree.sh" "$D/B" B +sh "$D/scripts/02_tree.sh" "$D/A" A "$TMO" "$NT" +sh "$D/scripts/02_tree.sh" "$D/B" B "$TMO" "$NT" diff --git a/docs/cladistics/issue-303/epa/01_run.sh b/docs/cladistics/issue-303/epa/01_run.sh index 3444470f..340563e6 100755 --- a/docs/cladistics/issue-303/epa/01_run.sh +++ b/docs/cladistics/issue-303/epa/01_run.sh @@ -3,8 +3,11 @@ # SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) # Experiment E8 (issue 303): evidence verdicts on placements made by the application's own # placement step (RAxML EPA). See ../README.md, section 8. -# usage: 01_run.sh epa_dir silva.bacteria.fasta -# epa_dir holds scripts/ (a copy of this directory), and receives work/ and raw/. -E=$1; SILVA=$2 -mkdir -p "$E/work/tree" "$E/raw" -julia -O2 "$E/scripts/01_select.jl" "$SILVA" "$E/work" 200 20261117 +# usage: 01_run.sh epa_dir silva.bacteria.fasta [N] +# epa_dir (absolute or relative) holds scripts/ (a copy of this directory), and receives work/ and raw/. +# N = number of references, default 200 (the published size). +set -eu +mkdir -p "$1/work/tree" "$1/raw" +E=$(cd "$1" && pwd) || exit 1 +SILVA=$2; N=${3:-200} +julia -O2 "$E/scripts/01_select.jl" "$SILVA" "$E/work" "$N" 20261117 diff --git a/docs/cladistics/issue-303/epa/01_select.jl b/docs/cladistics/issue-303/epa/01_select.jl index 0bfcb7ac..200edd71 100644 --- a/docs/cladistics/issue-303/epa/01_select.jl +++ b/docs/cladistics/issue-303/epa/01_select.jl @@ -34,6 +34,12 @@ names = ["s" * lpad(i, 3, '0') for i in 1:N] L = length(rows[1]); keep = [k for k in 1:L if count(r -> r[k] != 0x00, rows) >= 2] v4cols = [j for (j, k) in enumerate(keep) if V4S <= k <= V4E] A = zeros(UInt8, N, length(keep)); for i in 1:N, (j, k) in enumerate(keep); A[i, j] = rows[i][k]; end +# Fail loudly: eligibility was tested on whole rows, but dropping columns that are filled in fewer +# than two chosen rows can take a V4 base away. 03_prep.jl needs FRAG V4 bases in every row. +for i in 1:N + nv4 = count(j -> A[i, j] != 0x00, v4cols) + nv4 >= FRAG || error("selected row $(names[i]) ($(orig[i])) has only $nv4 V4 bases after column filtering, need $FRAG; no output written") +end ch = ['-', 'A', 'C', 'G', 'T'] mkpath(out) open(joinpath(out, "ref_full.fasta"), "w") do io diff --git a/docs/cladistics/issue-303/epa/02_tree.sh b/docs/cladistics/issue-303/epa/02_tree.sh index b975ba92..92d64a81 100755 --- a/docs/cladistics/issue-303/epa/02_tree.sh +++ b/docs/cladistics/issue-303/epa/02_tree.sh @@ -3,9 +3,12 @@ # SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) # Experiment E8 (issue 303): evidence verdicts on placements made by the application's own # placement step (RAxML EPA). See ../README.md, section 8. -# usage: 02_tree.sh epa_dir +# usage: 02_tree.sh epa_dir [threads] (threads default 4, the published setting) # Application: iqtree -s ALN -m MFP -b 100 -nt 4 -seed S -pre PFX -redo # Used here: iqtree2 -s ALN -m GTR+G4 -nt 4 -seed 1 -pre PFX -redo # (-m MFP aborted after 12 min at 90/286 models; -b 100 omitted: it does not change the ML tree) -E=$1 -cd $E/work/tree && iqtree2 -s $E/work/ref_full.fasta -m GTR+G4 -nt 4 -seed 1 -pre $E/work/tree/reference -redo > $E/raw/02_iqtree.stdout 2>&1 +set -eu +E=$(cd "$1" && pwd) || exit 1 +NT=${2:-4} +cd "$E/work/tree" +iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -nt "$NT" -seed 1 -pre "$E/work/tree/reference" -redo > "$E/raw/02_iqtree.stdout" 2>&1 diff --git a/docs/cladistics/issue-303/epa/03_prep.jl b/docs/cladistics/issue-303/epa/03_prep.jl index ffef9614..e9ceeaca 100644 --- a/docs/cladistics/issue-303/epa/03_prep.jl +++ b/docs/cladistics/issue-303/epa/03_prep.jl @@ -8,6 +8,9 @@ using Random, Printf include(joinpath(@__DIR__, "lib_nwk.jl")) E, Q, seed = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]) W = joinpath(E, "work") +# start from a clean query directory: stale q*/ variants (e.g. pos from an earlier run with more +# controls) must not survive into the scorer +rm(joinpath(W, "q"); recursive = true, force = true) names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) leafidx = Dict(n => i for (i, n) in enumerate(names)) seqs = Dict{String,String}(); cur = "" diff --git a/docs/cladistics/issue-303/epa/05_run_epa.sh b/docs/cladistics/issue-303/epa/05_run_epa.sh index 32bd356a..bb16ebf7 100755 --- a/docs/cladistics/issue-303/epa/05_run_epa.sh +++ b/docs/cladistics/issue-303/epa/05_run_epa.sh @@ -7,5 +7,12 @@ # Application command (src/pipeline/phylogeny.jl, step "place"), threads=4 -> -T max(2,threads) = -T 4 in the app; # here -T 2 with two runs in parallel on 4 cores (thread count does not change the result). # raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s -t -T N -w / -E=$1; K=$2 -ls -d "$E"/work/q/q*/"$K" | xargs -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ +set -eu +E=$(cd "$1" && pwd) || exit 1; K=$2 +# one NUL-terminated path per query directory, so paths with spaces stay whole; xargs exits 123 if any worker failed +n=0 +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && n=$((n + 1)); done +[ "$n" -gt 0 ] || { echo "05_run_epa.sh: no query directories for kind $K under $E/work/q" >&2; exit 1; } +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && printf "%s\0" "$D"; done | + xargs -0 -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ || + { echo "05_run_epa.sh: at least one EPA run failed (kind $K); see raxml.stdout in the query directories" >&2; exit 1; } diff --git a/docs/cladistics/issue-303/epa/08_by_depth.jl b/docs/cladistics/issue-303/epa/08_by_depth.jl index f4c1b6fd..c7c27c3e 100644 --- a/docs/cladistics/issue-303/epa/08_by_depth.jl +++ b/docs/cladistics/issue-303/epa/08_by_depth.jl @@ -6,6 +6,8 @@ # breakdown of wrong entailed verdicts by clade size. Written separately from 06_tables.jl as a # cross-check. usage: julia 08_by_depth.jl epa_dir using Printf, Statistics +# median as text; an empty collection (no clade reaches the threshold) reports n/a instead of throwing +medstr(x) = isempty(x) ? "n/a" : @sprintf("%.1f", median(x)) function load(kind) pc = [split(l, '\t') for l in readlines(joinpath(ARGS[1], "raw", "perclade_$kind.tsv"))[2:end]] C = [(parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1") for r in pc] @@ -17,14 +19,14 @@ for kind in ("amp", "pos", "neg") C, Q = load(kind) println("== $kind: $(length(C)) clade rows, $(length(Q)) queries") for a in (0.01, 0.05) - ent = [c for c in C if c[3] >= 1 - a]; ref = [c for c in C if c[3] <= a] + ent = [c for c in C if c[3] >= 1 - a - 1e-12]; ref = [c for c in C if c[3] <= a + 1e-12] ew = count(c -> !c[4], ent); rw = count(c -> c[4], ref) # smallest entailed clade per query sm = Dict{Int,Tuple{Int,Bool}}() for c in ent; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end wrongq = sort([q for (q, v) in sm if !v[2]]) - @printf(" alpha %.2f: entailed %d wrong %d | refuted %d wrong %d | smallest entailed: %d queries, median size %.1f, wrong %d\n", - a, length(ent), ew, length(ref), rw, length(sm), median([v[1] for v in values(sm)]), length(wrongq)) + @printf(" alpha %.2f: entailed %d wrong %d | refuted %d wrong %d | smallest entailed: %d queries, median size %s, wrong %d\n", + a, length(ent), ew, length(ref), rw, length(sm), medstr([v[1] for v in values(sm)]), length(wrongq)) if kind == "amp" println(" wrong-smallest queries (qi: clade size, edges apart, placements, best LWR): ", join(["$q: $(sm[q][1]), $(Q[q].apart), $(Q[q].n), $(round(Q[q].best, digits=3))" for q in wrongq], " | ")) @@ -37,7 +39,7 @@ for kind in ("amp", "pos", "neg") for k in (2, 4, 8, 16) s2 = Dict{Int,Tuple{Int,Bool}}() for c in ent; c[2] >= k || continue; (!haskey(s2, c[1]) || c[2] < s2[c[1]][1]) && (s2[c[1]] = (c[2], c[4])); end - @printf(" smallest entailed clade of size >= %2d: %3d queries, median size %.1f, wrong %d\n", k, length(s2), median([v[1] for v in values(s2)]), count(v -> !v[2], values(s2))) + @printf(" smallest entailed clade of size >= %2d: %3d queries, median size %s, wrong %d\n", k, length(s2), medstr([v[1] for v in values(s2)]), count(v -> !v[2], values(s2))) end # by number of placements reported for (lab, f) in (("1 placement", n -> n == 1), ("2+ placements", n -> n >= 2)) From 5c1aca0610bd6ef828a2d6b6322bb0931cc297d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 04:21:04 +0000 Subject: [PATCH 4/5] test(cladistics): regression test for the issue 303 selection check 01_select.jl stops when a selected row falls below 240 V4 bases after column filtering (e506a91). test_select.jl builds two synthetic two-row alignments and checks both directions: the script writes its output when both rows keep 240 bases, and exits non-zero with no output when each row loses one. The test fails against the script as it was before e506a91. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01J4QvDiW8qD4f2JDQJ4buGo --- docs/cladistics/issue-303/README.md | 6 +++ docs/cladistics/issue-303/epa/test_select.jl | 54 ++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 docs/cladistics/issue-303/epa/test_select.jl diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index a1e7bf98..6525f95d 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -465,6 +465,12 @@ julia WORK/scripts/06_tables.jl WORK amp pos neg julia WORK/scripts/08_by_depth.jl WORK ``` +The check that selection stops when a row loses a V4 base to column filtering: + +```bash +julia docs/cladistics/issue-303/epa/test_select.jl +``` + The drivers take optional trailing size arguments for a quick trial; without them they run the published sizes. `01_run.sh` takes the number of references (default 200), `02_tree.sh` the thread count (default 4), and `03_prep.jl` the number of queries. For experiment 9, diff --git a/docs/cladistics/issue-303/epa/test_select.jl b/docs/cladistics/issue-303/epa/test_select.jl new file mode 100644 index 00000000..d5b7bf7f --- /dev/null +++ b/docs/cladistics/issue-303/epa/test_select.jl @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): regression test for 01_select.jl. See ../README.md, section 8. +# +# usage: julia test_select.jl +# +# 01_select.jl tests eligibility (240 bases in the V4 window) on whole rows, then drops every +# column filled in fewer than two selected rows. A row can lose a V4 base that way. It must +# then stop with an error and write nothing, because 03_prep.jl needs 240 V4 bases in each row. +# Two synthetic two-row alignments: one where both rows keep 240 V4 bases, one where each row +# loses one. +const V4S = 11894; const LEN = 25400 +const SELECT = joinpath(@__DIR__, "01_select.jl") + +function row(v4_columns) + s = fill('-', LEN) + s[1:960] .= 'A' # 960 bases outside the V4 window + for k in v4_columns; s[k] = 'C'; end # 240 bases inside it: 1,200 in all + String(s) +end + +function select(dir, rows) + fasta = joinpath(dir, "in.fasta"); out = joinpath(dir, "out") + open(fasta, "w") do io + for (i, r) in enumerate(rows); println(io, ">row", i); println(io, r); end + end + log = joinpath(dir, "log.txt") + p = run(pipeline(ignorestatus(`$(Base.julia_cmd()) $SELECT $fasta $out 2 1`); stdout = log, stderr = log)) + p.exitcode, out, read(log, String) +end + +failures = String[] +check(ok, what) = (println(ok ? "ok " : "FAIL ", what); ok || push!(failures, what)) + +mktempdir() do dir + shared = collect(V4S:V4S+239) + code, out, log = select(dir, [row(shared), row(shared)]) + check(code == 0, "both rows keep 240 V4 bases: exit status 0") + check(isfile(joinpath(out, "ref_full.fasta")) && isfile(joinpath(out, "names.tsv")), "both rows keep 240 V4 bases: output written") + v4 = isfile(joinpath(out, "v4cols.txt")) ? length(split(strip(read(joinpath(out, "v4cols.txt"), String)), ",")) : 0 + check(v4 == 240, "both rows keep 240 V4 bases: 240 V4 columns kept (got $v4)") +end + +mktempdir() do dir + a = collect(V4S:V4S+239) # last column is filled in this row only + b = vcat(collect(V4S:V4S+238), V4S + 300) # one column filled in this row only + code, out, log = select(dir, [row(a), row(b)]) + check(code != 0, "each row loses one V4 base: non-zero exit status (got $code)") + check(occursin("has only 239 V4 bases after column filtering", log), "each row loses one V4 base: error names the count") + check(!isdir(out), "each row loses one V4 base: no output written") +end + +isempty(failures) || (println(length(failures), " check(s) failed"); exit(1)) +println("all checks passed") From dfe29751317e54fe7e5016b0db48f39934b02d9e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 04:24:10 +0000 Subject: [PATCH 5/5] docs(cladistics): quote the work directory in the issue 303 run commands The running section says the work directory may contain spaces, but its commands used a bare WORK placeholder that a shell would split. They now set WORK once and use "$WORK" throughout. The scoring, table and cross-check commands were run as written with a space in the path. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_01J4QvDiW8qD4f2JDQJ4buGo --- docs/cladistics/issue-303/README.md | 34 ++++++++++++++++------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 6525f95d..2ab79fca 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -436,33 +436,37 @@ julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP julia -O2 -t 4 docs/cladistics/issue-303/placement_evidence.jl /path/to/silva.bacteria.fasta ``` -Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. `WORK` is a directory +Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. `$WORK` is a directory of your choice, absolute or relative, and may contain spaces; the drivers resolve it to an -absolute path and stop with a non-zero status if a stage fails. Copy `epa/` to `WORK/scripts`, -then run its steps in order: +absolute path and stop with a non-zero status if a stage fails. Set it once in the shell, copy +`epa/` to `$WORK/scripts`, then run the steps in order: ```bash -sh WORK/scripts/01_run.sh WORK /path/to/silva.bacteria.fasta +WORK=/path/to/work ``` ```bash -sh WORK/scripts/02_tree.sh WORK +sh "$WORK/scripts/01_run.sh" "$WORK" /path/to/silva.bacteria.fasta ``` ```bash -julia WORK/scripts/03_prep.jl WORK 200 7 +sh "$WORK/scripts/02_tree.sh" "$WORK" ``` ```bash -(set -e; for k in amp pos neg; do sh WORK/scripts/05_run_epa.sh WORK $k; julia WORK/scripts/04_score.jl WORK $k; done) +julia "$WORK/scripts/03_prep.jl" "$WORK" 200 7 ``` ```bash -julia WORK/scripts/06_tables.jl WORK amp pos neg +(set -e; for k in amp pos neg; do sh "$WORK/scripts/05_run_epa.sh" "$WORK" $k; julia "$WORK/scripts/04_score.jl" "$WORK" $k; done) ``` ```bash -julia WORK/scripts/08_by_depth.jl WORK +julia "$WORK/scripts/06_tables.jl" "$WORK" amp pos neg +``` + +```bash +julia "$WORK/scripts/08_by_depth.jl" "$WORK" ``` The check that selection stops when a row loses a V4 base to column filtering: @@ -478,26 +482,26 @@ thread count (default 4), and `03_prep.jl` the number of queries. For experiment (default 1500) and the threads (default 4), and `run_rest.sh` takes the queries per draw (default 300) and the whole-sequence controls (default 60). -Experiment 9: copy `epa/` and then `epa-depth/` over it to `WORK/scripts`, then: +Experiment 9: copy `epa/` and then `epa-depth/` over it to `$WORK/scripts`, then: ```bash -sh WORK/scripts/run_trees.sh WORK /path/to/silva.bacteria.fasta +sh "$WORK/scripts/run_trees.sh" "$WORK" /path/to/silva.bacteria.fasta ``` ```bash -sh WORK/scripts/run_rest.sh WORK +sh "$WORK/scripts/run_rest.sh" "$WORK" ``` ```bash -mkdir -p WORK/raw && (set -e; for t in "A amp" "A pos" "B amp"; do set -- $t; julia WORK/scripts/04_score.jl WORK/$1 $1 $2; done) +mkdir -p "$WORK/raw" && (set -e; for t in "A amp" "A pos" "B amp"; do set -- $t; julia "$WORK/scripts/04_score.jl" "$WORK/$1" $1 $2; done) ``` ```bash -julia WORK/scripts/06_tables.jl new WORK/raw +julia "$WORK/scripts/06_tables.jl" new "$WORK/raw" ``` ```bash -julia WORK/scripts/08_crosscheck.jl WORK/raw +julia "$WORK/scripts/08_crosscheck.jl" "$WORK/raw" ``` The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its