diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 2722bf63..2ab79fca 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -30,7 +30,12 @@ tested reached 60%. An earlier reading of experiment 4, that zoomed-in views kee of well-supported branches, did not survive at 1,024 sequences (66%). Placing an amplicon-length fragment on a reference tree built from full-length sequences does work: clade verdicts from bootstrapped placements were right for 1,105 of 1,106 entailed and 57,116 of 57,128 refuted -verdicts, and pin a read to a clade of median size 18 among 399 references. +verdicts, and pin a read to a clade of median size 18 among 399 references. With the +application's own placement step (RAxML EPA) the picture is sharper and less safe: 2,375 of +2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more +references and 7 errors among 62 on clades of two or three. On a larger sample the smallest +confident clade was wrong for 32 of 448 queries (7.1%), falling to 4 of 448 (0.9%) when the +clade had to hold at least 64 of 399 references. ## Data @@ -203,15 +208,154 @@ control shows the score can fail. What it does not show is in the limits below. The counting rule that turns per-read verdicts into a per-sample verdict for a clade, with separate bounds for wrongly counted and missed reads, is proved in `metadatastician/MetaManifold-Evidence` (`agda/src/MetaManifold/Evidence/TwoSided.agda`, -not yet merged there). It also proves that an entailed verdict survives zooming out to any -enclosing clade and a refuted verdict survives zooming in. +not yet merged there). It also proves two zoom laws, each under a premise on the bounds. An +entailed verdict survives zooming out to an enclosing clade if the same bound on wrongly +counted reads holds for the larger clade. A refuted verdict survives zooming in to a +contained clade if that clade's bound on missed reads is no larger. An earlier version of +this page stated both without their premises; containment alone is not enough for either. + +### 8. The application's own placement step (`epa/`) + +Experiment 7 placed by least squares. This repeats it with the command the application runs +in its `place` step: `raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2` (RAxML 8.2.12). 200 +full-length sequences from the same SILVA alignment give a maximum-likelihood reference tree +(IQ-TREE 2.0.7, GTR+G4). Each of the 200 in turn is pruned from the tree and its 240-base V4 +fragment is placed back. For every clade of the pruned tree, the mass is the sum of the +likelihood weight ratios of the placements inside it, as RAxML reports them. The verdict is +entailed at mass 0.99 or more, refuted at 0.01 or less, unresolved between. + +| | Amplicon fragment | Whole sequence (positive control) | A different sequence placed (negative control) | +| --- | --- | --- | --- | +| Placed on exactly the true edge | 91 of 200 (45.5%) | 188 of 200 (94.0%) | 0 of 200 | +| Within two edges of it | 76.5% | 99.5% | 1.0% | +| Entailed verdicts that are right | 2,375 of 2,388 (99.46%) | 3,065 of 3,065 | 1,507 of 3,176 (47.45%) | +| Refuted verdicts that are right | 35,988 of 36,016 (99.92%) | 36,208 of 36,211 (99.99%) | 34,392 of 35,952 (95.66%) | +| Clades that truly hold the sequence: entailed / unresolved / wrongly refuted | 75.2% / 23.9% / 0.9% | 97.0% / 2.9% / 0.1% | 47.7% / 2.9% / 49.4% | +| Smallest entailed clade, median size of 199 | 4 | 4 | 3 | +| Smallest entailed clade holds the truth | 139 of 146 queries (95.2%) | 193 of 193 | 9 of 189 (4.8%) | +| Queries with no entailed clade | 54 | 7 | 11 | + +In this negative control the other sequence stays in the tree, so its fragment is placed +beside it; experiment 7 left it out. The two negative columns are not comparable. + +The errors of the amplicon fragment depend on how small the clade is: + +| Clade size (of 199 references) | Entailed verdicts | Wrong | +| --- | --- | --- | +| 2 to 3 | 62 | 7 | +| 4 to 7 | 132 | 1 | +| 8 to 15 | 228 | 1 | +| 16 to 49 | 451 | 4 | +| 50 to 199 | 1,515 | 0 | + +Verdicts on nested clades of one query are not independent, so the per-query count is the +honest one. Reporting the smallest entailed clade, 7 of 146 queries get a wrong clade. +Reporting the smallest entailed clade of at least 4, 8 or 16 references, 4, 3 or 2 of the +146 do, at median sizes 6.5, 13 and 23. + +RAxML reports few placements per query: a median of 3, at most 7, and a single one for 60 of +the 200. Four of the seven wrong queries had a single placement with weight 0.99 or more. The +mass is therefore lumpy. Clades given mass in (0, 0.05] hold the truth for 9 of 52, +(0.05, 0.25] 22 of 96, (0.25, 0.5] 24 of 46, (0.5, 0.75] 53 of 98, (0.75, 0.95] 280 of 324 +and (0.95, 1] 2,745 of 2,773. + +With the looser rule (entailed at 0.95 or more, refuted at 0.05 or less) 2,745 of 2,773 +entailed verdicts are right (98.99%), and the smallest entailed clade is wrong for 17 of 172 +queries. + +Three self-checks passed. An independent calculation of the truth from path distances on the +unpruned tree matched in all 10 tests (9 distinct queries). In all 600 runs the 395 edge +numbers of the placement file map to 395 distinct branches of the pruned tree. Clade mass +never decreased +from a clade to the clade containing it (39,200 pairs per column). `08_by_depth.jl` +recomputes the tables from the per-clade output separately from `06_tables.jl`; the two +agree. + +What this shows: with the application's placement step, "present in this clade" was right +in every case seen for clades of a quarter of the tree or more, and wrong about one time in +nine for clades of two or three references. At that depth the verdict is confidently wrong, +not unresolved. Experiment 9 repeats this on a larger sample and finds errors on large clades +too. What it does not show is in the limits below. + +### 9. Error by clade depth (`epa-depth/`) + +The same test as experiment 8 on a larger sample, to put a number on how the error depends +on clade size. Two independent draws of 400 references (13 sequences are in both), a +maximum-likelihood tree for each, and 300 amplicon queries per draw: 600 queries. As a +control, the whole sequence of 60 queries was placed: 60 of 60 landed on the exact edge and +none of their 1,134 entailed verdicts was wrong. + +For the amplicon fragment, 9,555 of 9,613 entailed verdicts were right. The 58 wrong ones by +clade size: + +| Clade size (of 399 references) | Entailed verdicts | Wrong | Queries with a wrong verdict | +| --- | --- | --- | --- | +| 2 to 3 | 215 | 15 | 13 | +| 4 to 7 | 417 | 12 | 11 | +| 8 to 15 | 507 | 11 | 9 | +| 16 to 31 | 748 | 7 | 6 | +| 32 to 63 | 999 | 6 | 2 | +| 64 to 127 | 431 | 2 | 1 | +| 128 to 399 | 6,296 | 5 | 3 | + +The five errors in the last row are on clades holding 357 to 372 of the 399 references. The +branch there cuts off a small group, and the sequence truly sits in that small group. Binned +by the smaller side of the branch instead, the wrong count falls from 15 of 444 at two or +three references to 2 of 1,271 at 64 to 127, and 0 of 1,119 at 128 or more. + +One result per query is the honest unit. The rule "report the smallest entailed clade with +at least k references" gave: + +| k | Queries given a clade (of 600) | Median clade size | Wrong | Rate (exact 95% interval) | +| --- | --- | --- | --- | --- | +| 2 | 448 | 4 | 32 | 7.1% (4.9 to 9.9) | +| 4 | 448 | 7 | 25 | 5.6% (3.6 to 8.1) | +| 8 | 448 | 14 | 17 | 3.8% (2.2 to 6.0) | +| 16 | 448 | 26 | 11 | 2.5% (1.2 to 4.4) | +| 32 | 448 | 46.5 | 6 | 1.3% (0.5 to 2.9) | +| 64 | 448 | 173 | 4 | 0.9% (0.2 to 2.3) | + +The other 152 queries had no entailed clade. The two draws agree within their intervals: +19 of 230 and 13 of 218 at k = 2. Of the 32 queries wrong at k = 2, 18 had a single reported +placement and 18 had a best weight of 0.99 or more. With the looser rule (0.95) the rates +are higher: 61 of 540 at k = 2 and 10 of 540 at k = 64. + +The intervals are Clopper-Pearson and treat queries as independent. The queries of one draw +share one reference tree, so the intervals cover the sampling of queries on these two trees, +not the variation between trees. + +Self-checks passed: the control above; an independent truth calculation matched for 10 +queries in each draw; all 793 edge numbers of every placement file map to distinct branches +of the pruned tree; clade mass never decreased from a clade to the clade containing it; and +the new table code reproduces experiment 8's counts from experiment 8's output. +`epa-depth/08_crosscheck.jl` recomputes the tables separately from `06_tables.jl`; the two +agree. + +What this shows: the error of a "present in this clade" verdict from an amplicon fragment +falls as the clade grows and does not reach zero. Requiring 64 references in the clade +brought it under 1% of queries, at the price of a median clade of 173 of 399. The whole +sequence, placed the same way, was never wrong in 60 queries. The truth was itself built +from the whole sequence, so that control shows the machinery is sound; it does not show the +reference tree is right at the finest depth. ## Reading of the results -- **Evidence on placements.** Experiment 7 is the one positive real-data result. Verdicts by - clade are sound at the rates above, and they give the "thumbnail then zoom" idea a form - that holds: the verdict is decided at a coarse clade and becomes undecided, not wrong, as - the clade shrinks. +- **Evidence on placements.** Experiments 7 and 8 are the positive real-data results. Verdicts + by clade are sound at the rates above for coarse clades, and they give the "thumbnail then + zoom" idea a form that holds at the coarse end: the verdict is decided at a coarse clade. + How it fails as the clade shrinks depends on where the mass comes from. With bootstrapped + least squares (experiment 7) it became undecided. With the likelihood weights of the + application's placement step (experiment 8) it became wrong for 7 of 62 verdicts on the + smallest clades. Experiment 9 puts numbers on it: a minimum clade size trades error for + resolution (7.1% of queries wrong with no minimum, 0.9% with a minimum of 64 references). + A rule for the application needs that minimum or a better-calibrated mass. Neither has + been chosen, and no rule has been tested inside the application. +- **Counting reads does not average this error away.** Each query in experiments 7 and 8 is + one sequence. Reads from the same organism share that sequence, so if it is misplaced they + are all misplaced together. The bound on wrongly counted reads that the counting rule + takes as input cannot be read off the number of reads; it has to come from the number of + distinct sequences and a per-depth error rate. Experiment 9 gives that rate for error-free + fragments on two trees; it has not been measured on real reads. - **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic number is enough. On this dataset exact arithmetic gives the same decisions as double precision. What it adds is order-independence and a provable comparison, not a different @@ -241,8 +385,22 @@ enclosing clade and a refuted verdict survives zooming in. verdict in 1,106 means "entailed" is a measured rate, not a proof. Verdicts for nested clades of one query are not independent: counted per query, the smallest entailed clade was wrong for 1 of 130 queries, and 20 further queries had no entailed clade at all. -- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with the application's - own placement step or with real reads, and any change to the application. +- Experiment 8: the truth is where the full-length sequence sits in a maximum-likelihood tree + that was not bootstrapped, so some errors on the smallest clades may be the reference tree's + own uncertainty; the two were not separated. The reference tree used a fixed model (GTR+G4) + with IQ-TREE 2.0.7, where the application selects a model, bootstraps and pins IQ-TREE 3.1.4. + The application's alignment and trimming steps (MAFFT, trimAl) were skipped: queries sit in + the columns of the SILVA alignment. Queries are error-free fragments of reference sequences. + 200 references and 200 queries, one draw. Experiment 9 shares every limit of experiment 8 + except size: 400 references, two draws, 600 queries. The second draw's tree was built + without bootstrap after the bootstrapped run exceeded 25 minutes; the bootstrap supports of + the first tree were saved and not analysed. Experiments 7 and 8 differ in the number of + references, the tree method, the source of the mass and the thresholds, so the difference + between them is not the effect of the placement method alone. A cheaper agent ran the + experiment; the tables were recomputed separately from its per-clade output. +- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with real reads, the + application's placement step with its speed heuristic turned off, and any change to the + application. ## Running @@ -278,6 +436,74 @@ julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP julia -O2 -t 4 docs/cladistics/issue-303/placement_evidence.jl /path/to/silva.bacteria.fasta ``` +Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. `$WORK` is a directory +of your choice, absolute or relative, and may contain spaces; the drivers resolve it to an +absolute path and stop with a non-zero status if a stage fails. Set it once in the shell, copy +`epa/` to `$WORK/scripts`, then run the steps in order: + +```bash +WORK=/path/to/work +``` + +```bash +sh "$WORK/scripts/01_run.sh" "$WORK" /path/to/silva.bacteria.fasta +``` + +```bash +sh "$WORK/scripts/02_tree.sh" "$WORK" +``` + +```bash +julia "$WORK/scripts/03_prep.jl" "$WORK" 200 7 +``` + +```bash +(set -e; for k in amp pos neg; do sh "$WORK/scripts/05_run_epa.sh" "$WORK" $k; julia "$WORK/scripts/04_score.jl" "$WORK" $k; done) +``` + +```bash +julia "$WORK/scripts/06_tables.jl" "$WORK" amp pos neg +``` + +```bash +julia "$WORK/scripts/08_by_depth.jl" "$WORK" +``` + +The check that selection stops when a row loses a V4 base to column filtering: + +```bash +julia docs/cladistics/issue-303/epa/test_select.jl +``` + +The drivers take optional trailing size arguments for a quick trial; without them they run the +published sizes. `01_run.sh` takes the number of references (default 200), `02_tree.sh` the +thread count (default 4), and `03_prep.jl` the number of queries. For experiment 9, +`run_trees.sh` takes the references per draw (default 400), the bootstrap timeout in seconds +(default 1500) and the threads (default 4), and `run_rest.sh` takes the queries per draw +(default 300) and the whole-sequence controls (default 60). + +Experiment 9: copy `epa/` and then `epa-depth/` over it to `$WORK/scripts`, then: + +```bash +sh "$WORK/scripts/run_trees.sh" "$WORK" /path/to/silva.bacteria.fasta +``` + +```bash +sh "$WORK/scripts/run_rest.sh" "$WORK" +``` + +```bash +mkdir -p "$WORK/raw" && (set -e; for t in "A amp" "A pos" "B amp"; do set -- $t; julia "$WORK/scripts/04_score.jl" "$WORK/$1" $1 $2; done) +``` + +```bash +julia "$WORK/scripts/06_tables.jl" new "$WORK/raw" +``` + +```bash +julia "$WORK/scripts/08_crosscheck.jl" "$WORK/raw" +``` + The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its own copy of the shared code because it is the script as it was run; the later scripts include `lib_tree.jl`. diff --git a/docs/cladistics/issue-303/epa-depth/02_tree.sh b/docs/cladistics/issue-303/epa-depth/02_tree.sh new file mode 100755 index 00000000..2274d30c --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/02_tree.sh @@ -0,0 +1,28 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: 02_tree.sh epa_dir draw [timeout_s] [threads] +# IQ-TREE GTR+G4 with UFBoot 1000 (default 4 threads); fallback without -B after timeout_s +# (default 1500 s = 25 min wall clock). Logs go to ../raw relative to epa_dir. +set -eu +mkdir -p "$1/work" +E=$(cd "$1" && pwd) || exit 1; DR=$2; TMO=${3:-1500}; NT=${4:-4} +T=$E/work/tree; mkdir -p "$T"; cd "$T" +RAW=$(cd "$E/.." && pwd)/raw +LOG=$RAW/tree_$DR.log +s=$(date +%s) +rc=0 +timeout "$TMO" iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -B 1000 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.stdout" 2>&1 || rc=$? +if [ "$rc" -ne 0 ] || [ ! -s "$T/reference.treefile" ]; then + echo "draw $DR: -B 1000 run rc=$rc after $(( $(date +%s) - s )) s; RERUN WITHOUT -B" > "$LOG" + s=$(date +%s) + iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.nob.stdout" 2>&1 + echo "draw $DR: rerun without -B took $(( $(date +%s) - s )) s" >> "$LOG" +else + echo "draw $DR: -B 1000 completed, $(( $(date +%s) - s )) s wall clock, rc=0" > "$LOG" +fi +[ -s "$T/reference.treefile" ] || { echo "02_tree.sh: no reference.treefile for draw $DR" >&2; exit 1; } +cp "$T/reference.treefile" "$T/reference_with_support.treefile" diff --git a/docs/cladistics/issue-303/epa-depth/03_prep.jl b/docs/cladistics/issue-303/epa-depth/03_prep.jl new file mode 100644 index 00000000..75afd6a7 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/03_prep.jl @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 03_prep.jl epa_dir Q seed npos (amp for all Q queries; pos for the first npos queries; no neg) +# Leave-one-out preparation: for each query q, prune, write pruned tree, truth, and three alignments (amp/pos/neg). +using Random, Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, Q, seed, NPOS = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]), parse(Int, ARGS[4]) +W = joinpath(E, "work") +# start from a clean query directory: stale q*/ variants (e.g. pos from an earlier run with more +# controls) must not survive into the scorer +rm(joinpath(W, "q"); recursive = true, force = true) +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +seqs = Dict{String,String}(); cur = "" +for l in eachline(joinpath(W, "ref_full.fasta")); startswith(l, ">") ? (global cur = l[2:end]) : (seqs[cur] = l); end +v4cols = parse.(Int, split(strip(read(joinpath(W, "v4cols.txt"), String)), ",")) +g0 = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g0.names[u] => u for u in eachindex(g0.names) if g0.names[u] != "") +@assert length(nid) == N +rng = MersenneTwister(seed) +queries = sort(shuffle(rng, 1:N)[1:Q]) +function fragment(s::String, cols) + v = collect(s); out = fill('-', length(v)); k = 0 + for j in cols; if v[j] != '-'; out[j] = v[j]; k += 1; k == 240 && break; end; end + @assert k == 240; String(out) +end +function newick(g::NG, root) + io = IOBuffer() + function w(u, p) + if isleaf(g, u); print(io, g.names[u]); return; end + print(io, "("); first = true + for (v, l) in g.adj[u]; v == p && continue; first || print(io, ","); first = false; w(v, u); print(io, ":", l); end + print(io, ")") + end + # root must be an internal node + print(io, "("); first = true + for (v, l) in g.adj[root]; first || print(io, ","); first = false; w(v, root); print(io, ":", l); end + print(io, ");"); String(take!(io)) +end +open(joinpath(W, "queries.tsv"), "w") do qio + println(qio, "qi\tquery") + for (qi, q) in enumerate(queries) + qn = names[q]; u = nid[qn] + adj = [copy(d) for d in g0.adj]; g = NG(copy(g0.names), adj, [Dict{Int,Int}() for _ in adj]) + @assert length(adj[u]) == 1 + p = first(keys(adj[u])); @assert length(adj[p]) == 3 "parent of query has degree $(length(adj[p]))" + delete!(adj[p], u); empty!(adj[u]) + (a, la), (b, lb) = collect(adj[p]) + delete!(adj[a], p); delete!(adj[b], p); empty!(adj[p]); adj[a][b] = la + lb; adj[b][a] = la + lb + refs = [n for n in names if n != qn]; rl = leafidx[minimum(refs)] + # truth: root at the alphabetically first leaf; the merged edge a-b, child side + par, kids, below, order = root_at(g, nid[names[rl]], leafidx) + child = par[a] == b ? a : (par[b] == a ? b : error("a,b not adjacent after merge")) + truth = sort([names[i] for i in below[child]]) + dir = joinpath(W, "q", @sprintf("q%03d", qi)); mkpath(dir) + root = first(u2 for u2 in eachindex(g.names) if !isleaf(g, u2) && length(g.adj[u2]) == 3) + write(joinpath(dir, "pruned.nwk"), newick(g, root) * "\n") + write(joinpath(dir, "truth.txt"), join(truth, "\n") * "\n") + write(joinpath(dir, "query.txt"), qn * "\n") + for (kind, row) in (qi <= NPOS ? [("amp", fragment(seqs[qn], v4cols)), ("pos", seqs[qn])] : [("amp", fragment(seqs[qn], v4cols))]) + kd = joinpath(dir, kind); mkpath(kd) + open(joinpath(kd, "aln.fasta"), "w") do io + for n in refs; println(io, ">", n); println(io, seqs[n]); end + println(io, ">qry"); println(io, row) + end + cp(joinpath(dir, "pruned.nwk"), joinpath(kd, "tree.nwk"); force=true) + end + println(qio, qi, "\t", qn) + end +end +println("prepared $Q queries") diff --git a/docs/cladistics/issue-303/epa-depth/04_score.jl b/docs/cladistics/issue-303/epa-depth/04_score.jl new file mode 100644 index 00000000..19c34a66 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/04_score.jl @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 04_score.jl epa_dir draw kind -> ../raw/perquery__.tsv, perclade__.tsv, checks__.txt +using Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, draw, kind = ARGS[1], ARGS[2], ARGS[3]; W = joinpath(E, "work"); R = joinpath(E, "..", "raw") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +pq = open(joinpath(R, "perquery_$(draw)_$kind.tsv"), "w"); pc = open(joinpath(R, "perclade_$(draw)_$kind.tsv"), "w") +ck = open(joinpath(R, "checks_$(draw)_$kind.txt"), "w") +println(pq, "qi\tquery\tstatus\tnplacements\ttotal_lwr\tbest_lwr\texact\tedges_apart") +println(pc, "qi\tclade_size\tmass\ttruth_inside\tn_refs") +nedges_ok = true; distinct_ok = true; mono_viol = 0; mono_checked = 0; nscored = 0 +for q in qs + qi = parse(Int, q[1]); dir = joinpath(W, "q", @sprintf("q%03d", qi)); kd = joinpath(dir, kind) + isdir(kd) || continue + jp = joinpath(kd, "RAxML_portableTree.epa.jplace") + if !isfile(jp); println(pq, qi, "\t", q[2], "\tMISSING\t0\t0\t0\t0\t0"); continue; end + g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String))) + truthleaves = split(strip(read(joinpath(dir, "truth.txt"), String)), "\n") + refs = sort([n for n in names if n != q[2]]); rl = leafidx[refs[1]] + nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") + par, kids, below, order = root_at(g, nid[refs[1]], leafidx) + # edge above node u (u != root leaf) <-> below[u]; root leaf's pendant edge == edge above its only child + root_child = first(kids[nid[refs[1]]]) + edgekey = Dict{BitSet,Int}() # bipartition (side without refs[1]) -> node + for u in order; u == nid[refs[1]] && continue; edgekey[below[u]] = u; end + ne = length(edgekey) + 0 # includes root_child's edge == pendant of root leaf + expected = 2 * (N - 1) - 3 + # jplace tree + tree, fields, rows = parse_jplace(jp) + jg = parse_nwk(tree) + jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "") + @assert !haskey(jn, "qry") "query leaf present in jplace tree" + jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx) + enum2node = Dict{Int,Int}(); jkey = Dict{Int,BitSet}() + for u in jorder; u == jn[refs[1]] && continue; e = jg.lab[u][jpar[u]]; enum2node[e] = u; jkey[e] = jbelow[u]; end + # the edge above the root leaf's only child and the pendant edge of root leaf are the same unrooted edge: jplace labels it once + nj = length(jkey) + if nj != expected; global nedges_ok = false; end + # edge-number -> bipartition uniqueness + if length(Set(values(jkey))) != length(jkey); global distinct_ok = false; end + println(ck, "q$qi: jplace edges=$(length(jkey)) distinct_bipartitions=$(length(Set(values(jkey)))) expected=$expected pruned_tree_edges=$(length(edgekey)) all_in_pruned=$(all(k -> haskey(edgekey, k), values(jkey)))") + ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields) + pls = [(Int(r[ei]), r[li]) for r in rows] + total = sum(p[2] for p in pls) + # best placement + bi = argmax([p[2] for p in pls]); bkey = jkey[pls[bi][1]] + tkey = BitSet(leafidx[n] for n in truthleaves) + # truth is the below-set of its child in the pruned tree rooted at refs[1] + @assert haskey(edgekey, tkey) "truth bipartition not an edge of pruned tree" + function apart(a, b) + a == b && return 0 + da = Dict{Int,Int}(); u = a; k = 0; while u != 0 && u != nid[refs[1]]; da[u] = k; u = par[u]; k += 1; end + u = b; k = 0; while u != 0 && u != nid[refs[1]]; haskey(da, u) && return da[u] + k; u = par[u]; k += 1; end + typemax(Int) + end + tnode = edgekey[tkey]; bnode = edgekey[bkey] + # root-leaf pendant edge: represent by root_child's node (same edge) + println(pq, qi, "\t", q[2], "\tOK\t", length(pls), "\t", total, "\t", pls[bi][2], "\t", Int(bkey == tkey), "\t", apart(bnode, tnode)) + # clades: internal nodes other than the root leaf, whose parent... internal non-leaf nodes of the rooted tree (root leaf excluded) + plnode = [(edgekey[jkey[e]], w) for (e, w) in pls] + clades = [u for u in order if u != nid[refs[1]] && !isleaf(g, u)] + masses = Dict{Int,Float64}() + for c in clades + S = below[c]; m = 0.0 + for (nd, w) in plnode; below[nd] != S && issubset(below[nd], S) && (m += w); end + masses[c] = m + println(pc, qi, "\t", length(S), "\t", m, "\t", Int(tkey != S && issubset(tkey, S)), "\t", N - 1) + end + # monotonicity under inclusion: mass(child) <= mass(parent) for parent clade strictly containing + for c in clades; p = par[c]; (p == nid[refs[1]] || isleaf(g, p)) && continue; global mono_checked += 1; masses[c] <= masses[p] + 1e-12 || (global mono_viol += 1); end + global nscored += 1 +end +println(ck, "SUMMARY scored=$nscored edge_count_ok=$nedges_ok distinct_bipartitions_ok=$distinct_ok monotone_pairs_checked=$mono_checked monotone_violations=$mono_viol") +close(pq); close(pc); close(ck) +println("scored $nscored; ", read(joinpath(R, "checks_$(draw)_$kind.txt"), String)[end-200:end]) diff --git a/docs/cladistics/issue-303/epa-depth/05_run_epa.sh b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh new file mode 100755 index 00000000..8aabec67 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/05_run_epa.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: 05_run_epa.sh epa_dir kind (kind = amp | pos); same command as the application's place step, -T 2, two jobs in parallel +set -eu +E=$(cd "$1" && pwd) || exit 1; K=$2 +RAW=$(cd "$E/.." && pwd)/raw +s=$(date +%s) +# one NUL-terminated path per query directory, so paths with spaces stay whole; xargs exits 123 if any worker failed +n=0 +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && n=$((n + 1)); done +[ "$n" -gt 0 ] || { echo "05_run_epa.sh: no query directories for kind $K under $E/work/q" >&2; exit 1; } +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && printf "%s\0" "$D"; done | + xargs -0 -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ || + { echo "05_run_epa.sh: at least one EPA run failed (kind $K); see raxml.stdout in the query directories" >&2; exit 1; } +echo "EPA $E $K: $n runs, wall clock $(( $(date +%s) - s )) s" >> "$RAW/epa_timing.txt" diff --git a/docs/cladistics/issue-303/epa-depth/06_tables.jl b/docs/cladistics/issue-303/epa-depth/06_tables.jl new file mode 100644 index 00000000..88918fd6 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/06_tables.jl @@ -0,0 +1,168 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: julia 06_tables.jl new RAW_DIR -> T1-T6 for draws A, B and pooled (amp), plus pos for A +# julia 06_tables.jl old OLD_RAW_DIR -> reproduces the N=200 headline numbers from ../epa/raw/perclade_amp.tsv (no n_refs column there) +using Printf, Statistics +# ---------- Clopper-Pearson (exact, two-sided) by bisection on the BigFloat binomial CDF ---------- +setprecision(BigFloat, 256) +function bcdf(x::Int, n::Int, p::BigFloat) # P(X <= x) + x < 0 && return big"0.0"; x >= n && return big"1.0" + s = big"0.0"; q = 1 - p + for i in 0:x; s += binomial(big(n), big(i)) * p^i * q^(n - i); end + s +end +function cp_interval(x::Int, n::Int; level = 0.05) + a = BigFloat(level) / 2 + lo = big"0.0"; hi = big"1.0" + if x > 0 # lower: P(X >= x | p) = a (increasing in p) + l, h = big"0.0", big"1.0" + for _ in 1:200; m = (l + h) / 2; (1 - bcdf(x - 1, n, m) < a) ? (l = m) : (h = m); end + lo = (l + h) / 2 + end + if x < n # upper: P(X <= x | p) = a (decreasing in p) + l, h = big"0.0", big"1.0" + for _ in 1:200; m = (l + h) / 2; (bcdf(x, n, m) > a) ? (l = m) : (h = m); end + hi = (l + h) / 2 + else hi = big"1.0" end + Float64(lo), Float64(hi) +end +# ---------- loading ---------- +struct DS; label::String; pq::Dict{Int,Vector{String}}; cl::Dict{Int,Vector{Tuple{Int,Float64,Bool}}}; nref::Int; end +function load(label, pqpath, pcpath; keyoff = 0, nref_default = 199) + pq = Dict{Int,Vector{String}}(); cl = Dict{Int,Vector{Tuple{Int,Float64,Bool}}}() + rows = [split(l, "\t") for l in eachline(pqpath)][2:end] + for r in rows; r[3] == "OK" && (pq[parse(Int, r[1]) + keyoff] = String.(r)); end + hdr = split(readline(pcpath), "\t"); nref = nref_default + for l in Iterators.drop(eachline(pcpath), 1) + r = split(l, "\t"); qi = parse(Int, r[1]) + keyoff + push!(get!(cl, qi, Tuple{Int,Float64,Bool}[]), (parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")) + length(r) >= 5 && (nref = parse(Int, r[5])) + end + DS(label, pq, cl, nref) +end +merge_ds(label, a::DS, b::DS) = DS(label, merge(a.pq, b.pq), merge(a.cl, b.cl), a.nref) +isent(m, alpha) = m >= 1 - alpha - 1e-12 +isref(m, alpha) = m <= alpha + 1e-12 +# ---------- T1 ---------- +function t1(d::DS) + ok = [d.pq[q] for q in sort(collect(keys(d.pq)))] + ex = [r[7] == "1" for r in ok]; ap = [parse(Int, r[8]) for r in ok]; tot = [parse(Float64, r[5]) for r in ok]; npl = [parse(Int, r[4]) for r in ok] + @printf(" queries scored %d; placed on exactly the true edge: %d of %d (%.1f%%); within 2 edges: %d of %d; median edges apart %g\n", length(ok), sum(ex), length(ok), 100mean(ex), count(<=(2), ap), length(ok), median(ap)) + @printf(" placements per query: median %g, max %d, queries with a single placement: %d of %d; total LWR per query: min %.4f, median %.4f\n", median(npl), maximum(npl), count(==(1), npl), length(ok), minimum(tot), median(tot)) + for alpha in (0.01, 0.05) + eo = eb = ro = rb = ui = uo = 0 + for cl in values(d.cl), (sz, m, t) in cl + if isent(m, alpha); t ? (eo += 1) : (eb += 1) + elseif isref(m, alpha); t ? (rb += 1) : (ro += 1) + else t ? (ui += 1) : (uo += 1) end + end + @printf(" alpha %.2f: entailed %d, wrong %d | refuted %d, wrong %d | unresolved %d (truth inside %d, truth not inside %d)\n", alpha, eo + eb, eb, ro + rb, rb, ui + uo, ui, uo) + end +end +# ---------- T2 ---------- +const BINS = [(2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 10^6)] +binlabel(b) = b[2] >= 10^6 ? @sprintf("%d-%d", b[1], 399) : @sprintf("%d-%d", b[1], b[2]) +function t2(d::DS, alpha) + println(" bin entailed verdicts wrong distinct queries with a wrong verdict") + for b in BINS + ne = nw = 0; wq = Set{Int}() + for (q, cl) in d.cl, (sz, m, t) in cl + (b[1] <= sz <= b[2] && isent(m, alpha)) || continue + ne += 1; if !t; nw += 1; push!(wq, q); end + end + @printf(" %-9s %9d %12d %12d\n", binlabel(b), ne, nw, length(wq)) + end +end +# ---------- T3 ---------- +function smallest_entailed(d::DS, alpha, k) # -> Dict qi => (size, truth_inside) + out = Dict{Int,Tuple{Int,Bool}}() + for (q, cl) in d.cl + best = nothing + for (sz, m, t) in cl + (sz >= k && isent(m, alpha)) || continue + (best === nothing || sz < best[1]) && (best = (sz, t)) + end + best !== nothing && (out[q] = best) + end + out +end +function t3(d::DS, alpha; ks = (2, 4, 8, 16, 32, 64)) + nq = length(d.pq) + println(" k queries with a clade (of $nq) median size wrong wrong rate exact 95% CI (Clopper-Pearson)") + for k in ks + s = smallest_entailed(d, alpha, k); n = length(s) + if n == 0; @printf(" %-4d %6d\n", k, 0); continue; end + w = count(v -> !v[2], values(s)); lo, hi = cp_interval(w, n) + @printf(" %-4d %6d %18g %14d %9.4f (%.4f, %.4f) [%d of %d]\n", k, n, median([v[1] for v in values(s)]), w, w / n, lo, hi, w, n) + end +end +# ---------- T4 ---------- +function t4(ds::Vector{DS}, pooled::DS, alpha) + for d in vcat(ds, [pooled]) + s = smallest_entailed(d, alpha, 2); wr = sort([q for (q, v) in s if !v[2]]) + npl1 = count(q -> d.pq[q][4] == "1", wr); b99 = count(q -> parse(Float64, d.pq[q][6]) >= 0.99, wr) + @printf(" %s: %d queries with an entailed clade, %d wrong; of the wrong: single reported placement %d of %d; best LWR >= 0.99: %d of %d\n", d.label, length(s), length(wr), npl1, length(wr), b99, length(wr)) + end + println(" list (key = draw+query index; pooled keys: A = qi, B = 1000+qi): key clade_size edges_best_to_truth placements best_lwr") + for q in sort([q for (q, v) in smallest_entailed(pooled, alpha, 2) if !v[2]]) + r = pooled.pq[q]; sz = smallest_entailed(pooled, alpha, 2)[q][1] + @printf(" %5d %6d %6s %6s %10.4f\n", q, sz, r[8], r[4], parse(Float64, r[6])) + end +end +# ---------- T5 ---------- +function t5(d::DS) + edges = [0.05, 0.25, 0.5, 0.75, 0.95]; hit = zeros(Int, 7); cnt = zeros(Int, 7) + for cl in values(d.cl), (sz, m, t) in cl + b = m <= 0 ? 1 : m > 0.95 + 1e-12 ? 7 : 1 + findfirst(x -> m <= x + 1e-12, edges) + cnt[b] += 1; hit[b] += t + end + labels = ["0", "(0, .05]", "(.05, .25]", "(.25, .5]", "(.5, .75]", "(.75, .95]", "(.95, 1]"] + for b in 1:7; cnt[b] > 0 && @printf(" mass %-11s clades %8d truth inside %8d (%.2f%%)\n", labels[b], cnt[b], hit[b], 100hit[b] / cnt[b]); end +end +# ---------- T6 ---------- +function t6(A::DS, B::DS) + for alpha in (0.01, 0.05) + @printf(" alpha %.2f k draw A: wrong/n rate CI | draw B: wrong/n rate CI\n", alpha) + for k in (2, 8, 32) + f(d) = begin s = smallest_entailed(d, alpha, k); n = length(s); w = count(v -> !v[2], values(s)); lo, hi = cp_interval(w, n); @sprintf("%d/%d %.4f (%.4f, %.4f)", w, n, w / n, lo, hi) end + @printf(" %-3d %s | %s\n", k, f(A), f(B)) + end + end +end +# ---------- main ---------- +mode, R = ARGS[1], ARGS[2] +if mode == "cptest" + for (x, n) in ((0, 100), (7, 146), (13, 2388), (5, 5), (0, 1)) + lo, hi = cp_interval(x, n); @printf("CP %d of %d: (%.4f, %.4f)\n", x, n, lo, hi) + end +elseif mode == "old" + d = load("old", joinpath(R, "perquery_amp.tsv"), joinpath(R, "perclade_amp.tsv")) + for alpha in (0.01, 0.05) + eo = eb = 0 + for cl in values(d.cl), (sz, m, t) in cl; isent(m, alpha) && (t ? (eo += 1) : (eb += 1)); end + @printf("OLD N=200 reproduction, alpha %.2f: entailed %d wrong %d\n", alpha, eo + eb, eb) + end + s = smallest_entailed(d, 0.01, 2); @printf("OLD smallest entailed clade alpha 0.01: queries with clade %d, wrong %d\n", length(s), count(v -> !v[2], values(s))) +else + A = load("A", joinpath(R, "perquery_A_amp.tsv"), joinpath(R, "perclade_A_amp.tsv")) + B = load("B", joinpath(R, "perquery_B_amp.tsv"), joinpath(R, "perclade_B_amp.tsv"), keyoff = 1000) + P = merge_ds("pooled", A, B) + Apos = load("A-pos", joinpath(R, "perquery_A_pos.tsv"), joinpath(R, "perclade_A_pos.tsv")) + println("Per-query unit note: verdicts on nested clades of one query are NOT independent; T3 (one result per query) is the statistically honest unit. T2 counts verdicts and is descriptive only.") + println("Clade size = number of references below the node; n_refs = $(A.nref) (pruned tree). Pooled = draw A + draw B queries (a reference present in both draws may be a query in both).\n") + for d in (A, B, P) + println("\n################ ", d.label, " (amp) ################"); println("T1"); t1(d) + for alpha in (0.01, 0.05) + @printf("\nT2 alpha %.2f (%s)\n", alpha, d.label); t2(d, alpha) + @printf("\nT3 alpha %.2f (%s)\n", alpha, d.label); t3(d, alpha) + end + println("\nT5 calibration (", d.label, ")"); t5(d) + end + println("\n################ A-pos (positive control: full-length sequence, first 60 queries of draw A) ################"); t1(Apos) + for alpha in (0.01, 0.05); @printf("\nT3 alpha %.2f (A-pos)\n", alpha); t3(Apos, alpha); end + for alpha in (0.01, 0.05); @printf("\nT4 alpha %.2f (smallest entailed clade, k=2, wrong)\n", alpha); t4([A, B], P, alpha); end + println("\nT6 agreement between draws"); t6(A, B) +end diff --git a/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl new file mode 100644 index 00000000..3af94564 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/08_crosscheck.jl @@ -0,0 +1,67 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# Cross-check of 06_tables.jl, written separately: the size-bin table, the per-query rule, the +# errors on the largest clades, and the same errors binned by the smaller side of the branch. +# usage: julia 08_crosscheck.jl RAW_DIR +using Printf, Statistics +# Same boundary tolerance as 06_tables.jl (isent), so a mass of 0.9899999999999999 counts at alpha 0.01. +const TOL = 1e-12 +medstr(x) = isempty(x) ? "n/a" : @sprintf("%.1f", median(x)) +function load(tag) + C = [(r = split(l, '\t'); (parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")) for l in readlines(joinpath(ARGS[1], "perclade_$tag.tsv"))[2:end]] + Q = Dict((r = split(l, '\t'); parse(Int, r[1]) => (n = parse(Int, r[4]), best = parse(Float64, r[6]), exact = r[7] == "1", apart = parse(Int, r[8]))) for l in readlines(joinpath(ARGS[1], "perquery_$tag.tsv"))[2:end]) + C, Q +end +A, QA = load("A_amp"); B, QB = load("B_amp"); P, QP = load("A_pos") +println("rows A $(length(A)) B $(length(B)); queries A $(length(QA)) B $(length(QB)); pos exact $(count(q -> q.exact, values(QP))) of $(length(QP))") +bins = ((2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 399)) +for a in (0.01, 0.05) + tot = 0; wr = 0 + for (lo, hi) in bins + s = ""; n = 0; w = 0; qs = Set{Tuple{Char,Int}}() + for (tag, C) in (('A', A), ('B', B)), c in C + (c[3] >= 1 - a - TOL && lo <= c[2] <= hi) || continue + n += 1; c[4] || (w += 1; push!(qs, (tag, c[1]))) + end + tot += n; wr += w + @printf(" alpha %.2f size %3d..%3d: entailed %5d wrong %2d (queries %d)\n", a, lo, hi, n, w, length(qs)) + end + println(" alpha $a total entailed $tot wrong $wr") + for k in (2, 4, 8, 16, 32, 64) + n = 0; w = 0; sizes = Int[] + for C in (A, B) + sm = Dict{Int,Tuple{Int,Bool}}() + for c in C; (c[3] >= 1 - a - TOL && c[2] >= k) || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + n += length(sm); w += count(v -> !v[2], values(sm)); append!(sizes, [v[1] for v in values(sm)]) + end + @printf(" alpha %.2f k>=%2d: queries %d median size %s wrong %d\n", a, k, n, medstr(sizes), w) + end +end +println("wrong entailed verdicts on clades of 128+ at alpha 0.01 (draw, query, clade size, edges apart, placements, best LWR):") +for (tag, C, Q) in (('A', A, QA), ('B', B, QB)), c in C + (c[3] >= 0.99 - TOL && c[2] >= 128 && !c[4]) && println(" $tag q$(c[1]) size $(c[2]) apart $(Q[c[1]].apart) n $(Q[c[1]].n) best $(round(Q[c[1]].best, digits=3)) mass $(round(c[3], digits=4))") +end +# How far is the best placement from the truth for wrong vs right smallest-entailed queries (k=2) +for (tag, C, Q) in (('A', A, QA), ('B', B, QB)) + sm = Dict{Int,Tuple{Int,Bool}}() + for c in C; c[3] >= 0.99 - TOL || continue; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + wq = [q for (q, v) in sm if !v[2]] + println(" draw $tag: wrong smallest-entailed queries $(length(wq)); edges apart: ", sort([Q[q].apart for q in wq])) +end + +# ---- by the smaller side of the branch: min(clade size, references outside it) ---- +rows2(tag) = [(r = split(l, '\t'); (tag, parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1", parse(Int, r[5]))) for l in readlines(joinpath(ARGS[1], "perclade_$tag.tsv"))[2:end]] +C2 = vcat(rows2("A_amp"), rows2("B_amp")) +for a in (0.01, 0.05) + for (lo, hi) in ((1, 1), (2, 3), (4, 7), (8, 15), (16, 31), (32, 63), (64, 127), (128, 200)) + n = 0; w = 0; qs = Set{Tuple{String,Int}}() + for c in C2 + m = min(c[3], c[6] - c[3]); (c[4] >= 1 - a - TOL && lo <= m <= hi) || continue + n += 1; c[5] || (w += 1; push!(qs, (c[1], c[2]))) + end + @printf(" alpha %.2f smaller side %3d..%3d: entailed %5d wrong %2d (queries %d)\n", a, lo, hi, n, w, length(qs)) + end +end diff --git a/docs/cladistics/issue-303/epa-depth/run_rest.sh b/docs/cladistics/issue-303/epa-depth/run_rest.sh new file mode 100755 index 00000000..e2b29dfc --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/run_rest.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: run_rest.sh WORK [Q [NPOS]] (after run_trees.sh has finished) +# Q amplicon queries per draw (selection seeds 7 and 8), default 300; whole-sequence control for +# the first NPOS queries of draw A, default 60. WORK may be relative. Any failed stage stops +# the script with a non-zero status. +set -eu +D=$(cd "$1" && pwd) || exit 1 +Q=${2:-300}; NPOS=${3:-60} +cd "$D" +julia -O2 scripts/03_prep.jl "$D/A" "$Q" 7 "$NPOS" > raw/03_prep_A.log 2>&1 +julia -O2 scripts/03_prep.jl "$D/B" "$Q" 8 0 > raw/03_prep_B.log 2>&1 +sh scripts/05_run_epa.sh "$D/A" amp +if [ "$NPOS" -gt 0 ]; then sh scripts/05_run_epa.sh "$D/A" pos; fi +sh scripts/05_run_epa.sh "$D/B" amp diff --git a/docs/cladistics/issue-303/epa-depth/run_trees.sh b/docs/cladistics/issue-303/epa-depth/run_trees.sh new file mode 100755 index 00000000..c286aab9 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/run_trees.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the +# application's placement step. See ../README.md, section 9. These files replace their +# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged. +# usage: run_trees.sh WORK /path/to/silva.bacteria.fasta [N [timeout_s [threads]]] +# WORK (absolute or relative) holds scripts/ (../epa/ with this directory copied over it) and +# receives A/, B/ and raw/. Defaults are the published sizes: N = 400 references per draw, +# UFBoot timeout 1500 s, 4 threads. Any failed stage stops the script with a non-zero status. +set -eu +mkdir -p "$1/raw" +D=$(cd "$1" && pwd) || exit 1 +SILVA=$2; N=${3:-400}; TMO=${4:-1500}; NT=${5:-4} +for X in A B; do + seed=20261117; [ "$X" = B ] && seed=20261118 + mkdir -p "$D/$X/work" + julia -O2 "$D/scripts/01_select.jl" "$SILVA" "$D/$X/work" "$N" "$seed" > "$D/raw/01_select_$X.log" 2>&1 +done +sh "$D/scripts/02_tree.sh" "$D/A" A "$TMO" "$NT" +sh "$D/scripts/02_tree.sh" "$D/B" B "$TMO" "$NT" diff --git a/docs/cladistics/issue-303/epa/01_run.sh b/docs/cladistics/issue-303/epa/01_run.sh new file mode 100755 index 00000000..340563e6 --- /dev/null +++ b/docs/cladistics/issue-303/epa/01_run.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 01_run.sh epa_dir silva.bacteria.fasta [N] +# epa_dir (absolute or relative) holds scripts/ (a copy of this directory), and receives work/ and raw/. +# N = number of references, default 200 (the published size). +set -eu +mkdir -p "$1/work/tree" "$1/raw" +E=$(cd "$1" && pwd) || exit 1 +SILVA=$2; N=${3:-200} +julia -O2 "$E/scripts/01_select.jl" "$SILVA" "$E/work" "$N" 20261117 diff --git a/docs/cladistics/issue-303/epa/01_select.jl b/docs/cladistics/issue-303/epa/01_select.jl new file mode 100644 index 00000000..200edd71 --- /dev/null +++ b/docs/cladistics/issue-303/epa/01_select.jl @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 01_select.jl silva.fasta outdir N seed +# Same eligibility / column rule as placement_evidence.jl load_alignment. +using Random +const V4S = 11894; const V4E = 25319; const FRAG = 240 +code(c::UInt8) = c == UInt8('A') ? 0x01 : c == UInt8('C') ? 0x02 : c == UInt8('G') ? 0x03 : (c == UInt8('T') || c == UInt8('U')) ? 0x04 : 0x00 +path, out, N, seed = ARGS[1], ARGS[2], parse(Int, ARGS[3]), parse(Int, ARGS[4]) +rng = MersenneTwister(seed) +eligible = Int[]; idx = 0 +open(path) do io + while !eof(io) + readline(io); s = codeunits(readline(io)); global idx += 1 + tot = 0; v4 = 0 + @inbounds for (k, c) in enumerate(s) + if code(c) != 0x00; tot += 1; (V4S <= k <= V4E) && (v4 += 1); end + end + (tot >= 1200 && v4 >= FRAG) && push!(eligible, idx) + end +end +chosen = sort(shuffle(rng, eligible)[1:N]); cs = Set(chosen) +names = String[]; rows = Vector{Vector{UInt8}}(); idx = 0; orig = String[] +open(path) do io + while !eof(io) + h = readline(io); s = readline(io); global idx += 1 + if idx in cs; push!(orig, h[2:end]); push!(rows, UInt8[code(c) for c in codeunits(s)]); end + end +end +# shuffle order so names s001.. are not in file order +perm = shuffle(rng, 1:N); orig = orig[perm]; rows = rows[perm] +names = ["s" * lpad(i, 3, '0') for i in 1:N] +L = length(rows[1]); keep = [k for k in 1:L if count(r -> r[k] != 0x00, rows) >= 2] +v4cols = [j for (j, k) in enumerate(keep) if V4S <= k <= V4E] +A = zeros(UInt8, N, length(keep)); for i in 1:N, (j, k) in enumerate(keep); A[i, j] = rows[i][k]; end +# Fail loudly: eligibility was tested on whole rows, but dropping columns that are filled in fewer +# than two chosen rows can take a V4 base away. 03_prep.jl needs FRAG V4 bases in every row. +for i in 1:N + nv4 = count(j -> A[i, j] != 0x00, v4cols) + nv4 >= FRAG || error("selected row $(names[i]) ($(orig[i])) has only $nv4 V4 bases after column filtering, need $FRAG; no output written") +end +ch = ['-', 'A', 'C', 'G', 'T'] +mkpath(out) +open(joinpath(out, "ref_full.fasta"), "w") do io + for i in 1:N; println(io, ">", names[i]); println(io, String([ch[A[i, j] + 1] for j in 1:size(A, 2)])); end +end +open(joinpath(out, "names.tsv"), "w") do io; for i in 1:N; println(io, names[i], "\t", orig[i]); end; end +open(joinpath(out, "v4cols.txt"), "w") do io; println(io, join(v4cols, ",")); end +println("eligible=$(length(eligible)) N=$N kept_columns=$(size(A,2)) v4_columns=$(length(v4cols)) seed=$seed") diff --git a/docs/cladistics/issue-303/epa/02_tree.sh b/docs/cladistics/issue-303/epa/02_tree.sh new file mode 100755 index 00000000..92d64a81 --- /dev/null +++ b/docs/cladistics/issue-303/epa/02_tree.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 02_tree.sh epa_dir [threads] (threads default 4, the published setting) +# Application: iqtree -s ALN -m MFP -b 100 -nt 4 -seed S -pre PFX -redo +# Used here: iqtree2 -s ALN -m GTR+G4 -nt 4 -seed 1 -pre PFX -redo +# (-m MFP aborted after 12 min at 90/286 models; -b 100 omitted: it does not change the ML tree) +set -eu +E=$(cd "$1" && pwd) || exit 1 +NT=${2:-4} +cd "$E/work/tree" +iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -nt "$NT" -seed 1 -pre "$E/work/tree/reference" -redo > "$E/raw/02_iqtree.stdout" 2>&1 diff --git a/docs/cladistics/issue-303/epa/03_prep.jl b/docs/cladistics/issue-303/epa/03_prep.jl new file mode 100644 index 00000000..e9ceeaca --- /dev/null +++ b/docs/cladistics/issue-303/epa/03_prep.jl @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 03_prep.jl epa_dir Q seed +# Leave-one-out preparation: for each query q, prune, write pruned tree, truth, and three alignments (amp/pos/neg). +using Random, Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, Q, seed = ARGS[1], parse(Int, ARGS[2]), parse(Int, ARGS[3]) +W = joinpath(E, "work") +# start from a clean query directory: stale q*/ variants (e.g. pos from an earlier run with more +# controls) must not survive into the scorer +rm(joinpath(W, "q"); recursive = true, force = true) +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +seqs = Dict{String,String}(); cur = "" +for l in eachline(joinpath(W, "ref_full.fasta")); startswith(l, ">") ? (global cur = l[2:end]) : (seqs[cur] = l); end +v4cols = parse.(Int, split(strip(read(joinpath(W, "v4cols.txt"), String)), ",")) +g0 = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g0.names[u] => u for u in eachindex(g0.names) if g0.names[u] != "") +@assert length(nid) == N +rng = MersenneTwister(seed) +queries = sort(shuffle(rng, 1:N)[1:Q]); negsrc = [rand(rng, setdiff(1:N, q)) for q in queries] +function fragment(s::String, cols) + v = collect(s); out = fill('-', length(v)); k = 0 + for j in cols; if v[j] != '-'; out[j] = v[j]; k += 1; k == 240 && break; end; end + @assert k == 240; String(out) +end +function newick(g::NG, root) + io = IOBuffer() + function w(u, p) + if isleaf(g, u); print(io, g.names[u]); return; end + print(io, "("); first = true + for (v, l) in g.adj[u]; v == p && continue; first || print(io, ","); first = false; w(v, u); print(io, ":", l); end + print(io, ")") + end + # root must be an internal node + print(io, "("); first = true + for (v, l) in g.adj[root]; first || print(io, ","); first = false; w(v, root); print(io, ":", l); end + print(io, ");"); String(take!(io)) +end +open(joinpath(W, "queries.tsv"), "w") do qio + println(qio, "qi\tquery\tnegsrc") + for (qi, q) in enumerate(queries) + qn = names[q]; u = nid[qn] + adj = [copy(d) for d in g0.adj]; g = NG(copy(g0.names), adj, [Dict{Int,Int}() for _ in adj]) + @assert length(adj[u]) == 1 + p = first(keys(adj[u])); @assert length(adj[p]) == 3 "parent of query has degree $(length(adj[p]))" + delete!(adj[p], u); empty!(adj[u]) + (a, la), (b, lb) = collect(adj[p]) + delete!(adj[a], p); delete!(adj[b], p); empty!(adj[p]); adj[a][b] = la + lb; adj[b][a] = la + lb + refs = [n for n in names if n != qn]; rl = leafidx[minimum(refs)] + # truth: root at the alphabetically first leaf; the merged edge a-b, child side + par, kids, below, order = root_at(g, nid[names[rl]], leafidx) + child = par[a] == b ? a : (par[b] == a ? b : error("a,b not adjacent after merge")) + truth = sort([names[i] for i in below[child]]) + dir = joinpath(W, "q", @sprintf("q%03d", qi)); mkpath(dir) + root = first(u2 for u2 in eachindex(g.names) if !isleaf(g, u2) && length(g.adj[u2]) == 3) + write(joinpath(dir, "pruned.nwk"), newick(g, root) * "\n") + write(joinpath(dir, "truth.txt"), join(truth, "\n") * "\n") + write(joinpath(dir, "query.txt"), qn * "\n") + for (kind, row) in (("amp", fragment(seqs[qn], v4cols)), ("pos", seqs[qn]), ("neg", fragment(seqs[names[negsrc[qi]]], v4cols))) + kd = joinpath(dir, kind); mkpath(kd) + open(joinpath(kd, "aln.fasta"), "w") do io + for n in refs; println(io, ">", n); println(io, seqs[n]); end + println(io, ">qry"); println(io, row) + end + cp(joinpath(dir, "pruned.nwk"), joinpath(kd, "tree.nwk"); force=true) + end + println(qio, qi, "\t", qn, "\t", names[negsrc[qi]]) + end +end +println("prepared $Q queries") diff --git a/docs/cladistics/issue-303/epa/04_score.jl b/docs/cladistics/issue-303/epa/04_score.jl new file mode 100644 index 00000000..982f7703 --- /dev/null +++ b/docs/cladistics/issue-303/epa/04_score.jl @@ -0,0 +1,79 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 04_score.jl epa_dir kind -> raw/perquery_.tsv, raw/perclade_.tsv, raw/checks_.txt +using Printf +include(joinpath(@__DIR__, "lib_nwk.jl")) +E, kind = ARGS[1], ARGS[2]; W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)) +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +pq = open(joinpath(E, "raw", "perquery_$kind.tsv"), "w"); pc = open(joinpath(E, "raw", "perclade_$kind.tsv"), "w") +ck = open(joinpath(E, "raw", "checks_$kind.txt"), "w") +println(pq, "qi\tquery\tstatus\tnplacements\ttotal_lwr\tbest_lwr\texact\tedges_apart") +println(pc, "qi\tclade_size\tmass\ttruth_inside") +nedges_ok = true; distinct_ok = true; mono_viol = 0; mono_checked = 0; nscored = 0 +for q in qs + qi = parse(Int, q[1]); dir = joinpath(W, "q", @sprintf("q%03d", qi)); kd = joinpath(dir, kind) + jp = joinpath(kd, "RAxML_portableTree.epa.jplace") + if !isfile(jp); println(pq, qi, "\t", q[2], "\tMISSING\t0\t0\t0\t0\t0"); continue; end + g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String))) + truthleaves = split(strip(read(joinpath(dir, "truth.txt"), String)), "\n") + refs = sort([n for n in names if n != q[2]]); rl = leafidx[refs[1]] + nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") + par, kids, below, order = root_at(g, nid[refs[1]], leafidx) + # edge above node u (u != root leaf) <-> below[u]; root leaf's pendant edge == edge above its only child + root_child = first(kids[nid[refs[1]]]) + edgekey = Dict{BitSet,Int}() # bipartition (side without refs[1]) -> node + for u in order; u == nid[refs[1]] && continue; edgekey[below[u]] = u; end + ne = length(edgekey) + 0 # includes root_child's edge == pendant of root leaf + expected = 2 * (N - 1) - 3 + # jplace tree + tree, fields, rows = parse_jplace(jp) + jg = parse_nwk(tree) + jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "") + @assert !haskey(jn, "qry") "query leaf present in jplace tree" + jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx) + enum2node = Dict{Int,Int}(); jkey = Dict{Int,BitSet}() + for u in jorder; u == jn[refs[1]] && continue; e = jg.lab[u][jpar[u]]; enum2node[e] = u; jkey[e] = jbelow[u]; end + # the edge above the root leaf's only child and the pendant edge of root leaf are the same unrooted edge: jplace labels it once + nj = length(jkey) + if nj != expected; global nedges_ok = false; end + # edge-number -> bipartition uniqueness + if length(Set(values(jkey))) != length(jkey); global distinct_ok = false; end + println(ck, "q$qi: jplace edges=$(length(jkey)) distinct_bipartitions=$(length(Set(values(jkey)))) expected=$expected pruned_tree_edges=$(length(edgekey)) all_in_pruned=$(all(k -> haskey(edgekey, k), values(jkey)))") + ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields) + pls = [(Int(r[ei]), r[li]) for r in rows] + total = sum(p[2] for p in pls) + # best placement + bi = argmax([p[2] for p in pls]); bkey = jkey[pls[bi][1]] + tkey = BitSet(leafidx[n] for n in truthleaves) + # truth is the below-set of its child in the pruned tree rooted at refs[1] + @assert haskey(edgekey, tkey) "truth bipartition not an edge of pruned tree" + function apart(a, b) + a == b && return 0 + da = Dict{Int,Int}(); u = a; k = 0; while u != 0 && u != nid[refs[1]]; da[u] = k; u = par[u]; k += 1; end + u = b; k = 0; while u != 0 && u != nid[refs[1]]; haskey(da, u) && return da[u] + k; u = par[u]; k += 1; end + typemax(Int) + end + tnode = edgekey[tkey]; bnode = edgekey[bkey] + # root-leaf pendant edge: represent by root_child's node (same edge) + println(pq, qi, "\t", q[2], "\tOK\t", length(pls), "\t", total, "\t", pls[bi][2], "\t", Int(bkey == tkey), "\t", apart(bnode, tnode)) + # clades: internal nodes other than the root leaf, whose parent... internal non-leaf nodes of the rooted tree (root leaf excluded) + plnode = [(edgekey[jkey[e]], w) for (e, w) in pls] + clades = [u for u in order if u != nid[refs[1]] && !isleaf(g, u)] + masses = Dict{Int,Float64}() + for c in clades + S = below[c]; m = 0.0 + for (nd, w) in plnode; below[nd] != S && issubset(below[nd], S) && (m += w); end + masses[c] = m + println(pc, qi, "\t", length(S), "\t", m, "\t", Int(tkey != S && issubset(tkey, S))) + end + # monotonicity under inclusion: mass(child) <= mass(parent) for parent clade strictly containing + for c in clades; p = par[c]; (p == nid[refs[1]] || isleaf(g, p)) && continue; global mono_checked += 1; masses[c] <= masses[p] + 1e-12 || (global mono_viol += 1); end + global nscored += 1 +end +println(ck, "SUMMARY scored=$nscored edge_count_ok=$nedges_ok distinct_bipartitions_ok=$distinct_ok monotone_pairs_checked=$mono_checked monotone_violations=$mono_viol") +close(pq); close(pc); close(ck) +println("scored $nscored; ", read(joinpath(E, "raw", "checks_$kind.txt"), String)[end-200:end]) diff --git a/docs/cladistics/issue-303/epa/05_run_epa.sh b/docs/cladistics/issue-303/epa/05_run_epa.sh new file mode 100755 index 00000000..bb16ebf7 --- /dev/null +++ b/docs/cladistics/issue-303/epa/05_run_epa.sh @@ -0,0 +1,18 @@ +#!/bin/sh +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: 05_run_epa.sh epa_dir kind (kind = amp | pos | neg) +# Application command (src/pipeline/phylogeny.jl, step "place"), threads=4 -> -T max(2,threads) = -T 4 in the app; +# here -T 2 with two runs in parallel on 4 cores (thread count does not change the result). +# raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s -t -T N -w / +set -eu +E=$(cd "$1" && pwd) || exit 1; K=$2 +# one NUL-terminated path per query directory, so paths with spaces stay whole; xargs exits 123 if any worker failed +n=0 +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && n=$((n + 1)); done +[ "$n" -gt 0 ] || { echo "05_run_epa.sh: no query directories for kind $K under $E/work/q" >&2; exit 1; } +for D in "$E"/work/q/q*/"$K"; do [ -d "$D" ] && printf "%s\0" "$D"; done | + xargs -0 -P 2 -n 1 sh -c 'D=$1; cd "$D" || exit 1; raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2 -n epa -s "$D/aln.fasta" -t "$D/tree.nwk" -T 2 -w "$D/" > "$D/raxml.stdout" 2>&1' _ || + { echo "05_run_epa.sh: at least one EPA run failed (kind $K); see raxml.stdout in the query directories" >&2; exit 1; } diff --git a/docs/cladistics/issue-303/epa/06_tables.jl b/docs/cladistics/issue-303/epa/06_tables.jl new file mode 100644 index 00000000..22c21dd5 --- /dev/null +++ b/docs/cladistics/issue-303/epa/06_tables.jl @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 06_tables.jl epa_dir kind [kind...] -> prints tables from raw/perquery_.tsv and raw/perclade_.tsv +using Printf, Statistics +E = ARGS[1] +function table(kind) + pq = [split(l, "\t") for l in eachline(joinpath(E, "raw", "perquery_$kind.tsv"))][2:end] + ok = filter(r -> r[3] == "OK", pq) + missing_n = length(pq) - length(ok) + pc = [split(l, "\t") for l in eachline(joinpath(E, "raw", "perclade_$kind.tsv"))][2:end] + byq = Dict{Int,Vector{Tuple{Int,Float64,Bool}}}() + for r in pc; push!(get!(byq, parse(Int, r[1]), Tuple{Int,Float64,Bool}[]), (parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1")); end + ex = [r[7] == "1" for r in ok]; ap = [parse(Int, r[8]) for r in ok]; tot = [parse(Float64, r[5]) for r in ok]; npl = [parse(Int, r[4]) for r in ok] + println("\n===== ", kind, " =====") + @printf("queries scored: %d (missing/failed: %d); placed on exactly the true edge: %.1f%% (%d); within 2 edges: %.1f%%; median edges apart: %g\n", + length(ok), missing_n, 100mean(ex), sum(ex), 100mean(ap .<= 2), median(ap)) + @printf("placements reported per query: median %g, max %d; queries with a single placement: %d; total LWR reported per query: min %.4f, median %.4f, max %.4f\n", + median(npl), maximum(npl), count(==(1), npl), minimum(tot), median(tot), maximum(tot)) + for alpha in (0.01, 0.05) + ent_ok = ent_bad = ref_ok = ref_bad = unr_in = unr_out = 0; pin = Int[]; pin_ok = 0; nopin = 0; q_wrong = 0 + for (qi, cl) in byq + best = 0; bsz = typemax(Int); bt = false + for (sz, m, t) in cl + if m >= 1 - alpha - 1e-12 + t ? (ent_ok += 1) : (ent_bad += 1) + sz < bsz && ((bsz, bt, best) = (sz, t, 1)) + elseif m <= alpha + 1e-12 + t ? (ref_bad += 1) : (ref_ok += 1) + else + t ? (unr_in += 1) : (unr_out += 1) + end + end + if best == 0; nopin += 1 else push!(pin, bsz); bt ? (pin_ok += 1) : (q_wrong += 1) end + end + @printf("\n alpha = %.2f: entailed if mass >= %.2f, refuted if mass <= %.2f\n", alpha, 1 - alpha, alpha) + @printf(" entailed %7d verdicts, wrong %5d (truth not inside) -> sound %.2f%%\n", ent_ok + ent_bad, ent_bad, 100ent_ok / max(ent_ok + ent_bad, 1)) + @printf(" refuted %7d verdicts, wrong %5d (truth inside) -> sound %.3f%%\n", ref_ok + ref_bad, ref_bad, 100ref_ok / max(ref_ok + ref_bad, 1)) + @printf(" unresolved %7d verdicts (%d truth inside, %d truth not inside)\n", unr_in + unr_out, unr_in, unr_out) + d = ent_ok + unr_in + ref_bad + @printf(" of the %d clades that truly contain the query: entailed %.1f%%, unresolved %.1f%%, wrongly refuted %.1f%%\n", d, 100ent_ok / d, 100unr_in / d, 100ref_bad / d) + if !isempty(pin) + @printf(" smallest entailed clade per query: median size %g (quartiles %g, %g) of %d references; contains truth for %.1f%% of %d queries; no entailed clade for %d queries\n", + median(pin), quantile(pin, 0.25), quantile(pin, 0.75), 199, 100pin_ok / length(pin), length(pin), nopin) + else + @printf(" no entailed clade for any query (%d)\n", nopin) + end + @printf(" queries whose smallest entailed clade is wrong: %d of %d (with an entailed clade)\n", q_wrong, length(pin)) + end + println("\n calibration of clade mass (share of clades that contain the truth)") + edges = [0.05, 0.25, 0.5, 0.75, 0.95]; hit = zeros(Int, 7); cnt = zeros(Int, 7) + for cl in values(byq), (sz, m, t) in cl + b = m <= 0 ? 1 : m > 0.95 + 1e-12 ? 7 : 1 + findfirst(x -> m <= x + 1e-12, edges) + cnt[b] += 1; hit[b] += t + end + labels = ["0", "(0, .05]", "(.05, .25]", "(.25, .5]", "(.5, .75]", "(.75, .95]", "(.95, 1]"] + for b in 1:7; cnt[b] > 0 && @printf(" mass %-11s clades %7d truth inside %6.2f%%\n", labels[b], cnt[b], 100hit[b] / cnt[b]); end +end +for k in ARGS[2:end]; table(k); end diff --git a/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl b/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl new file mode 100644 index 00000000..88c70513 --- /dev/null +++ b/docs/cladistics/issue-303/epa/07_selfcheck_truth.jl @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# usage: julia 07_selfcheck_truth.jl epa_dir -- independent truth: from leaf-to-leaf path distances in the UNPRUNED tree +# Rule: q hangs by pendant length l from p. For leaves x,y != q: g(x,y) = d(q,x)+d(q,y)-d(x,y). +# x,y on different sides of p (sides A/B = the two other neighbours' subtrees): g = 2l (the minimum); same side: g = 2(l + d(p, meet)) >= 2l. +# Leaves are clustered by linking pairs with g > min + tol; the two largest components must be the sides. +using Printf, Random +include(joinpath(@__DIR__, "lib_nwk.jl")) +E = ARGS[1]; W = joinpath(E, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))] +g = parse_nwk(strip(read(joinpath(W, "tree", "reference.treefile"), String))) +nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") +function dists(src) + d = fill(NaN, length(g.names)); d[src] = 0.0; st = [src] + while !isempty(st); u = pop!(st); for (v, l) in g.adj[u]; isnan(d[v]) && (d[v] = d[u] + l; push!(st, v)); end; end + d +end +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +rng = MersenneTwister(99); pick = shuffle(rng, 1:length(qs))[1:5] +big = [i for i in shuffle(rng, 1:length(qs)) if length(readlines(joinpath(W, "q", @sprintf("q%03d", parse(Int, qs[i][1])), "truth.txt"))) >= 5][1:5]; append!(pick, big) +allok = true +for i in pick + qi = parse(Int, qs[i][1]); qn = qs[i][2] + truth = Set(split(strip(read(joinpath(W, "q", @sprintf("q%03d", qi), "truth.txt"), String)), "\n")) + dq = dists(nid[qn]); others = [n for n in names if n != qn]; m = length(others) + D = Dict(n => dists(nid[n]) for n in others) + G = [dq[nid[x]] + dq[nid[y]] - D[x][nid[y]] for x in others, y in others] + mn = minimum(G[i2, j2] for i2 in 1:m, j2 in 1:m if i2 != j2) + tol = 1e-7 + comp = collect(1:m); find(x) = (while comp[x] != x; comp[x] = comp[comp[x]]; x = comp[x]; end; x) + for i2 in 1:m, j2 in i2+1:m; G[i2, j2] > mn + tol && (comp[find(i2)] = find(j2)); end + groups = Dict{Int,Vector{String}}(); for i2 in 1:m; push!(get!(groups, find(i2), String[]), others[i2]); end + gs = sort(collect(values(groups)); by = length, rev = true) + sides = [Set(x) for x in gs[1:min(2, length(gs))]] + # the "truth" side in the prep is the side not containing the alphabetically first reference leaf + first_leaf = minimum(others) + cand = [s for s in sides if !(first_leaf in s)] + ok = length(gs) == 2 && length(cand) == 1 && cand[1] == truth + global allok &= ok + @printf(" %s (q%03d): components=%d sizes=%s; independent side without %s has %d leaves; prepared truth has %d leaves; match=%s\n", qn, qi, length(gs), string(length.(gs)), first_leaf, isempty(cand) ? -1 : length(cand[1]), length(truth), ok) +end +println("SELF-CHECK (a): all ten match (5 random + 5 with truth side >= 5 leaves) = ", allok) diff --git a/docs/cladistics/issue-303/epa/08_by_depth.jl b/docs/cladistics/issue-303/epa/08_by_depth.jl new file mode 100644 index 00000000..c7c27c3e --- /dev/null +++ b/docs/cladistics/issue-303/epa/08_by_depth.jl @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# Independent recomputation of the tables from raw/perclade_*.tsv and raw/perquery_*.tsv, and the +# breakdown of wrong entailed verdicts by clade size. Written separately from 06_tables.jl as a +# cross-check. usage: julia 08_by_depth.jl epa_dir +using Printf, Statistics +# median as text; an empty collection (no clade reaches the threshold) reports n/a instead of throwing +medstr(x) = isempty(x) ? "n/a" : @sprintf("%.1f", median(x)) +function load(kind) + pc = [split(l, '\t') for l in readlines(joinpath(ARGS[1], "raw", "perclade_$kind.tsv"))[2:end]] + C = [(parse(Int, r[1]), parse(Int, r[2]), parse(Float64, r[3]), r[4] == "1") for r in pc] + pq = [split(l, '\t') for l in readlines(joinpath(ARGS[1], "raw", "perquery_$kind.tsv"))[2:end]] + Q = Dict(parse(Int, r[1]) => (n = parse(Int, r[4]), tot = parse(Float64, r[5]), best = parse(Float64, r[6]), exact = r[7] == "1", apart = parse(Int, r[8])) for r in pq) + C, Q +end +for kind in ("amp", "pos", "neg") + C, Q = load(kind) + println("== $kind: $(length(C)) clade rows, $(length(Q)) queries") + for a in (0.01, 0.05) + ent = [c for c in C if c[3] >= 1 - a - 1e-12]; ref = [c for c in C if c[3] <= a + 1e-12] + ew = count(c -> !c[4], ent); rw = count(c -> c[4], ref) + # smallest entailed clade per query + sm = Dict{Int,Tuple{Int,Bool}}() + for c in ent; (!haskey(sm, c[1]) || c[2] < sm[c[1]][1]) && (sm[c[1]] = (c[2], c[4])); end + wrongq = sort([q for (q, v) in sm if !v[2]]) + @printf(" alpha %.2f: entailed %d wrong %d | refuted %d wrong %d | smallest entailed: %d queries, median size %s, wrong %d\n", + a, length(ent), ew, length(ref), rw, length(sm), medstr([v[1] for v in values(sm)]), length(wrongq)) + if kind == "amp" + println(" wrong-smallest queries (qi: clade size, edges apart, placements, best LWR): ", + join(["$q: $(sm[q][1]), $(Q[q].apart), $(Q[q].n), $(round(Q[q].best, digits=3))" for q in wrongq], " | ")) + # entailed error by clade size + for (lo, hi) in ((2, 3), (4, 7), (8, 15), (16, 49), (50, 199)) + e = [c for c in ent if lo <= c[2] <= hi] + @printf(" entailed, clade size %3d..%3d: %5d verdicts, wrong %2d\n", lo, hi, length(e), count(c -> !c[4], e)) + end + # rule: smallest entailed clade of size >= k + for k in (2, 4, 8, 16) + s2 = Dict{Int,Tuple{Int,Bool}}() + for c in ent; c[2] >= k || continue; (!haskey(s2, c[1]) || c[2] < s2[c[1]][1]) && (s2[c[1]] = (c[2], c[4])); end + @printf(" smallest entailed clade of size >= %2d: %3d queries, median size %s, wrong %d\n", k, length(s2), medstr([v[1] for v in values(s2)]), count(v -> !v[2], values(s2))) + end + # by number of placements reported + for (lab, f) in (("1 placement", n -> n == 1), ("2+ placements", n -> n >= 2)) + qs = [q for q in keys(sm) if f(Q[q].n)] + @printf(" %s: %d queries with an entailed clade, wrong smallest %d\n", lab, length(qs), count(q -> !sm[q][2], qs)) + end + end + end +end diff --git a/docs/cladistics/issue-303/epa/lib_nwk.jl b/docs/cladistics/issue-303/epa/lib_nwk.jl new file mode 100644 index 00000000..46a3a7cc --- /dev/null +++ b/docs/cladistics/issue-303/epa/lib_nwk.jl @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): evidence verdicts on placements made by the application's own +# placement step (RAxML EPA). See ../README.md, section 8. +# Minimal Newick + jplace reading; standard library only. Data is read as data only. +# Graph = Vector{Dict{Int,Float64}} adjacency; names[i] = leaf name or "". +mutable struct NG; names::Vector{String}; adj::Vector{Dict{Int,Float64}}; lab::Vector{Dict{Int,Int}}; end +# lab[u][v] = jplace edge number of edge u-v (only filled by parse with {n}) +function parse_nwk(s::AbstractString) + names = String[]; adj = Dict{Int,Float64}[]; lab = Dict{Int,Int}[] + newnode!(n) = (push!(names, n); push!(adj, Dict{Int,Float64}()); push!(lab, Dict{Int,Int}()); length(names)) + s = strip(s); pos = 1; n = lastindex(s) + # returns (node, branchlength, edgenum) + function rd() + if s[pos] == '(' + pos += 1; kids = Tuple{Int,Float64,Int}[] + while true + push!(kids, rd()); c = s[pos]; pos += 1 + c == ',' && continue; c == ')' && break; error("bad newick at $pos") + end + lbl = "" + while s[pos] ∉ (':', ',', ')', ';', '{'); lbl *= s[pos]; pos += 1; end + u = newnode!("") + for (k, l, e) in kids; adj[u][k] = l; adj[k][u] = l; e >= 0 && (lab[u][k] = e; lab[k][u] = e); end + else + st = pos; while s[pos] ∉ (':', ',', ')', ';', '{'); pos += 1; end + u = newnode!(String(s[st:pos-1])) + end + bl = 0.0; en = -1 + if s[pos] == ':'; pos += 1; st = pos; while s[pos] ∉ (',', ')', ';', '{'); pos += 1; end; bl = parse(Float64, s[st:pos-1]); end + if s[pos] == '{'; pos += 1; st = pos; while s[pos] != '}'; pos += 1; end; en = parse(Int, s[st:pos-1]); pos += 1; end + (u, bl, en) + end + (root, _, _) = rd() + g = NG(names, adj, lab) + # suppress a degree-2 root + if length(adj[root]) == 2 && names[root] == "" + (a, la), (b, lb) = collect(adj[root]) + delete!(adj[a], root); delete!(adj[b], root); delete!(adj[root], a); delete!(adj[root], b) + adj[a][b] = la + lb; adj[b][a] = la + lb + if haskey(lab[root], a) && haskey(lab[root], b); lab[a][b] = lab[root][a]; lab[b][a] = lab[root][a]; end + end + g +end +isleaf(g::NG, u) = g.names[u] != "" +alive(g::NG, u) = !isempty(g.adj[u]) +# Root at leaf `rl`; returns parent, children, below (BitSet of leaf *indices in leafidx*), order (preorder) +function root_at(g::NG, rl::Int, leafidx::Dict{String,Int}) + n = length(g.names); parent = zeros(Int, n); order = Int[]; st = [rl]; seen = falses(n); seen[rl] = true + kids = [Int[] for _ in 1:n] + while !isempty(st) + u = pop!(st); push!(order, u) + for v in keys(g.adj[u]); if !seen[v]; seen[v] = true; parent[v] = u; push!(kids[u], v); push!(st, v); end; end + end + below = Vector{BitSet}(undef, n) + for u in reverse(order) + if isleaf(g, u); below[u] = BitSet([leafidx[g.names[u]]]) + else b = BitSet(); for c in kids[u]; union!(b, below[c]); end; below[u] = b; end + end + parent, kids, below, order +end +function parse_jplace(path) + txt = read(path, String) + m = match(r"\"tree\"\s*:\s*\"([^\"]*)\"", txt); tree = m.captures[1] + fm = match(r"\"fields\"\s*:\s*\[([^\]]*)\]", txt) + fields = [strip(replace(x, "\"" => "")) for x in split(fm.captures[1], ",")] + pm = match(r"\"p\"\s*:\s*\[(.*?)\]\s*,\s*\"n\""s, txt) + rows = Vector{Vector{Float64}}() + for rm in eachmatch(r"\[([^\[\]]+)\]", pm.captures[1] * "]") + push!(rows, parse.(Float64, split(rm.captures[1], ","))) + end + tree, fields, rows +end diff --git a/docs/cladistics/issue-303/epa/test_select.jl b/docs/cladistics/issue-303/epa/test_select.jl new file mode 100644 index 00000000..d5b7bf7f --- /dev/null +++ b/docs/cladistics/issue-303/epa/test_select.jl @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E8 (issue 303): regression test for 01_select.jl. See ../README.md, section 8. +# +# usage: julia test_select.jl +# +# 01_select.jl tests eligibility (240 bases in the V4 window) on whole rows, then drops every +# column filled in fewer than two selected rows. A row can lose a V4 base that way. It must +# then stop with an error and write nothing, because 03_prep.jl needs 240 V4 bases in each row. +# Two synthetic two-row alignments: one where both rows keep 240 V4 bases, one where each row +# loses one. +const V4S = 11894; const LEN = 25400 +const SELECT = joinpath(@__DIR__, "01_select.jl") + +function row(v4_columns) + s = fill('-', LEN) + s[1:960] .= 'A' # 960 bases outside the V4 window + for k in v4_columns; s[k] = 'C'; end # 240 bases inside it: 1,200 in all + String(s) +end + +function select(dir, rows) + fasta = joinpath(dir, "in.fasta"); out = joinpath(dir, "out") + open(fasta, "w") do io + for (i, r) in enumerate(rows); println(io, ">row", i); println(io, r); end + end + log = joinpath(dir, "log.txt") + p = run(pipeline(ignorestatus(`$(Base.julia_cmd()) $SELECT $fasta $out 2 1`); stdout = log, stderr = log)) + p.exitcode, out, read(log, String) +end + +failures = String[] +check(ok, what) = (println(ok ? "ok " : "FAIL ", what); ok || push!(failures, what)) + +mktempdir() do dir + shared = collect(V4S:V4S+239) + code, out, log = select(dir, [row(shared), row(shared)]) + check(code == 0, "both rows keep 240 V4 bases: exit status 0") + check(isfile(joinpath(out, "ref_full.fasta")) && isfile(joinpath(out, "names.tsv")), "both rows keep 240 V4 bases: output written") + v4 = isfile(joinpath(out, "v4cols.txt")) ? length(split(strip(read(joinpath(out, "v4cols.txt"), String)), ",")) : 0 + check(v4 == 240, "both rows keep 240 V4 bases: 240 V4 columns kept (got $v4)") +end + +mktempdir() do dir + a = collect(V4S:V4S+239) # last column is filled in this row only + b = vcat(collect(V4S:V4S+238), V4S + 300) # one column filled in this row only + code, out, log = select(dir, [row(a), row(b)]) + check(code != 0, "each row loses one V4 base: non-zero exit status (got $code)") + check(occursin("has only 239 V4 bases after column filtering", log), "each row loses one V4 base: error names the count") + check(!isdir(out), "each row loses one V4 base: no output written") +end + +isempty(failures) || (println(length(failures), " check(s) failed"); exit(1)) +println("all checks passed")