diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md index 2ab79fca..7c3a9580 100644 --- a/docs/cladistics/issue-303/README.md +++ b/docs/cladistics/issue-303/README.md @@ -35,7 +35,9 @@ application's own placement step (RAxML EPA) the picture is sharper and less saf 2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more references and 7 errors among 62 on clades of two or three. On a larger sample the smallest confident clade was wrong for 32 of 448 queries (7.1%), falling to 4 of 448 (0.9%) when the -clade had to hold at least 64 of 399 references. +clade had to hold at least 64 of 399 references. About two thirds of those errors were +judged against a branch the reference tree does not itself support; where it is firm, 6 of +155 queries were wrong. ## Data @@ -338,6 +340,37 @@ sequence, placed the same way, was never wrong in 60 queries. The truth was itse from the whole sequence, so that control shows the machinery is sound; it does not show the reference tree is right at the finest depth. +### 10. Placement error, or an uncertain reference tree? (`epa-depth/10_support.jl`) + +Experiment 9 judges each verdict against the reference tree built with the query's whole +sequence. That tree is itself an estimate. The tree of the first draw carries +ultrafast-bootstrap support on its branches: a median of 100, with 112 of its 397 internal +branches below 95. For each reported clade, one branch of that tree backs the truth. If the +truth says the query is inside the clade, it is the branch that holds the clade with the +query. If the truth says it is outside, it is the branch that holds the clade without it. The +support of that branch is the reference tree's own confidence in the truth for that verdict. +The thresholds 95 and 70 were fixed before the tables were computed. + +First draw, 230 queries given a clade, wrong verdicts by the support of the truth branch: + +| Smallest entailed clade of at least | Support 95 or more (exact 95% interval) | Support below 95 | +| --- | --- | --- | +| 2 references | 6 of 155 (3.9%, 1.4 to 8.2) | 13 of 75 (17.3%, 9.6 to 27.8) | +| 8 | 2 of 107 (1.9%, 0.2 to 6.6) | 8 of 123 (6.5%, 2.8 to 12.4) | +| 32 | 1 of 69 (1.4%, 0.0 to 7.8) | 3 of 161 (1.9%, 0.4 to 5.3) | + +With no minimum size, 13 of the 19 wrong verdicts were judged against a branch the reference +tree does not itself support at 95. The two errors on very large clades from this draw (357 +and 367 references) were against branches with support 62 and 65. The other 6 were against +well-supported branches, 5 of them with support 100. Those are placement errors by the +reference tree's own standard. + +What this shows: at the finest depth about two thirds of the measured error sits where the +reference tree is itself unsure, and about one third is error against a well-supported +branch. It splits experiment 9's error; it does not reduce it. In the first case the truth +may be what is wrong, and in the second the placement is. Where the reference tree is firm, +the smallest confident clade was wrong for 6 of 155 queries. + ## Reading of the results - **Evidence on placements.** Experiments 7 and 8 are the positive real-data results. Verdicts @@ -349,7 +382,10 @@ reference tree is right at the finest depth. smallest clades. Experiment 9 puts numbers on it: a minimum clade size trades error for resolution (7.1% of queries wrong with no minimum, 0.9% with a minimum of 64 references). A rule for the application needs that minimum or a better-calibrated mass. Neither has - been chosen, and no rule has been tested inside the application. + been chosen, and no rule has been tested inside the application. Experiment 10 adds that + the reference tree matters as much as the placement: most fine-depth errors fall on + branches the reference tree does not support, so showing a verdict only on supported + branches of the reference is a third option, also untested. - **Counting reads does not average this error away.** Each query in experiments 7 and 8 is one sequence. Reads from the same organism share that sequence, so if it is misplaced they are all misplaced together. The bound on wrongly counted reads that the counting rule @@ -394,7 +430,10 @@ reference tree is right at the finest depth. 200 references and 200 queries, one draw. Experiment 9 shares every limit of experiment 8 except size: 400 references, two draws, 600 queries. The second draw's tree was built without bootstrap after the bootstrapped run exceeded 25 minutes; the bootstrap supports of - the first tree were saved and not analysed. Experiments 7 and 8 differ in the number of + the first tree are analysed in experiment 10. Experiment 10 uses that one tree and 230 + queries. The support of one branch is a lower bound on the tree's confidence that a query + lies outside a clade, because the branch can fail in a bootstrap replicate for reasons + that do not involve the query. Experiments 7 and 8 differ in the number of references, the tree method, the source of the mass and the thresholds, so the difference between them is not the effect of the placement method alone. A cheaper agent ran the experiment; the tables were recomputed separately from its per-clade output. @@ -504,6 +543,12 @@ julia "$WORK/scripts/06_tables.jl" new "$WORK/raw" julia "$WORK/scripts/08_crosscheck.jl" "$WORK/raw" ``` +Experiment 10, on the first draw: + +```bash +julia "$WORK/scripts/10_support.jl" "$WORK/A" +``` + The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its own copy of the shared code because it is the script as it was run; the later scripts include `lib_tree.jl`. diff --git a/docs/cladistics/issue-303/epa-depth/10_support.jl b/docs/cladistics/issue-303/epa-depth/10_support.jl new file mode 100644 index 00000000..a6421947 --- /dev/null +++ b/docs/cladistics/issue-303/epa-depth/10_support.jl @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E10 (issue 303): are the wrong entailed verdicts of experiment 9 errors of the +# placement, or uncertainty in the reference tree? +# +# usage: julia 10_support.jl DRAW_DIR (DRAW_DIR = the experiment 9 directory of a draw whose +# tree was built with UFBoot, i.e. draw A) +# +# The truth in experiment 9 is where the full-length sequence q sits in the reference tree built +# WITH q. That tree carries ultrafast-bootstrap support on its branches. For a clade S of the +# pruned tree, the verdict "q is in S" is judged against one branch of the full tree: +# +# truth says q IS in S <=> the full tree has the clade S + q; its support backs the truth. +# truth says q is NOT in S <=> the full tree has the clade S (without q); its support backs it. +# +# So every reported clade gets a "truth support". A wrong verdict against a weakly supported +# branch may be the reference tree's uncertainty, not a placement error. A wrong verdict against +# a branch with support >= 95 is a placement error by the reference tree's own standard. +using Printf, Statistics +include(joinpath(@__DIR__, "lib_nwk.jl")) +D = ARGS[1]; W = joinpath(D, "work") +names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names) +leafidx = Dict(n => i for (i, n) in enumerate(names)); ALL = BitSet(1:N) + +# ---- full tree with support labels: every internal branch -> (leaf set below it, support) ---- +function supports(s::AbstractString) + out = Dict{BitSet,Float64}(); pos = Ref(1) + function rd() + if s[pos[]] == '(' + pos[] += 1; set = BitSet() + while true + union!(set, rd()); c = s[pos[]]; pos[] += 1 + c == ',' && continue; c == ')' && break; error("bad newick at $(pos[])") + end + st = pos[]; while s[pos[]] ∉ (':', ',', ')', ';'); pos[] += 1; end + lbl = s[st:pos[]-1] + if s[pos[]] == ':'; pos[] += 1; while s[pos[]] ∉ (',', ')', ';'); pos[] += 1; end; end + isempty(lbl) || (out[canon(set)] = parse(Float64, split(lbl, "/")[end])) + return set + else + st = pos[]; while s[pos[]] ∉ (':', ',', ')', ';'); pos[] += 1; end + nm = s[st:pos[]-1] + if s[pos[]] == ':'; pos[] += 1; while s[pos[]] ∉ (',', ')', ';'); pos[] += 1; end; end + return BitSet([leafidx[nm]]) + end + end + rd(); out +end +# a branch is a split of the leaves; name it by the side that does not hold leaf 1 +canon(set::BitSet) = 1 in set ? setdiff(ALL, set) : set +SUP = supports(strip(read(joinpath(W, "tree", "reference_with_support.treefile"), String))) +trivial(set) = (c = canon(set); length(c) <= 1 || length(c) >= N - 1) +"support of the full-tree branch with this leaf set on one side; 100 for a pendant branch; NaN if absent" +support(set::BitSet) = trivial(set) ? 100.0 : get(SUP, canon(set), NaN) +@printf("full tree: %d leaves, %d internal branches with a support value (median %.0f, %d below 95, %d below 70)\n", + N, length(SUP), median(values(SUP)), count(<(95), values(SUP)), count(<(70), values(SUP))) + +# ---- per query: the reported clade under the rule "smallest entailed clade of >= k references" ---- +qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end] +struct Rep; q::Int; k::Int; size::Int; wrong::Bool; sup::Float64; end +reps = Rep[]; absent = 0 +for q in qs + qi = parse(Int, q[1]); qn = q[2]; dir = joinpath(W, "q", @sprintf("q%03d", qi)) + jp = joinpath(dir, "amp", "RAxML_portableTree.epa.jplace"); isfile(jp) || continue + g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String))) + truth = BitSet(leafidx[n] for n in split(strip(read(joinpath(dir, "truth.txt"), String)), "\n")) + refs = sort([n for n in names if n != qn]) + nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "") + par, kids, below, order = root_at(g, nid[refs[1]], leafidx) + tree, fields, rows = parse_jplace(jp); jg = parse_nwk(tree) + jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "") + jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx) + jkey = Dict{Int,BitSet}(); for u in jorder; u == jn[refs[1]] && continue; jkey[jg.lab[u][jpar[u]]] = jbelow[u]; end + ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields) + pls = [(jkey[Int(r[ei])], r[li]) for r in rows] + clades = [below[u] for u in order if u != nid[refs[1]] && !isleaf(g, u)] + mass(S) = sum((w for (b, w) in pls if b != S && issubset(b, S)); init = 0.0) + ent = [S for S in clades if mass(S) >= 0.99 - 1e-12] + qix = leafidx[qn] + for k in (2, 8, 32) + c = [S for S in ent if length(S) >= k]; isempty(c) && continue + S = c[argmin(length.(c))] + inside = truth != S && issubset(truth, S) + # the full-tree branch that backs the truth for this clade + s = inside ? support(union(S, BitSet([qix]))) : support(S) + isnan(s) && (global absent += 1) + push!(reps, Rep(qi, k, length(S), !inside, s)) + end +end +println("reported clades whose truth branch was not found in the full tree: $absent") + +for k in (2, 8, 32) + R = [r for r in reps if r.k == k && !isnan(r.sup)] + println("\nrule: smallest entailed clade of at least $k references ($(length(R)) queries with a clade)") + @printf(" %-28s %8s %8s %10s\n", "truth branch support", "queries", "wrong", "rate") + for (lab, f) in (("100 (or pendant)", r -> r.sup >= 100), ("95 to 99", r -> 95 <= r.sup < 100), + ("70 to 94", r -> 70 <= r.sup < 95), ("below 70", r -> r.sup < 70), + ("95 or more (firm)", r -> r.sup >= 95), ("below 95 (not firm)", r -> r.sup < 95)) + X = filter(f, R); w = count(r -> r.wrong, X) + @printf(" %-28s %8d %8d %9s\n", lab, length(X), w, isempty(X) ? "n/a" : @sprintf("%.1f%%", 100w / length(X))) + end + Wq = [r for r in R if r.wrong] + println(" wrong queries: clade size / truth support: ", join(sort(["$(r.size)/$(round(Int, r.sup))" for r in Wq]), " ")) +end