Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions docs/cladistics/issue-303/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ application's own placement step (RAxML EPA) the picture is sharper and less saf
2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more
references and 7 errors among 62 on clades of two or three. On a larger sample the smallest
confident clade was wrong for 32 of 448 queries (7.1%), falling to 4 of 448 (0.9%) when the
clade had to hold at least 64 of 399 references.
clade had to hold at least 64 of 399 references. About two thirds of those errors were
judged against a branch the reference tree does not itself support; where it is firm, 6 of
155 queries were wrong.

## Data

Expand Down Expand Up @@ -338,6 +340,37 @@ sequence, placed the same way, was never wrong in 60 queries. The truth was itse
from the whole sequence, so that control shows the machinery is sound; it does not show the
reference tree is right at the finest depth.

### 10. Placement error, or an uncertain reference tree? (`epa-depth/10_support.jl`)

Experiment 9 judges each verdict against the reference tree built with the query's whole
sequence. That tree is itself an estimate. The tree of the first draw carries
ultrafast-bootstrap support on its branches: a median of 100, with 112 of its 397 internal
branches below 95. For each reported clade, one branch of that tree backs the truth. If the
truth says the query is inside the clade, it is the branch that holds the clade with the
query. If the truth says it is outside, it is the branch that holds the clade without it. The
support of that branch is the reference tree's own confidence in the truth for that verdict.
The thresholds 95 and 70 were fixed before the tables were computed.

First draw, 230 queries given a clade, wrong verdicts by the support of the truth branch:

| Smallest entailed clade of at least | Support 95 or more (exact 95% interval) | Support below 95 |
| --- | --- | --- |
| 2 references | 6 of 155 (3.9%, 1.4 to 8.2) | 13 of 75 (17.3%, 9.6 to 27.8) |
| 8 | 2 of 107 (1.9%, 0.2 to 6.6) | 8 of 123 (6.5%, 2.8 to 12.4) |
| 32 | 1 of 69 (1.4%, 0.0 to 7.8) | 3 of 161 (1.9%, 0.4 to 5.3) |

With no minimum size, 13 of the 19 wrong verdicts were judged against a branch the reference
tree does not itself support at 95. The two errors on very large clades from this draw (357
and 367 references) were against branches with support 62 and 65. The other 6 were against
well-supported branches, 5 of them with support 100. Those are placement errors by the
reference tree's own standard.

What this shows: at the finest depth about two thirds of the measured error sits where the
reference tree is itself unsure, and about one third is error against a well-supported
branch. It splits experiment 9's error; it does not reduce it. In the first case the truth
may be what is wrong, and in the second the placement is. Where the reference tree is firm,
the smallest confident clade was wrong for 6 of 155 queries.

## Reading of the results

- **Evidence on placements.** Experiments 7 and 8 are the positive real-data results. Verdicts
Expand All @@ -349,7 +382,10 @@ reference tree is right at the finest depth.
smallest clades. Experiment 9 puts numbers on it: a minimum clade size trades error for
resolution (7.1% of queries wrong with no minimum, 0.9% with a minimum of 64 references).
A rule for the application needs that minimum or a better-calibrated mass. Neither has
been chosen, and no rule has been tested inside the application.
been chosen, and no rule has been tested inside the application. Experiment 10 adds that
the reference tree matters as much as the placement: most fine-depth errors fall on
branches the reference tree does not support, so showing a verdict only on supported
branches of the reference is a third option, also untested.
- **Counting reads does not average this error away.** Each query in experiments 7 and 8 is
one sequence. Reads from the same organism share that sequence, so if it is misplaced they
are all misplaced together. The bound on wrongly counted reads that the counting rule
Expand Down Expand Up @@ -394,7 +430,10 @@ reference tree is right at the finest depth.
200 references and 200 queries, one draw. Experiment 9 shares every limit of experiment 8
except size: 400 references, two draws, 600 queries. The second draw's tree was built
without bootstrap after the bootstrapped run exceeded 25 minutes; the bootstrap supports of
the first tree were saved and not analysed. Experiments 7 and 8 differ in the number of
the first tree are analysed in experiment 10. Experiment 10 uses that one tree and 230
queries. The support of one branch is a lower bound on the tree's confidence that a query
lies outside a clade, because the branch can fail in a bootstrap replicate for reasons
that do not involve the query. Experiments 7 and 8 differ in the number of
references, the tree method, the source of the mass and the thresholds, so the difference
between them is not the effect of the placement method alone. A cheaper agent ran the
experiment; the tables were recomputed separately from its per-clade output.
Expand Down Expand Up @@ -504,6 +543,12 @@ julia "$WORK/scripts/06_tables.jl" new "$WORK/raw"
julia "$WORK/scripts/08_crosscheck.jl" "$WORK/raw"
```

Experiment 10, on the first draw:

```bash
julia "$WORK/scripts/10_support.jl" "$WORK/A"
```

The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its
own copy of the shared code because it is the script as it was run; the later scripts include
`lib_tree.jl`.
104 changes: 104 additions & 0 deletions docs/cladistics/issue-303/epa-depth/10_support.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# SPDX-License-Identifier: MPL-2.0
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
# Experiment E10 (issue 303): are the wrong entailed verdicts of experiment 9 errors of the
# placement, or uncertainty in the reference tree?
#
# usage: julia 10_support.jl DRAW_DIR (DRAW_DIR = the experiment 9 directory of a draw whose
# tree was built with UFBoot, i.e. draw A)
#
# The truth in experiment 9 is where the full-length sequence q sits in the reference tree built
# WITH q. That tree carries ultrafast-bootstrap support on its branches. For a clade S of the
# pruned tree, the verdict "q is in S" is judged against one branch of the full tree:
#
# truth says q IS in S <=> the full tree has the clade S + q; its support backs the truth.
# truth says q is NOT in S <=> the full tree has the clade S (without q); its support backs it.
#
# So every reported clade gets a "truth support". A wrong verdict against a weakly supported
# branch may be the reference tree's uncertainty, not a placement error. A wrong verdict against
# a branch with support >= 95 is a placement error by the reference tree's own standard.
using Printf, Statistics
include(joinpath(@__DIR__, "lib_nwk.jl"))
D = ARGS[1]; W = joinpath(D, "work")
names = [String(split(l, "\t")[1]) for l in eachline(joinpath(W, "names.tsv"))]; N = length(names)
leafidx = Dict(n => i for (i, n) in enumerate(names)); ALL = BitSet(1:N)

# ---- full tree with support labels: every internal branch -> (leaf set below it, support) ----
function supports(s::AbstractString)
out = Dict{BitSet,Float64}(); pos = Ref(1)
function rd()
if s[pos[]] == '('
pos[] += 1; set = BitSet()
while true
union!(set, rd()); c = s[pos[]]; pos[] += 1
c == ',' && continue; c == ')' && break; error("bad newick at $(pos[])")
end
st = pos[]; while s[pos[]] ∉ (':', ',', ')', ';'); pos[] += 1; end
lbl = s[st:pos[]-1]
if s[pos[]] == ':'; pos[] += 1; while s[pos[]] ∉ (',', ')', ';'); pos[] += 1; end; end
isempty(lbl) || (out[canon(set)] = parse(Float64, split(lbl, "/")[end]))
return set
else
st = pos[]; while s[pos[]] ∉ (':', ',', ')', ';'); pos[] += 1; end
nm = s[st:pos[]-1]
if s[pos[]] == ':'; pos[] += 1; while s[pos[]] ∉ (',', ')', ';'); pos[] += 1; end; end
return BitSet([leafidx[nm]])
end
end
rd(); out
end
# a branch is a split of the leaves; name it by the side that does not hold leaf 1
canon(set::BitSet) = 1 in set ? setdiff(ALL, set) : set
SUP = supports(strip(read(joinpath(W, "tree", "reference_with_support.treefile"), String)))
trivial(set) = (c = canon(set); length(c) <= 1 || length(c) >= N - 1)
"support of the full-tree branch with this leaf set on one side; 100 for a pendant branch; NaN if absent"
support(set::BitSet) = trivial(set) ? 100.0 : get(SUP, canon(set), NaN)
@printf("full tree: %d leaves, %d internal branches with a support value (median %.0f, %d below 95, %d below 70)\n",
N, length(SUP), median(values(SUP)), count(<(95), values(SUP)), count(<(70), values(SUP)))

# ---- per query: the reported clade under the rule "smallest entailed clade of >= k references" ----
qs = [String.(split(l, "\t")) for l in eachline(joinpath(W, "queries.tsv"))][2:end]
struct Rep; q::Int; k::Int; size::Int; wrong::Bool; sup::Float64; end
reps = Rep[]; absent = 0
for q in qs
qi = parse(Int, q[1]); qn = q[2]; dir = joinpath(W, "q", @sprintf("q%03d", qi))
jp = joinpath(dir, "amp", "RAxML_portableTree.epa.jplace"); isfile(jp) || continue
g = parse_nwk(strip(read(joinpath(dir, "pruned.nwk"), String)))
truth = BitSet(leafidx[n] for n in split(strip(read(joinpath(dir, "truth.txt"), String)), "\n"))
refs = sort([n for n in names if n != qn])
nid = Dict(g.names[u] => u for u in eachindex(g.names) if g.names[u] != "")
par, kids, below, order = root_at(g, nid[refs[1]], leafidx)
tree, fields, rows = parse_jplace(jp); jg = parse_nwk(tree)
jn = Dict(jg.names[u] => u for u in eachindex(jg.names) if jg.names[u] != "")
jpar, jkids, jbelow, jorder = root_at(jg, jn[refs[1]], leafidx)
jkey = Dict{Int,BitSet}(); for u in jorder; u == jn[refs[1]] && continue; jkey[jg.lab[u][jpar[u]]] = jbelow[u]; end
ei = findfirst(==("edge_num"), fields); li = findfirst(==("like_weight_ratio"), fields)
pls = [(jkey[Int(r[ei])], r[li]) for r in rows]
clades = [below[u] for u in order if u != nid[refs[1]] && !isleaf(g, u)]
mass(S) = sum((w for (b, w) in pls if b != S && issubset(b, S)); init = 0.0)
ent = [S for S in clades if mass(S) >= 0.99 - 1e-12]
qix = leafidx[qn]
for k in (2, 8, 32)
c = [S for S in ent if length(S) >= k]; isempty(c) && continue
S = c[argmin(length.(c))]
inside = truth != S && issubset(truth, S)
# the full-tree branch that backs the truth for this clade
s = inside ? support(union(S, BitSet([qix]))) : support(S)
isnan(s) && (global absent += 1)
push!(reps, Rep(qi, k, length(S), !inside, s))
end
end
println("reported clades whose truth branch was not found in the full tree: $absent")

for k in (2, 8, 32)
R = [r for r in reps if r.k == k && !isnan(r.sup)]
println("\nrule: smallest entailed clade of at least $k references ($(length(R)) queries with a clade)")
@printf(" %-28s %8s %8s %10s\n", "truth branch support", "queries", "wrong", "rate")
for (lab, f) in (("100 (or pendant)", r -> r.sup >= 100), ("95 to 99", r -> 95 <= r.sup < 100),
("70 to 94", r -> 70 <= r.sup < 95), ("below 70", r -> r.sup < 70),
("95 or more (firm)", r -> r.sup >= 95), ("below 95 (not firm)", r -> r.sup < 95))
X = filter(f, R); w = count(r -> r.wrong, X)
@printf(" %-28s %8d %8d %9s\n", lab, length(X), w, isempty(X) ? "n/a" : @sprintf("%.1f%%", 100w / length(X)))
end
Wq = [r for r in R if r.wrong]
println(" wrong queries: clade size / truth support: ", join(sort(["$(r.size)/$(round(Int, r.sup))" for r in Wq]), " "))
end
Loading