Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
244 changes: 235 additions & 9 deletions docs/cladistics/issue-303/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@ tested reached 60%. An earlier reading of experiment 4, that zoomed-in views kee
of well-supported branches, did not survive at 1,024 sequences (66%). Placing an amplicon-length
fragment on a reference tree built from full-length sequences does work: clade verdicts from
bootstrapped placements were right for 1,105 of 1,106 entailed and 57,116 of 57,128 refuted
verdicts, and pin a read to a clade of median size 18 among 399 references.
verdicts, and pin a read to a clade of median size 18 among 399 references. With the
application's own placement step (RAxML EPA) the picture is sharper and less safe: 2,375 of
2,388 entailed verdicts were right, with no error among 1,515 verdicts on clades of 50 or more
references and 7 errors among 62 on clades of two or three. On a larger sample the smallest
confident clade was wrong for 32 of 448 queries (7.1%), falling to 4 of 448 (0.9%) when the
clade had to hold at least 64 of 399 references.

## Data

Expand Down Expand Up @@ -203,15 +208,154 @@ control shows the score can fail. What it does not show is in the limits below.
The counting rule that turns per-read verdicts into a per-sample verdict for a clade, with
separate bounds for wrongly counted and missed reads, is proved in
`metadatastician/MetaManifold-Evidence` (`agda/src/MetaManifold/Evidence/TwoSided.agda`,
not yet merged there). It also proves that an entailed verdict survives zooming out to any
enclosing clade and a refuted verdict survives zooming in.
not yet merged there). It also proves two zoom laws, each under a premise on the bounds. An
entailed verdict survives zooming out to an enclosing clade if the same bound on wrongly
counted reads holds for the larger clade. A refuted verdict survives zooming in to a
contained clade if that clade's bound on missed reads is no larger. An earlier version of
this page stated both without their premises; containment alone is not enough for either.

### 8. The application's own placement step (`epa/`)

Experiment 7 placed by least squares. This repeats it with the command the application runs
in its `place` step: `raxmlHPC-PTHREADS-SSE3 -f v -m GTRCATI -G 0.2` (RAxML 8.2.12). 200
full-length sequences from the same SILVA alignment give a maximum-likelihood reference tree
(IQ-TREE 2.0.7, GTR+G4). Each of the 200 in turn is pruned from the tree and its 240-base V4
fragment is placed back. For every clade of the pruned tree, the mass is the sum of the
likelihood weight ratios of the placements inside it, as RAxML reports them. The verdict is
entailed at mass 0.99 or more, refuted at 0.01 or less, unresolved between.

| | Amplicon fragment | Whole sequence (positive control) | A different sequence placed (negative control) |
| --- | --- | --- | --- |
| Placed on exactly the true edge | 91 of 200 (45.5%) | 188 of 200 (94.0%) | 0 of 200 |
| Within two edges of it | 76.5% | 99.5% | 1.0% |
| Entailed verdicts that are right | 2,375 of 2,388 (99.46%) | 3,065 of 3,065 | 1,507 of 3,176 (47.45%) |
| Refuted verdicts that are right | 35,988 of 36,016 (99.92%) | 36,208 of 36,211 (99.99%) | 34,392 of 35,952 (95.66%) |
| Clades that truly hold the sequence: entailed / unresolved / wrongly refuted | 75.2% / 23.9% / 0.9% | 97.0% / 2.9% / 0.1% | 47.7% / 2.9% / 49.4% |
| Smallest entailed clade, median size of 199 | 4 | 4 | 3 |
| Smallest entailed clade holds the truth | 139 of 146 queries (95.2%) | 193 of 193 | 9 of 189 (4.8%) |
| Queries with no entailed clade | 54 | 7 | 11 |

In this negative control the other sequence stays in the tree, so its fragment is placed
beside it; experiment 7 left it out. The two negative columns are not comparable.

The errors of the amplicon fragment depend on how small the clade is:

| Clade size (of 199 references) | Entailed verdicts | Wrong |
| --- | --- | --- |
| 2 to 3 | 62 | 7 |
| 4 to 7 | 132 | 1 |
| 8 to 15 | 228 | 1 |
| 16 to 49 | 451 | 4 |
| 50 to 199 | 1,515 | 0 |

Verdicts on nested clades of one query are not independent, so the per-query count is the
honest one. Reporting the smallest entailed clade, 7 of 146 queries get a wrong clade.
Reporting the smallest entailed clade of at least 4, 8 or 16 references, 4, 3 or 2 of the
146 do, at median sizes 6.5, 13 and 23.

RAxML reports few placements per query: a median of 3, at most 7, and a single one for 60 of
the 200. Four of the seven wrong queries had a single placement with weight 0.99 or more. The
mass is therefore lumpy. Clades given mass in (0, 0.05] hold the truth for 9 of 52,
(0.05, 0.25] 22 of 96, (0.25, 0.5] 24 of 46, (0.5, 0.75] 53 of 98, (0.75, 0.95] 280 of 324
and (0.95, 1] 2,745 of 2,773.

With the looser rule (entailed at 0.95 or more, refuted at 0.05 or less) 2,745 of 2,773
entailed verdicts are right (98.99%), and the smallest entailed clade is wrong for 17 of 172
queries.

Three self-checks passed. An independent calculation of the truth from path distances on the
unpruned tree matched in all 10 tests (9 distinct queries). In all 600 runs the 395 edge
numbers of the placement file map to 395 distinct branches of the pruned tree. Clade mass
never decreased
from a clade to the clade containing it (39,200 pairs per column). `08_by_depth.jl`
recomputes the tables from the per-clade output separately from `06_tables.jl`; the two
agree.

What this shows: with the application's placement step, "present in this clade" was right
in every case seen for clades of a quarter of the tree or more, and wrong about one time in
nine for clades of two or three references. At that depth the verdict is confidently wrong,
not unresolved. Experiment 9 repeats this on a larger sample and finds errors on large clades
too. What it does not show is in the limits below.

### 9. Error by clade depth (`epa-depth/`)

The same test as experiment 8 on a larger sample, to put a number on how the error depends
on clade size. Two independent draws of 400 references (13 sequences are in both), a
maximum-likelihood tree for each, and 300 amplicon queries per draw: 600 queries. As a
control, the whole sequence of 60 queries was placed: 60 of 60 landed on the exact edge and
none of their 1,134 entailed verdicts was wrong.

For the amplicon fragment, 9,555 of 9,613 entailed verdicts were right. The 58 wrong ones by
clade size:

| Clade size (of 399 references) | Entailed verdicts | Wrong | Queries with a wrong verdict |
| --- | --- | --- | --- |
| 2 to 3 | 215 | 15 | 13 |
| 4 to 7 | 417 | 12 | 11 |
| 8 to 15 | 507 | 11 | 9 |
| 16 to 31 | 748 | 7 | 6 |
| 32 to 63 | 999 | 6 | 2 |
| 64 to 127 | 431 | 2 | 1 |
| 128 to 399 | 6,296 | 5 | 3 |

The five errors in the last row are on clades holding 357 to 372 of the 399 references. The
branch there cuts off a small group, and the sequence truly sits in that small group. Binned
by the smaller side of the branch instead, the wrong count falls from 15 of 444 at two or
three references to 2 of 1,271 at 64 to 127, and 0 of 1,119 at 128 or more.

One result per query is the honest unit. The rule "report the smallest entailed clade with
at least k references" gave:

| k | Queries given a clade (of 600) | Median clade size | Wrong | Rate (exact 95% interval) |
| --- | --- | --- | --- | --- |
| 2 | 448 | 4 | 32 | 7.1% (4.9 to 9.9) |
| 4 | 448 | 7 | 25 | 5.6% (3.6 to 8.1) |
| 8 | 448 | 14 | 17 | 3.8% (2.2 to 6.0) |
| 16 | 448 | 26 | 11 | 2.5% (1.2 to 4.4) |
| 32 | 448 | 46.5 | 6 | 1.3% (0.5 to 2.9) |
| 64 | 448 | 173 | 4 | 0.9% (0.2 to 2.3) |

The other 152 queries had no entailed clade. The two draws agree within their intervals:
19 of 230 and 13 of 218 at k = 2. Of the 32 queries wrong at k = 2, 18 had a single reported
placement and 18 had a best weight of 0.99 or more. With the looser rule (0.95) the rates
are higher: 61 of 540 at k = 2 and 10 of 540 at k = 64.

The intervals are Clopper-Pearson and treat queries as independent. The queries of one draw
share one reference tree, so the intervals cover the sampling of queries on these two trees,
not the variation between trees.

Self-checks passed: the control above; an independent truth calculation matched for 10
queries in each draw; all 793 edge numbers of every placement file map to distinct branches
of the pruned tree; clade mass never decreased from a clade to the clade containing it; and
the new table code reproduces experiment 8's counts from experiment 8's output.
`epa-depth/08_crosscheck.jl` recomputes the tables separately from `06_tables.jl`; the two
agree.

What this shows: the error of a "present in this clade" verdict from an amplicon fragment
falls as the clade grows and does not reach zero. Requiring 64 references in the clade
brought it under 1% of queries, at the price of a median clade of 173 of 399. The whole
sequence, placed the same way, was never wrong in 60 queries. The truth was itself built
from the whole sequence, so that control shows the machinery is sound; it does not show the
reference tree is right at the finest depth.

## Reading of the results

- **Evidence on placements.** Experiment 7 is the one positive real-data result. Verdicts by
clade are sound at the rates above, and they give the "thumbnail then zoom" idea a form
that holds: the verdict is decided at a coarse clade and becomes undecided, not wrong, as
the clade shrinks.
- **Evidence on placements.** Experiments 7 and 8 are the positive real-data results. Verdicts
by clade are sound at the rates above for coarse clades, and they give the "thumbnail then
zoom" idea a form that holds at the coarse end: the verdict is decided at a coarse clade.
How it fails as the clade shrinks depends on where the mass comes from. With bootstrapped
least squares (experiment 7) it became undecided. With the likelihood weights of the
application's placement step (experiment 8) it became wrong for 7 of 62 verdicts on the
smallest clades. Experiment 9 puts numbers on it: a minimum clade size trades error for
resolution (7.1% of queries wrong with no minimum, 0.9% with a minimum of 64 references).
A rule for the application needs that minimum or a better-calibrated mass. Neither has
been chosen, and no rule has been tested inside the application.
- **Counting reads does not average this error away.** Each query in experiments 7 and 8 is
one sequence. Reads from the same organism share that sequence, so if it is misplaced they
are all misplaced together. The bound on wrongly counted reads that the counting rule
takes as input cannot be read off the number of reads; it has to come from the number of
distinct sequences and a per-depth error rate. Experiment 9 gives that rate for error-free
fragments on two trees; it has not been measured on real reads.
- **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic
number is enough. On this dataset exact arithmetic gives the same decisions as double
precision. What it adds is order-independence and a provable comparison, not a different
Expand Down Expand Up @@ -241,8 +385,22 @@ enclosing clade and a refuted verdict survives zooming in.
verdict in 1,106 means "entailed" is a measured rate, not a proof. Verdicts for nested clades of
one query are not independent: counted per query, the smallest entailed clade was wrong for
1 of 130 queries, and 20 further queries had no entailed clade at all.
- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with the application's
own placement step or with real reads, and any change to the application.
- Experiment 8: the truth is where the full-length sequence sits in a maximum-likelihood tree
that was not bootstrapped, so some errors on the smallest clades may be the reference tree's
own uncertainty; the two were not separated. The reference tree used a fixed model (GTR+G4)
with IQ-TREE 2.0.7, where the application selects a model, bootstraps and pins IQ-TREE 3.1.4.
The application's alignment and trimming steps (MAFFT, trimAl) were skipped: queries sit in
the columns of the SILVA alignment. Queries are error-free fragments of reference sequences.
200 references and 200 queries, one draw. Experiment 9 shares every limit of experiment 8
except size: 400 references, two draws, 600 queries. The second draw's tree was built
without bootstrap after the bootstrapped run exceeded 25 minutes; the bootstrap supports of
the first tree were saved and not analysed. Experiments 7 and 8 differ in the number of
references, the tree method, the source of the mass and the thresholds, so the difference
between them is not the effect of the placement method alone. A cheaper agent ran the
experiment; the tables were recomputed separately from its per-clade output.
- Not done: any comparison with RapidNJ, SwiftNJ or FastTree, any test with real reads, the
application's placement step with its speed heuristic turned off, and any change to the
application.

## Running

Expand Down Expand Up @@ -278,6 +436,74 @@ julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP
julia -O2 -t 4 docs/cladistics/issue-303/placement_evidence.jl /path/to/silva.bacteria.fasta
```

Experiment 8 needs `raxmlHPC-PTHREADS-SSE3` and `iqtree2` on the path. `$WORK` is a directory
of your choice, absolute or relative, and may contain spaces; the drivers resolve it to an
absolute path and stop with a non-zero status if a stage fails. Set it once in the shell, copy
`epa/` to `$WORK/scripts`, then run the steps in order:

```bash
WORK=/path/to/work
```

```bash
sh "$WORK/scripts/01_run.sh" "$WORK" /path/to/silva.bacteria.fasta
```

```bash
sh "$WORK/scripts/02_tree.sh" "$WORK"
```

```bash
julia "$WORK/scripts/03_prep.jl" "$WORK" 200 7
```

```bash
(set -e; for k in amp pos neg; do sh "$WORK/scripts/05_run_epa.sh" "$WORK" $k; julia "$WORK/scripts/04_score.jl" "$WORK" $k; done)
```

```bash
julia "$WORK/scripts/06_tables.jl" "$WORK" amp pos neg
```

```bash
julia "$WORK/scripts/08_by_depth.jl" "$WORK"
```

The check that selection stops when a row loses a V4 base to column filtering:

```bash
julia docs/cladistics/issue-303/epa/test_select.jl
```

The drivers take optional trailing size arguments for a quick trial; without them they run the
published sizes. `01_run.sh` takes the number of references (default 200), `02_tree.sh` the
thread count (default 4), and `03_prep.jl` the number of queries. For experiment 9,
`run_trees.sh` takes the references per draw (default 400), the bootstrap timeout in seconds
(default 1500) and the threads (default 4), and `run_rest.sh` takes the queries per draw
(default 300) and the whole-sequence controls (default 60).

Experiment 9: copy `epa/` and then `epa-depth/` over it to `$WORK/scripts`, then:

```bash
sh "$WORK/scripts/run_trees.sh" "$WORK" /path/to/silva.bacteria.fasta
```

```bash
sh "$WORK/scripts/run_rest.sh" "$WORK"
```

```bash
mkdir -p "$WORK/raw" && (set -e; for t in "A amp" "A pos" "B amp"; do set -- $t; julia "$WORK/scripts/04_score.jl" "$WORK/$1" $1 $2; done)
```

```bash
julia "$WORK/scripts/06_tables.jl" new "$WORK/raw"
```

```bash
julia "$WORK/scripts/08_crosscheck.jl" "$WORK/raw"
```

The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its
own copy of the shared code because it is the script as it was run; the later scripts include
`lib_tree.jl`.
28 changes: 28 additions & 0 deletions docs/cladistics/issue-303/epa-depth/02_tree.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/sh
# SPDX-License-Identifier: MPL-2.0
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
# Experiment E9 (issue 303): error of entailed clade verdicts by clade depth, with the
# application's placement step. See ../README.md, section 9. These files replace their
# namesakes from ../epa/; 01_select.jl, 07_selfcheck_truth.jl and lib_nwk.jl are used unchanged.
# usage: 02_tree.sh epa_dir draw [timeout_s] [threads]
# IQ-TREE GTR+G4 with UFBoot 1000 (default 4 threads); fallback without -B after timeout_s
# (default 1500 s = 25 min wall clock). Logs go to ../raw relative to epa_dir.
set -eu
mkdir -p "$1/work"
E=$(cd "$1" && pwd) || exit 1; DR=$2; TMO=${3:-1500}; NT=${4:-4}
T=$E/work/tree; mkdir -p "$T"; cd "$T"
RAW=$(cd "$E/.." && pwd)/raw
LOG=$RAW/tree_$DR.log
s=$(date +%s)
rc=0
timeout "$TMO" iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -B 1000 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.stdout" 2>&1 || rc=$?
if [ "$rc" -ne 0 ] || [ ! -s "$T/reference.treefile" ]; then
echo "draw $DR: -B 1000 run rc=$rc after $(( $(date +%s) - s )) s; RERUN WITHOUT -B" > "$LOG"
s=$(date +%s)
iqtree2 -s "$E/work/ref_full.fasta" -m GTR+G4 -nt "$NT" -seed 1 -pre "$T/reference" -redo > "$RAW/02_iqtree_$DR.nob.stdout" 2>&1
echo "draw $DR: rerun without -B took $(( $(date +%s) - s )) s" >> "$LOG"
else
echo "draw $DR: -B 1000 completed, $(( $(date +%s) - s )) s wall clock, rc=0" > "$LOG"
fi
[ -s "$T/reference.treefile" ] || { echo "02_tree.sh: no reference.treefile for draw $DR" >&2; exit 1; }
cp "$T/reference.treefile" "$T/reference_with_support.treefile"
Loading
Loading