Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions docs/cladistics/issue-303/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
<!-- SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> -->

# Issue 303 experiments: exact Phred arithmetic and the lazy zoomable tree

Fork only. Nothing here is proposed upstream. These are the measurements behind the
second-opinion work that [issue 303](https://github.com/hyperpolymath/MetaManifold-WebUI/issues/303)
asks for before any algorithm change. They are experiments, not a method and not a
complexity claim. Run on 2026-11-16 and 2026-11-17 with Julia 1.12.7 on four threads.

Agents wrote the scripts and this page. The person answering for them is
Jonathan D. A. Jewell (The Open University).

## What was asked

1. Does Phred-scored uncertainty force the exact-arithmetic work to move from the rationals
to the reals?
2. Can tree building go from cubic towards linear by treating coarse views as "thumbnails"
and zooming in only where needed?

## What was found, in one paragraph

Phred arithmetic can be exact without the reals, and on real reads it changes no filter
decision. Asking for distances only when needed recovers an exactly tree-like input from
about n log n distances instead of n². On real 240-base amplicon reads, however, the full
neighbour-joining tree is not a stable object: a tree built on 768 of 1,024 sequences agrees
with the 1,024-sequence tree on only 78% of that tree's well-supported branches. Lazy views
cannot be validated against a target that disagrees with itself, and no thumbnail summary
tested reached 60%. An earlier reading of experiment 4, that zoomed-in views keep over 90%
of well-supported branches, did not survive at 1,024 sequences (66%).

## Data

Experiments 4 to 6 use the forward reads of the MiSeq_SOP dataset, the same study the
integration test uses. The archive is not committed.

```bash
curl -L -o miseqsopdata.zip https://mothur.s3.us-east-2.amazonaws.com/wiki/miseqsopdata.zip
```

```bash
sha256sum miseqsopdata.zip
```

Expected digest: `728f7544824f8ca0223618b18bb60fce6a5dc20ae3d476267dbd5cf5594a35d4`
(downloaded 2026-11-17). Unzip it and pass the `MiSeq_SOP` directory to the scripts.

Reads are truncated to 240 bases, dropped if their expected errors exceed 2 or they contain
`N`, then dereplicated. They are **not** DADA2 output. Where a script says "crude denoise" it
drops every sequence within one edit of a more abundant one, and every sequence seen once.

## Experiments

"Agreement" is always: of the branches of the larger tree, cut down to the leaves of a view,
what share also appear in the tree built on the view alone. "Supported" means the branch is at
least three substitutions long. There is no known true tree for real reads, so real-data
experiments measure agreement, never accuracy.

### 1. Exact Phred expected errors (`phred_exact.jl`)

Every term `10^(-Q/10)` is a rational times a power of θ = `10^(-1/10)`. θ is a root of
`10x¹⁰ − 1`, which is irreducible, so an expected-error sum is one element of ℚ(θ): ten
rational coefficients. Comparison with a rational threshold is exact by bisection on θ.

| Check | Result |
| --- | --- |
| Exact sign against 512-bit arithmetic, 60,000 comparisons on random reads | 0 disagreements |
| Twenty bases at Q10 (sum exactly 2), threshold 2, double precision | reports 2.0000000000000004 > 2: wrong |
| 2,000 reads constructed to sum to exactly 2 | double precision wrong on all 2,000 |
| Random reads, double precision against exact | 0 disagreements |

### 2. Distances on demand, exact tree-like input (`zoom_tree.jl`)

Leaves are inserted one at a time; each is placed by asking at most three distances per level
of a centroid search. Input is an additive tree metric with integer edge lengths.

| Leaves | Distances asked | Share of the full matrix | Asked / (n log₂ n) |
| --- | --- | --- | --- |
| 1,024 | 10,210 to 15,199 | 1.9% to 2.9% | 1.0 to 1.5 |
| 4,096 | 53,608 to 79,971 | 0.6% to 1.0% | 1.1 to 1.6 |
| 16,384 | 266,642 to 394,964 | 0.2% to 0.3% | 1.2 to 1.7 |

The lower figure is a random-split tree, the higher a caterpillar. Every reconstruction matched
the true tree in topology and edge lengths, and canonical neighbour joining agreed on 160 small
cases. n log n is the known lower bound for this query model, so this is not O(n). The
prototype's running time is quadratic in bookkeeping; only the distance count is measured.

### 3. Lazy views, simulated noisy sequences (`zoom_noisy.jl`)

512 leaves, Jukes-Cantor evolution on random trees, views of 64 leaves, 5 trees per row.

| Sites | View | Agreement, all branches |
| --- | --- | --- |
| 250 | zoomed-in neighbourhood | 93.8% |
| 250 | thumbnail (spread-out leaves) | 86.2% |
| 250 | random subset | 92.5% |
| 1,000 | zoomed-in / thumbnail | 99.3% / 97.4% |
| 5,000 | all views | 99.7% or better |

On exact tree-like distances every figure is 100%. The simulated data sat 9 to 175 times
outside the Atteson radius even where neighbour joining recovered every branch, so that
radius does not explain when lazy views agree.

### 4. Lazy views, real reads (`zoom_real.jl`, `zoom_real_1024.jl`)

Part A, the Phred filter on 152,360 real reads: double precision and the exact method agree
on every read. No read sums exactly to the threshold; the closest is 2.2e-6 away. For 98.1% of
reads the double-precision sum depends on the order of addition. The exact method (integer
counts per quality value, exact comparison) took 6.2 times as long as double precision.

Part B, views of 64 leaves, 40 views per row:

| Sequences | View | All branches | Supported branches (middle 95% of views) |
| --- | --- | --- | --- |
| 512, error variants kept | zoomed-in | 60.7% | 91.2% (71.4 to 100) |
| 512, error variants kept | random subset | 60.8% | 81.4% (70.8 to 95.8) |
| 512, error variants kept | thumbnail | 43.1% | 50.9% (39.3 to 61.3) |
| 384, crude denoise | zoomed-in | 53.6% | 94.3% (77.7 to 100) |
| 384, crude denoise | random subset | 55.4% | 82.1% (71.1 to 95.8) |
| 384, crude denoise | thumbnail | 50.4% | 69.8% (56.6 to 89.0) |
| 1,024, crude denoise | zoomed-in | 35.3% | 65.8% (0 to 100) |
| 1,024, crude denoise | random subset | 43.3% | 60.2% (48.1 to 80.7) |

Agreement with leaf labels shuffled is 0% throughout. The 1,024 row is why the zoomed-in
figures above it should not be quoted alone.

### 5. Can the thumbnail be fixed by a better summary? (`thumb_real.jl`)

1,024 sequences, 64 regions around random representatives, 30 draws. Target: the full tree cut
down to the representatives. Share of supported branches reproduced:

| Summary of each region | Supported branches | Pairwise alignments |
| --- | --- | --- |
| one representative | 55.9% | 2,016 |
| mean over 4 members, corrected | 44.6% | 28,441 |
| mean over 8 members, corrected | 46.8% | 89,253 |
| mean over all members, corrected | 46.5% | 523,776 (all) |
| 5-mer counts of the representative | 55.4% | none |
| mean 5-mer counts of the region | 51.0% | none |

Spread-out representatives and 32 regions give the same picture; no summary exceeds 56%.
Alignment-free 5-mer distances match aligned distances for this purpose at no alignment cost.

### 6. Is the target stable? (`ceiling_real.jl`)

The tree on all 1,024 sequences compared with a tree built on a random subset of them, both cut
down to the same 64 leaves, 30 draws per row.

| Tree built on | All branches | Supported branches (middle 95%) |
| --- | --- | --- |
| 64 of 1,024 | 43.1% | 59.6% (44.0 to 73.7) |
| 128 | 43.8% | 59.6% (37.7 to 74.9) |
| 256 | 47.6% | 65.9% (47.7 to 81.3) |
| 512 | 51.7% | 70.4% (55.1 to 89.2) |
| 768 | 62.1% | 78.4% (59.3 to 90.4) |

## Reading of the results

- **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic
number is enough. On this dataset exact arithmetic gives the same decisions as double
precision. What it adds is order-independence and a provable comparison, not a different
answer.
- **Linear time.** On tree-like input the distance count falls from n² to about n log n, which
is optimal for distance queries. Nothing here supports O(n) for de novo tree building, and no
use for the Veblen hierarchy in the algorithm was found.
- **Lazy views on amplicon reads.** Experiment 6 is the limiting result. At 240 bases the
neighbour-joining tree changes, including in branches it reports as well supported, when
sequences are added. Agreement with that tree is therefore the wrong contract, and experiment
5's failure is not a failure of the summaries. A stable overview has to come from outside the
amplicon data, which is what a reference tree with placement provides.

## Limits

- One dataset, one marker region, one read length; dereplicated reads, not denoised variants.
- Neighbour joining only, with Jukes-Cantor distances in double precision and first-minimum tie
breaking. Other distance corrections, tie policies and tree methods were not tried.
- "Supported" is a branch-length threshold, not a bootstrap or posterior support.
- Samples are small (30 to 40 views per row) and the ranges are wide.
- Experiments 2 and 3 use simulated input. Experiment 2 measures distance counts, not time.
- Not done: any proof, any comparison with RapidNJ, SwiftNJ or FastTree, any test of placement
on a reference tree, and any change to the application.

## Running

```bash
julia docs/cladistics/issue-303/phred_exact.jl
```

```bash
julia docs/cladistics/issue-303/zoom_tree.jl
```

```bash
julia -O2 docs/cladistics/issue-303/zoom_noisy.jl
```

```bash
julia -O2 -t 4 docs/cladistics/issue-303/zoom_real.jl /path/to/MiSeq_SOP
```

```bash
julia -O2 -t 4 docs/cladistics/issue-303/zoom_real_1024.jl /path/to/MiSeq_SOP
```

```bash
julia -O2 -t 4 docs/cladistics/issue-303/thumb_real.jl /path/to/MiSeq_SOP
```

```bash
julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP
```

The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its
own copy of the shared code because it is the script as it was run; the later scripts include
`lib_tree.jl`.
37 changes: 37 additions & 0 deletions docs/cladistics/issue-303/ceiling_real.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# SPDX-License-Identifier: MPL-2.0
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
# Experiment E6: is the TARGET itself stable? Real MiSeq_SOP reads.
#
# usage: julia -t 4 ceiling_real.jl <directory holding *_R1_001.fastq>
#
# E5 found no region summary reproduces more than about 56% of the well-supported branches of
# the full tree cut down to 64 representatives. Before blaming the summaries, ask how well the
# full tree agrees with ITSELF when the set of other sequences changes: build the tree on all n
# sequences and on a random subset T of them, cut both down to the same 64 leaves S (S inside T),
# and compare. If two "full" trees disagree about the same 64 leaves, no thumbnail can match both.
include(joinpath(@__DIR__, "lib_tree.jl"))

function run(dir; nmax = 1024, k = 64, draws = 30)
rng = MersenneTwister(20261117); seqs = load_set(dir, nmax); n = length(seqs)
D, _ = distance_matrix(seqs); key = rand(rng, UInt64, n); one = 1 / TRUNC
full, _, fblen = nj(D, collect(1:n))
@printf("n = %d sequences; views of %d leaves; %d draws per row\n", n, k, draws)
@printf(" %-34s %7s %9s %24s\n", "tree compared with the n-leaf tree", "all", "weighted", ">=3 substitutions")
for tsize in (k, 128, 256, 512, 768)
rows = NTuple{3,Float64}[]
for _ in 1:draws
T = sort(randperm(rng, n)[1:tsize]); S = sort(T[randperm(rng, tsize)[1:k]])
inS = fill(false, n); inS[S] .= true; target = splits(full, n, key, inS, fblen)
sub, m, sblen = nj(D, T); inST = [t in S for t in T] |> collect
a = splits(sub, m, key[T], Vector{Bool}(inST), sblen)
tot = length(target); sh = count(x -> haskey(a, x), keys(target))
w = sum(values(target)); wsh = sum((v for (x, v) in target if haskey(a, x)); init = 0.0)
t3 = [x for (x, v) in target if v >= 3one]
push!(rows, (sh / max(tot, 1), wsh / max(w, eps()), isempty(t3) ? NaN : count(x -> haskey(a, x), t3) / length(t3)))
end
c3 = filter(!isnan, [r[3] for r in rows])
@printf(" tree built on %4d of the %d %6.1f%% %8.1f%% %7.1f%% (%5.1f .. %5.1f)\n", tsize, n,
100mean(r[1] for r in rows), 100mean(r[2] for r in rows), 100mean(c3), 100quantile(c3, 0.025), 100quantile(c3, 0.975))
end
end
run(ARGS[1])
133 changes: 133 additions & 0 deletions docs/cladistics/issue-303/lib_tree.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# SPDX-License-Identifier: MPL-2.0
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
# Shared code for the issue-303 experiments: FASTQ reading, exact Phred expected errors, edit distance,
# canonical neighbour joining with branch lengths, and tree splits restricted to a leaf subset.
using Random, Printf, Statistics
const TRUNC = 240
const MAXEE = 2

# ---------- FASTQ ----------
function read_fastq(path)
seqs = String[]; quals = String[]
open(path) do io
while !eof(io)
readline(io); s = readline(io); readline(io); q = readline(io)
push!(seqs, s); push!(quals, q)
end
end
seqs, quals
end

# ---------- Part A: exact expected errors ----------
# EE = sum 10^(-Q/10). With Q = 10a + b the term is 10^(-a) * theta^b, theta = 10^(-1/10).
# Keep ten integer coefficients over the common denominator 10^9 (Q <= 99): pure integer counting.
const POW10 = [big(10)^k for k in 0:9]
function ee_coeffs(q::AbstractVector{UInt8}) # q are Phred values, not ASCII
c = zeros(Int64, 10)
@inbounds for Q in q
a, b = divrem(Int(Q), 10); c[b+1] += 10^(9 - a)
end
c
end
"sign of EE - T for integer threshold T, exact (bisection on theta; P is monotone in theta)."
function cmp_ee(c::Vector{Int64}, T::Int)
p = Rational{BigInt}.(c) .// big(10)^9; p[1] -= T
all(iszero, p) && return 0
all(iszero, @view p[2:end]) && return Int(sign(p[1]))
lo, hi = big(0) // 1, big(1) // 1
while true
slo = sign(foldr((ck, acc) -> ck + lo * acc, p)); shi = sign(foldr((ck, acc) -> ck + hi * acc, p))
(slo == shi && slo != 0) && return Int(slo)
mid = (lo + hi) / 2
if 10 * mid^10 < 1; lo = mid else hi = mid end
end
end
ee_float(q) = sum(10.0^(-Int(Q) / 10) for Q in q)

# ---------- distances ----------
function edit_distance(a::Vector{UInt8}, b::Vector{UInt8}, prev::Vector{Int32}, cur::Vector{Int32})
n = length(a); m = length(b)
@inbounds for j in 0:m; prev[j+1] = j; end
@inbounds for i in 1:n
cur[1] = i; ai = a[i]
for j in 1:m
sub = prev[j] + (ai != b[j]); del = prev[j+1] + 1; ins = cur[j] + 1
cur[j+1] = min(sub, del, ins)
end
prev, cur = cur, prev
end
Int(prev[m+1])
end
function distance_matrix(seqs::Vector{Vector{UInt8}})
n = length(seqs); D = zeros(n, n); E = zeros(Int, n, n)
Threads.@threads for i in 1:n
prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1)
for j in i+1:n
e = edit_distance(seqs[i], seqs[j], prev, cur); p = e / TRUNC
d = p >= 0.74 ? 5.0 : -0.75 * log(1 - 4p / 3)
D[i, j] = d; D[j, i] = d; E[i, j] = e; E[j, i] = e
end
end
D, E
end

# ---------- canonical neighbour joining (same code as the simulation, E3) ----------
function nj(D::Matrix{Float64}, idx::Vector{Int})
m = length(idx); d = D[idx, idx]; N = 2m - 2
d = vcat(hcat(d, zeros(m, N - m)), zeros(N - m, N)); children = fill((0, 0), m); active = collect(1:m); blen = zeros(N)
tsum = zeros(N); for i in 1:m; tsum[i] = sum(@view d[i, 1:m]); end
nxt = m
while length(active) > 2
r = length(active); bi = 0; bj = 0; bq = Inf
@inbounds for x in 1:r, y in x+1:r
i = active[x]; j = active[y]; q = (r - 2) * d[i, j] - tsum[i] - tsum[j]
if q < bq; bq = q; bi = i; bj = j; end
end
nxt += 1; u = nxt; push!(children, (bi, bj)); s = 0.0
li = d[bi, bj] / 2 + (tsum[bi] - tsum[bj]) / (2 * (r - 2)); blen[bi] = max(li, 0.0); blen[bj] = max(d[bi, bj] - li, 0.0)
@inbounds for k in active
(k == bi || k == bj) && continue
v = (d[bi, k] + d[bj, k] - d[bi, bj]) / 2; d[u, k] = v; d[k, u] = v; s += v
tsum[k] += v - d[bi, k] - d[bj, k]
end
tsum[u] = s; filter!(k -> k != bi && k != bj, active); push!(active, u)
end
push!(children, (active[1], active[2])); blen[active[1]] = max(d[active[1], active[2]], 0.0)
children, m, blen
end
function splits(children, nleaves, key::Vector{UInt64}, inS::Vector{Bool}, blen::Vector{Float64})
N = length(children); h = zeros(UInt64, N); c = zeros(Int, N)
for u in 1:nleaves; inS[u] && (h[u] = key[u]; c[u] = 1); end
for u in nleaves+1:N; (a, b) = children[u]; h[u] = h[a] ⊻ h[b]; c[u] = c[a] + c[b]; end
tot = c[N]; all = h[N]; out = Dict{UInt64,Float64}() # split => summed length of the edges that induce it
for u in 1:N-1; if 2 <= c[u] <= tot - 2; k = min(h[u], all ⊻ h[u]); out[k] = get(out, k, 0.0) + blen[u]; end; end
out
end


# ---------- real reads: filter, dereplicate, crude denoise ----------
"Distinct forward reads (truncated, EE-filtered, no N), most abundant first, dropping anything
within one edit of a kept sequence and anything seen once. At most nmax sequences."
function load_set(dir, nmax)
files = sort(filter(f -> endswith(f, "_R1_001.fastq"), readdir(dir; join = true)))
counts = Dict{String,Int}()
for f in files
seqs, quals = read_fastq(f)
for (s, qs) in zip(seqs, quals)
length(s) < TRUNC && continue
q = UInt8[UInt8(c) - 0x21 for c in codeunits(qs)[1:TRUNC]]
ee_float(q) > MAXEE && continue # E4: identical to the exact filter on every read here
st = s[1:TRUNC]; occursin('N', st) && continue
counts[st] = get(counts, st, 0) + 1
end
end
uniq = sort(collect(counts); by = x -> (-x[2], x[1]))
kept = Vector{Vector{UInt8}}(); prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1)
for x in uniq # crude denoise, as E4 set 2
x[2] < 2 && break
s = Vector{UInt8}(codeunits(x[1]))
any(t -> edit_distance(s, t, prev, cur) <= 1, kept) || push!(kept, s)
length(kept) >= nmax && break
end
kept
end
Loading
Loading