diff --git a/docs/cladistics/issue-303/README.md b/docs/cladistics/issue-303/README.md new file mode 100644 index 00000000..5c2974de --- /dev/null +++ b/docs/cladistics/issue-303/README.md @@ -0,0 +1,215 @@ + + + +# Issue 303 experiments: exact Phred arithmetic and the lazy zoomable tree + +Fork only. Nothing here is proposed upstream. These are the measurements behind the +second-opinion work that [issue 303](https://github.com/hyperpolymath/MetaManifold-WebUI/issues/303) +asks for before any algorithm change. They are experiments, not a method and not a +complexity claim. Run on 2026-11-16 and 2026-11-17 with Julia 1.12.7 on four threads. + +Agents wrote the scripts and this page. The person answering for them is +Jonathan D. A. Jewell (The Open University). + +## What was asked + +1. Does Phred-scored uncertainty force the exact-arithmetic work to move from the rationals + to the reals? +2. Can tree building go from cubic towards linear by treating coarse views as "thumbnails" + and zooming in only where needed? + +## What was found, in one paragraph + +Phred arithmetic can be exact without the reals, and on real reads it changes no filter +decision. Asking for distances only when needed recovers an exactly tree-like input from +about n log n distances instead of n². On real 240-base amplicon reads, however, the full +neighbour-joining tree is not a stable object: a tree built on 768 of 1,024 sequences agrees +with the 1,024-sequence tree on only 78% of that tree's well-supported branches. Lazy views +cannot be validated against a target that disagrees with itself, and no thumbnail summary +tested reached 60%. An earlier reading of experiment 4, that zoomed-in views keep over 90% +of well-supported branches, did not survive at 1,024 sequences (66%). + +## Data + +Experiments 4 to 6 use the forward reads of the MiSeq_SOP dataset, the same study the +integration test uses. The archive is not committed. + +```bash +curl -L -o miseqsopdata.zip https://mothur.s3.us-east-2.amazonaws.com/wiki/miseqsopdata.zip +``` + +```bash +sha256sum miseqsopdata.zip +``` + +Expected digest: `728f7544824f8ca0223618b18bb60fce6a5dc20ae3d476267dbd5cf5594a35d4` +(downloaded 2026-11-17). Unzip it and pass the `MiSeq_SOP` directory to the scripts. + +Reads are truncated to 240 bases, dropped if their expected errors exceed 2 or they contain +`N`, then dereplicated. They are **not** DADA2 output. Where a script says "crude denoise" it +drops every sequence within one edit of a more abundant one, and every sequence seen once. + +## Experiments + +"Agreement" is always: of the branches of the larger tree, cut down to the leaves of a view, +what share also appear in the tree built on the view alone. "Supported" means the branch is at +least three substitutions long. There is no known true tree for real reads, so real-data +experiments measure agreement, never accuracy. + +### 1. Exact Phred expected errors (`phred_exact.jl`) + +Every term `10^(-Q/10)` is a rational times a power of θ = `10^(-1/10)`. θ is a root of +`10x¹⁰ − 1`, which is irreducible, so an expected-error sum is one element of ℚ(θ): ten +rational coefficients. Comparison with a rational threshold is exact by bisection on θ. + +| Check | Result | +| --- | --- | +| Exact sign against 512-bit arithmetic, 60,000 comparisons on random reads | 0 disagreements | +| Twenty bases at Q10 (sum exactly 2), threshold 2, double precision | reports 2.0000000000000004 > 2: wrong | +| 2,000 reads constructed to sum to exactly 2 | double precision wrong on all 2,000 | +| Random reads, double precision against exact | 0 disagreements | + +### 2. Distances on demand, exact tree-like input (`zoom_tree.jl`) + +Leaves are inserted one at a time; each is placed by asking at most three distances per level +of a centroid search. Input is an additive tree metric with integer edge lengths. + +| Leaves | Distances asked | Share of the full matrix | Asked / (n log₂ n) | +| --- | --- | --- | --- | +| 1,024 | 10,210 to 15,199 | 1.9% to 2.9% | 1.0 to 1.5 | +| 4,096 | 53,608 to 79,971 | 0.6% to 1.0% | 1.1 to 1.6 | +| 16,384 | 266,642 to 394,964 | 0.2% to 0.3% | 1.2 to 1.7 | + +The lower figure is a random-split tree, the higher a caterpillar. Every reconstruction matched +the true tree in topology and edge lengths, and canonical neighbour joining agreed on 160 small +cases. n log n is the known lower bound for this query model, so this is not O(n). The +prototype's running time is quadratic in bookkeeping; only the distance count is measured. + +### 3. Lazy views, simulated noisy sequences (`zoom_noisy.jl`) + +512 leaves, Jukes-Cantor evolution on random trees, views of 64 leaves, 5 trees per row. + +| Sites | View | Agreement, all branches | +| --- | --- | --- | +| 250 | zoomed-in neighbourhood | 93.8% | +| 250 | thumbnail (spread-out leaves) | 86.2% | +| 250 | random subset | 92.5% | +| 1,000 | zoomed-in / thumbnail | 99.3% / 97.4% | +| 5,000 | all views | 99.7% or better | + +On exact tree-like distances every figure is 100%. The simulated data sat 9 to 175 times +outside the Atteson radius even where neighbour joining recovered every branch, so that +radius does not explain when lazy views agree. + +### 4. Lazy views, real reads (`zoom_real.jl`, `zoom_real_1024.jl`) + +Part A, the Phred filter on 152,360 real reads: double precision and the exact method agree +on every read. No read sums exactly to the threshold; the closest is 2.2e-6 away. For 98.1% of +reads the double-precision sum depends on the order of addition. The exact method (integer +counts per quality value, exact comparison) took 6.2 times as long as double precision. + +Part B, views of 64 leaves, 40 views per row: + +| Sequences | View | All branches | Supported branches (middle 95% of views) | +| --- | --- | --- | --- | +| 512, error variants kept | zoomed-in | 60.7% | 91.2% (71.4 to 100) | +| 512, error variants kept | random subset | 60.8% | 81.4% (70.8 to 95.8) | +| 512, error variants kept | thumbnail | 43.1% | 50.9% (39.3 to 61.3) | +| 384, crude denoise | zoomed-in | 53.6% | 94.3% (77.7 to 100) | +| 384, crude denoise | random subset | 55.4% | 82.1% (71.1 to 95.8) | +| 384, crude denoise | thumbnail | 50.4% | 69.8% (56.6 to 89.0) | +| 1,024, crude denoise | zoomed-in | 35.3% | 65.8% (0 to 100) | +| 1,024, crude denoise | random subset | 43.3% | 60.2% (48.1 to 80.7) | + +Agreement with leaf labels shuffled is 0% throughout. The 1,024 row is why the zoomed-in +figures above it should not be quoted alone. + +### 5. Can the thumbnail be fixed by a better summary? (`thumb_real.jl`) + +1,024 sequences, 64 regions around random representatives, 30 draws. Target: the full tree cut +down to the representatives. Share of supported branches reproduced: + +| Summary of each region | Supported branches | Pairwise alignments | +| --- | --- | --- | +| one representative | 55.9% | 2,016 | +| mean over 4 members, corrected | 44.6% | 28,441 | +| mean over 8 members, corrected | 46.8% | 89,253 | +| mean over all members, corrected | 46.5% | 523,776 (all) | +| 5-mer counts of the representative | 55.4% | none | +| mean 5-mer counts of the region | 51.0% | none | + +Spread-out representatives and 32 regions give the same picture; no summary exceeds 56%. +Alignment-free 5-mer distances match aligned distances for this purpose at no alignment cost. + +### 6. Is the target stable? (`ceiling_real.jl`) + +The tree on all 1,024 sequences compared with a tree built on a random subset of them, both cut +down to the same 64 leaves, 30 draws per row. + +| Tree built on | All branches | Supported branches (middle 95%) | +| --- | --- | --- | +| 64 of 1,024 | 43.1% | 59.6% (44.0 to 73.7) | +| 128 | 43.8% | 59.6% (37.7 to 74.9) | +| 256 | 47.6% | 65.9% (47.7 to 81.3) | +| 512 | 51.7% | 70.4% (55.1 to 89.2) | +| 768 | 62.1% | 78.4% (59.3 to 90.4) | + +## Reading of the results + +- **Exactness.** The rationals do not need replacing by the reals for Phred: one algebraic + number is enough. On this dataset exact arithmetic gives the same decisions as double + precision. What it adds is order-independence and a provable comparison, not a different + answer. +- **Linear time.** On tree-like input the distance count falls from n² to about n log n, which + is optimal for distance queries. Nothing here supports O(n) for de novo tree building, and no + use for the Veblen hierarchy in the algorithm was found. +- **Lazy views on amplicon reads.** Experiment 6 is the limiting result. At 240 bases the + neighbour-joining tree changes, including in branches it reports as well supported, when + sequences are added. Agreement with that tree is therefore the wrong contract, and experiment + 5's failure is not a failure of the summaries. A stable overview has to come from outside the + amplicon data, which is what a reference tree with placement provides. + +## Limits + +- One dataset, one marker region, one read length; dereplicated reads, not denoised variants. +- Neighbour joining only, with Jukes-Cantor distances in double precision and first-minimum tie + breaking. Other distance corrections, tie policies and tree methods were not tried. +- "Supported" is a branch-length threshold, not a bootstrap or posterior support. +- Samples are small (30 to 40 views per row) and the ranges are wide. +- Experiments 2 and 3 use simulated input. Experiment 2 measures distance counts, not time. +- Not done: any proof, any comparison with RapidNJ, SwiftNJ or FastTree, any test of placement + on a reference tree, and any change to the application. + +## Running + +```bash +julia docs/cladistics/issue-303/phred_exact.jl +``` + +```bash +julia docs/cladistics/issue-303/zoom_tree.jl +``` + +```bash +julia -O2 docs/cladistics/issue-303/zoom_noisy.jl +``` + +```bash +julia -O2 -t 4 docs/cladistics/issue-303/zoom_real.jl /path/to/MiSeq_SOP +``` + +```bash +julia -O2 -t 4 docs/cladistics/issue-303/zoom_real_1024.jl /path/to/MiSeq_SOP +``` + +```bash +julia -O2 -t 4 docs/cladistics/issue-303/thumb_real.jl /path/to/MiSeq_SOP +``` + +```bash +julia -O2 -t 4 docs/cladistics/issue-303/ceiling_real.jl /path/to/MiSeq_SOP +``` + +The scripts use only Julia's standard library and fixed random seeds. `zoom_real.jl` carries its +own copy of the shared code because it is the script as it was run; the later scripts include +`lib_tree.jl`. diff --git a/docs/cladistics/issue-303/ceiling_real.jl b/docs/cladistics/issue-303/ceiling_real.jl new file mode 100644 index 00000000..1896ab32 --- /dev/null +++ b/docs/cladistics/issue-303/ceiling_real.jl @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E6: is the TARGET itself stable? Real MiSeq_SOP reads. +# +# usage: julia -t 4 ceiling_real.jl +# +# E5 found no region summary reproduces more than about 56% of the well-supported branches of +# the full tree cut down to 64 representatives. Before blaming the summaries, ask how well the +# full tree agrees with ITSELF when the set of other sequences changes: build the tree on all n +# sequences and on a random subset T of them, cut both down to the same 64 leaves S (S inside T), +# and compare. If two "full" trees disagree about the same 64 leaves, no thumbnail can match both. +include(joinpath(@__DIR__, "lib_tree.jl")) + +function run(dir; nmax = 1024, k = 64, draws = 30) + rng = MersenneTwister(20261117); seqs = load_set(dir, nmax); n = length(seqs) + D, _ = distance_matrix(seqs); key = rand(rng, UInt64, n); one = 1 / TRUNC + full, _, fblen = nj(D, collect(1:n)) + @printf("n = %d sequences; views of %d leaves; %d draws per row\n", n, k, draws) + @printf(" %-34s %7s %9s %24s\n", "tree compared with the n-leaf tree", "all", "weighted", ">=3 substitutions") + for tsize in (k, 128, 256, 512, 768) + rows = NTuple{3,Float64}[] + for _ in 1:draws + T = sort(randperm(rng, n)[1:tsize]); S = sort(T[randperm(rng, tsize)[1:k]]) + inS = fill(false, n); inS[S] .= true; target = splits(full, n, key, inS, fblen) + sub, m, sblen = nj(D, T); inST = [t in S for t in T] |> collect + a = splits(sub, m, key[T], Vector{Bool}(inST), sblen) + tot = length(target); sh = count(x -> haskey(a, x), keys(target)) + w = sum(values(target)); wsh = sum((v for (x, v) in target if haskey(a, x)); init = 0.0) + t3 = [x for (x, v) in target if v >= 3one] + push!(rows, (sh / max(tot, 1), wsh / max(w, eps()), isempty(t3) ? NaN : count(x -> haskey(a, x), t3) / length(t3))) + end + c3 = filter(!isnan, [r[3] for r in rows]) + @printf(" tree built on %4d of the %d %6.1f%% %8.1f%% %7.1f%% (%5.1f .. %5.1f)\n", tsize, n, + 100mean(r[1] for r in rows), 100mean(r[2] for r in rows), 100mean(c3), 100quantile(c3, 0.025), 100quantile(c3, 0.975)) + end +end +run(ARGS[1]) diff --git a/docs/cladistics/issue-303/lib_tree.jl b/docs/cladistics/issue-303/lib_tree.jl new file mode 100644 index 00000000..cbae8ccf --- /dev/null +++ b/docs/cladistics/issue-303/lib_tree.jl @@ -0,0 +1,133 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Shared code for the issue-303 experiments: FASTQ reading, exact Phred expected errors, edit distance, +# canonical neighbour joining with branch lengths, and tree splits restricted to a leaf subset. +using Random, Printf, Statistics +const TRUNC = 240 +const MAXEE = 2 + +# ---------- FASTQ ---------- +function read_fastq(path) + seqs = String[]; quals = String[] + open(path) do io + while !eof(io) + readline(io); s = readline(io); readline(io); q = readline(io) + push!(seqs, s); push!(quals, q) + end + end + seqs, quals +end + +# ---------- Part A: exact expected errors ---------- +# EE = sum 10^(-Q/10). With Q = 10a + b the term is 10^(-a) * theta^b, theta = 10^(-1/10). +# Keep ten integer coefficients over the common denominator 10^9 (Q <= 99): pure integer counting. +const POW10 = [big(10)^k for k in 0:9] +function ee_coeffs(q::AbstractVector{UInt8}) # q are Phred values, not ASCII + c = zeros(Int64, 10) + @inbounds for Q in q + a, b = divrem(Int(Q), 10); c[b+1] += 10^(9 - a) + end + c +end +"sign of EE - T for integer threshold T, exact (bisection on theta; P is monotone in theta)." +function cmp_ee(c::Vector{Int64}, T::Int) + p = Rational{BigInt}.(c) .// big(10)^9; p[1] -= T + all(iszero, p) && return 0 + all(iszero, @view p[2:end]) && return Int(sign(p[1])) + lo, hi = big(0) // 1, big(1) // 1 + while true + slo = sign(foldr((ck, acc) -> ck + lo * acc, p)); shi = sign(foldr((ck, acc) -> ck + hi * acc, p)) + (slo == shi && slo != 0) && return Int(slo) + mid = (lo + hi) / 2 + if 10 * mid^10 < 1; lo = mid else hi = mid end + end +end +ee_float(q) = sum(10.0^(-Int(Q) / 10) for Q in q) + +# ---------- distances ---------- +function edit_distance(a::Vector{UInt8}, b::Vector{UInt8}, prev::Vector{Int32}, cur::Vector{Int32}) + n = length(a); m = length(b) + @inbounds for j in 0:m; prev[j+1] = j; end + @inbounds for i in 1:n + cur[1] = i; ai = a[i] + for j in 1:m + sub = prev[j] + (ai != b[j]); del = prev[j+1] + 1; ins = cur[j] + 1 + cur[j+1] = min(sub, del, ins) + end + prev, cur = cur, prev + end + Int(prev[m+1]) +end +function distance_matrix(seqs::Vector{Vector{UInt8}}) + n = length(seqs); D = zeros(n, n); E = zeros(Int, n, n) + Threads.@threads for i in 1:n + prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1) + for j in i+1:n + e = edit_distance(seqs[i], seqs[j], prev, cur); p = e / TRUNC + d = p >= 0.74 ? 5.0 : -0.75 * log(1 - 4p / 3) + D[i, j] = d; D[j, i] = d; E[i, j] = e; E[j, i] = e + end + end + D, E +end + +# ---------- canonical neighbour joining (same code as the simulation, E3) ---------- +function nj(D::Matrix{Float64}, idx::Vector{Int}) + m = length(idx); d = D[idx, idx]; N = 2m - 2 + d = vcat(hcat(d, zeros(m, N - m)), zeros(N - m, N)); children = fill((0, 0), m); active = collect(1:m); blen = zeros(N) + tsum = zeros(N); for i in 1:m; tsum[i] = sum(@view d[i, 1:m]); end + nxt = m + while length(active) > 2 + r = length(active); bi = 0; bj = 0; bq = Inf + @inbounds for x in 1:r, y in x+1:r + i = active[x]; j = active[y]; q = (r - 2) * d[i, j] - tsum[i] - tsum[j] + if q < bq; bq = q; bi = i; bj = j; end + end + nxt += 1; u = nxt; push!(children, (bi, bj)); s = 0.0 + li = d[bi, bj] / 2 + (tsum[bi] - tsum[bj]) / (2 * (r - 2)); blen[bi] = max(li, 0.0); blen[bj] = max(d[bi, bj] - li, 0.0) + @inbounds for k in active + (k == bi || k == bj) && continue + v = (d[bi, k] + d[bj, k] - d[bi, bj]) / 2; d[u, k] = v; d[k, u] = v; s += v + tsum[k] += v - d[bi, k] - d[bj, k] + end + tsum[u] = s; filter!(k -> k != bi && k != bj, active); push!(active, u) + end + push!(children, (active[1], active[2])); blen[active[1]] = max(d[active[1], active[2]], 0.0) + children, m, blen +end +function splits(children, nleaves, key::Vector{UInt64}, inS::Vector{Bool}, blen::Vector{Float64}) + N = length(children); h = zeros(UInt64, N); c = zeros(Int, N) + for u in 1:nleaves; inS[u] && (h[u] = key[u]; c[u] = 1); end + for u in nleaves+1:N; (a, b) = children[u]; h[u] = h[a] ⊻ h[b]; c[u] = c[a] + c[b]; end + tot = c[N]; all = h[N]; out = Dict{UInt64,Float64}() # split => summed length of the edges that induce it + for u in 1:N-1; if 2 <= c[u] <= tot - 2; k = min(h[u], all ⊻ h[u]); out[k] = get(out, k, 0.0) + blen[u]; end; end + out +end + + +# ---------- real reads: filter, dereplicate, crude denoise ---------- +"Distinct forward reads (truncated, EE-filtered, no N), most abundant first, dropping anything +within one edit of a kept sequence and anything seen once. At most nmax sequences." +function load_set(dir, nmax) + files = sort(filter(f -> endswith(f, "_R1_001.fastq"), readdir(dir; join = true))) + counts = Dict{String,Int}() + for f in files + seqs, quals = read_fastq(f) + for (s, qs) in zip(seqs, quals) + length(s) < TRUNC && continue + q = UInt8[UInt8(c) - 0x21 for c in codeunits(qs)[1:TRUNC]] + ee_float(q) > MAXEE && continue # E4: identical to the exact filter on every read here + st = s[1:TRUNC]; occursin('N', st) && continue + counts[st] = get(counts, st, 0) + 1 + end + end + uniq = sort(collect(counts); by = x -> (-x[2], x[1])) + kept = Vector{Vector{UInt8}}(); prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1) + for x in uniq # crude denoise, as E4 set 2 + x[2] < 2 && break + s = Vector{UInt8}(codeunits(x[1])) + any(t -> edit_distance(s, t, prev, cur) <= 1, kept) || push!(kept, s) + length(kept) >= nmax && break + end + kept +end diff --git a/docs/cladistics/issue-303/phred_exact.jl b/docs/cladistics/issue-303/phred_exact.jl new file mode 100644 index 00000000..ade3fc3f --- /dev/null +++ b/docs/cladistics/issue-303/phred_exact.jl @@ -0,0 +1,122 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E1: exact Phred expected-error arithmetic in Q(theta), theta = 10^(-1/10). +# +# 10^(-Q/10) = 10^(-a) * theta^b with Q = 10a + b, 0 <= b <= 9. +# theta is a root of 10 x^10 - 1, irreducible over Q (x^10 - 10 is Eisenstein at 2), +# so 1, theta, ..., theta^9 are linearly independent over Q. An expected-error sum +# EE = sum_i 10^(-Q_i/10) is therefore ONE element of Q(theta): ten rational coefficients. +# +# Comparison with a rational threshold T: P(x) = (C0 - T) + C1 x + ... + C9 x^9. +# - all coefficients zero -> EE == T exactly +# - otherwise P(theta) != 0 (linear independence), and since C1..C9 >= 0, P is +# monotone non-decreasing on x > 0, so a rational enclosure lo < theta < hi gives +# P(lo) <= P(theta) <= P(hi) with no interval arithmetic. Bisect until signs agree. +using Random, Printf + +const R = Rational{BigInt} + +"EE as ten exact rational coefficients of theta^0..theta^9." +function ee_exact(q::AbstractVector{<:Integer}) + c = zeros(R, 10) + for Q in q + a, b = divrem(Int(Q), 10) + c[b + 1] += R(1, big(10)^a) + end + c +end + +horner(c, x) = foldr((ck, acc) -> ck + x * acc, c) + +"Sign of EE - T, exact. Returns (-1 | 0 | +1, bisection steps used)." +function cmp_exact(c::Vector{R}, T::R) + p = copy(c); p[1] -= T + all(iszero, p) && return (0, 0) + @assert all(>=(0), @view p[2:end]) + if all(iszero, @view p[2:end]) # purely rational: no theta needed + return (Int(sign(p[1])), 0) + end + lo, hi = R(0), R(1) # 0 < theta < 1 + steps = 0 + while true + slo, shi = sign(horner(p, lo)), sign(horner(p, hi)) + slo == shi && slo != 0 && return (Int(slo), steps) + mid = (lo + hi) / 2 + if 10 * mid^10 < 1; lo = mid else hi = mid end # theta^10 = 1/10 + steps += 1 + steps > 4000 && error("no separation: contradicts linear independence") + end +end + +ee_float(q) = sum(10.0^(-Q / 10) for Q in q) # what a C/R double loop does +function ee_big(q; prec = 512) + setprecision(BigFloat, prec) do + sum(BigFloat(10)^(-BigFloat(Q) / 10) for Q in q) + end +end + +println("== 1. A filter decision that double precision gets wrong ==") +q = fill(10, 20) # twenty bases at Q10: EE is exactly 2 +T = R(2) +@printf("float EE = %.17g ; float says EE > 2 : %s\n", ee_float(q), ee_float(q) > 2.0) +println("exact EE vs 2 : sign = ", cmp_exact(ee_exact(q), T)[1], " (0 means exactly equal, so the read is kept under EE > maxEE)") + +println("\n== 2. Order dependence of the double-precision sum ==") +rng = MersenneTwister(20261116) +function order_dependent(rng, n, len, qs) + k = 0 + for _ in 1:n + q = rand(rng, qs, len) + e1 = ee_float(q); e2 = ee_float(sort(q)); e3 = ee_float(reverse(sort(q))) + k += !(e1 == e2 == e3) + end + k +end +for (name, qs) in (("full range Q0..41", 0:41), ("binned {2,12,23,37}", [2, 12, 23, 37])) + k = order_dependent(rng, 20_000, 250, qs) + @printf("%-22s reads whose float EE changes with base order: %d / 20000 (%.1f%%)\n", name, k, 100k / 20000) +end +println("exact coefficients are order-independent by construction (rational addition is associative).") + +println("\n== 3. Agreement of exact sign with 512-bit BigFloat, and float disagreements ==") +function agree(rng, n, len, qs, Ts) + bad_exact = 0; bad_float = 0; mingap = big(Inf); maxsteps = 0 + for _ in 1:n + q = rand(rng, qs, len) + c = ee_exact(q); eb = ee_big(q); ef = ee_float(q) + for T in Ts + s, st = cmp_exact(c, R(T)); maxsteps = max(maxsteps, st) + sb = Int(sign(eb - T)); sf = Int(sign(ef - T)) + bad_exact += (s != sb); bad_float += (s != sf) + mingap = min(mingap, abs(eb - T)) + end + end + (bad_exact, bad_float, Float64(mingap), maxsteps) +end +for (name, qs, len) in (("Q0..41, len 250", 0:41, 250), ("binned, len 250", [2, 12, 23, 37], 250), + ("Q30..41, len 150", 30:41, 150), ("near-threshold: Q8..12, len 20", 8:12, 20)) + be, bf, g, st = agree(rng, 5_000, len, qs, (1, 2, 5)) + @printf("%-32s exact!=BigFloat: %d float!=exact: %d min|EE-T|: %.3g max bisection steps: %d\n", name, be, bf, g, st) +end + +println("\n== 4. Adversarial ties: reads built to sit exactly on the threshold ==") +function ties(rng, n) + flips = 0; exact_ties = 0 + for _ in 1:n + # multiples of ten only -> EE rational; pick counts so EE == 2 exactly + k10 = rand(rng, 0:20); k20 = 10 * (20 - k10) # k10*0.1 + k20*0.01 = 2 + q = shuffle(rng, vcat(fill(10, k10), fill(20, k20))) + s, _ = cmp_exact(ee_exact(q), R(2)); exact_ties += (s == 0) + flips += (ee_float(q) > 2.0) + end + (exact_ties, flips) +end +et, fl = ties(rng, 2_000) +@printf("constructed reads with EE exactly 2: %d / 2000 ; float wrongly reports EE > 2 for %d of them (%.1f%%)\n", et, fl, 100fl / 2000) + +println("\n== 5. Cost ==") +qs = [rand(rng, 0:41, 250) for _ in 1:20_000] +ee_float(qs[1]); cmp_exact(ee_exact(qs[1]), R(2)) +tf = @elapsed for q in qs; ee_float(q) > 2.0; end +te = @elapsed for q in qs; cmp_exact(ee_exact(q), R(2)); end +@printf("20000 reads x 250 bases: float %.3f s, exact %.3f s (ratio %.0fx); exact = %.1f microseconds per read\n", tf, te, te / tf, 1e6te / 20000) diff --git a/docs/cladistics/issue-303/thumb_real.jl b/docs/cladistics/issue-303/thumb_real.jl new file mode 100644 index 00000000..20a17cec --- /dev/null +++ b/docs/cladistics/issue-303/thumb_real.jl @@ -0,0 +1,107 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E5: can the THUMBNAIL be fixed? Real MiSeq_SOP reads. +# +# usage: julia -t 4 thumb_real.jl +# +# E4 showed a thumbnail built from one representative sequence per region reproduces only +# 51-70% of the well-supported branches of the full tree. Here the leaves are partitioned into +# k regions and each region is summarised by MORE than one sequence. The target is unchanged: +# the full neighbour-joining tree cut down to the k region representatives. Estimators: +# +# one-rep distance between the two representatives (the E4 baseline) +# members-m mean distance over m sampled members of each region, minus the mean +# within-region distance (estimates the distance between the regions' roots) +# members-all the same over every member (not lazy: needs all distances; an upper bound) +# kmer-rep alignment-free: L1 distance between the representatives' 5-mer counts +# kmer-region alignment-free: L1 distance between the regions' MEAN 5-mer counts. +# Costs one pass over the sequences (linear) plus k^2 vector distances. +# +# Regions are assigned with the 5-mer sketch (nearest representative), so the whole pipeline +# for kmer-region never computes a pairwise alignment. +include(joinpath(@__DIR__, "lib_tree.jl")) + +const K5 = 5 +function sketch(s::Vector{UInt8}) + v = zeros(Float32, 4^K5); code = 0; valid = 0 + @inbounds for c in s + x = c == UInt8('A') ? 0 : c == UInt8('C') ? 1 : c == UInt8('G') ? 2 : c == UInt8('T') ? 3 : -1 + if x < 0; valid = 0; code = 0; continue; end + code = ((code << 2) | x) & (4^K5 - 1); valid += 1 + valid >= K5 && (v[code+1] += 1f0) + end + v +end +l1(a, b) = (s = 0f0; @inbounds @simd for i in eachindex(a); s += abs(a[i] - b[i]); end; s) +"turn an L1 distance between 5-mer count vectors into a substitution-like distance" +kmer_dist(x) = (p = Float64(x) / (2K5 * TRUNC); p >= 0.74 ? 5.0 : -0.75 * log(1 - 4p / 3)) + + +function recall(est_children, est_blen, m, estkey, target::Dict{UInt64,Float64}) + a = splits(est_children, m, estkey, fill(true, m), est_blen); one = 1 / TRUNC + tot = length(target); sh = count(k -> haskey(a, k), keys(target)) + w = sum(values(target)); wsh = sum((v for (k, v) in target if haskey(a, k)); init = 0.0) + t3 = [k for (k, v) in target if v >= 3one] + (sh / max(tot, 1), wsh / max(w, eps()), isempty(t3) ? NaN : count(k -> haskey(a, k), t3) / length(t3)) +end + +function run(dir; nmax = 1024, k = 64, draws = 30, scheme = :random) + rng = MersenneTwister(20261117) + seqs = load_set(dir, nmax); n = length(seqs) + D, _ = distance_matrix(seqs); key = rand(rng, UInt64, n); S = [sketch(s) for s in seqs] + full, _, fblen = nj(D, collect(1:n)) + names = ["one-rep", "members-4", "members-8", "members-all", "kmer-rep", "kmer-region"] + res = Dict(nm => NTuple{3,Float64}[] for nm in names); aligns = Dict(nm => 0 for nm in names); sizes = Int[] + for _ in 1:draws + if scheme == :random + reps = sort(randperm(rng, n)[1:k]) + else # spread out, chosen with the sketch only + reps = [rand(rng, 1:n)]; mind = [l1(S[i], S[reps[1]]) for i in 1:n] + while length(reps) < k; r = argmax(mind); push!(reps, r); for i in 1:n; mind[i] = min(mind[i], l1(S[i], S[r])); end; end + sort!(reps) + end + region = [argmin([l1(S[i], S[r]) for r in reps]) for i in 1:n] + for (j, r) in enumerate(reps); region[r] = j; end + members = [findall(==(j), region) for j in 1:k]; append!(sizes, length.(members)) + inS = fill(false, n); inS[reps] .= true + target = splits(full, n, key, inS, fblen); rk = key[reps] + function est(M) + c, m, b = nj(M, collect(1:k)) + recall(c, b, m, rk, target) + end + push!(res["one-rep"], est(D[reps, reps])); aligns["one-rep"] = k * (k - 1) ÷ 2 + function member_matrix(pick) + sel = [pick(members[j], reps[j]) for j in 1:k]; M = zeros(k, k) + within = [length(s) > 1 ? mean(D[a, b] for a in s for b in s if a < b) : 0.0 for s in sel] + for a in 1:k, b in a+1:k + v = mean(D[x, y] for x in sel[a] for y in sel[b]) - (within[a] + within[b]) / 2 + M[a, b] = v; M[b, a] = v + end + M, sum(length(sel[a]) * length(sel[b]) for a in 1:k for b in a+1:k) + sum(length(s) * (length(s) - 1) ÷ 2 for s in sel) + end + for (nm, m) in (("members-4", 4), ("members-8", 8)) + # the representative plus up to m-1 other members, never more than the region holds + M, na = member_matrix((mem, r) -> (u = unique(vcat(r, shuffle(rng, mem))); u[1:min(m, length(u))])) + push!(res[nm], est(M)); aligns[nm] = na + end + M, na = member_matrix((mem, r) -> mem); push!(res["members-all"], est(M)); aligns["members-all"] = na + Mk = zeros(k, k); for a in 1:k, b in a+1:k; v = kmer_dist(l1(S[reps[a]], S[reps[b]])); Mk[a, b] = v; Mk[b, a] = v; end + push!(res["kmer-rep"], est(Mk)) + means = [reduce(+, S[members[j]]) ./ Float32(length(members[j])) for j in 1:k] + Mr = zeros(k, k); for a in 1:k, b in a+1:k; v = kmer_dist(l1(means[a], means[b])); Mr[a, b] = v; Mr[b, a] = v; end + push!(res["kmer-region"], est(Mr)) + end + @printf("\nn = %d sequences, k = %d regions (%s representatives), %d draws; region size median %d, range %d..%d\n", + n, k, scheme == :random ? "random" : "spread-out", draws, median(sizes), minimum(sizes), maximum(sizes)) + @printf(" %-13s %7s %9s %24s %s\n", "summary", "all", "weighted", ">=3 substitutions", "pairwise alignments (full tree needs $(n * (n - 1) ÷ 2))") + for nm in names + xs = res[nm]; c3 = filter(!isnan, [x[3] for x in xs]) + @printf(" %-13s %6.1f%% %8.1f%% %7.1f%% (%5.1f .. %5.1f) %s\n", nm, 100mean(x[1] for x in xs), 100mean(x[2] for x in xs), + 100mean(c3), 100quantile(c3, 0.025), 100quantile(c3, 0.975), aligns[nm] == 0 ? "none" : string(aligns[nm])) + end +end + +dir = ARGS[1] +for (k, scheme) in ((64, :random), (64, :spread), (32, :random)) + run(dir; nmax = 1024, k = k, scheme = scheme) +end diff --git a/docs/cladistics/issue-303/zoom_noisy.jl b/docs/cladistics/issue-303/zoom_noisy.jl new file mode 100644 index 00000000..88041b21 --- /dev/null +++ b/docs/cladistics/issue-303/zoom_noisy.jl @@ -0,0 +1,168 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E3: does a LAZY ZOOMABLE tree show the same thing as the full tree, on noisy data? +# +# A lazy zoomable tree never builds the tree on all n leaves. It builds a tree on a subset S +# (a thumbnail of representatives, or the members of one region when the user zooms in). +# That view is trustworthy iff +# NJ(D restricted to S) == NJ(D) restricted to S ("zoom consistency") +# For exact tree-like distances this is automatic. Within the Atteson radius it is a theorem: +# restricting to S only merges edges, so the shortest edge cannot shrink, so the radius +# condition survives restriction and both sides equal the true tree on S. +# Real distances come from finite sequences and are far outside that radius. This script +# measures what actually happens: sequences evolve down a random tree (Jukes-Cantor), distances +# are estimated from L sites, and we compare the two sides, and each side against the truth. +using Random, Printf, Statistics + +# ---------- rooted binary trees as child lists ---------- +struct RTree + children::Vector{Tuple{Int,Int}} # (0,0) for a leaf + blen::Vector{Float64} # length of the edge above each node + nleaves::Int # leaves are nodes 1..nleaves + root::Int +end + +"Random topology by joining random pairs; edge lengths uniform in [lo, hi] substitutions per site." +function random_rtree(rng, n; lo = 0.005, hi = 0.05) + children = fill((0, 0), n); blen = [lo + (hi - lo) * rand(rng) for _ in 1:n]; active = collect(1:n) + while length(active) > 1 + i = rand(rng, 1:length(active)); a = active[i]; deleteat!(active, i) + j = rand(rng, 1:length(active)); b = active[j]; deleteat!(active, j) + push!(children, (a, b)); push!(blen, lo + (hi - lo) * rand(rng)); push!(active, length(children)) + end + RTree(children, blen, n, length(children)) +end + +"Evolve sequences of L sites down the tree under Jukes-Cantor; returns leaf sequences." +function evolve(rng, t::RTree, L) + seqs = Vector{Vector{UInt8}}(undef, length(t.children)); seqs[t.root] = rand(rng, UInt8(0):UInt8(3), L) + stack = [t.root] + while !isempty(stack) + u = pop!(stack); (a, b) = t.children[u]; a == 0 && continue + for c in (a, b) + p = 0.75 * (1 - exp(-4 * t.blen[c] / 3)); s = copy(seqs[u]) + @inbounds for k in 1:L + rand(rng) < p && (s[k] = (s[k] + rand(rng, UInt8(1):UInt8(3))) % UInt8(4)) + end + seqs[c] = s; push!(stack, c) + end + end + seqs[1:t.nleaves] +end + +function jc_matrix(seqs) + n = length(seqs); L = length(seqs[1]); D = zeros(n, n); ties = 0 + @inbounds for i in 1:n, j in i+1:n + h = 0; si = seqs[i]; sj = seqs[j] + for k in 1:L; h += si[k] != sj[k]; end + p = h / L; d = p >= 0.74 ? 5.0 : -0.75 * log(1 - 4p / 3); D[i, j] = d; D[j, i] = d + end + D +end + +"True path-length distances between leaves." +function true_matrix(t::RTree) + N = length(t.children); par = zeros(Int, N); for u in 1:N; (a, b) = t.children[u]; a != 0 && (par[a] = u; par[b] = u); end + depth = zeros(N); order = [t.root]; i = 1 + while i <= length(order); u = order[i]; i += 1; (a, b) = t.children[u]; a != 0 && (depth[a] = depth[u] + t.blen[a]; depth[b] = depth[u] + t.blen[b]; push!(order, a, b)); end + anc = [Int[] for _ in 1:t.nleaves] + for l in 1:t.nleaves; u = l; while u != 0; push!(anc[l], u); u = par[u]; end; end + n = t.nleaves; D = zeros(n, n) + for i in 1:n, j in i+1:n + si = Set(anc[i]); lca = first(u for u in anc[j] if u in si) + d = depth[i] + depth[j] - 2depth[lca]; D[i, j] = d; D[j, i] = d + end + D +end + +# ---------- canonical neighbour joining on a (sub)matrix: returns the tree as child lists ---------- +function nj(D::Matrix{Float64}, idx::Vector{Int}) + m = length(idx); d = D[idx, idx]; N = 2m - 2 + d = vcat(hcat(d, zeros(m, N - m)), zeros(N - m, N)); children = fill((0, 0), m); active = collect(1:m) + tsum = zeros(N); for i in 1:m; tsum[i] = sum(@view d[i, 1:m]); end + nxt = m + while length(active) > 2 + r = length(active); bi = 0; bj = 0; bq = Inf + @inbounds for x in 1:r, y in x+1:r + i = active[x]; j = active[y]; q = (r - 2) * d[i, j] - tsum[i] - tsum[j] + if q < bq; bq = q; bi = i; bj = j; end + end + nxt += 1; u = nxt; push!(children, (bi, bj)); s = 0.0 + @inbounds for k in active + (k == bi || k == bj) && continue + v = (d[bi, k] + d[bj, k] - d[bi, bj]) / 2; d[u, k] = v; d[k, u] = v; s += v + tsum[k] += v - d[bi, k] - d[bj, k] + end + tsum[u] = s; filter!(k -> k != bi && k != bj, active); push!(active, u) + end + push!(children, (active[1], active[2])) # final join closes an (arbitrary) root + children, m # leaves 1..m correspond to idx +end + +# ---------- splits of a tree restricted to a leaf subset ---------- +"Non-trivial bipartitions of leaf set S (given as key per leaf, 0 = not in S) induced by the tree." +function splits(children, nleaves, key::Vector{UInt64}, inS::Vector{Bool}) + N = length(children); h = zeros(UInt64, N); c = zeros(Int, N) + for u in 1:nleaves; inS[u] && (h[u] = key[u]; c[u] = 1); end + for u in nleaves+1:N; (a, b) = children[u]; h[u] = h[a] ⊻ h[b]; c[u] = c[a] + c[b]; end # children precede parents + tot = c[N]; all = h[N]; out = Set{UInt64}() + for u in 1:N; (2 <= c[u] <= tot - 2) && push!(out, min(h[u], all ⊻ h[u])); end + out +end + +# ---------- one replicate ---------- +function replicate(rng, n, L, k) + t = random_rtree(rng, n); seqs = evolve(rng, t, L); D = jc_matrix(seqs); DT = true_matrix(t) + key = rand(rng, UInt64, n); allin = fill(true, n) + mu = minimum(t.blen); atteson = maximum(abs.(D .- DT)) / (mu / 2) # < 1 means inside the radius + full_children, _ = nj(D, collect(1:n)) + truth_all = splits(t.children, n, key, allin); full_all = splits(full_children, n, key, allin) + acc_full = length(intersect(truth_all, full_all)) / (n - 3) + res = Dict{Symbol,Vector{NTuple{3,Float64}}}() + function view(S::Vector{Int}) + inS = fill(false, n); inS[S] .= true + sub_children, m = nj(D, S); subkey = key[S] + sub = splits(sub_children, m, subkey, fill(true, m)) # NJ(D|S) + restr = splits(full_children, n, key, inS) # NJ(D)|S + truth = splits(t.children, n, key, inS) # T|S + denom = length(S) - 3 + (length(intersect(sub, restr)) / denom, length(intersect(sub, truth)) / denom, length(intersect(restr, truth)) / denom) + end + # (a) random subset + res[:random] = [view(sort(randperm(rng, n)[1:k])) for _ in 1:4] + # (b) thumbnail: farthest-point representatives (spread out across the data) + reps = [rand(rng, 1:n)]; mind = copy(D[reps[1], :]) + while length(reps) < k; r = argmax(mind); push!(reps, r); mind .= min.(mind, @view D[r, :]); end + res[:thumbnail] = [view(sort(reps))] + # (c) zoom: the k leaves nearest to a random leaf (one neighbourhood) + res[:zoom] = [view(sort(sortperm(D[rand(rng, 1:n), :])[1:k])) for _ in 1:4] + (acc_full = acc_full, atteson = atteson, res = res) +end + +function main() + rng = MersenneTwister(20261116); n = 512; k = 64; reps = 5 + println("n = $n leaves, views of k = $k, $reps random trees per row, edge lengths 0.005-0.05 subs/site") + println("Columns: agree = splits shared by the lazy view NJ(D|S) and the full tree cut down to S;") + println(" lazy-true / full-true = share of the true splits on S each of them recovers.\n") + @printf("%-7s %-10s %8s %10s %10s %10s %s\n", "sites", "view", "agree", "lazy-true", "full-true", "full NJ", "max error / Atteson radius") + for L in (250, 1000, 5000, 50000) + rs = [replicate(rng, n, L, k) for _ in 1:reps] + for v in (:random, :thumbnail, :zoom) + xs = reduce(vcat, [r.res[v] for r in rs]) + @printf("%-7d %-10s %7.1f%% %9.1f%% %9.1f%% %9.1f%% %.1f\n", L, v, + 100mean(x[1] for x in xs), 100mean(x[2] for x in xs), 100mean(x[3] for x in xs), + 100mean(r.acc_full for r in rs), mean(r.atteson for r in rs)) + end + end + println("\n== control: exact tree-like distances (no noise) must give 100% everywhere ==") + t = random_rtree(rng, n); DT = true_matrix(t); key = rand(rng, UInt64, n) + fc, _ = nj(DT, collect(1:n)); S = sort(randperm(rng, n)[1:k]); inS = fill(false, n); inS[S] .= true + sc, m = nj(DT, S) + a = splits(sc, m, key[S], fill(true, m)); b = splits(fc, n, key, inS); c = splits(t.children, n, key, inS) + @printf("agree %.1f%% lazy-true %.1f%% full-true %.1f%% (full NJ vs truth %.1f%%)\n", 100length(intersect(a, b)) / (k - 3), + 100length(intersect(a, c)) / (k - 3), 100length(intersect(b, c)) / (k - 3), + 100length(intersect(splits(fc, n, key, fill(true, n)), splits(t.children, n, key, fill(true, n)))) / (n - 3)) + println("\ndistance look-ups: full tree needs ", n * (n - 1) ÷ 2, "; one view of $k needs ", k * (k - 1) ÷ 2, + " (", round(100 * (k * (k - 1) ÷ 2) / (n * (n - 1) ÷ 2), digits = 2), "%); choosing spread-out representatives costs ", k * n, " more") +end +main() diff --git a/docs/cladistics/issue-303/zoom_real.jl b/docs/cladistics/issue-303/zoom_real.jl new file mode 100644 index 00000000..5018a3fb --- /dev/null +++ b/docs/cladistics/issue-303/zoom_real.jl @@ -0,0 +1,217 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E4: the lazy zoomable tree and the exact Phred filter, on REAL reads (MiSeq_SOP). +# +# usage: julia zoom_real.jl +# +# Part A Phred on real quality strings: exact expected-error filter vs double precision. +# Part B Zoom consistency on real sequences: does a tree built lazily on a subset S agree with the +# full neighbour-joining tree cut down to S? There is no known true tree for real data, so +# only agreement is measured, never accuracy. +# +# Honest limits, stated up front: +# - sequences are dereplicated forward reads (240 bases), NOT DADA2-denoised ASVs. A second set +# removes anything within one edit of a more abundant sequence, a crude stand-in for denoising. +# - distances are Jukes-Cantor corrections of edit distance / length, in double precision. +using Random, Printf, Statistics + +const TRUNC = 240 +const MAXEE = 2 + +# ---------- FASTQ ---------- +function read_fastq(path) + seqs = String[]; quals = String[] + open(path) do io + while !eof(io) + readline(io); s = readline(io); readline(io); q = readline(io) + push!(seqs, s); push!(quals, q) + end + end + seqs, quals +end + +# ---------- Part A: exact expected errors ---------- +# EE = sum 10^(-Q/10). With Q = 10a + b the term is 10^(-a) * theta^b, theta = 10^(-1/10). +# Keep ten integer coefficients over the common denominator 10^9 (Q <= 99): pure integer counting. +const POW10 = [big(10)^k for k in 0:9] +function ee_coeffs(q::AbstractVector{UInt8}) # q are Phred values, not ASCII + c = zeros(Int64, 10) + @inbounds for Q in q + a, b = divrem(Int(Q), 10); c[b+1] += 10^(9 - a) + end + c +end +"sign of EE - T for integer threshold T, exact (bisection on theta; P is monotone in theta)." +function cmp_ee(c::Vector{Int64}, T::Int) + p = Rational{BigInt}.(c) .// big(10)^9; p[1] -= T + all(iszero, p) && return 0 + all(iszero, @view p[2:end]) && return Int(sign(p[1])) + lo, hi = big(0) // 1, big(1) // 1 + while true + slo = sign(foldr((ck, acc) -> ck + lo * acc, p)); shi = sign(foldr((ck, acc) -> ck + hi * acc, p)) + (slo == shi && slo != 0) && return Int(slo) + mid = (lo + hi) / 2 + if 10 * mid^10 < 1; lo = mid else hi = mid end + end +end +ee_float(q) = sum(10.0^(-Int(Q) / 10) for Q in q) + +# ---------- distances ---------- +function edit_distance(a::Vector{UInt8}, b::Vector{UInt8}, prev::Vector{Int32}, cur::Vector{Int32}) + n = length(a); m = length(b) + @inbounds for j in 0:m; prev[j+1] = j; end + @inbounds for i in 1:n + cur[1] = i; ai = a[i] + for j in 1:m + sub = prev[j] + (ai != b[j]); del = prev[j+1] + 1; ins = cur[j] + 1 + cur[j+1] = min(sub, del, ins) + end + prev, cur = cur, prev + end + Int(prev[m+1]) +end +function distance_matrix(seqs::Vector{Vector{UInt8}}) + n = length(seqs); D = zeros(n, n); E = zeros(Int, n, n) + Threads.@threads for i in 1:n + prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1) + for j in i+1:n + e = edit_distance(seqs[i], seqs[j], prev, cur); p = e / TRUNC + d = p >= 0.74 ? 5.0 : -0.75 * log(1 - 4p / 3) + D[i, j] = d; D[j, i] = d; E[i, j] = e; E[j, i] = e + end + end + D, E +end + +# ---------- canonical neighbour joining (same code as the simulation, E3) ---------- +function nj(D::Matrix{Float64}, idx::Vector{Int}) + m = length(idx); d = D[idx, idx]; N = 2m - 2 + d = vcat(hcat(d, zeros(m, N - m)), zeros(N - m, N)); children = fill((0, 0), m); active = collect(1:m); blen = zeros(N) + tsum = zeros(N); for i in 1:m; tsum[i] = sum(@view d[i, 1:m]); end + nxt = m + while length(active) > 2 + r = length(active); bi = 0; bj = 0; bq = Inf + @inbounds for x in 1:r, y in x+1:r + i = active[x]; j = active[y]; q = (r - 2) * d[i, j] - tsum[i] - tsum[j] + if q < bq; bq = q; bi = i; bj = j; end + end + nxt += 1; u = nxt; push!(children, (bi, bj)); s = 0.0 + li = d[bi, bj] / 2 + (tsum[bi] - tsum[bj]) / (2 * (r - 2)); blen[bi] = max(li, 0.0); blen[bj] = max(d[bi, bj] - li, 0.0) + @inbounds for k in active + (k == bi || k == bj) && continue + v = (d[bi, k] + d[bj, k] - d[bi, bj]) / 2; d[u, k] = v; d[k, u] = v; s += v + tsum[k] += v - d[bi, k] - d[bj, k] + end + tsum[u] = s; filter!(k -> k != bi && k != bj, active); push!(active, u) + end + push!(children, (active[1], active[2])); blen[active[1]] = max(d[active[1], active[2]], 0.0) + children, m, blen +end +function splits(children, nleaves, key::Vector{UInt64}, inS::Vector{Bool}, blen::Vector{Float64}) + N = length(children); h = zeros(UInt64, N); c = zeros(Int, N) + for u in 1:nleaves; inS[u] && (h[u] = key[u]; c[u] = 1); end + for u in nleaves+1:N; (a, b) = children[u]; h[u] = h[a] ⊻ h[b]; c[u] = c[a] + c[b]; end + tot = c[N]; all = h[N]; out = Dict{UInt64,Float64}() # split => summed length of the edges that induce it + for u in 1:N-1; if 2 <= c[u] <= tot - 2; k = min(h[u], all ⊻ h[u]); out[k] = get(out, k, 0.0) + blen[u]; end; end + out +end + +ci(xs) = (mean(xs), quantile(xs, 0.025), quantile(xs, 0.975)) + +function zoom_test(rng, name, seqs, k; nviews = 40) + n = length(seqs); D, E = distance_matrix(seqs); key = rand(rng, UInt64, n) + full, _, fblen = nj(D, collect(1:n)) + offdiag = [E[i, j] for i in 1:n for j in i+1:n] + @printf("\n%s: n = %d sequences; edit distance median %d, range %d..%d; %.1f%% of pairs within 3 edits\n", + name, n, median(offdiag), minimum(offdiag), maximum(offdiag), 100mean(offdiag .<= 3)) + one = 1 / TRUNC # one substitution, as a distance + # For a view S: every split of the full tree cut down to S, with the length of the branch that + # carries it. Report how much of that structure the lazy view reproduces: + # all every split counted equally (includes zero-length, i.e. unsupported, branches) + # weighted each split counted by its branch length + # >=1, >=3 only splits whose branch is at least 1 or 3 substitutions long + function view(S, fullkey) + inS = fill(false, n); inS[S] .= true + sub, m, sblen = nj(D, S); a = splits(sub, m, key[S], fill(true, m), sblen); b = splits(full, n, fullkey, inS, fblen) + tot = length(b); shared = count(kk -> haskey(a, kk), keys(b)) + wtot = sum(values(b)); wsh = sum((v for (kk, v) in b if haskey(a, kk)); init = 0.0) + b1 = [kk for (kk, v) in b if v >= one]; b3 = [kk for (kk, v) in b if v >= 3one] + (shared / max(tot, 1), wsh / max(wtot, eps()), isempty(b1) ? NaN : count(kk -> haskey(a, kk), b1) / length(b1), + isempty(b3) ? NaN : count(kk -> haskey(a, kk), b3) / length(b3), length(b1) / max(tot, 1), length(b3) / max(tot, 1)) + end + views = Dict{String,Vector{NTuple{6,Float64}}}() + views["random subset"] = [view(sort(randperm(rng, n)[1:k]), key) for _ in 1:nviews] + views["zoomed-in neighbourhood"] = [view(sort(sortperm(D[rand(rng, 1:n), :])[1:k]), key) for _ in 1:nviews] + th = NTuple{6,Float64}[] + for _ in 1:nviews + reps = [rand(rng, 1:n)]; mind = copy(D[reps[1], :]) + while length(reps) < k; r = argmax(mind); push!(reps, r); mind .= min.(mind, @view D[r, :]); end + push!(th, view(sort(reps), key)) + end + views["thumbnail (spread out)"] = th + keyp = key[randperm(rng, n)] + views["chance (labels shuffled)"] = [view(sort(randperm(rng, n)[1:k]), keyp) for _ in 1:nviews] + @printf(" %-26s %9s %9s %16s %16s %s\n", "view", "all", "weighted", ">=1 subst", ">=3 subst", "share of splits with a >=1 / >=3 branch") + for label in ("random subset", "thumbnail (spread out)", "zoomed-in neighbourhood", "chance (labels shuffled)") + xs = views[label]; col(i) = filter(!isnan, [x[i] for x in xs]) + f(i) = (c = col(i); isempty(c) ? " n/a" : @sprintf("%5.1f%%", 100mean(c))) + g(i) = (c = col(i); isempty(c) ? "n/a" : @sprintf("%5.1f%% (%4.1f..%5.1f)", 100mean(c), 100quantile(c, 0.025), 100quantile(c, 0.975))) + @printf(" %-26s %9s %9s %16s %16s %s / %s\n", label, f(1), f(2), g(3), g(4), f(5), f(6)) + end + @printf(" distance look-ups: full tree %d, one view of %d: %d (%.2f%%)\n", n * (n - 1) ÷ 2, k, k * (k - 1) ÷ 2, + 100 * (k * (k - 1) ÷ 2) / (n * (n - 1) ÷ 2)) +end + +function main(dir) + rng = MersenneTwister(20261117) + files = sort(filter(f -> endswith(f, "_R1_001.fastq"), readdir(dir; join = true))) + println("forward-read files: ", length(files)) + counts = Dict{String,Int}(); nreads = 0; kept = 0; short = 0; hasN = 0 + qhist = zeros(Int, 100); float_wrong = 0; exact_ties = 0; minabs = Inf; order_dep = 0; over = 0 + t_exact = 0.0; t_float = 0.0 + for f in files + seqs, quals = read_fastq(f) + for (s, qs) in zip(seqs, quals) + nreads += 1 + if length(s) < TRUNC; short += 1; continue; end + st = s[1:TRUNC] + q = UInt8[UInt8(c) - 0x21 for c in codeunits(qs)[1:TRUNC]] + for Q in q; qhist[Q+1] += 1; end + t_exact += @elapsed (sgn = cmp_ee(ee_coeffs(q), MAXEE)) + t_float += @elapsed (ef = ee_float(q)) + sf = ef > MAXEE ? 1 : (ef < MAXEE ? -1 : 0) + float_wrong += (sf != sgn); exact_ties += (sgn == 0); minabs = min(minabs, abs(ef - MAXEE)) + order_dep += (ee_float(sort(q)) != ef) + if sgn > 0; over += 1; continue; end + if occursin('N', st); hasN += 1; continue; end + kept += 1; counts[st] = get(counts, st, 0) + 1 + end + end + println("\n== Part A: Phred expected-error filter on real reads (truncated to $TRUNC, maxEE = $MAXEE) ==") + @printf("reads %d; shorter than %d: %d; EE > %d (exact): %d; containing N: %d; kept: %d\n", nreads, TRUNC, short, MAXEE, over, hasN, kept) + println("distinct quality values seen: ", join([Q - 1 for Q in 1:100 if qhist[Q] > 0], " ")) + @printf("double precision disagrees with exact on %d of %d reads; exact ties with the threshold: %d\n", float_wrong, nreads - short, exact_ties) + @printf("closest any read came to the threshold: |EE - %d| = %.3g\n", MAXEE, minabs) + @printf("reads whose double-precision EE changes if the bases are summed in sorted order: %d (%.1f%%)\n", order_dep, 100order_dep / (nreads - short)) + @printf("time: exact (integer counts + exact compare) %.2f s, double %.2f s (ratio %.1fx)\n", t_exact, t_float, t_exact / t_float) + + println("\n== Part B: zoom consistency on real sequences ==") + uniq = sort(collect(counts); by = x -> -x[2]) + @printf("distinct sequences: %d; with abundance >= 2: %d; >= 8: %d\n", length(uniq), count(x -> x[2] >= 2, uniq), count(x -> x[2] >= 8, uniq)) + tobytes(s) = Vector{UInt8}(codeunits(s)) + top = [tobytes(x[1]) for x in uniq[1:min(512, length(uniq))]] + zoom_test(rng, "Set 1 (512 most abundant distinct reads, error variants included)", top, 64) + # crude denoise: walk down by abundance, drop anything within one edit of a kept sequence + keptseqs = Vector{Vector{UInt8}}(); prev = zeros(Int32, TRUNC + 1); cur = zeros(Int32, TRUNC + 1) + for x in uniq + x[2] < 2 && break + s = tobytes(x[1]); near = false + for t in keptseqs + if edit_distance(s, t, prev, cur) <= 1; near = true; break; end + end + near || push!(keptseqs, s) + length(keptseqs) >= 384 && break + end + zoom_test(rng, "Set 2 (crude denoise: nothing within one edit of a more abundant sequence)", keptseqs, 64) +end +main(ARGS[1]) diff --git a/docs/cladistics/issue-303/zoom_real_1024.jl b/docs/cladistics/issue-303/zoom_real_1024.jl new file mode 100644 index 00000000..cb9948c2 --- /dev/null +++ b/docs/cladistics/issue-303/zoom_real_1024.jl @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Check: does the zoomed-in result from E4 survive at n = 1024? (E4 used 512 and 384.) +# usage: julia -t 4 zoom_real_1024.jl +include(joinpath(@__DIR__, "lib_tree.jl")) +function run(dir; nmax = 1024, k = 64, draws = 40) + rng = MersenneTwister(20261117); seqs = load_set(dir, nmax); n = length(seqs) + D, _ = distance_matrix(seqs); key = rand(rng, UInt64, n); one = 1 / TRUNC + full, _, fblen = nj(D, collect(1:n)) + function view(S) + inS = fill(false, n); inS[S] .= true; target = splits(full, n, key, inS, fblen) + sub, m, sblen = nj(D, S); a = splits(sub, m, key[S], fill(true, m), sblen) + t3 = [x for (x, v) in target if v >= 3one]; w = sum(values(target)) + (count(x -> haskey(a, x), keys(target)) / max(length(target), 1), sum((v for (x, v) in target if haskey(a, x)); init = 0.0) / max(w, eps()), + isempty(t3) ? NaN : count(x -> haskey(a, x), t3) / length(t3), length(t3) / max(length(target), 1)) + end + @printf("n = %d sequences, views of %d, %d draws\n %-26s %7s %9s %24s %s\n", n, k, draws, "view", "all", "weighted", ">=3 substitutions", "share of branches >=3") + for (label, pick) in (("zoomed-in neighbourhood", () -> sort(sortperm(D[rand(rng, 1:n), :])[1:k])), ("random subset", () -> sort(randperm(rng, n)[1:k]))) + rows = [view(pick()) for _ in 1:draws]; c3 = filter(!isnan, [r[3] for r in rows]) + @printf(" %-26s %6.1f%% %8.1f%% %7.1f%% (%5.1f .. %5.1f) %.1f%%\n", label, 100mean(r[1] for r in rows), 100mean(r[2] for r in rows), + 100mean(c3), 100quantile(c3, 0.025), 100quantile(c3, 0.975), 100mean(r[4] for r in rows)) + end +end +run(ARGS[1]) diff --git a/docs/cladistics/issue-303/zoom_tree.jl b/docs/cladistics/issue-303/zoom_tree.jl new file mode 100644 index 00000000..8b4a7565 --- /dev/null +++ b/docs/cladistics/issue-303/zoom_tree.jl @@ -0,0 +1,231 @@ +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# Experiment E2: "thumbnail then zoom" tree reconstruction from distance QUERIES. +# +# Input model: the distance matrix is never materialised. d(i,j) is an oracle call +# (in practice: compare two sequences), and we count distinct calls. +# Contract: the distances are an additive tree metric with positive integer edge lengths +# (exact arithmetic, no floats). On such input canonical neighbour joining returns the +# true tree, so "same tree as NJ" and "the true tree" coincide; the NJ baseline below +# checks that on small n. +# +# Method: insert leaves one at a time. The current tree is the thumbnail; to place a new +# leaf, pick a centroid of the region still in play, ask <= degree(centroid) distances to +# learn which side of the centroid the leaf hangs on, discard the rest, repeat (zoom). +using Random, Printf + +# ---------- trees ---------- +mutable struct Tree + adj::Vector{Vector{Tuple{Int,Int}}} # vertex -> [(neighbour, edge length)] + leafid::Vector{Int} # vertex -> leaf label, 0 for internal +end +Tree() = Tree(Vector{Vector{Tuple{Int,Int}}}(), Int[]) +function addvertex!(t::Tree, leaf = 0); push!(t.adj, Tuple{Int,Int}[]); push!(t.leafid, leaf); length(t.adj); end +function addedge!(t::Tree, u, v, w); push!(t.adj[u], (v, w)); push!(t.adj[v], (u, w)); end +function deledge!(t::Tree, u, v); filter!(e -> e[1] != v, t.adj[u]); filter!(e -> e[1] != u, t.adj[v]); end +nverts(t::Tree) = length(t.adj) + +"Random binary tree on n leaves. shape = :yule (random splits) or :caterpillar (worst depth)." +function random_tree(rng, n; shape = :yule, wmax = 9) + t = Tree(); a = addvertex!(t, 1); b = addvertex!(t, 2); addedge!(t, a, b, rand(rng, 1:wmax) + rand(rng, 1:wmax)) + edges = [(a, b)]; last_leaf = b + for k in 3:n + (u, v) = shape == :caterpillar ? (first(t.adj[last_leaf])[1], last_leaf) : edges[rand(rng, 1:length(edges))] + w = first(e[2] for e in t.adj[u] if e[1] == v) + # an edge of length 1 cannot be split into two positive integers: lengthen it first + if w < 2; deledge!(t, u, v); w = 2; addedge!(t, u, v, w); end + w1 = rand(rng, 1:w-1) + deledge!(t, u, v); m = addvertex!(t); x = addvertex!(t, k) + addedge!(t, u, m, w1); addedge!(t, m, v, w - w1); addedge!(t, m, x, rand(rng, 1:wmax)) + filter!(e -> e != (u, v) && e != (v, u), edges); push!(edges, (u, m), (m, v), (m, x)); last_leaf = x + end + t +end + +"Distances from src to every vertex (iterative, tree)." +function dists_from(t::Tree, src) + d = fill(-1, nverts(t)); d[src] = 0; st = [src] + while !isempty(st) + u = pop!(st) + for (v, w) in t.adj[u]; d[v] < 0 && (d[v] = d[u] + w; push!(st, v)); end + end + d +end + +# ---------- oracle with query counting ---------- +mutable struct Oracle + truth::Tree; leafvert::Vector{Int}; rows::Dict{Int,Vector{Int}}; asked::Set{Tuple{Int,Int}} +end +function Oracle(t::Tree, n) + lv = zeros(Int, n); for v in 1:nverts(t); t.leafid[v] > 0 && (lv[t.leafid[v]] = v); end + Oracle(t, lv, Dict{Int,Vector{Int}}(), Set{Tuple{Int,Int}}()) +end +function query!(o::Oracle, i, j) + i == j && return 0 + push!(o.asked, minmax(i, j)) + row = get!(() -> dists_from(o.truth, o.leafvert[i]), o.rows, i) # cache is bookkeeping, not a query + row[o.leafvert[j]] +end + +# ---------- zoom insertion ---------- +"A leaf reachable from `nb` without passing through `from`; also its distance from `from`." +function leaf_beyond(t::Tree, from, nb, w0) + prev, cur, dist = from, nb, w0 + while t.leafid[cur] == 0 + (nx, w) = first(e for e in t.adj[cur] if e[1] != prev) + prev, cur, dist = cur, nx, dist + w + end + (t.leafid[cur], dist) +end + +"Vertices of the region reached from edge (c -> start), never expanding through blocked vertices." +function region(t::Tree, blocked::BitVector, c, start) + order = [c, start]; parent = Dict(c => 0, start => c) + if !blocked[start] + i = 2 + while i <= length(order) + u = order[i]; i += 1 + (blocked[u] && u != start) && continue + for (v, _) in t.adj[u] + haskey(parent, v) && continue + parent[v] = u; push!(order, v) + end + end + end + order, parent +end + +"Centroid of a region among its unblocked internal vertices, or 0 if the region is one edge." +function centroid(t::Tree, blocked::BitVector, order, parent) + size = Dict(v => 1 for v in order) + for v in Iterators.reverse(order); p = parent[v]; p != 0 && (size[p] += size[v]); end + total = length(order); best = 0; bestcost = typemax(Int) + for v in order + (blocked[v] || t.leafid[v] > 0) && continue + cost = total - size[v] + for (u, _) in t.adj[v]; get(parent, u, -1) == v && (cost = max(cost, size[u])); end + cost < bestcost && ((best, bestcost) = (v, cost)) + end + best +end + +"Insert leaf x into the reconstruction `t` using oracle queries only. Returns levels zoomed." +function insert_leaf!(t::Tree, o::Oracle, x, blocked::BitVector) + fill!(blocked, false); length(blocked) < nverts(t) + 2 && resize!(blocked, 2nverts(t) + 8) ; blocked[nverts(t)+1:end] .= false + c0 = 1; start = first(t.adj[c0])[1]; levels = 0 + order, parent = region(t, blocked, c0, start) + while true + c = centroid(t, blocked, order, parent) + if c == 0 # region is a single edge u-v: place x on it + u, v = order[1], order[2] + w = first(e[2] for e in t.adj[u] if e[1] == v) + (a, da) = t.leafid[u] > 0 ? (t.leafid[u], 0) : leaf_beyond(t, v, u, 0) |> r -> (r[1], r[2]) + # da = distance from u to leaf a (0 if u is the leaf itself) + if t.leafid[u] == 0 + nb = first(e for e in t.adj[u] if e[1] != v); (a, da) = leaf_beyond(t, u, nb[1], nb[2]) + end + (b, db) = t.leafid[v] > 0 ? (t.leafid[v], 0) : + (nb = first(e for e in t.adj[v] if e[1] != u); leaf_beyond(t, v, nb[1], nb[2])) + dax = query!(o, a, x); dbx = query!(o, b, x); dab = da + w + db + g2 = dax + dab - dbx; @assert iseven(g2); pos = g2 ÷ 2 - da + @assert 0 < pos < w "leaf attaches at a vertex: not a binary tree with positive edges" + deledge!(t, u, v); m = addvertex!(t); nx = addvertex!(t, x) + addedge!(t, u, m, pos); addedge!(t, m, v, w - pos); addedge!(t, m, nx, dax - g2 ÷ 2) + return levels + end + levels += 1 + nbs = t.adj[c] + (a, D) = leaf_beyond(t, c, nbs[1][1], nbs[1][2]); dax = query!(o, a, x) + (l2, D2) = leaf_beyond(t, c, nbs[2][1], nbs[2][2]); d2x = query!(o, l2, x) + g2 = dax + (D + D2) - d2x; @assert iseven(g2); g = g2 ÷ 2 # dist(a, branch point of x on a..l2) + side = 0 + if g < D; side = 1 + elseif g > D; side = 2 + else + for j in 3:length(nbs) + (lj, Dj) = leaf_beyond(t, c, nbs[j][1], nbs[j][2]); djx = query!(o, lj, x) + h2 = d2x + (D2 + Dj) - djx + if h2 ÷ 2 > D2; side = j; break; end + end + @assert side != 0 "leaf attaches exactly at an existing vertex (multifurcation)" + end + blocked[c] = true + order, parent = region(t, blocked, c, nbs[side][1]) + end +end + +function reconstruct(o::Oracle, n, perm) + t = Tree(); a = addvertex!(t, perm[1]); b = addvertex!(t, perm[2]); addedge!(t, a, b, query!(o, perm[1], perm[2])) + blocked = falses(4n); maxlev = 0 + for k in 3:n; maxlev = max(maxlev, insert_leaf!(t, o, perm[k], blocked)); end + t, maxlev +end + +# ---------- comparison: bipartitions with lengths ---------- +function splits(t::Tree, keys::Vector{UInt64}) + root = findfirst(>(0), t.leafid); order = [root]; parent = Dict(root => (0, 0)); i = 1 + while i <= length(order) + u = order[i]; i += 1 + for (v, w) in t.adj[u]; haskey(parent, v) || (parent[v] = (u, w); push!(order, v)); end + end + h = Dict(v => (t.leafid[v] > 0 ? keys[t.leafid[v]] : UInt64(0)) for v in order) + for v in Iterators.reverse(order); p = parent[v][1]; p != 0 && (h[p] ⊻= h[v]); end + all = h[root] + Set((min(h[v], all ⊻ h[v]), parent[v][2]) for v in order if parent[v][1] != 0) +end + +# ---------- canonical neighbour joining (baseline, exact rationals, O(n^3)) ---------- +function nj_splits(D::Matrix{Rational{BigInt}}, keys) + n = size(D, 1); active = collect(1:n); hk = Dict(i => keys[i] for i in 1:n); all = reduce(⊻, keys) + D = copy(D); out = Set{UInt64}(); nxt = n + Dd = Dict((i, j) => D[i, j] for i in 1:n, j in 1:n) + while length(active) > 3 + r = length(active); tsum = Dict(i => sum(Dd[(i, k)] for k in active) for i in active) + best = nothing; bq = nothing + for x in 1:r, y in x+1:r + i, j = active[x], active[y]; q = (r - 2) * Dd[(i, j)] - tsum[i] - tsum[j] + (bq === nothing || q < bq) && ((best, bq) = ((i, j), q)) + end + (i, j) = best; nxt += 1; u = nxt; hk[u] = hk[i] ⊻ hk[j]; push!(out, min(hk[u], all ⊻ hk[u])) + for k in active + (k == i || k == j) && continue + duk = (Dd[(i, k)] + Dd[(j, k)] - Dd[(i, j)]) / 2; Dd[(u, k)] = duk; Dd[(k, u)] = duk + end + Dd[(u, u)] = 0 // 1; filter!(k -> k != i && k != j, active); push!(active, u) + end + out +end + +# ---------- runs ---------- +function run(rng, n, shape; check_nj = false) + truth = random_tree(rng, n; shape); o = Oracle(truth, n) + rec, maxlev = reconstruct(o, n, randperm(rng, n)) + keys = rand(rng, UInt64, n); st = splits(truth, keys); sr = splits(rec, keys) + rf = length(symdiff(st, sr)) # 0 means identical topology AND edge lengths + njok = missing + if check_nj + lv = o.leafvert; D = [Rational{BigInt}(dists_from(truth, lv[i])[lv[j]]) for i in 1:n, j in 1:n] + topo(s) = Set(k for (k, _) in s if count_ones(0) == 0) # keys only + internal = Set(k for (k, _) in st) ; leafkeys = Set(min(keys[i], reduce(⊻, keys) ⊻ keys[i]) for i in 1:n) + njok = issubset(nj_splits(D, keys), setdiff(internal, leafkeys)) + end + (queries = length(o.asked), full = n * (n - 1) ÷ 2, rf = rf, maxlev = maxlev, nj = njok) +end + +rng = MersenneTwister(20261116) +println("== correctness: reconstruction equals the true tree (splits and edge lengths), and NJ agrees ==") +for shape in (:yule, :caterpillar), n in (8, 16, 32, 64) + bad = 0; njbad = 0 + for _ in 1:20 + r = run(rng, n, shape; check_nj = true); bad += (r.rf != 0); njbad += (r.nj !== true) + end + @printf("%-12s n=%3d 20 trees: mismatches vs truth %d, NJ joins not in the true tree %d\n", shape, n, bad, njbad) +end + +println("\n== scaling: distance queries used vs the full matrix ==") +@printf("%-12s %7s %12s %14s %9s %12s %8s %6s\n", "shape", "n", "queries", "full matrix", "fraction", "q/(n log2 n)", "levels", "RF") +for shape in (:yule, :caterpillar), n in (256, 1024, 4096, 16384) + t0 = time(); r = run(rng, n, shape); dt = time() - t0 + @printf("%-12s %7d %12d %14d %8.3f%% %12.3f %8d %6d (%.1fs)\n", shape, n, r.queries, r.full, + 100r.queries / r.full, r.queries / (n * log2(n)), r.maxlev, r.rf, dt) +end