Skip to content

osbench: the benches are each other's LAN peer, and the internet is an explicit evidence run - #1065

Merged
Japabu merged 5 commits into
mainfrom
bench-net
Oct 24, 2026
Merged

Japabu merged 5 commits into
mainfrom
bench-net

Conversation

@Japabu

@Japabu Japabu commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

osbench's lan rows assumed a peer on "any host on the link", which in practice meant the development Mac. The Mac travels and stays on Wi-Fi, so it can't serve as a reference. The lan group also ran inside the full suite's nine-minute boot, so every full-suite run needed a peer. Finally, nothing measured the internet path, and nothing turned repeated runs into a comparison with a stated noise floor.

Resulting behaviour

  • The benches are each other's peer. One bench runs the system under test, ToyOS or Linux. The other bench runs osbench-peer under Linux as the fixed reference, and the roles swap for the other direction. The peer's discovery answer says which of its links it answered over (kind, driver, and wired speed or Wi-Fi signal). The bench records that as lan.peer_link beside its own lan.link. No address, MAC or interface name is printed. Nothing depends on the Mac: every figure is taken on the clock of the bench under test.

  • lan and internet run only by name. They are out of osbench all and the full-suite boot, which no longer needs the other bench; its list bound drops by lan's 110 s, from 540 s to 430 s. Each group has a named-only metal row, osbench_lan and osbench_internet. When one pass names both rows they share one netbenchcase boot of benchcase.toml.

  • LAN measurement. The same four rows as before: tcp.connect (50 samples), tcp.rtt.64 (2,000 samples), and upload and download, now five timed 3 s rounds after one unread round. Linux link reading moves into osbench-lan::link so the peer can use it too. An address on an interface with no device is now refused instead of being called wired.

  • Internet rows are exploratory evidence, not regression tests. The rows are internet.tcp.connect, internet.https.rtt (cdn-cgi/trace on a kept connection), and internet.https.download and internet.https.upload (four 10 MB transfers each way, each timed whole in ns; the first session's 25 MB downloads drew Cloudflare's 429 rate limit). The endpoint is speed.cloudflare.com, Cloudflare's speed test: the endpoints Cloudflare's own measurement engine drives (https://github.com/cloudflare/speedtest). The user agent is toyos-build (https://github.com/ToyOSOrg/ToyOS). TLS is ureq and rustls on toyos-crypto's provider on both systems, so the cryptography is the same code. The answers name the client's address and location, so nothing reads or prints a header or body.

  • osbench-compare is a new host program and replaces the earlier scratch Python tables. It reads run files, either a Linux run's stdout or a raw ToyOS boot log. For each machine and benchmark it prints:

    • each system's per-run p50s;
    • each system's median and run-to-run range, (max−min)/median. Linux's range describes the path's noise; the rank test, not the range, decides;
    • the ratio ToyOS/Linux;
    • the exact two-sided Mann–Whitney p over the runs, with ties as half wins, and the verdict at 0.05.

    The least p is read from the same exact distribution, so ties count. With four runs against four and no ties it is 2/70 = 0.029; with three against four it is 2/35 = 0.057, unless one system's runs tie. A row whose counts cannot reach 0.05 says "too few runs to resolve (least p …)". Better or worse is judged only for ns and MB/s; other units get "ToyOS higher/lower". Each p is per row and uncorrected for the table's other rows. internet.* and lan.* rows are compared only across runs that agree on the kind of link (and, for LAN, the peer's driver).

Links and paths (from the earlier Linux identity runs; the run plan gives read-only commands to confirm them)

path under test peer ToyOS Linux
A T14, wired I219 (e1000e) AMD Linux, Wi-Fi AX200 measured measured
A′ T14, Wi-Fi AX210 AMD Linux, Wi-Fi lan.wifi.* unsupported: no Wi-Fi driver not run: the T14's Wi-Fi is down and the benches stay as they are
B AMD, Wi-Fi AX200 T14 Linux, wired every lan.* and internet.* row unsupported: ToyOS drives no card the AMD bench has measured

The AMD bench has no Ethernet, so path A has a Wi-Fi hop at the far end. Linux's runs on it give its ceiling and its run-to-run range; whether a ToyOS–Linux difference is resolved is the rank test's call, not the range's. In the first session the Wi-Fi hop did not limit the path: Linux moved 113 MB/s both ways, the gigabit link's rate.

The first paired session (coordinator-run, at d99ef5a)

Evidence, masked: https://github.com/ToyOSOrg/evidence/tree/cd7124802c7be399d17636d516c2a3a5a6e1585f/r3 (the session's files, compare-r3.md, and a README with the timeline).

row, T14 path A Linux (4 runs) ToyOS (3 runs) ToyOS/Linux exact p verdict
lan.wired.tcp.download 112, 110, 109, 112 MB/s 112, 112, 109 MB/s 1.01 1.000 parity
lan.wired.tcp.rtt.64 2.02 ms median 1.99 ms median 0.99 0.057 too few runs: ToyOS 1.4% lower in every run, unresolved at 3 against 4; the Wi-Fi hop likely dominates
lan.wired.tcp.upload 113, 113, 114, 113 MB/s 41, 48, 31 MB/s 0.36 0.029 ToyOS worse, no round overlapping
lan.wired.tcp.connect 1.46 ms median 2.19 ms median 1.50 0.057 too few runs (Linux range 58%)
internet.tcp.connect 6.09 ms 7.01 ms 1.15 0.057 exploratory
internet.https.rtt 8.55 ms 7.58 ms 0.89 0.057 exploratory
internet.https.upload (10 MB) 127 ms 178 ms 1.40 0.057 exploratory
internet.https.download (25 MB) 429 in every run 244 ms - - Linux unmeasured

Noise floor on path A, from Linux's run-to-run range: round trip 0.5%, upload 0.9%, download 2.7%, connect 58%.

Path B, Linux only (the AMD bench over its Wi-Fi to the T14's wired peer): 114 MB/s down, 110 MB/s up, 2.00 ms round trip. ToyOS drives no card the AMD bench has.

Caveats:

  • ToyOS has 3 runs, not 4. These are T2–T4 of the first attempt. T1's readback was lost.
  • The first attempt lost Linux L2–L4. The binaries were staged in /tmp, and the reboots wiped them.
  • The Linux runs are not interleaved with ToyOS's. In the rerun the T14's two identical sticks swapped places after a reboot, so the runner refused the ToyOS boots. The Linux runs are from 10–15 minutes later.
  • Every Linux internet.https.download got HTTP 429. The downloads that ran first succeeded: ToyOS's three and the first attempt's Linux L1 (257 ms per 25 MB). The 429s began after at least four runs of six 25 MB downloads from one address. The code now moves 40 MB down per run instead of 150 MB.

Findings recorded:

  • issues/netstack-sends-tcp-at-a-third-of-what-the-t14s-wired-link-carries.md: new defect.
    • Every ToyOS run's slowest round was 31 MB/s and its fastest 58–59 MB/s, with p50s of 41, 48 and 31 MB/s.
    • The cause is unmeasured, and the candidates are named.
    • Linux sent 113 MB/s at the same ~1.5 GHz clock readings, so the clock does not explain the gap.
  • issues/toyos-uses-what-modern-hardware-offers-for-speed.md, the T14's clock-ceiling record: cpu.clock's maximum was 3789 MHz under ToyOS against 4188–4190 MHz under Linux. Its p50, ToyOS 1.5–3.2 against Linux 3.8–4.2 GHz, reads idle and ramp policy, not the ceiling.

Checks

Evidence r1–r4 on branch pr-1065: https://github.com/ToyOSOrg/evidence/tree/15491ecbd03fd217ca4a31ae30ebbe3ed1f35c2f (r4 corrects r3's README).

  • cargo run -- --ci host at ec8b41d: exit 0, 103 green. 48c9d05 after it changes only issue records and one doc comment. At 548bef4, the last code change: exit 0. Earlier heads: c3b124b and d99ef5a were green; e60ee7a88 was red on one clippy lint, which I fixed.
  • Linux musl build at 548bef4: exit 0.
  • QEMU TCG smoke of osbench internet on ToyOS, at round 1's code; the later edits are constants and the comparison tool:
    • with a NIC, every row measured;
    • with no NIC, every row unsupported.
  • Negative controls, each a checked patch, applied and restored. Each turned the tests red:
    • m1: ties counted as whole wins. It passed the first time; I added a discriminating case, and it then failed.
    • m2: an unbounded answer length.
    • m3: the better/worse direction flipped.
    • m4: every unit judged better or worse.
    • m5: no least-p gate.
    • m6: a range over a median of 0.
    • m7: the least p computed from the counts alone.
    • m8: a tied complete separation left unresolved. This one is broad.
  • Not run since the session:
    • a metal boot of 548bef4. Its guest-side change is the download size and count, and the session at d99ef5a already exercised that path;
    • any Linux run of the smaller downloads.

Unresolved risks

  • The comparison is 3 ToyOS runs against 4 Linux runs, not interleaved.
    • Only the upload difference resolves, and only under the centre-distance two-sided rule, which osbench-compare names (doubling the one-sided p would give 2/35). Its strength is that no ToyOS round reached any Linux round.
    • The other rows read "too few runs". Their Linux ranges are tight, so the round trip and download are at parity in fact, but the test cannot say so.
    • A fourth ToyOS run would let rows without ties resolve at 2/70. None is scheduled.
  • No Linux download under the smaller volume has run. Whether 40 MB per run stays under Cloudflare's unpublished limit is untested.
  • The internet rows depend on Cloudflare's speed test keeping its current behaviour: __up accepting POSTs, and cdn-cgi/trace answering at the edge.
  • Not measured, by name:
    • the T14's lan.wifi.* under ToyOS (there is no Wi-Fi driver; issues/there-is-no-wifi.md);
    • every AMD bench lan.* and internet.* row under ToyOS;
    • the T14's Wi-Fi path under Linux (its Wi-Fi is down as the bench stands).

🤖 Generated with Claude Code

…n explicit evidence run

The development Mac moves and is on Wi-Fi, so it is no peer and no part of
a measurement. `osbench lan` now runs on one bench against the
`osbench-peer` the other bench's Linux runs as a fixed reference, each way;
ToyOS against Linux on one path is the comparison, and Linux's own runs give
the path's ceiling and noise.

- lan: the peer's answer says which of its links it answered over (kind,
  driver, and speed or Wi-Fi signal), refused past 160 printable bytes; the
  bench records it as `lan.peer_link`. Link reading moves from osbench's
  Linux layer into `osbench-lan::link`, so the peer uses it too, and an
  address on an interface with no device is refused rather than called
  wired. Transfers are five timed 3 s rounds after one unread.
- `lan` and the new `internet` leave `all` and the full suite's boot: they
  need another party, so each runs only by name, and the full suite no longer
  needs the other bench. Its list bound drops by `lan`'s 110 s.
- internet: TCP connect, an HTTPS round trip on a kept connection, 25 MB
  downloads and 10 MB uploads against Cloudflare's speed test
  (speed.cloudflare.com, the endpoints Cloudflare's own engine drives), over
  ureq and rustls on toyos-crypto's provider on both systems. Answers'
  headers and bodies name the client's address; nothing reads or prints
  them. A ToyOS netstack answers NotConnected until its lease lands and
  announces none, so the lookup is asked again each second for up to 60 s.
- Metal rows `osbench_lan` and `osbench_internet`, named-only, on one
  `netbenchcase` boot of benchcase.toml when a pass names both.
- osbench-compare: reads run files (a Linux run's stdout or a ToyOS boot's
  log) and prints per machine and benchmark each system's per-run p50s,
  median, run-to-run range, ToyOS/Linux and the exact two-sided
  Mann-Whitney p over the runs, with ties as half wins. LAN rows whose runs'
  peers were on different links are not compared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Review r1 of #1065 at c3b124b

Architecture

The ownership is right. The bench under test does all the timing on its own clock. The peer is a fixed Linux reference on the other bench and does no more than answer. Link reading now lives in osbench-lan::link, the one crate both ends already share. lan and internet run only by name, through BY_NAME and named_only rows, so the full-suite boot no longer needs a second party. The development Mac is gone from the code, the comments, the issue record and the run plan: no path still assumes a peer "on any host".

The statistical unit is right too. Each value is one run's p50, so there is no pseudo-replication over samples. The test is the exact two-sided permutation (Mann–Whitney) test: it enumerates every split of the pooled values, ties count half, and the distribution is conditional on the ties observed. I checked the table values by hand: 2/70, 0.1, 2/252, 4/70, and the half-win case [2,2] against [1,2].

The new code is about 280 production lines in osbench-compare, 136 in internet.rs and 125 in link.rs, which is mostly moved code. It adds no new crate: ureq, rustls, getrandom and webpki-roots are already in the lock.

Privacy passes.

  • The discovery answer is a kind, a driver and a speed or signal, filtered to printable ASCII and capped at 160 bytes. Its error strings carry no address.
  • lan.peer_link and internet.link name no interface, address or MAC.
  • internet.rs sends every body to io::sink. Headers stay inside ureq. A status failure prints only its code.
  • In the published evidence (r1), I grepped every file for IPv4 addresses, MACs, home paths and SSIDs. Nothing matched.

Endpoint. Cloudflare publishes cloudflare/speedtest so that third parties can drive these endpoints. The use here is limited:

  • the user agent names the project;
  • one run is about 210 MB;
  • the rows are reachable only through a named-only metal row or osbench internet typed by hand;
  • no host test, QEMU registration or all touches it.

This is consent-appropriate and stays outside automation.

Registration. Both new rows say which machines they run on (the T14, and why the AMD bench is excluded). Nightly does not run metal, so taking lan out of the full suite removes a peer dependency without hiding anything. The issue record states that the rows run by name.

Findings

BLOCKER tests/guest/osbench/compare/src/lib.rs:176-179 — lower_is_better treats only ns as lower-is-better, so the verdict's better/worse direction is wrong for most of osbench's other units.

  • osbench also emits mW (energy.rs:77), us (tail.rs:141), C, seconds, permille and MHz.
  • With the current code, higher idle package power reads "ToyOS better", and a cpu.clock difference gets a better/worse verdict that means nothing.
  • The module and the PR present osbench-compare as the general comparison of osbench runs, the replacement for the scratch tables over the whole suite. It is not limited to lan and internet.
  • Control m3 covers only ns and MB/s.
  • Fix: report the direction without judging it ("ToyOS higher" / "ToyOS lower"). Alternatively, judge only units whose direction is known, ns and MB/s, and print the bare direction for the rest. One host case with mW holds it.

BLOCKER tests/guest/osbench/compare/src/lib.rs:255-262 — "not resolved" is printed even when the run counts cannot reach 0.05 at all.

  • With 4 runs against 3, complete separation gives p = 2/C(7,3) = 2/35 = 0.057. With 3 against 3 it gives 0.1. Both are reported as "not resolved", the same words as overlapping runs.
  • The < 2 guard is arbitrary. The real limit is the least p the counts can attain.
  • On the first metal session, one refused ToyOS run of a row is enough to turn a clean separation into "not resolved". That would record a gap in the design as a null finding.
  • Fix: when 2 / C(n+m, n) > RESOLVED, print "too few runs to resolve (least p …)". Add one host case at 4 against 3.

BLOCKER tests/toyos.rs:1638-1653 — the bound arithmetic does not add up.

  • OSBENCH_LAN's comment counts the list's base 60 s as its own ("150 s leaves it 30 s"). OSBENCH_INTERNET adds only 60 s for a group that "measures for under 95 s".
  • In the shared netbenchcase boot the plan uses, the bound is 210 s. The worst case is lan's 60 s discovery, about 57.5 s of measuring, then internet's 95 s, which is about 212 s. So "leaves it 30 s" is false.
  • Named alone, internet gets 120 s for 95 s of measuring plus up to 60 s of LEASE retries.
  • The 3 MB/s download floor its span assumes for ToyOS TLS on metal has not been measured. QEMU TCG moved 0.2 MB/s. Label it an assumption.
  • Fix: give internet its own span plus a lease margin in adds_ms, and correct both comments. A hang ceiling sized below a legitimate run wastes a coordinated metal session.

BLOCKER (evidence) — the PR's material claims have not been run yet. These are:

  • ToyOS discovers the AMD peer across the T14's I219 and the AP's Wi-Fi;
  • ToyOS measures lan.wired.* and internet.* on metal;
  • the moved Linux link::of / toward code works on both benches;
  • four interleaved pairs on path A give a comparison.

The draft states this gap honestly, and run-plan.md is the right check: L1 T1 … L4 T4, then four AMD Linux runs, then osbench-compare. Landing needs those readbacks, the Linux .txt/.err files, the peer logs and the comparison table, published with the same masking. The image build in the evidence README predates the commit. The metal staging at c3b124b rebuilt it, so that is covered.

SUGGESTION compare/src/lib.rs table — each p is per row and uncorrected. Path A will compare about 8 measured rows at 4 against 4. If there were no real differences, one of them would still read "resolved" about one time in five (1−(1−2/70)^8; the rows are correlated, so this is rough). One line under each table saying "p per row, uncorrected" keeps the verdict honest without needing more runs.

SUGGESTION (PR body, for the orchestrator) — "a ToyOS–Linux difference smaller than Linux's run-to-run range on that path cannot be resolved" is not what the tool does. Complete separation can occur with a median difference below Linux's (max−min). The range is descriptive and the rank test decides. Say that instead.

SUGGESTION compare/src/lib.rs:127-130 — range divides by a median of 0. A permille p50 of 0 is plausible and would print inf% or NaN%.

SUGGESTION compare/src/lib.rs:207 — internet.* rows get no equivalent of the lan.peer_link check. If one Linux run left by a different link (it should not, since the T14's Wi-Fi is down), nothing would flag it. internet.link's kind could gate the row the way lan.peer_link does.

SUGGESTION tests/guest/osbench/src/internet.rs:116-136 — SystemRandom and the X25519-only provider are now a third copy, after build/release.rs and apps/doom/build.rs. A curve policy tied to one issue now has to be changed in three places when that issue closes. Exposing it from toyos-crypto would remove the copies. This is optional and outside this PR's subject.

SEND BACK

…ns are too few, and the net rows' bounds add up

- A verdict says better or worse only for `ns` (lower) and `MB/s`
  (higher); any other unit, a power or a clock, is told which way ToyOS
  differs. Counts whose complete separation cannot reach 0.05, 4 runs
  against 3 among them, say "too few runs to resolve (least p ...)"
  rather than "not resolved". A median of 0 has no range. Each table says
  its p is per row and uncorrected.
- `internet.*` rows are compared only across runs that left by one kind of
  link, as `lan.*` rows are across one kind of peer link. ToyOS's link
  description now starts with its kind, `wired`.
- The network rows' bounds: each arm adds its own worst span and the
  list's 60 s stays the boot's margin. `lan` 120 s (60 s discovery, under
  60 s measuring); `internet` 155 s (60 s for a lease, under 95 s at an
  assumed 3 MB/s down and 1.5 MB/s up, which no metal reading confirms).
  The shared boot's bound is 335 s.
- The crypto issue's consumer table names osbench's guest HTTPS client and
  the third copy of the provider's random source and group filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Review r1 response, head d99ef5a. Evidence: https://github.com/ToyOSOrg/evidence/tree/8f27129bd4eccbbc55a4c09f177d6946d1d81ffc/r2

  • B1 (direction). The verdict now says better or worse only where the better direction is known: ns is better lower, MB/s better higher. For any other unit it says "ToyOS higher" or "ToyOS lower". New host case: mW. Control m4, which judges every unit, turns it red.

  • B2 (least p). When complete separation cannot reach 0.05 (least p = 2/C(n+m,n)), the verdict is "too few runs to resolve (least p …)". This replaces the old < 2 guard. New host cases: 4 against 3 (least p 0.057), and least_p against the exact test. Control m5 turns it red.

  • B3 (bounds). Each arm now adds its own worst span, and the list's 60 s stays the boot's margin.

    • lan: 120 s (60 s of discovery, under 60 s of measuring).
    • internet: 155 s (60 s for the lease, under 95 s of measuring).
    • The shared boot's bound is 335 s, with a 670 s deadline.

    The 3 MB/s down and 1.5 MB/s up rates are labelled as an assumption that no metal reading confirms.

  • B4 (metal evidence). The coordinator's. The run plan, the four re-staged images and the Linux binaries are at d99ef5a.

  • Suggestions taken:

    • Each table now ends "Each p is its row's alone, uncorrected for the table's other rows."
    • range returns None for a median of 0; control m6 turns that red.
    • internet.* rows are compared only across runs whose internet.link has one kind. ToyOS's description now starts with wired.
  • Curve-policy copies. Not deduplicated. The consumer table in issues/toyos-crypto-is-in-the-tree-on-par-with-ring-and-secure.md now names osbench as a guest HTTPS client and records the three copies.

cargo run -- --ci host at d99ef5a: exit 0, all 103 steps green.

Japabu and others added 2 commits October 24, 2026 14:14
… downloads stay under the endpoint's limit, and the first paired session's findings

- osbench-compare: the least attainable p is read from the same exact
  distribution as p, the farthest splits' share, rather than assumed to be
  2/C(n+m,n). A difference whose p reaches 0.05 is resolved whatever the
  counts. The session's T14 upload, three ToyOS runs against four Linux
  runs three of which tie, has p = least = 1/35, and the old count-based
  gate called it "too few runs".
- internet: the downloads are four of 10 MB rather than six of 25 MB.
  Every Linux download in the session drew Cloudflare's 429 after four
  runs of the larger volume had passed. Its bound goes to 105 s.
- issues: netstack's TCP send on the T14's I219 runs at 0.36x Linux's on the
  same path while receive is at parity (new defect); the session's
  cpu.clock readings join the idle-clock issue; the comparison issue
  points at the session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… record

The network session's cpu.clock readings were ToyOS 1.5-3.2 GHz against
Linux 3.8-4.2 GHz per group. They are the same unexplained clock as the
3,788 MHz single-thread ceiling and the 2.67 GHz download windows, so they
belong with that record in toyos-uses-what-modern-hardware-offers-for-speed.
They leave the idle-clock issue, whose subject is the gauge and not the
clock. The upload defect points at the record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review r2 of #1065 at ec8b41d

This round covers d99ef5a, 548bef4 and ec8b41d. ec8b41d changes only issue records. So ci host at 548bef4 (r3/logs/ci-host-4.log, exit 0, 103 green) still covers the code.

Architecture

Nothing changed in ownership since r1. The changes are confined to osbench-compare, two constants in internet.rs, the two arms' adds_ms, and the issue records. Since r1 the branch adds about 120 production lines (most of them in osbench-compare) and about 90 test lines. That is proportionate to the three fixes.

r1 blockers

  • B1, the direction of the judged unit: fixed.
    • better() judges only ns (lower is better) and MB/s (higher is better).
    • Every other unit gets "ToyOS higher" or "ToyOS lower".
    • The mW host case holds this, and control m4 turns it red.
  • B2, too few runs: fixed, and better than what r1 asked for.
    • least is now the share of the farthest split in the same exact distribution, not 2/C(n+m,n).
    • Because p for a split is monotone in its distance from the centre, the farthest split's share is the smallest p those pooled values can reach. So the gate is right with ties as well as without them.
    • The 4-against-3 case (0.057) and the tied case hold it. Controls m5 and m7 are red.
  • B3, the bounds: fixed. The arms sum, plus the list's 60 s (metal.rs:677, list_bound_ms).
    • lan: 60 s discovery plus about 57.5 s of measuring (21.6 s of round trips and 36 s of transfers), within its 120 s.
    • internet: 60 s lease plus about 40 s at the assumed floor (13.3 s down, 26.7 s up), plus 22 + 55 = 77 short exchanges, within its 105 s.
    • The shared boot is 285 s. On metal it measured far under that: lan took secs=54 including 13.6 s of discovery, and internet took 3–5 s.
    • The comment labels the 3 / 1.5 MB/s floor as assumed.

The least p with ties

p = least = 1/35 for the upload is correct. I enumerated the splits.

  • The observed split puts ToyOS {31, 41, 48} under all four Linux values: 2U = 0, distance 12 from the centre of 12.
  • The mirror split puts {113, 113, 114} above the rest, but 113 ties a 113 left on the other side. That gives 2U = 22, distance 10.
  • So exactly one of the 35 splits is that far out.

The test is valid as a conditional permutation test. Integer-MB/s quantization creates the ties, but that does not make it invalid.

The result does depend on the two-sided convention. The code uses |2U − mn| ≥ observed, the centre-distance definition. Doubling the one-sided p would give 2/35 here, which is not resolved. The finding's real strength is that no ToyOS round reached any Linux round, and the issue rightly leads with that. See S1.

The 40 MB download and the 105 s bound

DOWN = 10 MB and ROUNDS = 3 (one unread plus three timed) give 40 MB per run instead of 150 MB. That stays below the volume that preceded the first 429: about 750 MB across L1 and T1–T4 before 11:50.

  • ROUNDS also cuts the uploads to four. That is harmless and is what the bound assumes.
  • The DOWN => Ok(()) length check still holds.
  • Inspection is enough for a change of two constants. The record honestly says that no Linux run at the smaller volume has happened yet.
  • The 105 s bound is covered above.

B4, the evidence

Resolved for landing the program. The metal claims r1 listed are now established, at d99ef5a, whose guest code differs from the head only in the two constants:

  • ToyOS found the AMD peer across the I219 and the AP's Wi-Fi in 13.6 s (lan.peer_found_after_ms=13585, peer_link=wireless … iwlwifi).
  • ToyOS measured every lan.wired.* and internet.* row in three boots, all judged passed.
  • The moved link::of code reads wired e1000e 1000 on the T14 and wireless iwlwifi on the AMD bench, as both bench and peer (step0 agrees).
  • osbench-compare produced the table. compare-r3.md, made at the head, matches the README's figures.

The interleaving gap weakens the measured findings, not the program. Two of those findings matter.

  • Upload: it survives the gap.
    • The ratio is 0.36x, and no round overlaps.
    • In the same boots, ToyOS's download ran at 112 MB/s. That shows the path was healthy while ToyOS ran.
    • Linux's later upload readings (range 0.9%) agree with path B.
    • A confound would have to hit only ToyOS's upload direction, and only for five minutes.
  • The other rows: they are labelled "too few runs" or exploratory. That is the right label.
  • The swapped sticks: they are an identity-guard event. They do not change a network workload.
  • The 429: it hit only Linux's downloads, and the row is recorded as unmeasured rather than compared.

Nothing more is owed on metal. A fourth interleaved ToyOS run is a follow-up for the findings, not a condition of landing.

Privacy

Privacy passes. I grepped every file under r3/ for the following:

  • IPv4 and IPv6 addresses;
  • MACs;
  • /Users, /home and /root;
  • SSID, hostname and .local/.lan names;
  • user names.

The only matches are @_2.40GHz in the CPU model and an SSID test name in the CI log. The conditions carry kind, driver, speed or signal, and the distro and kernel, nothing identifying. The ToyOS logs are cut to osbench's and the runner's lines.

Findings

BLOCKER issues/netstack-sends-tcp-at-a-third-of-what-the-t14s-wired-link-carries.md:26,42-44, issues/toyos-uses-what-modern-hardware-offers-for-speed.md:73-77 — the records state three things the evidence does not show.

  1. "ToyOS's rounds cluster near 31 and 58 MB/s" (line 26). osbench prints only min, p50, max and mean, and those rule the cluster out.
    • Run 2 is min=31 p50=41 max=58 mean=43, so its rounds are 31, x, 41, y, 58 with x + y = 85.
    • Run 3 is 31, ≤34, 48, ≥48, 59.
    • Only run 4 (31, 31, 31, 33, 59) looks bimodal. Two of the three p50s, 41 and 48, are near neither value.
    • "Suggests a cap" then steers whoever investigates first.
    • Fix: say what the data show (every run's min 31, max 58–59, p50s 41, 48 and 31), or delete the bullet.
  2. "The sending CPU's clock read 1.6 to 3.8 GHz … it is the T14's clock ceiling" (lines 42-44).
    • Readings of 1.6 GHz are not the ceiling.
    • The record leaves out the reading that bears directly on the question. Linux's uploads, which moved 113 MB/s, read mhz=1495 after every run, and mhz_settled of 1495 or 4186. ToyOS's read mhz of about 1595, and mhz_settled of 1595, 3787 or 1794.
    • Both systems sent at the same clock readings, so the clock does not separate them. At most, the clock matters through ToyOS's higher cost per byte.
    • Fix: state both systems' readings and drop "it is the T14's clock ceiling".
  3. "read the same" (toyos-uses…:74). The record's paragraph is about ToyOS's effective clock under load: 2.67 GHz over the download windows.
    • cpu.clock's p50 is 200 probes from idle, the gauge that osbench-times-cpus-leaving-idle… says reads idle and ramp policy.
    • The comparable part of this session is the ceiling: cpu.clock max was 3789 MHz under ToyOS in every run that reached it, against 4186–4190 under Linux.
    • Fix: name the gauge and give the max beside the p50. Alternatively, keep the p50 sighting with the gauge's own issue.

The published r3/README.md still points the cpu.clock readings at osbench-times-cpus-leaving-idle-and-not-their-settled-clock.md, which ec8b41d emptied of them. It needs a one-line correction in the evidence repository.

SUGGESTION S1 tests/guest/osbench/compare/src/lib.rs:137-142 — name the two-sided convention ("p is the share of splits whose |2U − mn| is at least the observed"). The only resolved row depends on it: doubling the one-sided p would give 2/35. Then replace "runs tied apart from the other system's can lower it", which is hard to parse, with the mechanism: ties can make the mirror-image split less extreme, so fewer splits reach the farthest distance.

SUGGESTION tests/guest/osbench/src/internet.rs:29-31 — "answers 429 past a download volume" is an inference. The session cannot tell a volume limit from a rate limit or a time window. "A volume or rate it does not publish" is no longer and is true.

SUGGESTION (PR body and no-published-run-compares…:41-43, for the orchestrator) — four corrections:

  • The 64 B round trip is not "at parity in fact". The two sets separate completely: ToyOS ran 1.984–1.994 ms and Linux 2.010–2.020 ms, so ToyOS was 1.4% lower in every run, and the result is unresolved at 3 against 4.
    • Every path, both systems and path B included, sits at about 2.00 ms with ranges of 0.2–0.5%. That suggests the Wi-Fi hop's fixed latency dominates this row on path A. If so, the row says little about either stack's latency.
    • "Within 1.4%, unresolved" is the honest reading.
  • The PR body's "Findings recorded" bullet is stale since ec8b41d. It still names the idle-clock issue.
  • The PR body repeats the "cluster" claim.
  • The README gives ToyOS's window as 11:44–11:49. That is when the readbacks completed. The boots' own lines run 11:42–11:48. This is immaterial to the comparison.

SEND BACK

…act test names its two-sided rule

- netstack upload defect: the "cluster near 31 and 58 MB/s" claim goes; the
  runs' min, p50 and max do not show it. The clock paragraph now gives both
  systems' readings: Linux sent 113 MB/s at the same ~1.5 GHz readings, so
  the clock does not explain the gap.
- The T14's clock-ceiling record compares cpu.clock's maximum: 3789 MHz
  under ToyOS against 4188-4190 under Linux. It no longer compares the
  probes' idle-ramp p50.
- osbench-compare's doc names its two-sided p, the share of splits whose
  |2U - nm| is at least the observed one, and says how ties lower the least p.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu
Japabu marked this pull request as ready for review October 24, 2026 12:31
@Japabu
Japabu added this pull request to the merge queue Oct 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 24, 2026
@Japabu
Japabu merged commit aeba59f into main Oct 24, 2026
8 checks passed
@Japabu
Japabu deleted the bench-net branch October 24, 2026 13:27
Japabu added a commit that referenced this pull request Oct 27, 2026
The C-state track is assigned: it is being built on branch deep-idle,
and that branch's PR carries its design. The AML track's 2026-10-12
ordering ruling is marked superseded, because the owner's direction of
2026-10-27 releases the plan's order. The track's exit gains a cpu.clock
reading, which the orchestrator accepted under that direction. The exit
names its statistic and scope: the maximum at ratio 42 in every group
where Linux's, on the same T14, reaches 42.

Where each reading of the old T14 clock bullet lives now:
- The cause, MSR_TURBO_RATIO_LIMIT 38 against 42 with what was ruled out
  and "at most 3789 under hlt", stays in the speed track's measured list.
- The scout's figures, and its work.build_shaped.j1 p50 stated as an
  unresolved reading with its spread, live in the C-state track only.
- Cores never below C1/C1E, package never leaving PC0 and the 3.16 W idle
  package move to the C-state track, whose exit measures idle energy.
- The internet_download windows' 2.67 (and 2.53) GHz median stays in
  the speed track as unexplained: the bin bounds it, it does not explain it.
- Intel's part specification gives way to the MSR it summarises. The
  5.65 W against PL1 and #1065's network-session maxima are subsumed by
  "no power limit bit logged" and the full session's at most 3789. Both
  remain in #866 and on evidence branch pr-1065.

The AMD sentence states the one-thread crunch.single ratios, parse
excepted, rather than a cpu.clock p50. The scheduler-clocks track no
longer says idle-state awareness waits for an AML interpreter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant