Repository navigation
Conversation
…n explicit evidence run The development Mac moves and is on Wi-Fi, so it is no peer and no part of a measurement. `osbench lan` now runs on one bench against the `osbench-peer` the other bench's Linux runs as a fixed reference, each way; ToyOS against Linux on one path is the comparison, and Linux's own runs give the path's ceiling and noise. - lan: the peer's answer says which of its links it answered over (kind, driver, and speed or Wi-Fi signal), refused past 160 printable bytes; the bench records it as `lan.peer_link`. Link reading moves from osbench's Linux layer into `osbench-lan::link`, so the peer uses it too, and an address on an interface with no device is refused rather than called wired. Transfers are five timed 3 s rounds after one unread. - `lan` and the new `internet` leave `all` and the full suite's boot: they need another party, so each runs only by name, and the full suite no longer needs the other bench. Its list bound drops by `lan`'s 110 s. - internet: TCP connect, an HTTPS round trip on a kept connection, 25 MB downloads and 10 MB uploads against Cloudflare's speed test (speed.cloudflare.com, the endpoints Cloudflare's own engine drives), over ureq and rustls on toyos-crypto's provider on both systems. Answers' headers and bodies name the client's address; nothing reads or prints them. A ToyOS netstack answers NotConnected until its lease lands and announces none, so the lookup is asked again each second for up to 60 s. - Metal rows `osbench_lan` and `osbench_internet`, named-only, on one `netbenchcase` boot of benchcase.toml when a pass names both. - osbench-compare: reads run files (a Linux run's stdout or a ToyOS boot's log) and prints per machine and benchmark each system's per-run p50s, median, run-to-run range, ToyOS/Linux and the exact two-sided Mann-Whitney p over the runs, with ties as half wins. LAN rows whose runs' peers were on different links are not compared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r1 of #1065 at c3b124bArchitectureThe ownership is right. The bench under test does all the timing on its own clock. The peer is a fixed Linux reference on the other bench and does no more than answer. Link reading now lives in The statistical unit is right too. Each value is one run's p50, so there is no pseudo-replication over samples. The test is the exact two-sided permutation (Mann–Whitney) test: it enumerates every split of the pooled values, ties count half, and the distribution is conditional on the ties observed. I checked the table values by hand: 2/70, 0.1, 2/252, 4/70, and the half-win case The new code is about 280 production lines in Privacy passes.
Endpoint. Cloudflare publishes
This is consent-appropriate and stays outside automation. Registration. Both new rows say which machines they run on (the T14, and why the AMD bench is excluded). Nightly does not run metal, so taking FindingsBLOCKER
BLOCKER
BLOCKER
BLOCKER (evidence) — the PR's material claims have not been run yet. These are:
The draft states this gap honestly, and SUGGESTION SUGGESTION (PR body, for the orchestrator) — "a ToyOS–Linux difference smaller than Linux's run-to-run range on that path cannot be resolved" is not what the tool does. Complete separation can occur with a median difference below Linux's (max−min). The range is descriptive and the rank test decides. Say that instead. SUGGESTION SUGGESTION SUGGESTION SEND BACK |
…ns are too few, and the net rows' bounds add up - A verdict says better or worse only for `ns` (lower) and `MB/s` (higher); any other unit, a power or a clock, is told which way ToyOS differs. Counts whose complete separation cannot reach 0.05, 4 runs against 3 among them, say "too few runs to resolve (least p ...)" rather than "not resolved". A median of 0 has no range. Each table says its p is per row and uncorrected. - `internet.*` rows are compared only across runs that left by one kind of link, as `lan.*` rows are across one kind of peer link. ToyOS's link description now starts with its kind, `wired`. - The network rows' bounds: each arm adds its own worst span and the list's 60 s stays the boot's margin. `lan` 120 s (60 s discovery, under 60 s measuring); `internet` 155 s (60 s for a lease, under 95 s at an assumed 3 MB/s down and 1.5 MB/s up, which no metal reading confirms). The shared boot's bound is 335 s. - The crypto issue's consumer table names osbench's guest HTTPS client and the third copy of the provider's random source and group filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review r1 response, head d99ef5a. Evidence: https://github.com/ToyOSOrg/evidence/tree/8f27129bd4eccbbc55a4c09f177d6946d1d81ffc/r2
|
… downloads stay under the endpoint's limit, and the first paired session's findings - osbench-compare: the least attainable p is read from the same exact distribution as p, the farthest splits' share, rather than assumed to be 2/C(n+m,n). A difference whose p reaches 0.05 is resolved whatever the counts. The session's T14 upload, three ToyOS runs against four Linux runs three of which tie, has p = least = 1/35, and the old count-based gate called it "too few runs". - internet: the downloads are four of 10 MB rather than six of 25 MB. Every Linux download in the session drew Cloudflare's 429 after four runs of the larger volume had passed. Its bound goes to 105 s. - issues: netstack's TCP send on the T14's I219 runs at 0.36x Linux's on the same path while receive is at parity (new defect); the session's cpu.clock readings join the idle-clock issue; the comparison issue points at the session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… record The network session's cpu.clock readings were ToyOS 1.5-3.2 GHz against Linux 3.8-4.2 GHz per group. They are the same unexplained clock as the 3,788 MHz single-thread ceiling and the 2.67 GHz download windows, so they belong with that record in toyos-uses-what-modern-hardware-offers-for-speed. They leave the idle-clock issue, whose subject is the gauge and not the clock. The upload defect points at the record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2 of #1065 at ec8b41dThis round covers d99ef5a, 548bef4 and ec8b41d. ec8b41d changes only issue records. So ArchitectureNothing changed in ownership since r1. The changes are confined to r1 blockers
The least p with tiesp = least = 1/35 for the upload is correct. I enumerated the splits.
The test is valid as a conditional permutation test. Integer-MB/s quantization creates the ties, but that does not make it invalid. The result does depend on the two-sided convention. The code uses The 40 MB download and the 105 s bound
B4, the evidenceResolved for landing the program. The metal claims r1 listed are now established, at d99ef5a, whose guest code differs from the head only in the two constants:
The interleaving gap weakens the measured findings, not the program. Two of those findings matter.
Nothing more is owed on metal. A fourth interleaved ToyOS run is a follow-up for the findings, not a condition of landing. PrivacyPrivacy passes. I grepped every file under
The only matches are FindingsBLOCKER
The published SUGGESTION S1 SUGGESTION SUGGESTION (PR body and
SEND BACK |
…act test names its two-sided rule - netstack upload defect: the "cluster near 31 and 58 MB/s" claim goes; the runs' min, p50 and max do not show it. The clock paragraph now gives both systems' readings: Linux sent 113 MB/s at the same ~1.5 GHz readings, so the clock does not explain the gap. - The T14's clock-ceiling record compares cpu.clock's maximum: 3789 MHz under ToyOS against 4188-4190 under Linux. It no longer compares the probes' idle-ramp p50. - osbench-compare's doc names its two-sided p, the share of splits whose |2U - nm| is at least the observed one, and says how ties lower the least p. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The C-state track is assigned: it is being built on branch deep-idle, and that branch's PR carries its design. The AML track's 2026-10-12 ordering ruling is marked superseded, because the owner's direction of 2026-10-27 releases the plan's order. The track's exit gains a cpu.clock reading, which the orchestrator accepted under that direction. The exit names its statistic and scope: the maximum at ratio 42 in every group where Linux's, on the same T14, reaches 42. Where each reading of the old T14 clock bullet lives now: - The cause, MSR_TURBO_RATIO_LIMIT 38 against 42 with what was ruled out and "at most 3789 under hlt", stays in the speed track's measured list. - The scout's figures, and its work.build_shaped.j1 p50 stated as an unresolved reading with its spread, live in the C-state track only. - Cores never below C1/C1E, package never leaving PC0 and the 3.16 W idle package move to the C-state track, whose exit measures idle energy. - The internet_download windows' 2.67 (and 2.53) GHz median stays in the speed track as unexplained: the bin bounds it, it does not explain it. - Intel's part specification gives way to the MSR it summarises. The 5.65 W against PL1 and #1065's network-session maxima are subsumed by "no power limit bit logged" and the full session's at most 3789. Both remain in #866 and on evidence branch pr-1065. The AMD sentence states the one-thread crunch.single ratios, parse excepted, rather than a cpu.clock p50. The scheduler-clocks track no longer says idle-state awareness waits for an AML interpreter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
osbench's
lanrows assumed a peer on "any host on the link", which in practice meant the development Mac. The Mac travels and stays on Wi-Fi, so it can't serve as a reference. Thelangroup also ran inside the full suite's nine-minute boot, so every full-suite run needed a peer. Finally, nothing measured the internet path, and nothing turned repeated runs into a comparison with a stated noise floor.Resulting behaviour
The benches are each other's peer. One bench runs the system under test, ToyOS or Linux. The other bench runs
osbench-peerunder Linux as the fixed reference, and the roles swap for the other direction. The peer's discovery answer says which of its links it answered over (kind, driver, and wired speed or Wi-Fi signal). The bench records that aslan.peer_linkbeside its ownlan.link. No address, MAC or interface name is printed. Nothing depends on the Mac: every figure is taken on the clock of the bench under test.lanandinternetrun only by name. They are out ofosbench alland the full-suite boot, which no longer needs the other bench; its list bound drops bylan's 110 s, from 540 s to 430 s. Each group has a named-only metal row,osbench_lanandosbench_internet. When one pass names both rows they share onenetbenchcaseboot ofbenchcase.toml.LAN measurement. The same four rows as before:
tcp.connect(50 samples),tcp.rtt.64(2,000 samples), and upload and download, now five timed 3 s rounds after one unread round. Linux link reading moves intoosbench-lan::linkso the peer can use it too. An address on an interface with no device is now refused instead of being called wired.Internet rows are exploratory evidence, not regression tests. The rows are
internet.tcp.connect,internet.https.rtt(cdn-cgi/traceon a kept connection), andinternet.https.downloadandinternet.https.upload(four 10 MB transfers each way, each timed whole in ns; the first session's 25 MB downloads drew Cloudflare's 429 rate limit). The endpoint isspeed.cloudflare.com, Cloudflare's speed test: the endpoints Cloudflare's own measurement engine drives (https://github.com/cloudflare/speedtest). The user agent istoyos-build (https://github.com/ToyOSOrg/ToyOS). TLS is ureq and rustls on toyos-crypto's provider on both systems, so the cryptography is the same code. The answers name the client's address and location, so nothing reads or prints a header or body.osbench-compareis a new host program and replaces the earlier scratch Python tables. It reads run files, either a Linux run's stdout or a raw ToyOS boot log. For each machine and benchmark it prints:(max−min)/median. Linux's range describes the path's noise; the rank test, not the range, decides;The least p is read from the same exact distribution, so ties count. With four runs against four and no ties it is 2/70 = 0.029; with three against four it is 2/35 = 0.057, unless one system's runs tie. A row whose counts cannot reach 0.05 says "too few runs to resolve (least p …)". Better or worse is judged only for
nsandMB/s; other units get "ToyOS higher/lower". Each p is per row and uncorrected for the table's other rows.internet.*andlan.*rows are compared only across runs that agree on the kind of link (and, for LAN, the peer's driver).Links and paths (from the earlier Linux identity runs; the run plan gives read-only commands to confirm them)
lan.wifi.*unsupported: no Wi-Fi driverlan.*andinternet.*row unsupported: ToyOS drives no card the AMD bench hasThe AMD bench has no Ethernet, so path A has a Wi-Fi hop at the far end. Linux's runs on it give its ceiling and its run-to-run range; whether a ToyOS–Linux difference is resolved is the rank test's call, not the range's. In the first session the Wi-Fi hop did not limit the path: Linux moved 113 MB/s both ways, the gigabit link's rate.
The first paired session (coordinator-run, at d99ef5a)
Evidence, masked: https://github.com/ToyOSOrg/evidence/tree/cd7124802c7be399d17636d516c2a3a5a6e1585f/r3 (the session's files,
compare-r3.md, and a README with the timeline).lan.wired.tcp.downloadlan.wired.tcp.rtt.64lan.wired.tcp.uploadlan.wired.tcp.connectinternet.tcp.connectinternet.https.rttinternet.https.upload(10 MB)internet.https.download(25 MB)Noise floor on path A, from Linux's run-to-run range: round trip 0.5%, upload 0.9%, download 2.7%, connect 58%.
Path B, Linux only (the AMD bench over its Wi-Fi to the T14's wired peer): 114 MB/s down, 110 MB/s up, 2.00 ms round trip. ToyOS drives no card the AMD bench has.
Caveats:
/tmp, and the reboots wiped them.internet.https.downloadgot HTTP 429. The downloads that ran first succeeded: ToyOS's three and the first attempt's Linux L1 (257 ms per 25 MB). The 429s began after at least four runs of six 25 MB downloads from one address. The code now moves 40 MB down per run instead of 150 MB.Findings recorded:
issues/netstack-sends-tcp-at-a-third-of-what-the-t14s-wired-link-carries.md: new defect.issues/toyos-uses-what-modern-hardware-offers-for-speed.md, the T14's clock-ceiling record:cpu.clock's maximum was 3789 MHz under ToyOS against 4188–4190 MHz under Linux. Its p50, ToyOS 1.5–3.2 against Linux 3.8–4.2 GHz, reads idle and ramp policy, not the ceiling.Checks
Evidence r1–r4 on branch
pr-1065: https://github.com/ToyOSOrg/evidence/tree/15491ecbd03fd217ca4a31ae30ebbe3ed1f35c2f (r4 corrects r3's README).cargo run -- --ci hostat ec8b41d: exit 0, 103 green. 48c9d05 after it changes only issue records and one doc comment. At 548bef4, the last code change: exit 0. Earlier heads: c3b124b and d99ef5a were green; e60ee7a88 was red on one clippy lint, which I fixed.osbench interneton ToyOS, at round 1's code; the later edits are constants and the comparison tool:unsupported.Unresolved risks
osbench-comparenames (doubling the one-sided p would give 2/35). Its strength is that no ToyOS round reached any Linux round.__upaccepting POSTs, andcdn-cgi/traceanswering at the edge.lan.wifi.*under ToyOS (there is no Wi-Fi driver;issues/there-is-no-wifi.md);lan.*andinternet.*row under ToyOS;🤖 Generated with Claude Code