Repository navigation
crypto: ChaCha20-Poly1305 and OpenSSH in one stitched AVX2 pass, eight blocks a batch - #1056
Conversation
On AVX2 without AVX-512VL (the AMD bench, Zen 2), seal 64 KiB ran 1.14x ring and open 1.35x: six blocks a batch, open in two passes, block 0 apart from the batches. - Eight blocks a batch, in the word-per-register layout AVX-512 already used, for the stream and the AEAD. Sixteen registers spill a few words a round, but the layout needs none of the row layout's diagonal shuffles, which share Zen 2's two shuffle pipes with its one shift pipe. Measured against four pairs of rows on the AMD bench, it won at every size. - One pass for seal and open. Block 0, the Poly1305 key, comes out of the first batch: with seven data blocks when sealing, or with one when an open runs past it, since an open's MAC waits for the key. The scalar Poly1305 runs three blocks in each of the ten double rounds and the rest after them. Sealing absorbs a batch's ciphertext beside the next batch, opening beside its own before the keystream overwrites it. Tails are stitched too, and short batches run one or two pairs. - The stitched code compiles with BMI2, so its multiplies are MULX. - OpenSSH's MAC goes through the same pass: its blocks begin 12 bytes into the payload, an open reads up to 12 bytes past its batch, and a partial last block ends in a one byte. - A refused ChaCha20-Poly1305 record and a refused OpenSSH payload are wiped, as AES-GCM's record is since #1054. The one-pass open has written plaintext before the tag is known. Every test of the refusal contract, the guest's parity instrument's included, now asks for zeros. - The 16-bit rotation's mask passes through an empty asm block. Otherwise LLVM lowers the VPSHUFB as VPSHUFLW and VPSHUFHW, two shuffle uops for one. - Unit tests call the AVX2 kernels directly, so they run on any AVX2 CPU whatever the dispatch picks. An OpenSSH unit test composes the packet from portable blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2c00ebb to
035c36d
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r1: #1056 at 4c853eb (code 035c36d, on main 3a19907)ArchitectureThe design matches the analyst's spec (C1–C4) and stays inside its boundary:
One kernel serves both RFC 8439 and OpenSSH. The only difference is the Size: the crate's sources grow by about +360 lines, of which about 165 are the new unit tests. That leaves about +200 net production lines. For that, the AMD bench moves open 64 KiB from 1.35 to 1.04–1.10 of ring and seal 1 KiB from 1.47 to 1.25, and OpenSSH open 16 KiB from 0.51 to 0.40. I accept the size for that gain under the hot-path direction. Correctness: traced, no defect foundI traced
Refusal wipe. I found no path that can leak plaintext:
Constant time by inspection. Every branch and index depends only on
Dispatch. The AVX2 entry requires Evidence
BLOCKERS
SUGGESTIONS
Open items as recorded
SEND BACK |
instrument reaches the stitched paths - The one-pass unit test checked AVX2 alone before calling a kernel compiled with BMI2, and claimed every AVX2 CPU has BMI2. ToyOS's own QEMU CPU doesn't. It now requires both and says so; on AVX2 without BMI2 it fails by name, where it died on an invalid opcode before. - crypto_constant_time timed ChaCha20-Poly1305 and OpenSSH at 64 B only, which never reaches the stitched batch: it now times both at 1 KiB too, as AES-GCM's stitched paths already were. - The track issue's AVX2 exception describes the one-pass open from the guest table at 035c36d, and drops the superseded two-pass breakdown. - The module header says each layout in its own sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response to review r1 (head ca676da, code commit 035c36d plus ca676da)Blockers
Suggestions
Evidence: https://github.com/ToyOSOrg/evidence/tree/a276e825e3e297b9ebe2e133f1af10676c4fa6d0 Checks at ca676da:
Requesting focused review of |
…e one pass Review r1 of #1056 asked whether short records cost more through the one-pass kernel. On the AMD bench (Linux probe, head and variant alternated twice) the short path first takes 64 B seal from 193 to 183 ns, open from 188 to 176 and reject from 183 to 171 (1.29, 1.19 and 1.10 of ring to 1.23, 1.11 and 1.04); 0 B and the 1 KiB control are unchanged. The T14, whose AVX-512 dispatch declines the one pass, is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r2: #1056 at 79764c8 (since r1: ca676da, 79764c8)ArchitectureNeither commit changes the design. 79764c8 moves 1–192 B ahead of the kernel in r1's blockers
The short-path-first switchI traced each boundary in both directions:
Accepted suggestions
BLOCKERS
SUGGESTIONS
Declined suggestionsThe SEND BACK |
- The track issue's AVX2 exception says the one pass serves records past 192 B; its per-message rows give the landing code's readings alone. - crypto_constant_time's header says the AEADs and OpenSSH are timed at 64 B and 1 KiB; RECORD's doc says why in one line. - The forged-record test's doc gives its bound: past 192 bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Response to review r2 (head 9c68ba5, text only over 79764c8)Blockers:
Suggestions, all taken:
9c68ba5 changes comments and the track issue only. No build, test or boot is affected; the AMD |
Problem
On AVX2-only CPUs, the common case (every AMD before Zen 4, and Intel desktops since Alder Lake), toyos-crypto's
ChaCha20-Poly1305 trailed the fastest library. On the AMD bench (Ryzen 7 4800U, Zen 2), crypto-oracles r2 measured
seal 64 KiB at 1.15× AWS-LC, open 64 KiB at 1.34× ring and seal 1 KiB at 1.48× ring. AVX2 ran 6 ChaCha20 blocks a
batch. Open made two passes, the vector Poly1305 then the keystream. Block 0, the Poly1305 key, was a separate
scalar block. The seal stitch left 2 of every 10 double rounds and the tail unstitched. OpenSSH's construction made
two passes in both directions.
Change
AVX2 machines without AVX-512VL only. The T14's AVX-512 dispatch is untouched.
register per state word of all eight blocks. Sixteen registers spill a few words a round, but the layout needs
none of the row layout's diagonal shuffles, which share Zen 2's two shuffle pipes (FP1/FP2) with its one shift pipe
(FP2) (uops.info, Zen 2). Round 1 measured the four-pair row layout too, and it lost (table). It is gone. The
stream (
xor, which SSH also uses) takes the same layout.arch::x86_64::chacha::avx2::chacha20_poly1305:when an open runs past the batch, since an open's MAC can't start before the key.
thirty after them. Sealing absorbs each batch's ciphertext beside the next batch; opening absorbs a batch beside
its own rounds, before the keystream overwrites it.
aead::chacha_open, throughsecret::wipe), because the one-pass open has written plaintext before the tag is known. Paths without the onepass (AVX-512, portable, AArch64, and 1–192 B on AVX2) still decrypt only an authentic record and zero a refused one. AES-GCM's refusal
zeroes since toyos-crypto: AES-GCM in one pass, 48 blocks to a reduction on VAES-512 #1054, so
aead's module contract is now one rule for both. Every test of it (differential,Wycheproof, the RFC 8439 vector, the rustls adapter, the guest parity instrument) asks for zeros.
A refused OpenSSH open zeroes the payload (
sshmodule contract).VPSHUFLW+VPSHUFHW, two FP1/2 uops for one. Round 1's control without it was slower on the AMD at every size.
chacha::seal,arch::chacha_seal, the six-block kernels andpairs_of.Poly1305::partsgets anexpect(dead_code)on targets other than hosted x86-64, its only user. The attribute names an architecture in ageneric module but holds no architecture code, as
field25519.rs's selector does.Measurements
Linux probes on each bench's own Linux (crypto-oracles' probe: ring 0.17.14, OpenSSL 3.5.5, AWS-LC 5.11.0;
toyos-crypto built incrementally in
[profile.toyos]settings). Ratios are ours over ring; above 1 is slower.The clock line reads 0.234 ns a cycle on the AMD (4.27 GHz).
Round 1, AMD:
AEAD: 22.3 µs).
64 KiB: 8.21 → 8.80 µs, the wipe of the refused 64 KiB buffer, still 0.33 of ring. That figure is 2c00ebb's,
whose wipe was a fill; since the rebase it is toyos-crypto: AES-GCM in one pass, 48 blocks to a reduction on VAES-512 #1054's 16-byte volatile
wipe, which costs more (on the AMD benchreject 64 KiB reads 0.75 µs above open at 035c36d, against 0.58 µs below it at 2c00ebb).
decrypt and a wipe, as every other library's does. This is the approved C1 tradeoff.
open 1 KiB barely moved, 1.426 → 1.409, because it hashed 448 bytes unstitched while waiting for the key); one pair
for short first and tail batches (round 1's 64 B rows rose); and OpenSSH.
Round 2: main b48c7dd against the head 2c00ebb, both benches, two alternated runs each (ours over ring):
rather than four pairs for the first batch.
The rebase. The head was then rebased onto main (035c36d) past #1054, whose AES-GCM refusal wipe makes the
contract the same for both AEADs. The ChaCha code is 2c00ebb's, but its refusal now goes through #1054's
secret::wipe(16-byte volatile stores) rather than a fill. The probe numbers above are 2c00ebb's.Round 3: the head 035c36d (run twice) and two variants of it, AMD bench, ours over ring:
the 64-byte tail, which costs far less than the 600 ns that implies.
round 2, with ring unchanged (22.0–22.4 µs) and the same clock (0.234 ns a cycle). The bulk loop is the same
4,503 bytes in both builds and differs only in alignment.
bulk rows read 1.04 to 1.10 of ring at these two heads.
Round 4 (review r1's suggestion): 1–192 B through the short path first, AMD bench, head ca676da against the
variant, alternated twice (ours over ring):
cryptocase;one pass, so its paths are unchanged.
cryptocaseand one AMDcryptoctcaseat 79764c8 are staged; the orchestrator decideswhether they are owed.
Checks and evidence
picks (the one-pass test asks for AVX2 and BMI2, and refuses by name without them):
arch::x86_64::chacha::tests::the_avx2_stream_matches_one_block_at_a_time: every length 0..1600 and batch edges,across counter carries;
...::the_avx2_one_pass_matches_both_macs_composed: RFC 8439's MAC and OpenSSH's, sealing and opening, everylength 1..1600 and up to 16 KiB, against portable blocks and
poly1305::tag.aead::tests::a_forged_chacha_record_opens_to_zeros;ssh::tests::packets_match_the_construction_composed_from_blocks.8439 vector test and the rustls adapter test follow the new contract.
tail emit, the rotation mask, the open's read-ahead, the seal's lag, the pair choice, the wipe, the OpenSSH
header order).
input.
r4/inspect/): 20 MULX and 4 IMUL (h2·r0) in each ofchacha20_poly1305andkeystream, and no MUL. The full-batch pathcalls nothing. Inside the bulk loop's range, the only calls are
keystreamandblock::xoron the tail branch, onceper message;
keystreamalso runs once for the first batch.cargo run -- --ci host: 103 steps, all green, exit 0, at ca676da and at 79764c8. clippy's three toyos-crypto shapes on x86_64 and aarch64: clean.cargo run -- --build-only: exit 0 at 035c36d, ca676da and 79764c8.cryptocase(crypto_instrument) andcryptoctcase(crypto_secret_timing) on the T14 and the AMD bench:round 2 at 2c00ebb: every boot rc=0.
crypto_secret_timingpassed on both benches, a pass that rests only on the orchestrator's judge summary ("1 passed, 1 failed, 2 boot(s)"): round 2's readbacks were deleted before they were published and are lost.crypto_instrumentfailed on both: the guest parity test still asked a refused ChaCha20-Poly1305 record to be left as it was, and it panicked before timing any row. That test now asks for zeros, for OpenSSH too, and passes under QEMU (78 rows in each build).cryptocaseat 035c36d passedcrypto_instrumenton both benches (round 3, "1 passed, 0 failed, 1 boot(s)" each; readbacks published).crypto_constant_timetimed ChaCha20-Poly1305 and OpenSSH at 64 B only. That never reaches thestitched batch, and OpenSSH at 64 B never reaches the kernel at all, so no timing boot so far covers the new path.
cryptoctcaseat ca676da:crypto_secret_timingpassed on the T14 and the AMD bench, 1/1 each.Every ChaCha20-Poly1305 and OpenSSH row passed at 64 B and at 1 KiB, max |t| 0.59–2.93; the leaky control leaks
and the null control passes.
len,seal,padorprefix.len(), the tag goes throughct::equal, and the wipe's length is the record's.uses, and the crate's BMI2 Montgomery kernels already use MULX on the AMD bench under
crypto_secret_timing_curves. That last point is an inference, not a measurement.pr-1056).Also in this branch
issues/toyos-crypto-is-in-the-tree-on-par-with-ring-and-secure.md:035c36d, with round 3's per-build spread;
issues/a-metal-judge-says-on-the-t14-for-every-bench.md(new, tooling):Readback::job_passedwords everybench's failure as "on the T14". Round 2's AMD judge line said so.
Risks
Per-build spread. The AVX2 bulk rows vary by about 0.06 of ring from build to build with the same vector loop
(round 3). An unrelated change elsewhere may move them either way.
Reject cost on AVX2. One-pass open makes reject cost an open, as above.
Inlining.
batchandeightare cost-based#[inline]#[target_feature]functions, as everywhere in thecrate. Each closure type gives each generic instance one call site, which is what inlines them today. A future
second caller should be checked in the machine code.
Spills. The word layout's spills are LLVM's choice. A different LLVM could spill differently on Zen 2.
Intel AVX2-only CPUs (Alder Lake and later desktops) are unmeasured: no such bench. Their shift ports differ
from Zen 2's, so the layout choice is Zen 2's measurement.
🤖 Generated with Claude Code