Skip to content

Cache snapshot memory hashes and add checked saving - #2079

Merged
jprendes merged 3 commits into
hyperlight-dev:mainfrom
jprendes:perf/snapshot-save-hash-cache
Nov 20, 2026
Merged

jprendes merged 3 commits into
hyperlight-dev:mainfrom
jprendes:perf/snapshot-save-hash-cache

Conversation

@jprendes

Copy link
Copy Markdown
Contributor

Summary

Avoid repeatedly hashing immutable snapshot memory layers and rereading existing content-addressed blobs during ordinary saves, while retaining a fully verifying save option.

  • Cache each memory layer's SHA-256 across snapshots that share that layer; seed the cache during checked_load verification.
  • Make Snapshot::save trust existing regular blob files with the expected content-addressed name and length, without reading their contents.
  • Add Snapshot::checked_save to recompute memory-layer hashes and verify existing blobs before reuse, atomically replacing mismatched contents.
  • Document the immutable-store assumption and add regression tests measuring hashed bytes and checking corrupted-blob repair.

Behavior and trust boundary

Ordinary save no longer detects or repairs same-length corruption in existing memory blobs. Callers must keep loaded and published memory blobs immutable and outside writable guest mounts. Use checked_save for save-time verification or checked_load to verify persisted contents. Neither method protects against concurrent mutation of loaded backing files.

The OCI format, tag semantics, and metadata/transport write paths are unchanged. There is no backing-file cache. This PR is independent of #2078 and does not remove synchronous file flushes.

Validation

  • cargo test -p hyperlight-host --locked --lib save -- --test-threads=1 — 24 tests passed, including VM-backed round trips, cache reuse, full checked-save hashing, and corrupted-blob repair.
  • cargo clippy -p hyperlight-host --locked --lib -- -D warnings
  • cargo +nightly fmt --all -- --check
  • git diff --check

No isolated latency improvement is claimed; tests verify the reduction in bytes hashed.

Make save reuse content-addressed blobs without rereading them and add checked_save for full hashing and existing-blob verification.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 22:11
@jprendes jprendes added the C-enhancement For PRs adding features, improving functionality, docs, tests, etc. label Nov 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An Unreleased changelog entry is missing to explain that save trusts same-length blobs and callers needing verification and corruption repair should use checked_save.

1 open issue
What changed in this PR

This PR reduces repeated hashing during snapshot persistence by caching memory-layer digests and trusting existing blobs during ordinary saves. Snapshot::checked_save provides explicit verification and repair under the documented immutable-store assumption.

Changes:

  • Cache digests on shared memory layers and seed them during checked_load.
  • Let ordinary saves reuse same-length regular blobs without reading their contents. Checked saves recompute hashes and atomically replace mismatched blobs.
  • Document the trust boundary and add regression tests for hash reuse and corruption repair.
File Description
src/​hyperlight_host/​src/​sandbox/​snapshot/​memory/​mod.rs Stores a digest cache on each shared memory blob.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​mod.rs Implements checked saving, cached hashes, and cache seeding during verified loads.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​fsutil.rs Supports trusted blob reuse or verified atomic replacement.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​digest.rs Tracks hashed bytes in test builds.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file_tests.rs Tests hash reuse, checked-save hashing, corruption repair, and new destinations.
docs/​snapshot-oci-format.md Explains immutable blob reuse and verification guarantees.

Review effort: Balanced

Explain trusted blob reuse in save and direct callers needing verification and corruption repair to checked_save.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
@hyperlight-gh-bot

This comment has been minimized.

Preserve both immutable-blob and durability documentation after the sync removal merged upstream. Clarify that checked_save does not add file or directory synchronization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: 362ebc5b3643
Baseline commit: e15e196fb844

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 270.45 ns (➖ 1.03x slower)
vec_bytes 82.94 ns (➖ 1.05x slower)
371.93 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
65536 141.54 ns (➖ 1.00x faster)
262144 521.39 ns (➖ 1.03x faster)

sandboxes

create_initialized_and_drop
medium 12.86 ms (➖ 1.05x faster)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
7.68 ns (➖ 1.01x faster) 7.70 ns (➖ 1.00x faster) 7.68 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 97.85 µs (➖ 1.02x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 1.89 µs (➖ 1.09x faster) 2.02 µs (➖ 1.05x slower)
262144 6.98 µs (➖ 1.01x faster) 7.37 µs (➖ 1.03x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 6.10 µs (➖ 1.05x faster) 6.24 µs (➖ 1.07x faster)
262144 28.03 µs (➖ 1.01x slower)
8192 1.03 µs (➖ 1.00x slower) 1.12 µs (➖ 1.09x slower)
kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 260.06 ns (➖ 1.02x slower)
vec_bytes 82.06 ns (➖ 1.02x slower)
679.67 µs (➖ 1.01x slower)

payload_allocation

slot_pool_segmented
65536 171.95 ns (➖ 1.00x slower)
262144 493.92 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 14.65 ms (➖ 1.03x faster)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
6.76 ns (➖ 1.00x faster) 6.76 ns (➖ 1.00x faster) 6.87 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 48.77 µs (➖ 1.02x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 2.11 µs (➖ 1.00x slower) 2.11 µs (➖ 1.00x faster)
262144 7.50 µs (➖ 1.00x faster) 7.47 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 7.07 µs (➖ 1.00x slower) 7.00 µs (➖ 1.02x faster)
262144 29.97 µs (➖ 1.02x slower)
8192 706.67 ns (➖ 1.02x faster) 723.69 ns (➖ 1.01x slower)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 354.77 ns (➖ 1.11x faster)
vec_bytes 104.35 ns (➖ 1.03x slower)
351.35 µs (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
65536 194.98 ns (➖ 1.32x faster)
262144 756.86 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 67.56 ms (➖ 1.30x slower)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
9.57 ns (➖ 1.00x faster) 9.72 ns (➖ 1.00x slower) 9.71 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 105.94 µs (➖ 1.01x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 2.21 µs (➖ 1.01x faster) 2.40 µs (➖ 1.03x faster)
262144 10.06 µs (➖ 1.05x faster) 10.56 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 8.40 µs (➖ 1.25x faster) 8.44 µs (➖ 1.15x faster)
262144 36.07 µs (➖ 1.10x faster)
8192 1.25 µs (➖ 1.02x slower) 1.40 µs (➖ 1.08x slower)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 375.70 ns (➖ 1.03x slower)
vec_bytes 85.88 ns (➖ 1.05x faster)
667.16 µs (➖ 1.07x faster)

payload_allocation

slot_pool_segmented
65536 259.49 ns (➖ 1.01x slower)
262144 675.55 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 71.88 ms (➖ 1.31x slower)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
8.70 ns (➖ 1.00x faster) 8.69 ns (➖ 1.01x faster) 9.17 ns (➖ 1.05x slower)

snapshot_files

load_snapshot_unverified
small 55.35 µs (➖ 1.00x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 2.28 µs (➖ 1.00x slower) 2.30 µs (➖ 1.02x faster)
262144 7.97 µs (➖ 1.00x slower) 8.06 µs (➖ 1.00x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 7.41 µs (➖ 1.01x faster) 7.48 µs (➖ 1.04x faster)
262144 38.97 µs (➖ 1.07x faster)
8192 912.84 ns (➖ 1.00x faster) 918.06 ns (➖ 1.01x faster)
hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 579.73 ns (➖ 1.04x slower)
vec_bytes 153.49 ns (➖ 1.04x faster)
1.99 ms (➖ 1.02x slower)

payload_allocation

slot_pool_segmented
65536 230.15 ns (➖ 1.00x faster)
262144 785.54 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 12.74 ms (➖ 1.01x slower)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
9.97 ns (➖ 1.00x faster) 9.91 ns (➖ 1.06x faster) 10.87 ns (➖ 1.09x slower)

snapshot_files

load_snapshot_unverified
small 850.15 µs (➖ 1.10x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 2.31 µs (➖ 1.00x faster) 2.31 µs (➖ 1.01x faster)
262144 9.40 µs (➖ 1.00x faster) 9.20 µs (➖ 1.02x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 8.83 µs (➖ 1.01x slower) 9.04 µs (➖ 1.07x faster)
262144 38.61 µs (➖ 1.15x faster)
8192 1.24 µs (➖ 1.04x faster) 1.25 µs (➖ 1.03x faster)
hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 544.63 ns (➖ 1.03x faster)
vec_bytes 158.25 ns (➖ 1.04x faster)
2.97 ms (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
65536 209.57 ns (➖ 1.00x slower)
262144 741.00 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 14.25 ms (➖ 1.00x faster)

slot_pool

alloc_dealloc_4096 alloc_dealloc_1500 alloc_dealloc_128
10.09 ns (➖ 1.00x slower) 10.29 ns (➖ 1.04x slower) 9.76 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 681.19 µs (➖ 1.00x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
65536 2.28 µs (➖ 1.02x faster) 2.24 µs (➖ 1.02x faster)
262144 7.75 µs (➖ 1.00x slower) 7.45 µs (➖ 1.03x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
65536 8.28 µs (➖ 1.07x faster) 8.32 µs (➖ 1.00x faster)
262144 42.14 µs (➖ 1.02x faster)
8192 1.04 µs (➖ 1.12x faster) 1.10 µs (➖ 1.03x slower)

Reported by cargo ci bench-report --candidate run:44712586657 --config-file bench_report.toml.

@andreiltd andreiltd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jprendes
jprendes merged commit 209f9e7 into hyperlight-dev:main Nov 20, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-enhancement For PRs adding features, improving functionality, docs, tests, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants