[snapshot] Record the measured payload size instead of the nominal PVC size - #466
Open
kkozoriz wants to merge 7 commits into
Open
[snapshot] Record the measured payload size instead of the nominal PVC size#466kkozoriz wants to merge 7 commits into
kkozoriz wants to merge 7 commits into
Conversation
Header-only pass over data.tar's PAX metadata, no decode required. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
For zstd, reads only frame headers. Other codecs still need a streaming pass, but no output is buffered. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
VolumeInfo gains RawSizeBytes/StoredSizeBytes alongside the existing nominal Size, under a new format version. Size stays what it always was: the PVC quantity used to provision the scratch volume on re-import, not the payload's actual byte size — a thin-provisioning backend rounds the device up from it, so the two can differ. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
MeasurePayload reads the actual data.bin[.<ext>]/data.tar on disk and is called once per node at finalize time, since a resumed or re-published node may not have downloaded anything in this run. Test fixtures in pipeline/manifest_worker switch from placeholder bytes to real zstd/tar payloads, since finalize now parses them. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
An already-staged file with no source MD5 to compare against was trusted on sight. Decode it and compare the real plaintext size against the declared one instead; a mismatch re-fetches, same as an MD5 mismatch already did. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
Upload's declared block-transfer size used to come from the nominal PVC quantity for every codec, block included. A thin-provisioning backend rounds the underlying device up from that quantity, so a compressed payload's real decoded size can exceed it — upload then rejected valid archives as size-mismatched. Now resolveBlockPayloadSize trusts the size recorded in the archive (from the previous commits) when present, falling back to measuring the payload itself for older archives. Also adds an X-Device-Size preflight so a target device provably too small fails before the first PUT instead of mid-transfer. Filesystem upload's progress total is now reported once, up front, from the preflight tar scan's exact sum — instead of growing as a running total that only reaches completeness at the last entry. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
kkozoriz
changed the title
Shortens overly long doc/inline comments this branch added across production and test files, keeping the same meaning, to reduce diff size and ease review. No behavior or test-logic changes. Signed-off-by: Konstantin Kozoriz <konstantin.kozoriz@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every size a snapshot reported for a volume's payload came from the nominal PVC quantity (
VolumeSnapshotContent.status.restoreSize), not from the payload itself. A thin-provisioning backend (LINSTOR/DRBD) rounds the underlying device up from that quantity and a filesystem volume is usually far emptier than its PVC, so the two genuinely differ. Consequences: download progress totals were wrong, andd8 snapshot uploadrejected valid archives as size-mismatched when the real decoded payload exceeded the nominal size.Fix
snapshot.yamlasVolumeInfo.rawSizeBytes/storedSizeBytes.Sizekeeps its old meaning: the nominal quantity used to provision the scratch volume on re-import.data.tar.X-Device-Sizepreflight so a provably too-small target device fails before the first PUT rather than mid-transfer.Before / After
Real 4-volume snapshot (
sds-replicated-thin-r3, 2 block + 2 filesystem), all four PVCs nominally 1Gi:Size(PVC)rawSizeBytes/storedSizeBytesstoredSizeBytesmatched the actual on-disk file size (stat) exactly for all four volumes.Download progress total for that snapshot: 3.4 GiB (real measured) instead of 4.0 GiB (4 x 1Gi nominal).
Validation
Numbers above come from a full
d8 snapshot download/uploadcycle against a live test cluster on a build containing this branch; recorded sizes were cross-checked againststaton the produced archive files.Tests
internal/snapshot/archive/snapshot_yaml_test.go- format version 3, round-trip, version 2 read compatibilityinternal/snapshot/archive/fsmetadata_test.go- tar raw-size suminternal/snapshot/compress/decode_test.go- decoded-size measurement per codecinternal/snapshot/volume/payload_size_test.go,manifest_worker_test.go,progress_test.go- measurement at finalization and progress accountinginternal/snapshot/volume/fs_test.go- resumed staging verified by size without a source MD5internal/snapshot/snapimport/{volume,fs,plan}_test.go- upload trusts the recorded size, device-size preflight, up-front filesystem total