[snapshot] Add --publish transport for upload/download, with TLS hardening and transfer reliability fixes

[kkozoriz]

Problem

d8 snapshot upload/download could only stream volume bytes to/from the exporter/importer pod's in-cluster service, so pulling or pushing a snapshot required direct network access to the cluster's internal service network — not possible from a machine that only has the cluster's public API URL.

Separately, three things made that transfer path fragile even when reachable:

• The shared HTTP client used for it (SafeClient.SetTLSCAData) merged a trust pool of CA certificates but never reset an inherited insecure-skip-tls-verify, so certificate verification was silently a no-op regardless of how good the trust pool was — the kubeconfig bearer token could be handed to whatever endpoint answered that address.
• Volume payload size was derived from the nominal PVC quantity, not the actual payload — a thin-provisioning backend rounds the underlying device up from that quantity, so download progress totals were wrong and upload sometimes rejected perfectly valid archives as size-mismatched.
• A single transient transport failure inside one chunk's Range GET failed the whole block or filesystem volume, even though a byte-exact durable resume checkpoint already existed on disk — a flaky link threw away an entire in-progress multi-gigabyte transfer.

Fix

• --publish on both upload and download: stream volume bytes through the cluster's published Ingress endpoint (status.publicURL) instead of requiring direct service-network reachability. Unset autodetects whether the in-cluster endpoint is reachable; explicit --publish=false keeps the old path. Works only with a bearer-token kubeconfig — the Ingress terminates TLS with its own certificate and doesn't forward a client certificate to the pod.
• TLS verification forced on unconditionally in SafeClient.SetTLSCAData: Insecure is cleared on both the rest.Config and the already-built transport clone (client-go bakes the flag into the base transport before WrapTransport runs), a previously installed WrapTransport is chained instead of clobbered, and a nil-panic on non-*http.Transport types is fixed.
• Measured payload size recorded at capture time (snapshot.yaml, manifest format version 3) instead of trusted from the nominal PVC quantity: upload now trusts the recorded size when present, with an X-Device-Size preflight so a genuinely too-small target device fails before the first PUT instead of mid-transfer. Filesystem upload reports its progress total up front from a preflight tar scan instead of growing a running total that only completed at the last entry. Version 2 archives are still read (readers fall back to measuring).
• Chunk retry from the durable offset: a broken Range GET is retried from the byte offset the interrupted attempt already persisted, with bounded exponential backoff and a no-progress guard, instead of discarding the whole volume. One retrier per volume, shared across chunk goroutines, with a single aggregate warning if retries were absorbed — filesystem volumes get this for free through the same path.

Validation

Build. Over every touched package (./internal/snapshot/..., ./pkg/libsaferequest/...):

Check	Result
go build	clean
go vet	clean
go test	24 packages ok
go test -race	24 packages ok, no data races
golangci-lint run	0 issues

Real-cluster e2e.

• Data integrity, byte-for-byte. A 4-volume snapshot (2 block + 2 filesystem, sds-replicated-thin-r3) downloaded with --publish=true from a machine with no route into the cluster network produced an archive byte-identical by sha256, across all 4 volumes, to both an internal-network download of the same snapshot and the live volume data on the cluster.
• Publish end-to-end, both directions. The same snapshot uploaded back with --publish=true from an external machine holding only a bearer-token kubeconfig: completed in 23.6 minutes, all 4 DataImport objects reached Completed, and the restored volumes' contents were confirmed by downloading them back and comparing sha256.
• Retry proven live, not only in a probe. That download hit 8 genuine transient interruptions (unexpected EOF) on the two block volumes' chunked transfer, 4 per volume. Every one recovered on the first retry (attempt=1), the per-volume summary logged retries=4, and the final archive's sha256 still matched two independently obtained copies of the same data. Roughly 458 progress log lines showed no double-counted bytes.
• Measured sizes cross-checked. Recorded storedSizeBytes matched stat on the produced archive files exactly for all four volumes, and the download progress total became a real 3.4 GiB instead of a nominal 4.0 GiB.
• TLS token leak reproduced and closed. The leak was demonstrated with a probe built from the same source in two isolated worktrees, one at each commit, and reproduced by hand against both trees during review.

download --publish=true from outside the cluster network:

upload --publish=true from outside the cluster network:

Measured vs. nominal payload sizes, cross-checked against stat:

Tests

• internal/snapshot/cmd/download, internal/snapshot/cmd/snapimport — flag defaults, autodetection, examples
• internal/snapshot/exporter/{export,dataexport,http,retry} — publish endpoint selection, 401 hinting, transient-failure classification
• internal/snapshot/transport, pkg/libsaferequest/client — TLS verification forced on, WrapTransport chaining, invalid CA data, client-cert survival, non-transport pass-through
• internal/snapshot/pipeline — publish threaded through pipeline config
• internal/snapshot/archive — manifest format version 3, round-trip, version-2 read compatibility
• internal/snapshot/volume — chunk retry, backoff, no-progress guard, concurrent retry, context cancellation, payload-size measurement, resumed staging verified by size
• internal/snapshot/snapimport — upload trusts recorded size, device-size preflight, up-front filesystem progress total

All of it passes under -race on the combined result, as shown above.