Skip to content

s3.md

Latest commit

 

History

History
119 lines (97 loc) · 5.75 KB

s3.md

File metadata and controls

119 lines (97 loc) · 5.75 KB
Raw

S3 buckets

A saved connection can be an S3-compatible bucket instead of a server: AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO, DigitalOcean Spaces, rclone serve s3, anything that speaks the S3 API. Once saved, a bucket is used exactly like a server: in the CLI, in both TUI panes and in the desktop app, with the same preview, the same jobs and the same mirror safety.

# Save it. The secret access key is asked for (never typed on the command line)
diskpush connections add backups s3://site-backups \
  --endpoint https://<ACCOUNT_ID>.r2.cloudflarestorage.com --access-key-id <KEY_ID>
diskpush connections test backups        # endpoint, bucket, credentials, top level

diskpush ls backups:/                    # browse
diskpush ./site/ backups:/site/          # local  -> bucket
diskpush backups:/site/ ./restore/       # bucket -> local
diskpush mirror ./site/ backups:/site/   # previews, then deletes only what you confirmed
diskpush tui ./site backups:/site        # one pane each

Saving a bucket

diskpush connections add NAME s3://BUCKET[/default/path] \
  [--endpoint URL] [--region REGION] [--path-style | --virtual-hosted] \
  [--access-key-id ID] [--secret-stdin | --no-secret] [--no-check]
Provider --endpoint --region
AWS S3 (omit) the bucket's region, e.g. eu-west-1
Cloudflare R2 https://<ACCOUNT_ID>.r2.cloudflarestorage.com auto (the default for R2 endpoints)
Backblaze B2 https://s3.<region>.backblazeb2.com e.g. us-west-004
Wasabi https://s3.<region>.wasabisys.com e.g. us-east-1
MinIO, rclone, others your server's URL whatever it was configured with, usually us-east-1

Addressing is path-style (endpoint/bucket/key) for every custom endpoint and virtual-hosted (bucket.s3.amazonaws.com/key) on AWS, except for bucket names with dots, which AWS can only serve path-style over TLS. --path-style or --virtual-hosted overrides it.

The connection is checked before it is saved: a wrong key, secret, region or bucket name is reported now (S3 SignatureDoesNotMatch … Check the secret access key, and the region.), not halfway through a backup. --no-check saves it anyway.

In the desktop app: Servers → New, choose S3 bucket, pick a provider to fill in the endpoint pattern, then the bucket and keys.

Credentials

The access key id is saved with the connection. The secret access key is not: it goes in the same owner-only (0600) settings store that plugin sign-ins use, never in the connections table, is never shown again, and is never sent to the desktop's renderer (the editor only knows whether one is saved). connections remove deletes it. See security.md.

To keep the key out of DiskPush entirely, save the connection with --no-secret (and no --access-key-id), and set the standard variables when you run it: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and, for temporary credentials, AWS_SESSION_TOKEN. That is the way to feed it from a vault.

Ad hoc, without saving anything

s3://bucket/path works anywhere an endpoint does. It uses a saved connection for that bucket if there is one, and otherwise the AWS variables, plus AWS_ENDPOINT_URL (or AWS_ENDPOINT_URL_S3) and AWS_REGION, exactly as the AWS CLI reads them:

AWS_ENDPOINT_URL=https://s3.us-west-004.backblazeb2.com AWS_REGION=us-west-004 \
  diskpush ./photos/ s3://family-photos/2026/

How transfers behave

S3 transfers run on DiskPush's own engine rather than rsync, and keep rsync's meaning wherever S3 can hold it:

  • A trailing slash on the source copies its contents; without one, the directory itself (./site lands in backups:/site/site/).
  • Unchanged files are skipped. The quick check is size and modification time. Uploads store each file's time in x-amz-meta-mtime, and downloads set it back, so timestamps survive a round trip and a re-run moves nothing. --checksum compares MD5s instead.
  • Large files go up as multipart uploads in 16 MiB parts (larger for files big enough to need it); a failed or cancelled upload is aborted, not left half-billed in the bucket. Downloads land in a hidden .<name>.diskpush-partial and are renamed into place only when complete.
  • Filters: --exclude, --include, exclude presets, --files-from and --only, --min-size/--max-size, --update, --ignore-existing, --existing, --dry-run.
  • Mirror deletes destination objects (or local files) that are not in the source, only after the preview has been confirmed, never paths matched by an exclude, and never anything outside the destination path.
  • Exit codes follow rsync: 0, 23 when some files failed (the rest were transferred; run it again), 20 when cancelled.

What S3 cannot hold is said, not faked: symbolic links and empty directories are skipped on upload with a line saying so, and permissions, owners, ACLs, extended attributes and hard links do not exist in a bucket. Pass-through rsync arguments (-- --foo) do not apply and are reported as ignored.

Between a server and a bucket there is no direct path: it would mean relaying every byte through this machine, or depending on a tool installed on the server. DiskPush refuses it and says so; sync to this machine first, or run DiskPush on the server.

Browsing

A bucket has no real directories: a folder is a key prefix ending in /. New folder creates the empty name/ marker object every S3 console creates; an empty folder disappears when its last object does, unless it has a marker. Rename is copy-then-delete (objects up to 5 GB), and delete of a folder deletes every object under it. A bucket root can never be emptied from the file browser. Fleet does not list buckets: there is no shell on one to run a command in.