Add optimole export: export the library and download its archives

README.md

@@ -1,11 +1,12 @@
 # optimole
 
 `optimole` uploads local files into an [Optimole](https://optimole.com) media library, into the folders you
-choose. It is a single binary with no dependencies.
+choose, and exports the library back as ZIP archives. It is a single binary with no dependencies.
 
 ```sh
 optimole login                                   # paste your API key once
 optimole upload -r ./photos --folder Campaigns   # photos/2024/a.jpg -> Campaigns/2024
+optimole export start                            # export the library, download ./optimole-export-ID
 ```
 
 ## Install
@@ -67,7 +68,7 @@ the storage URL the file bytes go to.
 
 **Folder-scoped keys** (the ones starting with `optml-s`) work for uploads into the folders they were
 granted (and their subfolders): always pass `--folder` (or `--folder-id`). They cannot create folders
-(`folder_creation_forbidden`), upload into the library root (`folder_required`), or list folders.
+(`folder_creation_forbidden`), upload into the library root (`folder_required`), list folders, or export.
 
 The API base URL defaults to `https://dashboard.optimole.com/api`; override it with `--api-url` or
 `OPTIMOLE_API_URL` (for example to try another environment). Only `https` URLs are
@@ -242,6 +243,109 @@ ID  PATH
 
 Folder-scoped keys cannot list folders (the API answers 403); `upload --folder PATH` still works with them.
 
+### `optimole export <command>`
+
+Exports the original files of the whole library and downloads them. An export covers every whitelisted
+domain of the account, plus `directUpload` for the files uploaded straight to the library; the server zips
+each domain into archives of about 1 GB. Exports need a full-access API key.
+
+| Command | What it does |
+|---|---|
+| `optimole export start [flags]` | Starts an export, follows it and downloads each archive as soon as it is ready. If an export is already running, follows that one instead. `--no-wait` only starts it and prints its ID. |
+| `optimole export download [ID\|latest] [flags]` | Downloads an existing export (default: the latest), waiting while it runs. Run it again to resume. |
+| `optimole export status [ID\|latest] [--json]` | Shows the export's status and each domain's status, archives and size. |
+| `optimole export list [--json]` | Lists the account's exports, newest first. |
+| `optimole export delete ID` | Deletes a finished export and its archives on the server (local files are not touched). |
+
+`start` and `download` take:
+
+| Flag | Meaning |
+|---|---|
+| `-o`, `--output DIR` | Where the export folder goes (default: the current folder). |
+| `--domain NAME` | Only download this domain's archives (repeatable, case-insensitive). The export itself always covers every domain. |
+| `-c`, `--concurrency N` | Archives downloaded in parallel, 1-8 (default 2). |
+| `--timeout DURATION` | Give up after this long, e.g. `90m` or `2h` (default: no limit). |
+| `--json` | One JSON object per event and a final summary object. |
+
+```sh
+optimole export start                          # start, wait, download everything
+optimole export start --no-wait                # start it now, download later
+optimole export download                       # the latest export, into ./optimole-export-ID
+optimole export download 42 -o /backups --domain example.com
+```
+
+#### Where the files go
+
+```text
+optimole-export-42/
+  example.com/images-1.zip
+  example.com/images-2.zip
+  directUpload/images-1.zip
+  export.json
+```
+
+Each domain gets a folder named after it. Domain names come from the server, so the CLI keeps only letters,
+digits, `.`, `-` and `_` (anything else becomes `_`, so `*.example.com` becomes `_.example.com`), drops leading
+and trailing dots, avoids Windows device names, and adds `-2`, `-3`... when two names differ only in case.
+`export.json` maps each domain to its folder and lists its archives with their size and whether they are
+downloaded. It holds no download URLs. The archives are not unpacked.
+
+#### How downloads work
+
+- While the export runs, the CLI checks it every 5 seconds (less often after a few minutes) and downloads each
+  archive as soon as the server lists it.
+- An archive is written to `images-N.zip.<size>.part` and renamed into place only when its size matches what
+  the server listed and every file in it reads back with its checksum (files in a compression format Go's ZIP
+  reader lacks are not read; Optimole's archives use deflate). A download that gets cut is resumed
+  where it stopped; one that stalls for 2 minutes is restarted. An archive that fails the check (a resume that
+  joined two versions of a rewritten archive, for example) is downloaded once more from scratch, then reported
+  as failed. An archive whose size neither the server nor storage gives is not downloaded.
+- One command at a time downloads into an export folder: it holds a lock on `.optimole-export.lock`, and a
+  second command into the same folder stops with an error. Links planted inside the export folder are not
+  followed.
+- Running `download` again skips the archives already in place and resumes the partial ones. If the server
+  listed an archive with a new size (it rewrote it), that archive is downloaded again.
+- Download URLs are valid for a day; when storage refuses one, the CLI asks the server for a fresh one.
+  Download URLs never get the API key and are never printed, and redirects are refused.
+- Ctrl-C and `--timeout` stop only the CLI, never the export on the server, and the command prints how to
+  resume (`optimole export download ID`). There is no way to cancel an export from the API.
+
+#### Output
+
+```text
+Started export 42.
+  example.com: running
+  directUpload: running
+Export 42: running
+downloading optimole-export-42/example.com/images-1.zip (1.0 GB)
+downloaded  optimole-export-42/example.com/images-1.zip (1.0 GB in 41s)
+  directUpload: completed, no files
+...
+Export 42: completed
+
+Export 42 completed: 2 downloaded, 0 already there, 0 failed (1.8 GB written) in optimole-export-42.
+```
+
+With `--json`:
+
+```json
+{"event":"started","id":42,"created":true}
+{"event":"domain","domain":"example.com","status":"running","archives":0}
+{"event":"export","id":42,"status":"running"}
+{"event":"downloading","domain":"example.com","part":1,"path":"optimole-export-42/example.com/images-1.zip","size":1073741824}
+{"event":"archive","domain":"example.com","part":1,"path":"optimole-export-42/example.com/images-1.zip","status":"downloaded","size":1073741824,"seconds":41}
+{"summary":{"id":42,"status":"completed","dir":"optimole-export-42","downloaded":2,"present":0,"failed":0,"bytes":1932735283,"failed_domains":[],"interrupted":false,"timed_out":false}}
+```
+
+Each look at the export reports its domains first, then the export's own status. An archive's `status` is
+`downloaded`, `present` (already on disk) or `failed` (with `error`). `created` is `false` when the command
+follows an export that was already running.
+
+An export is `queued`, `running`, `completed`, `failed` or `cancelled`. Once every domain has finished, it is
+`failed` if one of them failed; the other domains' archives are still downloaded, and the command exits 1. An account keeps at most 5
+exports (`export_limit_reached`); delete one with `optimole export delete ID`, or wait: finished exports are
+deleted automatically 10 days after they finish.
+
 ### `optimole login`, `optimole logout`
 
 `login` reads the key from stdin: type or paste it at the prompt, or pipe it (`optimole login < key.txt`;
@@ -258,9 +362,9 @@ Prints the version, commit and build date.
 
 | Code | Meaning |
 |---|---|
-| 0 | Every selected file was uploaded (skipped files do not count). |
-| 1 | At least one file failed, there was nothing to upload, or the output could not be written. |
-| 2 | Usage or configuration error (bad flags, a directory without `-r`, missing key, non-https URL). Nothing was uploaded. |
+| 0 | Every selected file was uploaded (skipped files do not count). For `export start` and `export download`: the export completed and every selected archive is on disk (`export start --no-wait`: the export was started, or one was already running). |
+| 1 | At least one file failed, there was nothing to upload, or the output could not be written. For exports: the export failed or was cancelled, an archive failed, `--timeout` passed, or the account keeps the most exports it may. |
+| 2 | Usage or configuration error (bad flags, a directory without `-r`, an unknown `--domain`, missing key, non-https URL). Nothing was uploaded or downloaded. |
 | 130 | Interrupted with Ctrl-C. |
 
 ## Limits
@@ -271,6 +375,8 @@ Prints the version, commit and build date.
 - The API allows 300 requests per minute per account; each file takes 2 API requests plus the storage upload,
   so very large batches may see `429` answers, which the CLI waits out.
 - A presigned upload URL is valid for 15 minutes.
+- An account runs one export at a time and keeps at most 5. Export archives are about 1 GB each (a part can be
+  larger when its first files are), and their download URLs are valid for a day.
 
 ## Development
 

cmd/optimole/main.go

@@ -1,4 +1,5 @@
-// Command optimole uploads local files into an Optimole media library.
+// Command optimole uploads local files into an Optimole media library and
+// exports the library as ZIP archives.
 package main
 
 import (

internal/api/client.go

@@ -1,6 +1,7 @@
 // Package api is a small client for the parts of the Optimole dashboard API
 // the CLI uses: reserving an asset, sending its bytes to the presigned storage
-// URL, completing it, and listing folders.
+// URL, completing it, listing folders, and running asset exports and
+// downloading their archives.
 package api
 
 import (
@@ -108,9 +109,10 @@ func New(opts Options) (*Client, error) {
 			// GET or send the key somewhere unexpected.
 			CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
 		},
-		// No overall timeout: a 100 MB body can take a while. The context and
-		// the transport's header timeout bound it instead. Redirects are
-		// refused: following one would replay the file to another host.
+		// No overall timeout: a 100 MB upload or a 1 GB archive download can
+		// take a while. The context and the transport's header timeout bound
+		// it instead. Redirects are refused: following one would replay the
+		// file to another host.
 		storage: &http.Client{
 			Transport:     transport,
 			CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },

internal/api/errors.go

@@ -23,6 +23,11 @@ const (
 	CodeFolderRequired = "folder_required"
 	// CodeFolderCreationForbidden: a folder-scoped key asked to create folders.
 	CodeFolderCreationForbidden = "folder_creation_forbidden"
+	// CodeExportInProgress: an export is already queued or running (its ID
+	// comes in ExportID).
+	CodeExportInProgress = "export_in_progress"
+	// CodeExportLimitReached: the account keeps the most exports it may.
+	CodeExportLimitReached = "export_limit_reached"
 )
 
 // Error is a non-2xx answer from the dashboard API.
@@ -34,8 +39,10 @@ type Error struct {
 	Message    string
 	// Fields holds the validation messages of a 422, keyed by field.
 	Fields map[string][]string
-	// Limit and Count come with upload_limit_reached.
+	// Limit and Count come with upload_limit_reached and export_limit_reached.
 	Limit, Count FlexInt
+	// ExportID comes with export_in_progress.
+	ExportID FlexInt
 	// Expected and Actual come with upload_size_mismatch.
 	Expected, Actual FlexInt
 	// RetryAfter is the parsed Retry-After header, if any.
@@ -56,6 +63,9 @@ func (e *Error) Error() string {
 	if e.Limit.Valid || e.Count.Valid {
 		details = append(details, fmt.Sprintf("limit %s, used %s", flexString(e.Limit), flexString(e.Count)))
 	}
+	if e.ExportID.Valid {
+		details = append(details, fmt.Sprintf("export %d", e.ExportID.Value))
+	}
 	if e.Expected.Valid || e.Actual.Valid {
 		details = append(details, fmt.Sprintf("expected %s bytes, storage has %s", flexString(e.Expected), flexString(e.Actual)))
 	}
@@ -116,14 +126,16 @@ func (e *NetworkError) Error() string {
 
 func (e *NetworkError) Unwrap() error { return e.Err }
 
-// StorageError is a failed PUT to the presigned storage URL.
+// StorageError is a failed request to a presigned storage URL.
 type StorageError struct {
+	// Op names the request; empty means the upload PUT.
+	Op string
 	// StatusCode is 0 when no HTTP answer arrived (network error).
 	StatusCode int
 	Code       string
 	Message    string
 	Err        error
-	// Attempts is how many PUTs were made to the URL.
+	// Attempts is how many requests were made to the URL.
 	Attempts int
 	// Uncertain is set when an attempt got no answer or a 5xx, so the bytes
 	// may have been stored under this asset after all.
@@ -156,10 +168,14 @@ func (e *StorageError) Error() string {
 	if e.Expired {
 		extra = append(extra, "the upload URL expired")
 	}
+	op := e.Op
+	if op == "" {
+		op = "upload to storage"
+	}
 	if len(extra) == 0 {
-		return "upload to storage: " + msg
+		return op + ": " + msg
 	}
-	return fmt.Sprintf("upload to storage: %s (%s)", msg, strings.Join(extra, "; "))
+	return fmt.Sprintf("%s: %s (%s)", op, msg, strings.Join(extra, "; "))
 }
 
 func (e *StorageError) Unwrap() error { return e.Err }
@@ -193,6 +209,7 @@ func parseError(op string, resp *http.Response, body []byte, now time.Time) *Err
 		Count    FlexInt                    `json:"count"`
 		Expected FlexInt                    `json:"expected"`
 		Actual   FlexInt                    `json:"actual"`
+		ExportID FlexInt                    `json:"export_id"`
 	}
 	if err := json.Unmarshal(body, &payload); err != nil {
 		ct := resp.Header.Get("Content-Type")
@@ -206,6 +223,7 @@ func parseError(op string, resp *http.Response, body []byte, now time.Time) *Err
 	e.Code = payload.Code
 	e.Limit, e.Count = payload.Limit, payload.Count
 	e.Expected, e.Actual = payload.Expected, payload.Actual
+	e.ExportID = payload.ExportID
 	if len(payload.Errors) > 0 {
 		e.Fields = make(map[string][]string, len(payload.Errors))
 		for field, raw := range payload.Errors {