diff --git a/README.md b/README.md index 97f14e3..6878579 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,7 @@ # Livepeer Node -Livepeer Node contains services and command-line tools for the Livepeer network. -It includes an orchestrator for Livepeer runners, standalone discovery, -a remote signer for network payments and discovery filtering, and a protocol CLI -for managing wallets, staking, earnings, funding, governance, and local signing. +Livepeer Node provides runner orchestration, discovery, payment signing, and +protocol commands for the Livepeer network. | Executable | Role | | --- | --- | @@ -13,21 +11,34 @@ for managing wallets, staking, earnings, funding, governance, and local signing. | `livepeer-signer` | Signs identities and payments, discovers orchestrators, and optionally publishes accounting events. | | `livepeer-chain` | Inspects accounts and contracts, manages wallets, staking, earnings, funds, and governance, and signs files locally. | -The orchestrator supports persistent and single-shot sessions, HTTP streaming, -server-sent events (SSE), WebSockets, and Trickle channels. It can run locally -without payments or use Ethereum payment tickets. The signer requires chain -configuration. +## Install -## Build - -Install the Go version declared in [go.mod](go.mod) or newer, then run from the repository root: +Linux releases include all executables and need no Go installation. This example +uses the GitHub CLI, `tar`, and `sha256sum`; choose a release and architecture: ```sh -make build -bin/livepeer version -bin/livepeer orchestrator --help +set -eu +version=v1.0.5 +arch=amd64 # or arm64 +release_dir="$HOME/.local/share/livepeer/$version" +( + mkdir -p "$release_dir" + cd "$release_dir" + gh release download "$version" --repo livepeer/node \ + --pattern "livepeer-$version-linux-$arch.tar.gz" --pattern SHA256SUMS + sha256sum --check --ignore-missing SHA256SUMS + tar -xzf "livepeer-$version-linux-$arch.tar.gz" --strip-components=1 + sha256sum --check SHA256SUMS +) +export PATH="$release_dir/bin:$PATH" +livepeer version ``` +Keep the executables together: `livepeer` dispatches to its sibling components. +Persist the PATH setting in your shell or use absolute executable paths in your +service configuration. For other platforms or source builds, see +[development](docs/development.md). + ## Run a local example This example forwards a request to a small HTTP service. It runs off-chain. @@ -41,10 +52,10 @@ while true; do done ``` -In a second terminal, from the repository root, start the orchestrator: +In a second terminal, from the repository root with `livepeer` on PATH, run: ```sh -bin/livepeer orchestrator --config configs/orchestrator/local.example.toml +livepeer orchestrator --config configs/orchestrator/local.example.toml ``` In a third terminal, check readiness, discovery, and forwarding. Allow up to @@ -62,28 +73,11 @@ runner and orchestrator with Ctrl-C in their terminals. ## Configure a deployment -Use [configuration and secrets](docs/configuration.md) for credentials, encrypted -Ethereum keystores and password files, and probe addresses. Then follow the -component guides: - -- [Orchestrator](docs/orchestrator.md): dynamic/static runners, networking, TLS, - proxy URLs, limits, and payments. -- [Discovery](docs/discovery.md): standalone inventory collection, caching, and peer advertising. -- [Signer](docs/signer.md): chain access, price limits, authorization, and Kafka - accounting. -- [Chain commands](docs/chain.md): inspecting accounts, previewing and submitting - transactions, and signing files locally. -- [Payment recovery](docs/payment-recovery.md): backups, restart, and uncertain - transaction recovery. - -Chain state changes perform a dry run by default. Add `--submit` to broadcast -and wait for successful inclusion. Use `--no-wait` for a single transaction -without replacements. Trickle retains five segments per channel, with a 10 MB -limit per segment and no global byte cap. - -For tests, SDK fixtures, and packaging, see [development](docs/development.md). -The [documentation index](docs/README.md) links the operating guides and protocol -references. +Start with [configuration and secrets](docs/configuration.md), then choose a +component guide from the [documentation index](docs/README.md). Use +[payment recovery](docs/payment-recovery.md) for backups, upgrades, rollback, and +transaction reconciliation; [development](docs/development.md) covers tests, +SDK fixtures, and release packaging. Imported or adapted code retains its original MIT notice in [MIT notice](LICENSE). diff --git a/docs/README.md b/docs/README.md index d5c4940..3bdc3a3 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,7 @@ # Documentation -Start with the [local example](../README.md#run-a-local-example) to build the -executables and send a request through an orchestrator. +Start with [installation](../README.md#install) and the +[local example](../README.md#run-a-local-example). ## Operating a node @@ -12,7 +12,7 @@ executables and send a request through an orchestrator. | [Discovery](discovery.md) | Collect and advertise orchestrator inventories without local runners or payments. | | [Signer](signer.md) | Configure price limits, authorization, discovery, and Kafka accounting. | | [Chain commands](chain.md) | Inspect accounts, manage stake and funds, and submit transactions. | -| [Payment recovery](payment-recovery.md) | Back up state, recover accounting events, and reconcile ticket redemptions. | +| [Payment recovery](payment-recovery.md) | Back up state, upgrade or roll back, and recover accounting/redemption work. | ## Developing and integrating diff --git a/docs/chain.md b/docs/chain.md index f4cb58a..d32543b 100644 --- a/docs/chain.md +++ b/docs/chain.md @@ -1,32 +1,25 @@ # Chain commands -The `chain` command manages stake, earnings, deposits, and other on-chain state -related to the Livepeer protocol. Use it to check balances, stake LPT, register -an orchestrator, withdraw funds, or vote on protocol proposals. - -Run it as `bin/livepeer chain` or `bin/livepeer-chain`. Configure the blockchain -connection (RPC endpoint) using a +Run `livepeer chain` or `livepeer-chain` to inspect protocol state, manage funds +and stake, register orchestrators, and vote. Set RPC access in a [chain config](../configs/chain/config.example.toml) or environment variables. -Accounts can be inferred from the shared keystore. Use `--account` or -`LIVEPEER_CHAIN_ACCOUNT` to select one; see [configuration and secrets](configuration.md). +Accounts are inferred from the keystore; use `--account` or +`LIVEPEER_CHAIN_ACCOUNT` to select one. See [configuration](configuration.md). ```sh -bin/livepeer chain --config /etc/livepeer/chain.toml account get -bin/livepeer chain --config /etc/livepeer/chain.toml stake bond ADDRESS --amount 10 +livepeer chain --config /etc/livepeer/chain.toml account get +livepeer chain --config /etc/livepeer/chain.toml stake bond ADDRESS --amount 10 ``` -Transaction commands perform a dry run by default. Add `--submit` to sign and broadcast transactions and wait for -inclusion in a block. Submissions require a keystore matching the selected -account; reads and dry runs need no password. +Transactions dry-run by default. Add `--submit` to sign, broadcast, and wait for +block inclusion. Submissions need a keystore matching the selected account; +reads and dry runs need no password. -Prefix all commands below by `bin/livepeer chain`. Add `--config FILE` to load a configuration file, or `--help` to see a command's flags and defaults. +Prefix commands below with `livepeer chain`; add `--config FILE` or `--help` as needed. ## Accounts and protocol state -`account create --keystore-password-file /path/to/password` creates an encrypted -account in the shared keystore and prints its address. See -[keystore setup](configuration.md#prepare-a-keystore) for password preparation -and file overrides. +See [keystore setup](configuration.md#prepare-a-keystore) for account creation. | Command | Purpose | | --- | --- | @@ -66,12 +59,11 @@ stake withdraw --lock-id ID Use `stake get` to inspect stake, pending earnings, and the current orchestrator. -Bonding deposits liquid LPT from the wallet and includes any required ERC20 approval transaction. Redelegation +Bonding deposits wallet LPT, including any required ERC20 approval. Redelegation moves existing stake; choose one mode. Unbonding creates a withdrawal lock. -Cancel-unbond restores the locked LPT to stake. If the account is still staked, -`ADDRESS` is optional: the LPT returns to the current orchestrator. If fully unbonded, `ADDRESS` is -required to choose an orchestrator. +Cancel-unbond restores locked LPT to the current orchestrator if still staked; +otherwise `ADDRESS` is required to choose one. List outstanding locks with `stake locks`, filtering by either `--withdrawable` or `--locked`. Paginate with `--from-id` and `--limit` (default 100, maximum 1000). @@ -90,14 +82,11 @@ orchestrator reward-caller set ADDRESS orchestrator reward-caller unset ``` -Use `orchestrator get` for registration status and settings, and `orchestrator list` -for pool members. - -`register` registers the configured account as an orchestrator. Unless the account is already -self-bonded, choose one staking source: wallet LPT, existing stake, or an unbonding -lock. Omitting the optional service URI preserves its current value; supply one -when none is configured. Supplied URIs -must be absolute HTTP/HTTPS URLs without credentials, query strings, or fragments. +`get` shows registration/settings; `list` shows pool members. To `register` the +configured account, choose one staking source unless already self-bonded: +wallet LPT, existing stake, or an unbonding lock. An omitted service URI preserves +the current value; supply one if unset. URIs must be absolute HTTP/HTTPS without +credentials, queries, or fragments. `set-config` preserves omitted values; unchanged settings need no transaction. Commission changes require an unlocked round. Calling `reward-caller set` authorizes another account to call diff --git a/docs/configuration.md b/docs/configuration.md index 58467a7..2490a29 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,16 +1,18 @@ # Configuration and secrets -Services read TOML (`.toml`) or JSON (`.json`). Unknown keys are rejected; -migration and recovery commands read needed settings and ignore unrelated keys. -Use the [orchestrator](../configs/orchestrator/config.example.toml), +Services accept TOML or JSON and reject unknown keys. Migration and recovery +commands read only their relevant settings. Use the +[orchestrator](../configs/orchestrator/config.example.toml), [signer](../configs/signer/config.example.toml), [discovery](../configs/discovery/config.example.toml), and [chain](../configs/chain/config.example.toml) examples as starting points. -Configuration precedence is **CLI > environment > config > default**. Select an -explicit file with `--config FILE` or the component's `CONFIG` environment -variable; the flag wins. An explicit config can set both `Network` and `DataDir`. -`--config=""` disables discovery. Empty environment values are ignored. +## Select configuration and storage + +Precedence is **CLI > environment > config > default**; empty environment values +are ignored. Select a file with `--config FILE` or the component's `CONFIG` +environment variable. An explicit config can set `Network` and `DataDir`. +`--config=""` disables config discovery. `Network` defaults to `arbitrum-one-mainnet` for chain, signer, and the orchestrator's `redemptions` and `migrate` commands. The orchestrator service @@ -19,26 +21,22 @@ defaults. Custom networks require explicit chain and Controller settings. Signers and paid orchestrators also need a feed or fixed conversion rate. Paid orchestrators must select an on-chain network. RPC chain IDs are checked. -By default, component config and storage use `~/.lpData//`; -accounts use the shared `~/.lpData//keystore`. Network names must be -nonempty directory names, excluding `.` and `..`. - -A supplied `DataDir` is used directly, even when it equals the calculated default. -Discovery looks for `config.toml` there; database defaults are `payments.sqlite` -(orchestrator) and `events.sqlite` (signer). Accounts use `/keystore`. +Config and storage default to `~/.lpData//`; accounts use +`~/.lpData//keystore`. Explicit `DataDir` values are used directly and +move accounts to `/keystore`, even if the value matches the default. +Database filenames default to `payments.sqlite` (orchestrator) and `events.sqlite` +(signer). Network names must be nonempty directory names other than `.` or `..`. -Explicit configs determine the effective network before default directories are -calculated. Without `--config`, each command loads `config.toml` from its data -directory if present. That file can't set `DataDir`, and its `Network` must match -the network selected by flag, environment or default. For example, a paid -orchestrator using discovery needs `--network arbitrum-one-mainnet`. -Missing discovered configs are skipped; explicit configs must exist, even when -their path matches the discovery path. +Without `--config`, commands load `/config.toml` if present. That file +cannot set `DataDir`; its `Network` must match the flag/environment/default. +A paid orchestrator using config discovery therefore needs +`--network arbitrum-one-mainnet`. Explicit configs must exist and determine the +network before default directories are calculated. -Relative config and input paths from CLI/environment use the working directory. -Input paths in config files use that file's directory. Relative database paths -use the effective datadir. Absolute paths stay unchanged. These rules also apply -to account creation, migrations, and recovery commands. +Relative input paths use the working directory for CLI/environment values and +the config file's directory for file values. Relative database paths use the +effective datadir; absolute paths stay unchanged. This includes account, +migration, and recovery commands. ## Names and environment variables @@ -53,20 +51,16 @@ the component prefix and underscores: | Signer | `LIVEPEER_SIGNER_` | `LIVEPEER_SIGNER_RPC_URL` | | Chain | `LIVEPEER_CHAIN_` | `LIVEPEER_CHAIN_RPC_URL` | -Run `bin/livepeer COMPONENT --help` for flags and defaults. The orchestrator, -discovery, and chain support `--print-config` to print configuration -with secrets omitted: +Run `livepeer COMPONENT --help` for flags and defaults. Orchestrator, discovery, +and chain support `--print-config`: ```sh -bin/livepeer orchestrator --config /etc/livepeer/orchestrator.toml --print-config -bin/livepeer chain --config /etc/livepeer/chain.toml --print-config -bin/livepeer discovery --config /etc/livepeer/discovery.toml --print-config +livepeer orchestrator --config /etc/livepeer/orchestrator.toml --print-config ``` -Printed configuration also omits secret file paths and the static runner file -path. Chain action inputs, transaction switches, `--output`, and `--print-config` -must be supplied on the command line. Orchestrator `--print-config` and -redemption `--submit` are also command-line-only controls. +Output omits secrets, secret file paths, and the static runner file path. +Chain action inputs, transaction switches, `--output`, and `--print-config` are +CLI-only, as are orchestrator `--print-config` and redemption `--submit`. ## Destination grants @@ -79,11 +73,10 @@ Grants are checked against each resolved IP at connection time. ## Credential files -RPC URLs, bootstrap credentials, keystore passwords, webhook URLs and headers, -and Kafka credentials are secrets. Supply each through its file -option or direct environment variable. Direct secret flags and TOML values are -rejected, and a direct environment secret cannot be set together with its file -option. +Supply RPC URLs, bootstrap credentials, keystore passwords, webhook URLs/headers, +and Kafka credentials through file options or direct environment variables. +Direct secret flags and TOML values are rejected. An environment secret cannot +be combined with its file option. Credential files are read as exact bytes, including trailing newlines. For example, create a bootstrap credential without a newline: @@ -93,41 +86,29 @@ umask 077 openssl rand -hex 32 | tr -d '\n' > runner-bootstrap.secret ``` -Set `BootstrapSecretFile` to that file's absolute path and give the same -credential to runners during initial registration. Secret-manager mounts must -also contain exactly the intended bytes. URL credential files must contain a -valid URL without a trailing newline. +Set `BootstrapSecretFile` to its absolute path and use the same credential for +initial runner registration. Secret-manager mounts must preserve exact bytes; +URL files require a valid URL without a trailing newline. ## Ethereum keystores -All three apps discover encrypted geth accounts in the shared network keystore -when `DataDir` is absent, or `/keystore/` when it is supplied. -A sole account is selected automatically. Use `--account` (`Account` in TOML) to -select among accounts, or `--keystore-file` to select a file directly and bypass -discovery. - -| Keystore TOML key | Password-file TOML key | -| --- | --- | -| `KeystoreFile` | `KeystorePasswordFile` | +Chain, signer, and paid orchestrators discover encrypted geth accounts in the +keystore directory described above. A sole account is selected automatically; +use `Account` / `--account` to select among accounts or `KeystoreFile` / +`--keystore-file` to bypass discovery. -Signing requires the selected keystore's password. -Supply the password in a file (`--keystore-password-file`) or an environment -variable. Environment variables use the app prefix, such as -`LIVEPEER_SIGNER_KEYSTORE_PASSWORD_FILE` for a password file or -`LIVEPEER_SIGNER_KEYSTORE_PASSWORD` for the password itself. CLI values override -environment values, which override TOML. +Signing also requires `KeystorePasswordFile` / `--keystore-password-file` or an +environment password. For signer, these are +`LIVEPEER_SIGNER_KEYSTORE_PASSWORD_FILE` and `LIVEPEER_SIGNER_KEYSTORE_PASSWORD`. Keystore and password files must be regular files readable only by their owner; `0400` and `0600` are accepted. Secret-manager symlinks are accepted when their -target files meet these requirements. Password files are read as exact bytes, -including whitespace and trailing newlines. Supply the actual password without -an unintended newline. To use a keystore without a password, supply an empty -password file. +target files meet these requirements. Password files follow the exact-byte +rules above; supply an empty file for a keystore without a password. -The signer requires a keystore and password at startup. The orchestrator requires -both when payments are configured. The chain requires both for submissions and -local signing. Changing the keystore or password requires a service restart, or -a new chain invocation. +Signer and paid orchestrators require these credentials at startup; chain needs +them for submissions and local signing. Credential changes require a restart +or a new chain invocation. The signer account is the payer, supplying the TicketBroker deposit and reserve. The orchestrator's keystore account signs redemptions and pays gas. Ticket payments @@ -138,23 +119,19 @@ tracking. ### Prepare a keystore -Create an encrypted account locally with the chain CLI. First prepare an -owner-only password file in an existing directory: +Create an account without RPC access; choose a password path in an existing directory: ```sh umask 077 openssl rand -hex 32 | tr -d '\n' > /path/to/password -bin/livepeer chain account create \ +livepeer chain account create \ --keystore-password-file /path/to/password ``` -Creation requires no RPC or account address. It encrypts a freshly generated key -with geth's standard scrypt settings and writes the keystore with `0600` -permissions in the shared network keystore, or `/keystore/` when -`DataDir` is supplied. An optional `--keystore-file` chooses a -new file in an existing parent directory; existing paths are never overwritten. The -command prints the new public address; add `--output json` for scripts. The -password can also come from `LIVEPEER_CHAIN_KEYSTORE_PASSWORD`. +The command uses geth's standard scrypt settings, saves the keystore with `0600` +permissions, and prints its address (`--output json` for scripts). +`--keystore-file` selects a new path in an existing parent directory; existing +files are never overwritten. No account address is required. ## Local storage and probes diff --git a/docs/development.md b/docs/development.md index 99b2216..06e6b76 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,16 +1,18 @@ # Development -The minimum Go version is declared in [go.mod](../go.mod). CI and release metadata -use that version; keep the digest-pinned Go builder in `docker/Dockerfile` aligned -with it. Build all executables with: +Install the Go version in [go.mod](../go.mod) or newer. From the repository root: ```sh make build -bin/livepeer version -bin/livepeer orchestrator --help -bin/livepeer completion orchestrator bash +export PATH="$PWD/bin:$PATH" +livepeer version +livepeer orchestrator --help +livepeer completion orchestrator bash ``` +CI and release metadata use the declared Go version; keep the digest-pinned +builder in `docker/Dockerfile` aligned with it. + ## Releases Set the [VERSION](../VERSION) file to the next `MAJOR.MINOR.PATCH` number without @@ -49,9 +51,9 @@ Integration tests use these pinned revisions: | Go: `livepeer/golang-runner` | `c3be5a14a91f8a3437133419becaced324d804fe` | `GO_RUNNER_SDK_DIR` | | Python: `livepeer/livepeer-python-gateway` | `76a1a1c58656ef2ad0c28542ea5f99086b76db69` | `PYTHON_RUNNER_SDK_DIR` | -SDK tests always export the pinned revision, regardless of the checkout's branch -or local edits, and fetch it into a temporary repository if it is missing locally. -To provision fixtures in a temporary directory (requires Git, Python 3.12, and `uv`): +Tests export the pinned committed trees, leaving development checkouts unchanged, +and fetch missing revisions into a temporary repository. To provision fixtures +(requires Git, Python 3.12, and `uv`): ```sh livepeer_fixture_dir=$(mktemp -d) @@ -68,9 +70,8 @@ go test -race ./... Without explicit paths, tests look for sibling `golang-runner` and `python-runner` checkouts and the Python checkout's `.venv/bin/python`. SDK -coverage can be skipped when the checkouts or dependencies are absent, so read -the test logs before reporting a compatibility result. Both SDK fixtures use -their pinned committed trees; development checkouts are left unchanged. +coverage can be skipped when checkouts or dependencies are absent; check the +logs before reporting compatibility. The fixtures cover persistent and single-shot sessions, proxy modes, discovery, runner callbacks, Trickle, and Python media/control round trips. Payment tests diff --git a/docs/discovery.md b/docs/discovery.md index 45c3174..4e088fe 100644 --- a/docs/discovery.md +++ b/docs/discovery.md @@ -7,13 +7,12 @@ Copy the [configuration example](../configs/discovery/config.example.toml), set `Nodes` to your source base URLs, and run: ```sh -bin/livepeer discovery --config /etc/livepeer/discovery.toml +livepeer discovery --config /etc/livepeer/discovery.toml ``` -The standalone `bin/livepeer-discovery` accepts the same arguments. -Configuration follows the [shared conventions](configuration.md), with the -`LIVEPEER_DISCOVERY_` environment prefix and default directory -`~/.lpData/offchain/discovery`. `--print-config` prints the effective TOML settings. +`livepeer-discovery` accepts the same arguments. See +[shared configuration](configuration.md) for environment variables, paths, and +`--print-config`; the default directory is `~/.lpData/offchain/discovery`. ## Sources and cache @@ -29,14 +28,11 @@ including when empty. Failures retain the cached inventory and last successful query time until `MaxFailedRefreshes` consecutive failures (default `6`). Set `0` to evict on the first failure. Polling continues after eviction. -Keep the source topology acyclic. For example, if A queries an orchestrator and -B, while B queries A, both discovery services can keep circulating a runner -after its orchestrator withdraws it or goes offline. Peer queries still succeed, -so failure-based eviction does not remove those copies; `observed_at` is preserved -but has no age limit. This also applies to longer cycles and self-references. -Have discovery services query orchestrators directly, or arrange aggregation in -one direction with no path back to an earlier collector. Advertised node metadata -does not create these connections; only configured `Nodes` are queried. +Keep configured `Nodes` acyclic, including no self-references. If A queries an +orchestrator and B, while B queries A, peers can circulate a withdrawn runner +indefinitely: peer queries succeed and preserved `observed_at` has no age limit. +Query orchestrators directly or aggregate in one direction. Advertised node +metadata creates no connections. ## HTTP and advertising diff --git a/docs/http-api.md b/docs/http-api.md index a9810d0..597f985 100644 --- a/docs/http-api.md +++ b/docs/http-api.md @@ -28,13 +28,11 @@ See [orchestrator setup](orchestrator.md), [signer setup](signer.md), and SDK revisions, fixture setup, and compatibility test coverage are documented in [development](development.md#sdk-integration-fixtures). -`RunnerServiceURL` can advertise a separate runner-facing base URL for the -heartbeat `orchestrator` and orchestrator-to-runner (O2R) control channel, -session callback header, and trickle -`internal_url`. Public discovery, session, proxy and channel URLs use -`ServiceURL`. The default generated proxy path is `/run/{proxy}`; -`/proxy/{proxy}` remains an accepted alias. O2R channels send periodic -`{"keep":"alive"}` messages while the runner is registered. +Registration defaults capacity to one and trims/lowercases price units and +currencies. See [public URLs](orchestrator.md#public-urls-and-tls) for +`RunnerServiceURL`, and [sessions and proxies](orchestrator.md#sessions-and-proxies) +for pricing, release behavior, and proxy routes. O2R channels send +`{"keep":"alive"}` periodically while the runner is registered. ## Discovery HTTP @@ -114,6 +112,9 @@ Success returns `payment`, `segCreds`, and `state`. Send the first two as the orchestrator's `Livepeer-Payment` and `Livepeer-Segment` headers. State is bound to the ticket recipient, application, payment type, and manifest ID. Earlier state can be replayed; HTTP retries can generate separate tickets. +Parameter refresh preserves the state sequence and resets the ticket nonce for +new randomness. Compatible signer replicas can continue client-held state; +recipients reject duplicate tickets. Requests are limited to 1 MiB, with a five-second body deadline, a 30-second request deadline, and 64 concurrent requests. Oversized payment request bodies @@ -150,12 +151,9 @@ the provenance of an arbitrary challenge. ### Authorization webhook -To approve or reject payments, configure an authorization webhook. The signer -sends it a `POST` with `Content-Type: application/json` before signing. The body -contains `headers` (the client's request headers, with an array of values per -name) and `state` (the proposed payment state). Use these to identify the caller -and evaluate the payment against your policy. The ticket batch is not included. -Example request, showing selected state fields: +The [configured webhook](signer.md#authorization) receives a JSON `POST` before +signing. Its body contains client `headers` (arrays of values) and proposed +payment `state`, without the ticket batch. Example with selected state fields: ```json { @@ -209,19 +207,7 @@ entries. Names are case-insensitive; repeated names retain multiple values. Quote an entire entry using CSV syntax when its value contains commas, for example `"X-List: a,b",Authorization: Bearer credential`. -`pm/wire` retains only the consumed fields and original field numbers: -`OrchestratorInfo` 1–4 and 6, `TicketParams` 1–7, `PriceInfo` 1–2, -`AuthToken` 1–3, `TicketExpirationParams` 1–2, `Payment` 1–5, payer params -1–2, and `SegData` 1, 3, 5 and 8. Other fields are skipped on decode. -Binary fixtures produced by the pinned Python runner check the wire -format in `pm/wire/wire_test.go`. - -## Lifecycle and response codes - -Registration defaults omitted capacity to one and normalizes price unit and -currency whitespace and case. See [sessions and proxies](orchestrator.md#sessions-and-proxies) -for session pricing and release behavior, and the [Trickle protocol](../trickle/README.md) -for channel behavior. +### Response codes | Signer status | Client action | | --- | --- | @@ -235,18 +221,6 @@ for channel behavior. | 502 | Restore the authorization webhook or correct its response. | | 503 | Retry after payment dependencies, discovery sources, or accounting storage recover. | -Healthy empty discovery returns HTTP 200 with `[]`. Rate and readiness policy -are described in [the signer guide](signer.md). Signed state preserves its -sequence through parameter refresh and resets the ticket nonce for new -randomness. Replicas with the same key and webhook configuration can continue -that state, but there is no server-side replay database: earlier state can be -reused and retries can receive different tickets. Recipients reject duplicates. - -Signed state acts as a bearer credential during cached authorization. The -webhook receives the proposed payment state; payment data is returned only -after approval. See [authorization](signer.md#authorization) for identity and -cache behavior. - ## Accounting events Optional Kafka accounting emits `create_signed_ticket` JSON events with @@ -268,46 +242,17 @@ or exhausted capacity returns 503. Kafka outages allow signing while the outbox has capacity. See [recovery](payment-recovery.md#signer-accounting-recovery) for backups and replay after restart. -The metrics listener (default `127.0.0.1:8938`) exposes: - -- `livepeer_signer_up` -- `livepeer_signer_kafka_pending_events` -- `livepeer_signer_kafka_pending_bytes` -- `livepeer_signer_kafka_oldest_event_age_seconds` -- `livepeer_signer_kafka_outbox_healthy` -- `livepeer_signer_kafka_publish_errors_total` -- `livepeer_signer_kafka_enqueue_errors_total` -- `livepeer_signer_kafka_storage_errors_total` - -`/healthz` reports process liveness. `/readyz` requires chain/funds access, a -positive deposit and total reserve, no withdrawal scheduled at or before the -next round, a fresh rate, and healthy enabled outbox storage with capacity. -Kafka connectivity alone does not make readiness fail. Recipient-specific -claimable reserve is checked during payment generation. - -Kafka settings use `[Kafka]` in TOML, `--kafka-*` flags, and -`LIVEPEER_SIGNER_KAFKA_*` environment variables. Supplying any setting enables -accounting and requires `Broker` and `Topic`. A broker accepts one `host[:port]` -(bracket IPv6), with TLS/system trust by default; `kafkas://` selects TLS and -`kafka://` selects plaintext. Credentials, paths, queries, and fragments are -rejected. Advertised cluster brokers must support the same transport and auth. - -Paired username/password secrets enable SASL; use files or -`LIVEPEER_SIGNER_KAFKA_USERNAME` and `LIVEPEER_SIGNER_KAFKA_PASSWORD`. -`AuthMethod` accepts `plain`, `scram-sha-256`, or `scram-sha-512` (default). -Without credentials, SASL is disabled. An explicit port overrides these signer -conventions: - -| Connection | Default port | -| --- | --- | -| Plaintext without SASL | 9092 | -| TLS without SASL | 9093 | -| SASL/SCRAM, with or without TLS | 9096 | -| SASL/PLAIN, with or without TLS | 9094 | +See signer [monitoring](signer.md#readiness-and-monitoring) and +[Kafka configuration](signer.md#kafka-accounting-and-budgets). ## Codec checks -The selected-field Protobuf decoder is compared with the generated runtime for -merged message fields, unknown and wrong-wire fields, UTF-8, and bounds. -Ticket nonces outside uint32 are rejected. See [development](development.md) -for fixtures and test commands. +`pm/wire` retains consumed fields with their original Protobuf numbers: +`OrchestratorInfo` 1–4 and 6, `TicketParams` 1–7, `PriceInfo` 1–2, +`AuthToken` 1–3, `TicketExpirationParams` 1–2, `Payment` 1–5, payer params +1–2, and `SegData` 1, 3, 5 and 8. Other fields are skipped. + +Python fixtures in `pm/wire/wire_test.go` check binary compatibility. Differential +tests compare the decoder with the generated runtime for merged messages, +unknown/wrong-wire fields, UTF-8, and bounds. Nonces outside uint32 are rejected. +See [development](development.md) for fixtures and test commands. diff --git a/docs/orchestrator.md b/docs/orchestrator.md index de61288..104087b 100644 --- a/docs/orchestrator.md +++ b/docs/orchestrator.md @@ -1,17 +1,15 @@ # Running an orchestrator -The orchestrator registers HTTP runners, advertises them through discovery, -reserves sessions, and forwards application traffic. It can run in on-chain or -off-chain mode, supporting persistent sessions and single-shot requests, HTTP -streaming, server-sent events (SSE), WebSockets, callbacks, and -[Trickle channels](../trickle/README.md). +The orchestrator registers and advertises runners, reserves sessions, and +forwards HTTP, SSE, WebSocket, and [Trickle](../trickle/README.md) traffic. +It supports persistent sessions and single-shot requests, on-chain or off-chain. Start with the [local example](../README.md#run-a-local-example). For deployment, copy the [configuration example](../configs/orchestrator/config.example.toml), set its paths and destinations, and run: ```sh -bin/livepeer orchestrator --config /etc/livepeer/orchestrator.toml +livepeer orchestrator --config /etc/livepeer/orchestrator.toml ``` ## Register runners @@ -24,28 +22,22 @@ heartbeats and unregister requests. Set `BootstrapSecretFile` as described in [configuration and secrets](configuration.md#credential-files). If you use only dynamic runners, omit `RunnerConfig`. -Static runners are loaded from `RunnerConfig` using `[[Runners]]` tables with -Go field names, including `RunnerURL`, `HealthURL`, and `HealthCode`. Optional -GPU and price tables use `[Runners.GPU]` and `[Runners.PriceInfo]`. Use the -[runner example](../configs/orchestrator/runners.example.toml) to set each -runner's ID, URL, application, mode, capacity, and health check. If you use only -static runners, omit `BootstrapSecretFile`. Their health URLs are checked -every `HeartbeatInterval` (five seconds by default). A runner with a health URL -is unavailable until its first successful check; three consecutive failed checks -release its sessions. +Static runners load from `RunnerConfig`: `[[Runners]]` tables with Go field +names such as `RunnerURL`, `HealthURL`, and `HealthCode`, plus optional +`[Runners.GPU]` and `[Runners.PriceInfo]`. Follow the +[runner example](../configs/orchestrator/runners.example.toml); omit +`BootstrapSecretFile` for static-only operation. Health URLs are checked every +`HeartbeatInterval` (default five seconds). A runner with a health URL is unavailable +until its first successful check; three consecutive failures release its sessions. -Discovery at `GET /discovery` lists available runners. Off-chain operation -accepts quoted runner prices but advertises free service. Enable the payment -settings below to charge clients. - -The orchestrator advertises its own runners. Multiple orchestrators can -be aggregated with a standalone [discovery service](discovery.md). +`GET /discovery` lists this orchestrator's available runners; aggregate multiple +orchestrators with [discovery](discovery.md). Off-chain operation accepts quoted +prices but advertises free service. Enable payments below to charge clients. ## Allow local and private destinations -Runner forwarding, callback-created session proxies, and static health checks -have separate [destination grants](configuration.md#destination-grants). Use -the corresponding setting to allow connections to local or private addresses: +Use separate [destination grants](configuration.md#destination-grants) for local +or private connections: | Setting | Permits private destinations for | | --- | --- | @@ -83,9 +75,9 @@ its HTTPS address. The separate metrics listener must remain on loopback; see ## Sessions and proxies -Each session keeps its reserved price and paid status. Later runner heartbeats or -exchange-rate updates affect new sessions. Every release immediately frees -capacity, revokes the session token, and removes session channels and proxies. +Each session keeps its reserved price and paid status; heartbeat and exchange-rate +updates affect new sessions. Releasing a session immediately frees capacity, +revokes its token, and removes its channels and proxies. A runner stop gives in-flight HTTP, SSE, and WebSocket requests up to one second to finish; all other releases cancel them immediately. This applies to both persistent and single-shot sessions. @@ -145,10 +137,7 @@ The registry accepts up to 256 runners and 10,000 sessions, with 25 generated proxies per session. Control requests have 64 admission slots and a five-second body deadline. Streaming requests have 256 separate slots. -The session channel API checks a 1,024-entry channel quota with 256 entries -reserved for runner controls. This is not a global Trickle memory limit. -Trickle retains five segments per channel and limits each segment to 10 MB -(10,000,000 bytes). There is no global byte cap or rolling window that allows a -segment to exceed that limit. Size memory for the channels and segments in use. -See the [protocol reference](../trickle/README.md) for missing-segment and -streaming behavior. +The session channel quota is 1,024 entries, with 256 reserved for runner controls. +Trickle retains five segments per channel, each capped at 10 MB (10,000,000 bytes). +There is no global byte cap; size memory for active channels and segments. See +the [protocol reference](../trickle/README.md) for streaming behavior. diff --git a/docs/payment-recovery.md b/docs/payment-recovery.md index 63dc252..786de99 100644 --- a/docs/payment-recovery.md +++ b/docs/payment-recovery.md @@ -1,32 +1,80 @@ # Payment state and transaction recovery -The orchestrator database contains winning ticket signatures and the recipient -randomness needed to redeem those winners. Keep it owner-only and backed up. -Payment parameters, authentication keys, session balances, and ticket nonce -replay guards are not persisted. Restarting rotates the keys and requires new -paid sessions; stored winning tickets remain redeemable. +SQLite preserves winning tickets and redemption transactions for +[paid orchestrators](orchestrator.md#enable-payments), and accounting events for +[Kafka-enabled signers](signer.md#kafka-accounting-and-budgets). Orchestrator +sessions, payment parameters, balances, and replay guards are process-local; +restarts rotate temporary keys and start fresh sessions. Stored winners remain +redeemable. Also start fresh sessions after restoring recipient state or +migrating legacy signed state. -Clients carry the signer's signed payment state, which is a bearer credential -while authorization is cached. When Kafka is enabled, each signer also owns a -durable accounting-event outbox. Back up that outbox and the clearinghouse's -accounting state independently. Start fresh client sessions after restoring -recipient state or migrating from legacy signed state. +## Back up SQLite -See [orchestrator setup](orchestrator.md#enable-payments) and -[signer setup](signer.md#kafka-accounting-and-budgets) for configuration. +Run as the service owner. Use a persistent, owner-only directory (`0700`) and +keep databases and WAL/SHM sidecars owner-only. The orchestrator database contains +ticket signatures and redemption randomness; signer sidecars can contain +accounting payloads. Signer startup rejects permissive or nonregular files. -## Signer accounting recovery +With the `sqlite3` CLI, back up either database while it is running. Set `db` to +the actual `RedeemerDB` or `Kafka.OutboxDB` path: + +```sh +( + set -eu + umask 077 + db="$HOME/.lpData/arbitrum-one-mainnet/orchestrator/payments.sqlite" + test -f "$db" + backup_dir=$(mktemp -d "$HOME/livepeer-backup.XXXXXX") + sqlite3 "$db" ".timeout 5000" ".backup '$backup_dir/state.sqlite'" + test "$(sqlite3 "$backup_dir/state.sqlite" 'PRAGMA integrity_check;')" = ok + printf 'Backup: %s\n' "$backup_dir/state.sqlite" +) +``` + +The backup API includes committed WAL contents; no sidecar copies are needed. +For a file copy instead, stop the service and copy the database plus any WAL/SHM +files together. Preserve ownership and permissions on restore. Back up the +clearinghouse's accounting state independently of each signer's outbox. + +## Upgrades and rollback + +Keep the previous executable and config. [Install](../README.md#install) the new +bundle in a separate directory, stop the affected service through its supervisor +(Ctrl-C for foreground examples), wait for exit, and back up its current state. +Startup applies pending migrations; inspect them before starting. This example +uses an unchanged config and a paid orchestrator: + +```sh +previous="$HOME/.local/share/livepeer/v1.0.4/bin/livepeer" +candidate="$HOME/.local/share/livepeer/v1.0.5/bin/livepeer" +config=/etc/livepeer/orchestrator.toml +"$candidate" orchestrator migrate --config "$config" status && + "$candidate" orchestrator --config "$config" +``` + +Check [probes](configuration.md#local-storage-and-probes), discovery, and a +representative application/payment request. To roll back, stop the candidate, +then check the previous binary against the **current** database: + +```sh +"$previous" orchestrator migrate --config "$config" status && + "$previous" orchestrator --config "$config" +``` -Keep `Kafka.OutboxDB` on persistent local storage, with one file per signer replica. -Missing parent directories are created at startup. New files have mode 0600; -existing files must be owner-only regular files. SQLite uses WAL and FULL synchronous -commits. Back up with SQLite's backup API or stop the signer before copying the database -and any WAL/SHM files. Monitor both pending payload bytes and actual disk usage. +Use `signer` and its config for an enabled outbox; omit migration checks for +off-chain orchestrators or signers without Kafka. Restore the previous config +if it changed, keeping network/database paths consistent. Check release notes: +migration status verifies history, not all application compatibility. Unknown +versions or changed migration checksums block downgrade. `migrate down` destroys +data, including payment/accounting tables in the initial migrations. If downgrade +is incompatible, retain the latest state for a compatible fix; restoring an old +snapshot can lose events, winners, and transaction identities. + +## Signer accounting recovery -Use a dedicated owner-only directory (mode 0700) and preserve owner-only -permissions on the database, `-wal`, and `-shm` files during backup and restore. -Startup rejects nonregular or permissive database and sidecar files. The WAL -can contain accounting payloads. +Keep `Kafka.OutboxDB` on persistent local storage, one file per replica. It uses +WAL and FULL synchronous commits; new files are `0600`. Monitor pending payload +bytes and actual disk use. Restart with the same signer key, Kafka broker, topic and outbox path. Pending events resume automatically with their original IDs and exact payloads. A nonempty outbox @@ -35,11 +83,10 @@ settings; credential rotation does not change its destination binding. Broker addresses learned from Kafka metadata can change without changing the bootstrap configuration. Never delete pending rows or replace the outbox to clear an outage. -Only acknowledged events are removed. A crash after Kafka accepts a batch but before -local deletion causes replay; the consumer must deduplicate event IDs. A partial or -uncertain write retains the entire batch. The outbox is a delivery queue, not an audit -archive: acknowledged payloads are deleted. Restoring a backup can replay old events -and omit events created after that backup, so preserve the latest outbox during rollback. +Only acknowledged events are removed. A crash before local deletion, or a partial +or uncertain write, replays the batch; consumers must deduplicate event IDs. +The outbox is a delivery queue, not an audit archive. Restoring a backup can replay +old events and omit newer ones, so preserve the latest outbox during rollback. Kafka outages allow signing to continue until the outbox runs out of capacity. Failed local persistence returns 503 without payment credentials. Events are saved before the @@ -102,13 +149,11 @@ session activity current. Expiry is monotonic within a recipient lifetime so a backward chain observation cannot revive parameters after their replay guards are removed. Auth-token refresh never resets nonce tracking. Winning tickets and uncertain transactions are retained for audit/reconciliation, so back up -and monitor database disk use independently of these control-state quotas. +and monitor database disk use independently of these quotas. Redemption planning uses geth's dynamic-fee selection and requires a base fee in the RPC header; a missing base fee leaves the ticket queued with an error. `RedeemerMaxFeePerGas` optionally caps the fee in wei per gas; a plan above that ceiling remains queued without signing or broadcasting. Prepared and -submitted identities reserve their nonces, including after restart. Keep the -redemption key exclusive to this database/worker; local nonce tracking does not -coordinate another process using the same key. Explicit retry always resends -the stored transaction without changing its fees. +submitted identities reserve their nonces across restarts. Explicit retry +resends the stored transaction without changing its fees. diff --git a/docs/signer.md b/docs/signer.md index 3d12d44..552485e 100644 --- a/docs/signer.md +++ b/docs/signer.md @@ -1,23 +1,8 @@ # Remote signer -The remote signer authorizes payments for work on the Livepeer network. It -holds an Ethereum key and signs payment tickets on behalf of clients, so an -application can pay for work without managing private keys or connecting to -Ethereum itself. - -Running the signer as a separate service keeps keys out of the application and -media processing path. It also lets a payment operator manage funding, price -limits, and authorization independently of the applications using the network. -Clients can run in browsers, on mobile devices, or in backend services while -payment signing stays with the signer. - -The signer can also provide **discovery**: finding orchestrators that offer the -application a client wants to run, along with their available runners and -prices. Runners execute the work; orchestrators make those runners available -to clients and collect payment. - -At a high level, the client uses the signer to find work providers and arrange -payment, then sends its application requests to the chosen orchestrator: +The signer holds an Ethereum key, authorizes and signs client payments, and +discovers orchestrators and their runners. Clients need neither private keys +nor RPC access; application traffic goes directly to the orchestrator: ```mermaid sequenceDiagram @@ -26,15 +11,12 @@ sequenceDiagram participant Orchestrator opt Find a runner through discovery Client->>Signer: Find runners for an application - Signer->>Orchestrator: Query available runners and prices - Signer-->>Client: Matching runners, prices, and URLs + Signer-->>Client: Cached matching runners, prices, and URLs end Client->>Orchestrator: Request work Orchestrator-->>Client: Payment requirements Client->>Signer: Request payment - Note over Signer: Check limits and draft payment - Note over Signer: Authorize new payment state - Note over Signer: Sign payment and record for accounting + Note over Signer: Check limits, authorize, sign, and record accounting Signer-->>Client: Signed payment Client->>Orchestrator: Submit payment and use the application ``` @@ -57,14 +39,12 @@ defaults are for Arbitrum mainnet; override them for another chain. See Before making payments, fund the signer's account with both a TicketBroker deposit and reserve using the [chain CLI](chain.md). -Start the signer: +Start the signer; check readiness from another terminal: ```sh -bin/livepeer signer --config /etc/livepeer/signer.toml +livepeer signer --config /etc/livepeer/signer.toml ``` -Check readiness from another terminal: - ```sh curl -fsS http://127.0.0.1:8938/readyz ``` @@ -75,10 +55,6 @@ and configure TLS termination at the proxy. ## Configure discovery -Discovery helps a client choose where to send work. A client can ask which -orchestrators have runners for a particular application or GPU, inspect their -advertised prices and capacity, and use a returned runner URL to start work. - Set `Orchestrators` to orchestrator or standalone [discovery](discovery.md) base URLs to query. Both expose the same inventory API. Local or private sources also need [destination grants](configuration.md#destination-grants). For example: @@ -95,29 +71,17 @@ the last inventory until `DiscoveryMaxFailedRefreshes` consecutive failures (default `6`, nonnegative) remove it; `0` evicts on the first failure. Recovered sources reappear on subsequent polls. -Query the signer to find runners: +Query by application and GPU; omit either filter to broaden the results: ```sh -# List all advertised runners -curl -fsS 'http://127.0.0.1:8937/discover-orchestrators' - -# Find runners for an application -curl -fsS 'http://127.0.0.1:8937/discover-orchestrators?app=example' - -# Also require a particular GPU name curl -fsS 'http://127.0.0.1:8937/discover-orchestrators?app=example&gpu=H100' ``` -Results include the orchestrator service URL and each matching runner's -application, URL, mode, capacity, GPU information when available, and advertised -price. Both filters must match when `app` and `gpu` are supplied together. - -Discovery reports what the sources advertise. Price limits are checked when -the signer generates a payment, so an advertised runner may still exceed your -configured limit. Finding a runner also does not reserve its capacity; the -client must request a session or submit work at the returned URL. See the -[discovery reference](http-api.md#discovery-filters-and-results) for result -fields and filtering details. +Results describe matching runners, prices, capacity, and application URLs. +Discovery neither reserves capacity nor applies payment ceilings; clients must +request work, and signing enforces price limits. See the +[discovery reference](http-api.md#discovery-filters-and-results) for fields and +filter rules. ## Orchestrator discovery webhook @@ -136,34 +100,21 @@ GET at the webhook URL returns `[{"address":"https://orch.example.com/discovery" Addresses are complete HTTP(S) discovery URLs without credentials, queries, or fragments. Invalid addresses and extra fields are ignored. -The webhook has a three-second timeout and 1 MiB response limit; redirects -are rejected. - -The signer fetches at startup and periodically (positive interval, default `5m`). -Until the first successful list, discovery requests also trigger or await a fetch; -failures return 503. Later requests use the cached list without waiting for refreshes. -On refresh failure, the last cached list is used. A successful empty response clears it. - -Sources returned by the webhook use the inventory polling settings above. +Fetches run at startup and every positive `OrchestratorWebhookRefreshInterval` +(default `5m`), with a three-second timeout, 1 MiB limit, and no redirects. +Before the first successful list, discovery requests trigger or await a fetch; +failures return 503. Later requests use the cache during refreshes and failures; +a successful empty response clears it. Returned sources use the inventory +polling settings above. ## Connect a client -Configure the client with the signer's base URL, such as -`https://signer.example.com`, and the credentials required by your authenticated -proxy. Include that authentication on every request to the signer. The client -needs network access to both the signer and the orchestrator it will use. - -The client can discover runners through the signer or use a known runner URL. -When the orchestrator requests payment, the client passes its payment -requirements to the signer and submits the resulting signed payment to the -orchestrator. Application traffic goes to the orchestrator; the signer handles -payment requests separately. - -For a client integration, follow the [signer HTTP API](http-api.md#remote-signer-http) -for request and response formats. Clients must retain the signed payment state -returned by the signer and pass it unchanged on subsequent payment calls for -the same session. The [SDK integration fixtures](development.md#sdk-integration-fixtures) -document the client versions and flows tested with this node. +Configure the signer's base URL and proxy credentials; authenticate every signer +request. Clients need access to both signer and orchestrator, found through +discovery or a known runner URL. Follow the +[signer HTTP API](http-api.md#remote-signer-http), retaining returned signed state +unchanged for subsequent payments in the same session. The +[SDK fixtures](development.md#sdk-integration-fixtures) list tested versions and flows. ## Set payment limits @@ -190,20 +141,13 @@ retries can generate separate payments. ## Authorization -An optional authorization webhook lets your service decide who may spend from -the signer's account and under what conditions. For example, it can reject a -payment when a customer's budget is exhausted or apply a customer-specific -price ceiling. - -Set `AuthWebhookFile` to a file containing the webhook URL. If the webhook -requires authentication, set `AuthWebhookHeadersFile` to a file containing -comma-separated headers such as `Authorization: Bearer credential`. -Omit the webhook settings to disable it. - -The signer sends the webhook the original client headers and proposed payment -state. It returns payment credentials to the client only after approval. See -the [webhook reference](http-api.md#authorization-webhook) for request examples, -approval and rejection responses, and optional price ceilings. +Use an authorization webhook to enforce caller budgets and price ceilings. +Set `AuthWebhookFile` to a URL file and, for authentication, +`AuthWebhookHeadersFile` to a comma-separated header file, such as +`Authorization: Bearer credential`. Omit these settings to disable the webhook. +The signer sends client headers and proposed payment state and returns payment +credentials only after approval. See the +[webhook reference](http-api.md#authorization-webhook) for requests and decisions. Omit `expiry` or return zero to authorize every payment. A future Unix timestamp caches approval and its price ceiling until expiry, delaying revocation and @@ -213,12 +157,8 @@ A proxy must authenticate requests and overwrite untrusted `Signer-` headers. ## Kafka accounting and budgets -The signer can publish signing events to Kafka so an accounting service can -track payments. For a clearinghouse that uses those events to enforce budgets, -configure Kafka together with the authorization webhook. - -Add `[Kafka]` with a bootstrap broker and topic. `OutboxDB` stores events waiting -for delivery; its [default directory](configuration.md) is created automatically: +Configure Kafka with the authorization webhook when a clearinghouse uses signing +events to enforce budgets. `[Kafka]` requires a bootstrap broker and topic: ```toml [Kafka] @@ -229,30 +169,56 @@ PasswordFile = "/run/secrets/kafka-password" OutboxDB = "/var/lib/livepeer/signer-events.sqlite" ``` -Credentials enable SASL/SCRAM-SHA-512 by default. Set the broker's port explicitly -and choose `AuthMethod` to match its listener. Omit all Kafka settings to disable -accounting. Give each replica its own outbox. +Settings also accept `--kafka-*` flags and `LIVEPEER_SIGNER_KAFKA_*` variables. +Any Kafka setting enables accounting; omit all to disable it. `Broker` accepts +one `host[:port]` (bracket IPv6): TLS with system trust by default, `kafkas://` +for TLS or `kafka://` for plaintext. Credentials, paths, queries, and fragments +are rejected. Advertised brokers must use the same transport and authentication. + +Paired username/password secrets enable SASL; use files or +`LIVEPEER_SIGNER_KAFKA_USERNAME` / `LIVEPEER_SIGNER_KAFKA_PASSWORD`. +`AuthMethod` accepts `plain`, `scram-sha-256`, or `scram-sha-512` (default). +Without credentials SASL is disabled. Set the port explicitly or use these defaults: -Events are persisted before the payment response and delivered at least once; -consumers must deduplicate event IDs. Separate signing requests produce separate -events, including requests whose responses are interrupted. Accounting lag can -allow a budget to be exceeded before authorization stops further payment. +| Connection | Port | +| --- | --- | +| Plaintext without SASL | 9092 | +| TLS without SASL | 9093 | +| SASL/SCRAM, with or without TLS | 9096 | +| SASL/PLAIN, with or without TLS | 9094 | + +Give each replica its own durable outbox. Events are persisted before payment +responses and delivered at least once; consumers must deduplicate IDs. Separate +signing requests produce separate events, even if responses are interrupted. +Accounting lag can allow budgets to be exceeded before authorization stops +payment. See [event formats and limits](http-api.md#accounting-events) and +[backup/recovery](payment-recovery.md#signer-accounting-recovery). ## Readiness and monitoring -Use the metrics listener (default `127.0.0.1:8938`) for `/healthz`, `/readyz`, -and `/metrics`. -Readiness checks chain access, payer funds, exchange-rate freshness, and enabled -outbox storage. Monitor pending events, oldest event age, publish errors, and -filesystem space. Kafka connectivity alone does not determine readiness. +Use the metrics listener (default `127.0.0.1:8938`): -Replicas with the same key and authorization configuration can continue -client-held signed state. Keep a separate durable outbox for each replica. +```sh +curl -fsS http://127.0.0.1:8938/healthz +curl -fsS http://127.0.0.1:8938/metrics +``` + +`/healthz` checks liveness. `/readyz` requires chain access, positive deposit and +total reserve, no withdrawal scheduled at or before the next round, a fresh +exchange rate, and healthy enabled outbox storage with capacity. Kafka +connectivity alone does not affect readiness. Recipient-specific claimable +reserve is checked during payment generation. -## Reference +Monitor backlog, errors, and filesystem space. `/metrics` exposes: -- [HTTP API](http-api.md#remote-signer-http): endpoints, fields, discovery filters, and payment limits. -- [Authorization webhook](http-api.md#authorization-webhook): decision fields and failure behavior. -- [Accounting events](http-api.md#accounting-events): delivery limits and metrics. -- [Configuration example](../configs/signer/config.example.toml): settings, defaults, and Kafka options. -- [Payment recovery](payment-recovery.md#signer-accounting-recovery): backup, restart, and backlog handling. +- `livepeer_signer_up` +- `livepeer_signer_kafka_pending_events` +- `livepeer_signer_kafka_pending_bytes` +- `livepeer_signer_kafka_oldest_event_age_seconds` +- `livepeer_signer_kafka_outbox_healthy` +- `livepeer_signer_kafka_publish_errors_total` +- `livepeer_signer_kafka_enqueue_errors_total` +- `livepeer_signer_kafka_storage_errors_total` + +Replicas with the same key and authorization configuration can continue +client-held signed state. Keep a separate durable outbox for each replica.