Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,11 @@ registrations and balances are already present. A fork that is started again res
stopped, as long as the data came from the bundle being spawned. See `docs/FORK.md`.

Only previewnet can start from genesis. Every other network (paseo-next-v2, devnet, kusama,
polkadot) is fork-only, defined by a descriptor in `networks/<name>.json` —
polkadot, westend) is fork-only, defined by a descriptor in `networks/<name>.json` —
see `networks/README.md` for the schema and per-network status. Polkadot forks relay, Asset Hub,
People and Bulletin with the fellowship's runtimes authorized at bite time; `docs/POLKADOT-FORK.md`
is the runbook for a dedicated machine.
is the runbook for a dedicated machine. Westend forks relay, Asset Hub and People: the relay and
Asset Hub have sudo and People doesn't, so a People runtime under test is authorized at bite time.

A descriptor states, explicitly, which binary each chain runs and from which release,
which runtime it starts from (genesis networks), its services, tools and bite tool.
Expand Down Expand Up @@ -120,7 +121,8 @@ networks/ # Network descriptors (see networks/README.md)
├── paseo-next-v2.json # Fork-only: paseo relay, previewnet para-id band
├── devnet.json # Fork-only: paseo relay, system-chain para ids (1000+)
├── kusama.json # Fork-only, no sudo, on-the-fly bite: relay + Asset Hub
└── polkadot.json # Fork-only, no sudo, on-the-fly bite: relay + Asset Hub + People + Bulletin
├── polkadot.json # Fork-only, no sudo, on-the-fly bite: relay + Asset Hub + People + Bulletin
└── westend.json # Fork-only, on-the-fly bite: relay + Asset Hub (sudo) + People (no sudo)

config/
├── ports.env # Port configuration for all services
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
# FRESH_BITE=1 # With FORK=1: bite the live network now instead of
# # downloading the latest published bundle
# UPGRADES="<chain>=<wasm> ..." # With a bite: runtimes to authorize at import, for a
# # network without sudo (kusama, polkadot). See docs/FORK.md.
# # chain without sudo (kusama, polkadot, westend's people). See docs/FORK.md.
#
# Examples:
# make start # Native, persistent, from genesis
Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,13 @@ what you can do with it afterwards:
| `devnet` | the Polkadot Products Devnet on public Paseo, system-chain para ids | yes | a bite of the live network |
| `kusama` | Kusama relay and Asset Hub | no | a bite, with the runtime under test authorized at import |
| `polkadot` | Polkadot relay, Asset Hub, People and Bulletin | no | a bite, with the runtime under test authorized at import |
| `westend` | Westend relay, Asset Hub and People | relay and Asset Hub; People doesn't | a bite of the live network, with People's runtime under test authorized at import |

**Forking a testnet with sudo.** The bundle is bitten for you every night, so a start is a
download. Once the fork runs, sudo is `//Alice`, and anything that needs root works as it
does on genesis: runtime upgrades, HRMP, core assignment, the People grants.
**Forking a testnet with sudo.** Previewnet's and paseo-next-v2's bundles are bitten for you
every night, so a start is a download; devnet and westend are bitten on the fly. Once the fork
runs, sudo is `//Alice`, and anything that needs root works as it does on genesis: runtime
upgrades, HRMP, core assignment, the People grants (off on westend). Westend's People doesn't have a Sudo pallet, so a
People runtime under test is authorized at bite: `make bite NETWORK=westend UPGRADES="people=<wasm>"`.

```bash
ppn start --fork # previewnet, from last night's bundle
Expand Down Expand Up @@ -141,7 +144,7 @@ ppn upgrade people --ws wss://my-host/people ./people_runtime.wasm # a remote
```

Sudo dispatches the authorization, so this is for genesis and for forks of a sudo network. On
a fork of Kusama or Polkadot the authorization comes from the bite, as described under
a fork of Kusama, Polkadot or Westend's People the authorization comes from the bite, as described under
"Forking a mainnet without sudo" above. See
[docs/RUNTIME-UPGRADE.md](docs/RUNTIME-UPGRADE.md).

Expand Down
4 changes: 3 additions & 1 deletion docs/FORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ bundle, not the environment.

Networks without sudo (Kusama, Polkadot) are not pre-bitten by CI — their bundles would be
stale by the time anyone used one, and biting a public chain nightly is load nobody asked for.
They are bitten on demand instead, and their system-chain specs come from the parachain binary
Westend is not pre-bitten either: its bundle is 3.3 GB, and its People doesn't have a Sudo pallet, so a
People runtime under test is authorized at bite like on Kusama and Polkadot. These are bitten on
demand instead, and their system-chain specs come from the parachain binary
(`builtin:asset-hub-kusama`, `builtin:people-polkadot`), so nothing has to publish one — except
Polkadot's Bulletin, which is not built in and runs from the spec Parity's own RPC nodes use.
Non-previewnet networks keep their own binaries (`bin/<network>/`, release pinned by the
Expand Down
8 changes: 4 additions & 4 deletions networks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ or `PPN_NETWORK=<name>` in the environment of the scripts.
| --- | --- |
| `name` | Network id; must match the filename. |
| `genesis` | May be spawned from genesis. Previewnet only — the genesis machinery (runtimes, chain-spec presets, DotNS genesis) is previewnet's. |
| `sudo` | The live network has sudo. Networks without it (Kusama, Polkadot) cannot be upgraded after the fork, so a runtime under test must be injected at bite time — which is why they cannot be pre-bitten by CI. |
| `sudo` | The live relay has sudo. Networks without it (Kusama, Polkadot) cannot be upgraded after the fork, so a runtime under test must be injected at bite time — which is why they cannot be pre-bitten by CI. It also gates the processes that dispatch People's `Sudo.sudo` (the attestation allowances and invites), so a network whose People doesn't have a Sudo pallet (Westend) turns those off in `services`, and a People runtime under test there still needs `--upgrade` at bite. |
| `bite.prebaked` | CI bites this network on release and publishes `fork-bundle-<name>.tar.gz` (`fork-bundle-previewnet.tar.gz` for previewnet). Only meaningful for sudo networks with a real `bite.source`. |
| `bite.source` | Base URL of the live network to bite. Per-chain `rpc` values that are paths resolve against it. When every `rpc` is an absolute URL (public networks like devnet), it serves as provenance (`manifest.source`) and as the origin `pin-bulletin-products` fetches product bytes from (`<source>/ipfs/<cid>`), so point it at the network's IPFS gateway. |
| `releases` | The version table: `{ "<your-name>": { "repo": "...", "tag": "..." } }`. Every binary and runtime reference names one of these, so a version is written once per network. Names are yours — there are no reserved or special-cased names, and nothing resolves against `config/versions.env`. An unused entry is an error. |
Expand All @@ -44,7 +44,7 @@ or `PPN_NETWORK=<name>` in the environment of the scripts.
| `tools` | Local tools this network needs that are neither a chain nor a service — previewnet's `chain-spec-builder`, for instance, which only a genesis network uses. |
| `bite.doppelganger` | `{repo, tag}` of the bite tool. Per network, because the build has to be able to execute the runtimes of the network being bitten. |
| *(bin layout)* | Only node binaries are network-versioned: previewnet's live in `bin/`, every other network's in `bin/<name>/` (plus `dg/` for bites). Shared tooling with no per-network dimension — zombie-cli, ipfs, postgres, identity, design families — lives in plain `bin/` and is pinned in `config/versions.env`. Collators run through `scripts/omni-node.sh` regardless of binary; it carries the libp2p fix and picks binary/dir from `PPN_COLLATOR_BINARY`/`PPN_BIN_DIR`. |
| `relay.chain` | Built-in chain name doppelganger runs with (`ZOMBIE_CHAIN`), e.g. `paseo`, `kusama`. |
| `relay.chain` | Built-in chain name doppelganger runs with (`ZOMBIE_CHAIN`), e.g. `paseo`, `kusama`. `westend` also sets `ZOMBIE_IS_WESTEND=1`, which makes the relay bite skip GRANDPA justification checks: Westend's GRANDPA history has a forced authority change at block 4356716. |
| `relay.spec` | Basename of the relay spec inside the bundle (`specs/<spec>.json`). |
| `relay.rpc`, `parachains[].rpc` | RPC endpoint: an absolute `ws(s)://`/`http(s)://` URL, or a path resolved against `bite.source`. |
| `rpcAlternates` | Informational: other public providers for the same chain, for when the primary is re-homed. Not consumed by tooling (yet). |
Expand Down Expand Up @@ -85,7 +85,7 @@ One thing to know:
## Forking a shared relay

`bite.sharedRelay` marks a network whose live relay carries parachains we do not run — Paseo,
Kusama, Polkadot. Previewnet's relay is ours end to end, so it inherits state that is already
Kusama, Polkadot, Westend. Previewnet's relay is ours end to end, so it inherits state that is already
correct and the bite leaves it alone. A shared relay inherits three things that are wrong for six
dev validators, and all three are silent:

Expand Down Expand Up @@ -113,4 +113,4 @@ dial collators, a resolved address is the only way that substream ever opens.
| devnet | no | yes | once its `_todo` clears | sources filled from docs.polkadotcommunity.foundation; one open question — whether doppelganger can warp-sync Asset Hub from the published smol (stateRootHash) spec. Verify with `make bite NETWORK=devnet`, then drop the `_todo`. It shares a relay with other parachains, so it needs `bite.sharedRelay` too — already set |
| kusama | no | on-the-fly only | no (no sudo) | wired. Asset Hub's spec comes from the parachain binary (`builtin:asset-hub-kusama`). A runtime under test is authorized at bite time — `ppn bite --upgrade asset-hub=<wasm>` — since nothing can dispatch root after the fork; see docs/FORK.md. Upstream equivalent: [zombie-bite#127](https://github.com/paritytech/zombie-bite/issues/127) |
| polkadot | no | on-the-fly only | no (no sudo) | relay + Asset Hub 1000 + People 1004 + Bulletin 1010, all live at runtimes 2.4; the fellowship's 2.5 (Individuality on People and Asset Hub) is what a fork of it is for — `make bite NETWORK=polkadot UPGRADES="asset-hub=<wasm> people=<wasm>"` authorizes the blobs at import. All six HRMP channels between the three exist live and survive the bite (reset, not cleared). ~50 registered parachains makes the core remapping above load-bearing rather than a nicety. Runbook: docs/POLKADOT-FORK.md |
| westend | no | on-the-fly only | no | relay + Asset Hub 1000 + People 1004, the chains polkadot-sdk releases Westend runtimes for. The relay and Asset Hub have sudo and upgrade after the fork; People has none, so its runtime is authorized at bite time (`UPGRADES="people=<wasm>"`). The Individuality processes are off, since Westend runs none of those pallets |
| westend | no | on-the-fly only | no | relay + Asset Hub 1000 + People 1004, the chains polkadot-sdk releases Westend runtimes for. The relay and Asset Hub have sudo and upgrade after the fork; People doesn't, so its runtime is authorized at bite time (`UPGRADES="people=<wasm>"`). The Individuality processes and the identity backend are off, since Westend doesn't run any of those pallets |
3 changes: 2 additions & 1 deletion networks/westend.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@
"increase-people-lite-attestation-allowance": false,
"increase-dotns-attestation-allowance": false,
"grant-invites": false,
"_note": "Westend runs none of the Individuality pallets these processes grant from."
"dub": false,
"_note": "Westend doesn't run the Individuality pallets these processes and the identity backend use."
},
"stack": {
"tld": "westend",
Expand Down
2 changes: 1 addition & 1 deletion packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,7 @@ export function buildProgram(): Command {
'Download the binaries, runtimes and tooling the selected network declares.\n' +
'What gets fetched comes from networks/<name>.json — see `ppn show`.'
)
.option('--if-needed', 'do nothing when everything the descriptor declares is present')
.option('--if-needed', 'skip the downloads when everything the descriptor declares is present (file: pins are still copied)')
.option('--force', 're-download every artifact, even ones the last stamp proves current')
).action(
async (
Expand Down
50 changes: 21 additions & 29 deletions packages/cli/src/commands/bite.ts
Original file line number Diff line number Diff line change
Expand Up @@ -125,17 +125,17 @@ function biteLogDir(out: string): string {
*
* Resolves whatever the exit code is: doppelganger deliberately ends an essential task to stop
* the node once it has captured the state, so a non-zero exit is the normal path. Success is
* judged by readBiteBlock.
* judged by readBiteBlock, from this run's log alone: the file starts empty.
*/
function runBiteNode(
export function runBiteNode(
cmd: string,
args: string[],
env: NodeJS.ProcessEnv,
logFile: string,
label = path.basename(logFile, '.log')
): Promise<void> {
fs.mkdirSync(path.dirname(logFile), { recursive: true });
const fd = fs.openSync(logFile, 'a');
const fd = fs.openSync(logFile, 'w');
fs.writeSync(fd, `=== ${cmd} ${args.join(' ')}\n`);
return new Promise((resolve) => {
const child = spawn(cmd, args, { env, stdio: ['ignore', fd, fd] });
Expand Down Expand Up @@ -204,14 +204,15 @@ function logTail(logFile: string, lines: number, filter?: RegExp): string[] {
}

/**
* The block a bite node captured, or null when it captured none. doppelganger writes the info
* file whether or not importing that block succeeded, so the import result in its log decides.
* The block a bite node captured. Fails with the end of its log when the import didn't
* succeed: doppelganger writes the info file either way.
*/
function readBiteBlock(infoFile: string, logFile: string): number | null {
const log = fs.existsSync(logFile) ? fs.readFileSync(logFile, 'utf-8') : '';
if (!log.includes('Block import done! : Ok(') || !fs.existsSync(infoFile)) return null;
const block = fs.readFileSync(infoFile, 'utf-8').trim();
return block ? Number(block) : null;
export function readBiteBlock(label: string, infoFile: string, logFile: string): number {
const isImportOk = fs.existsSync(logFile) && fs.readFileSync(logFile, 'utf-8').includes('Block import done! : Ok(');
const block = isImportOk && fs.existsSync(infoFile) ? fs.readFileSync(infoFile, 'utf-8').trim() : '';
if (block) return Number(block);
const hint = logTail(logFile, 5, /Block import done|not present on the host|error|panic/i);
throw new Error(`${label}: bite FAILED (log: ${logFile})\n${hint.map((line) => ' ' + line).join('\n')}`);
}

export interface BiteOptions {
Expand Down Expand Up @@ -602,16 +603,12 @@ export async function run(args: string[], opts: BiteOptions = {}): Promise<void>
const biteBlocks: Record<string, number> = {};
for (const para of parachains) {
const id = String(para.paraId);
const log = path.join(biteLogDir(out), `${id}.log`);
const block = readBiteBlock(path.join(out, 'work', id, 'info.txt'), log);
if (block === null) {
const hint = logTail(log, 3, /Block import done|not present on the host|error|panic/i);
throw new Error(
`para ${id}: bite FAILED (log: ${log})\n${hint.map((l) => ' ' + l).join('\n')}`
);
}
biteBlocks[id] = block;
console.log(` para ${id} bitten at block ${block}`);
biteBlocks[id] = readBiteBlock(
`para ${id}`,
path.join(out, 'work', id, 'info.txt'),
path.join(biteLogDir(out), `${id}.log`)
);
console.log(` para ${id} bitten at block ${biteBlocks[id]}`);
}

console.log('=== 5/6 biting relay with parachain heads injected ===');
Expand Down Expand Up @@ -645,21 +642,16 @@ export async function run(args: string[], opts: BiteOptions = {}): Promise<void>
ZOMBIE_CHAIN: net.relay.chain,
ZOMBIE_RC_EPOCH_DURATION: String(manifest.epochDuration),
RUST_LOG: 'doppelganger=info',
// Westend's GRANDPA history carries a forced authority change (#4356716), so no warp
// proof from genesis verifies up to the live head. doppelganger skips the justification
// checks under this variable, as zombie-bite does for Westend.
// Westend's GRANDPA history has a forced authority change at block 4356716, so a warp
// proof from genesis doesn't verify up to the live head. doppelganger skips the
// justification checks under this variable, as zombie-bite does for Westend.
...(net.relay.chain === 'westend' ? { ZOMBIE_IS_WESTEND: '1' } : {}),
},
relayLog,
`relay (${net.relay.chain})`
);

const relayBlock = readBiteBlock(path.join(relayWork, 'info.txt'), relayLog);
if (relayBlock === null) {
const tail = logTail(relayLog, 5, /Block import done|error|panic/i);
throw new Error(`relay bite FAILED (log: ${relayLog})\n${tail.map((l) => ' ' + l).join('\n')}`);
}
biteBlocks.relay = relayBlock;
biteBlocks.relay = readBiteBlock('relay', path.join(relayWork, 'info.txt'), relayLog);

console.log('=== 6/6 packing snapshots ===');
const snapshotBytes: Record<string, number> = {};
Expand Down
Loading
Loading