Skip to content

qol: allow a custom wasmModuleRoot and base directory - #214

Merged
ganeshvanahalli merged 4 commits into
releasefrom
qol-custom-wmr-and-base-dir
Nov 18, 2026
Merged

ganeshvanahalli merged 4 commits into
releasefrom
qol-custom-wmr-and-base-dir

Conversation

@TucksonDev

Copy link
Copy Markdown
Contributor

This PR adds 2 qol improvements:

  1. Allow starting the chain with a custom wasmModuleRoot (without this change, nitro loads the latest wasmModuleRoot)
  2. Allow running the testnode from any base directory (without this change, the testnode is expected to run from a nitro-testnode directory, and using a different directory doesn't properly clean up the containers)

@ganeshvanahalli ganeshvanahalli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both features work on a fresh --init. CI's l3node variants run the validation node through /config/machines, because the L3 node's staker turns block validation on. But the validation-node change breaks existing state restarted without --init. Details inline.

Checked and fine:

  • The locator follows the symlinks and matches each link name against module-root.txt.
  • The 0x* glob skips the legacy latest, which still exists and points at v40.
  • -n lets a current machine replace a legacy one with the same root.
  • /config is chowned to 1000 before the link step.

Comment thread docker-compose.yaml Outdated
volumes:
- "config:/config"
command: --conf.file /config/validation_node_config.json
command: --conf.file /config/validation_node_config.json --validation.wasm.root-path /config/machines

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag is always passed, but only --init creates /config/machines. When the path is missing, NewMachineLocator logs a warning and returns an empty locator, so nitro-val starts with no module roots. l3node (its staker turns on block validation) and validator (--validate) then fail to start with cannot validate WasmModuleRoot. Anyone who pulls this and reruns ./test-node.bash on existing volumes with an L3 or --validate gets a broken setup.

The links are also written into the config volume at init time. If the image changes without a re-init, roots other than latest go stale.

Building the directory when the container starts avoids both, and the init step in test-node.bash can go. Something like this, replacing the current entrypoint:

    entrypoint: ["sh", "-c", "mkdir -p /tmp/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /tmp/machines/ && exec /usr/local/bin/nitro-val \"$$@\"", "--"]
    command: --conf.file /config/validation_node_config.json --validation.wasm.root-path /tmp/machines

$$ stops compose from interpolating $@.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a701f6 , the validation node now builds the machine directory on every start with the specified entrypoint, and the init step in test-node.bash has been removed.

Comment thread test-node.bash Outdated
# The validation node serves the machines of one directory, so link the image's legacy and current machines into one.
# That way it can validate a rollup that starts on an older root, and the consensus-v61 root after an upgrade
echo == Linking legacy and current machines for the validation node
docker compose run --rm --entrypoint sh validation_node -c "mkdir -p /config/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /config/machines/"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has --validate been run end to end with one of the legacy roots?

  • The v3.11.3 image's target/machines already has v50, v51, v51.1 and v61. So the default nitro-val could already validate a chain that starts on v50/v51 and upgrades to v61.
  • This link only adds v30, v31, v32, v40, v41 and v50-alpha.1, copied from v3.7.6.
  • Nitro's own image only checks target/machines against the current prover (validate-wasm-module-root.sh). Its validator entrypoints only serve target/machines; the legacy roots appear only in allowed-wasm-module-roots.

So it isn't clear the v3.11.3 JIT and prover can validate, say, a v40 chain, and CI never sets WASM_MODULE_ROOT or --validate.

Also, the L3 deploy reuses $wasmroot, so the L3 starts on the custom root too. Is that intended?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, on v3.11.3 with the consensus-v40 root the L2 validator and the L3 node both validated blocks through JIT, and kept doing so after a restart without --init. Challenges would fail on a legacy root because the arbitrator needs machine.v2.wavm.br, which those directories lack. The L3 starting on the custom root is intended.

Comment thread test-node.bash Outdated

mydir=`dirname $0`
cd "$mydir"
project=$(basename "$PWD")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compose lowercases the directory name, strips characters outside [a-z0-9_-], and honours COMPOSE_PROJECT_NAME; basename does none of that. In a directory like Nitro-Testnode or testnode.v2, the filters still find nothing, so every run re-inits and the cleanup misses leftover containers. That's not a regression, but asking compose covers all of these cases:

project=$(docker compose config | sed -n 's/^name: //p')

Quoting $project in the filters would also help.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a701f6, the script now takes the name from docker compose config, plus --no-env-resolution (for the case where the blockscout submodule is not present).

Comment thread test-node.bash Outdated
l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'`
wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`
# WASM_MODULE_ROOT sets the rollup's initial root, which otherwise is the image's latest machine
wasmroot=${WASM_MODULE_ROOT:-`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: nothing checks that a custom root is in the image. A typo deploys fine, and then l3node or validator fails later with cannot validate WasmModuleRoot. Checking that /home/user/target/machines/$wasmroot or /home/user/nitro-legacy/machines/$wasmroot exists would make the script fail straight away.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a701f6, the script now exits with an error before deploying if WASM_MODULE_ROOT doesn't match a module-root.txt in the image.

@ganeshvanahalli

Copy link
Copy Markdown
Collaborator

posted a review from claude

@ganeshvanahalli ganeshvanahalli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round two. Both features work; the comments below are about what the legacy roots can and cannot do, one quoting case in the project name, and the error handling around the new root check.

Nit, take or leave: the label filter is spelled at four sites and could be one variable next to project=.

Comment thread docker-compose.yaml
entrypoint: /usr/local/bin/nitro-val
# Serve the image's legacy and current machines from one directory, so the node can validate a rollup that starts
# on an older root and the latest root after an upgrade. Built on every start, so it follows the image
entrypoint: ["sh", "-c", "mkdir -p /tmp/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /tmp/machines/ && exec /usr/local/bin/nitro-val \"$$@\"", "--"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you confirmed in the --validate thread, challenges cannot run on the roots this adds: those directories hold machine.wavm.br, and the v3.12.2 arbitrator opens machine.v2.wavm.br. The image never verifies them with its prover either. There is a second cost: every nitro service logs unable to find validator machine directory for the on-chain WASM module root at startup, because the entrypoint: /usr/local/bin/nitro overrides drop the image's --validation.wasm.allowed-wasm-module-roots, so each node's own locator only sees target/machines.

The roots that work fully (v50, v51, v51.1, v61) are already in target/machines, and the validation node serves them with no change here. So I would limit WASM_MODULE_ROOT to /home/user/target/machines/*/module-root.txt and drop this entrypoint. If you want the legacy roots anyway: say in the comment and in the error message that they are JIT-only, and add validation.wasm.allowed-wasm-module-roots: ["/home/user/nitro-legacy/machines", "/home/user/target/machines"] to the base node config in scripts/config.ts, so the nodes stop logging that error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to keep the legacy roots to, for example, test upgrades from older chains (I'm currently testing an upgrade from a v32 chain, which is what triggered this PR). I added a comment in 7807d37 to specify that the legacy machines only validate through JIT and can't run challenges, and the base node config in scripts/config.ts sets validation.wasm.allowed-wasm-module-roots to both directories. The --help text of the new --wasm-module-root option also says the nitro-legacy machines can't run challenges. I put that there instead of in the error message, because the error only fires for a root that isn't in the image at all.

Comment thread test-node.bash Outdated
num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l`
# Env files are skipped since blockscout's is missing when its submodule isn't checked out; compose before v2.35 has
# no flag for that and needs the submodule. Compose quotes project names that look like numbers or booleans
project=$({ docker compose config --no-env-resolution 2>/dev/null || docker compose config; } | sed -n 's/^name: //p' | tr -d '"')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tr -d '"' strips double quotes only. Compose single-quotes names that are YAML booleans: with a directory or COMPOSE_PROJECT_NAME of yes, no, on, off, y or n, docker compose config prints name: 'yes' (checked on compose v5.7.0). $project then keeps its quotes, every filter matches nothing, the script re-inits on each run and the cleanup misses the real containers. tr -d "\"'" covers both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7807d37

Comment thread test-node.bash Outdated
num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l`
# Env files are skipped since blockscout's is missing when its submodule isn't checked out; compose before v2.35 has
# no flag for that and needs the submodule. Compose quotes project names that look like numbers or booleans
project=$({ docker compose config --no-env-resolution 2>/dev/null || docker compose config; } | sed -n 's/^name: //p' | tr -d '"')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the blockscout submodule the fallback only moves the failure: on compose 2.35 or newer this line passes and the next compose command dies on the required env file, and on older compose the hidden unknown flag error is replaced by the env-file error. Either drop the fallback and let compose's own error explain the missing file, or make the file optional (required: false on the env_file entry, compose 2.24 or newer) and refuse --blockscout when it is missing.

Since this also runs before the option loop, --help now pays the config call and fails in that setup. Moving the project and volume lookup after the loop avoids that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On recent compose the commands the script runs next don't fail without the submodule. Since v2.34 compose only loads the env files of the services a command selects, with the same fix for run in v2.35. I checked on v5.7.0 without the submodule: ps, down and a run --no-deps of validation_node work, dry runs of build scripts and up -d redis get past env-file loading, and up -d blockscout reports the missing file. config still needs the file, which is why the lookup passes --no-env-resolution.

I kept the fallback because --no-env-resolution only exists since v2.35, so without it the lookup would fail on older compose even with the submodule checked out. If the submodule is missing on those versions, compose's own env-file error is printed, and those versions need the submodule anyway.

Besides that, the empty-project check is moved below the option loop, so --help prints even when the lookup fails. The lookup itself stays above the loop because the --init prompt reads force_init, which comes from the volume count. Done in 7807d37

Comment thread test-node.bash Outdated
sequenceraddress=`run_script print-address --account sequencer | tail -n 1 | tr -d '\r\n'`
l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'`
wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`
# WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth saying here: the machine must be at least as new as the chain's starting ArbOS version, which is 40 by default and 60 with --l2-tx-filtering (scripts/config.ts:505). An older replay cannot run a newer ArbOS, so a v30 root, or a v50 root with filtering on, passes the check and then diverges on the first validated block, and with failure-is-fatal the validator restarts in a loop.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed on the requirement. That comment went away with the env var, so in 7807d37 it's in the --help text of --wasm-module-root: the root needs a machine that supports the ArbOS version the chains start on. I left the numbers out so the text doesn't drift from scripts/config.ts. With the default ArbOS 40 that rules out validating with the v30, v31 and v32 machines. We can also have a new parameter to specify the arbOS version to start with, but that should be done in a separate PR.

One correction: the nitro services have no restart policy in the compose file, so a validator that hits a fatal validation error stops instead of restarting in a loop.

Comment thread test-node.bash Outdated
l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'`
wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`
# WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine
if [ -n "${WASM_MODULE_ROOT:-}" ]; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only runs under init, so on an existing chain a set WASM_MODULE_ROOT is ignored without a word, and the variable shows up in no --help line or README. A --wasm-module-root <hex> option with a "requires --init" check, like the existing --nowait requires --detach one, covers both. At minimum, warn when it is set and no init runs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7807d37, WASM_MODULE_ROOT is replaced by a --wasm-module-root option that is listed in --help, and the script exits with an error when the option is passed on a run that won't initialize. The check uses force_init, so the first run of a fresh project, which initializes without --init, accepts it too.

Comment thread test-node.bash Outdated
wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`
# WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine
if [ -n "${WASM_MODULE_ROOT:-}" ]; then
wasmroot=$WASM_MODULE_ROOT

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check comes late. By this line the old data has been removed (line 482), the volumes pruned, geth started and the L1 traffic loop forked into the background (line 549). A typo in WASM_MODULE_ROOT therefore costs the previous chain, leaves geth and the traffic generator running, and since new volumes now exist, the next plain ./test-node.bash skips init and the nodes fail for lack of l2_chain_info.json. The image is ready at line 465, so the check can run right after it and before "Removing old data", with docker run --rm --entrypoint sh nitro-node-dev-testnode -c … so it does not pull in the sequencer's depends_on: geth.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7807d37, the check now runs right after the nitro image is tagged and before "Removing old data".

Comment thread test-node.bash Outdated
# WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine
if [ -n "${WASM_MODULE_ROOT:-}" ]; then
wasmroot=$WASM_MODULE_ROOT
docker compose run --rm --entrypoint sh sequencer -c 'grep -qsxiF "$1" /home/user/target/machines/*/module-root.txt /home/user/nitro-legacy/machines/*/module-root.txt' sh "$wasmroot" || { echo "Error: WASM_MODULE_ROOT $wasmroot is not a machine of the nitro image"; exit 1; }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here. Any failure of the docker compose run itself, say the image is not built or geth fails to start, also lands in the || branch and prints "is not a machine of the nitro image", hiding the cause. Checking the exit code first (1 is no match, anything else a run failure) keeps them apart.

The check also tests membership, not shape: grep -i lets 0X… through, which ethers rejects later as an invalid hex value, after L1 is already funded, and a value with a newline matches one of its lines and then word-splits at the unquoted -e WASM_MODULE_ROOT=$wasmroot. A ^0x[0-9a-fA-F]{64}$ check in bash before the container run covers both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7807d37. The ^0x[0-9a-fA-F]{64}$ check runs in the option parser, so 0X…, a value with a newline or a missing value is rejected before anything is built, pulled or deleted. For the run failure I went with less code than an exit-code check. The roots are read with a plain assignment from docker run, so if that command fails the script stops under set -e with docker's own error, and "is not a machine of the nitro image" only prints when the read worked and the root isn't in the list.

@ganeshvanahalli ganeshvanahalli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ganeshvanahalli
ganeshvanahalli merged commit 3e2bbc2 into release Nov 18, 2026
30 checks passed
@ganeshvanahalli
ganeshvanahalli deleted the qol-custom-wmr-and-base-dir branch November 18, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants