Repository navigation
qol: allow a custom wasmModuleRoot and base directory - #214
Conversation
ganeshvanahalli
left a comment
There was a problem hiding this comment.
Both features work on a fresh --init. CI's l3node variants run the validation node through /config/machines, because the L3 node's staker turns block validation on. But the validation-node change breaks existing state restarted without --init. Details inline.
Checked and fine:
- The locator follows the symlinks and matches each link name against
module-root.txt. - The
0x*glob skips the legacylatest, which still exists and points at v40. -nlets a current machine replace a legacy one with the same root./configis chowned to 1000 before the link step.
| volumes: | ||
| - "config:/config" | ||
| command: --conf.file /config/validation_node_config.json | ||
| command: --conf.file /config/validation_node_config.json --validation.wasm.root-path /config/machines |
There was a problem hiding this comment.
This flag is always passed, but only --init creates /config/machines. When the path is missing, NewMachineLocator logs a warning and returns an empty locator, so nitro-val starts with no module roots. l3node (its staker turns on block validation) and validator (--validate) then fail to start with cannot validate WasmModuleRoot. Anyone who pulls this and reruns ./test-node.bash on existing volumes with an L3 or --validate gets a broken setup.
The links are also written into the config volume at init time. If the image changes without a re-init, roots other than latest go stale.
Building the directory when the container starts avoids both, and the init step in test-node.bash can go. Something like this, replacing the current entrypoint:
entrypoint: ["sh", "-c", "mkdir -p /tmp/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /tmp/machines/ && exec /usr/local/bin/nitro-val \"$$@\"", "--"]
command: --conf.file /config/validation_node_config.json --validation.wasm.root-path /tmp/machines$$ stops compose from interpolating $@.
There was a problem hiding this comment.
Done in 8a701f6 , the validation node now builds the machine directory on every start with the specified entrypoint, and the init step in test-node.bash has been removed.
| # The validation node serves the machines of one directory, so link the image's legacy and current machines into one. | ||
| # That way it can validate a rollup that starts on an older root, and the consensus-v61 root after an upgrade | ||
| echo == Linking legacy and current machines for the validation node | ||
| docker compose run --rm --entrypoint sh validation_node -c "mkdir -p /config/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /config/machines/" |
There was a problem hiding this comment.
Has --validate been run end to end with one of the legacy roots?
- The v3.11.3 image's
target/machinesalready has v50, v51, v51.1 and v61. So the defaultnitro-valcould already validate a chain that starts on v50/v51 and upgrades to v61. - This link only adds v30, v31, v32, v40, v41 and v50-alpha.1, copied from v3.7.6.
- Nitro's own image only checks
target/machinesagainst the current prover (validate-wasm-module-root.sh). Its validator entrypoints only servetarget/machines; the legacy roots appear only inallowed-wasm-module-roots.
So it isn't clear the v3.11.3 JIT and prover can validate, say, a v40 chain, and CI never sets WASM_MODULE_ROOT or --validate.
Also, the L3 deploy reuses $wasmroot, so the L3 starts on the custom root too. Is that intended?
There was a problem hiding this comment.
Yes, on v3.11.3 with the consensus-v40 root the L2 validator and the L3 node both validated blocks through JIT, and kept doing so after a restart without --init. Challenges would fail on a legacy root because the arbitrator needs machine.v2.wavm.br, which those directories lack. The L3 starting on the custom root is intended.
|
|
||
| mydir=`dirname $0` | ||
| cd "$mydir" | ||
| project=$(basename "$PWD") |
There was a problem hiding this comment.
Compose lowercases the directory name, strips characters outside [a-z0-9_-], and honours COMPOSE_PROJECT_NAME; basename does none of that. In a directory like Nitro-Testnode or testnode.v2, the filters still find nothing, so every run re-inits and the cleanup misses leftover containers. That's not a regression, but asking compose covers all of these cases:
project=$(docker compose config | sed -n 's/^name: //p')Quoting $project in the filters would also help.
There was a problem hiding this comment.
Done in 8a701f6, the script now takes the name from docker compose config, plus --no-env-resolution (for the case where the blockscout submodule is not present).
| l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'` | ||
| wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"` | ||
| # WASM_MODULE_ROOT sets the rollup's initial root, which otherwise is the image's latest machine | ||
| wasmroot=${WASM_MODULE_ROOT:-`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"`} |
There was a problem hiding this comment.
Nit: nothing checks that a custom root is in the image. A typo deploys fine, and then l3node or validator fails later with cannot validate WasmModuleRoot. Checking that /home/user/target/machines/$wasmroot or /home/user/nitro-legacy/machines/$wasmroot exists would make the script fail straight away.
There was a problem hiding this comment.
Done in 8a701f6, the script now exits with an error before deploying if WASM_MODULE_ROOT doesn't match a module-root.txt in the image.
|
posted a review from claude |
ganeshvanahalli
left a comment
There was a problem hiding this comment.
Round two. Both features work; the comments below are about what the legacy roots can and cannot do, one quoting case in the project name, and the error handling around the new root check.
Nit, take or leave: the label filter is spelled at four sites and could be one variable next to project=.
| entrypoint: /usr/local/bin/nitro-val | ||
| # Serve the image's legacy and current machines from one directory, so the node can validate a rollup that starts | ||
| # on an older root and the latest root after an upgrade. Built on every start, so it follows the image | ||
| entrypoint: ["sh", "-c", "mkdir -p /tmp/machines && ln -sfn /home/user/nitro-legacy/machines/0x* /home/user/target/machines/* /tmp/machines/ && exec /usr/local/bin/nitro-val \"$$@\"", "--"] |
There was a problem hiding this comment.
As you confirmed in the --validate thread, challenges cannot run on the roots this adds: those directories hold machine.wavm.br, and the v3.12.2 arbitrator opens machine.v2.wavm.br. The image never verifies them with its prover either. There is a second cost: every nitro service logs unable to find validator machine directory for the on-chain WASM module root at startup, because the entrypoint: /usr/local/bin/nitro overrides drop the image's --validation.wasm.allowed-wasm-module-roots, so each node's own locator only sees target/machines.
The roots that work fully (v50, v51, v51.1, v61) are already in target/machines, and the validation node serves them with no change here. So I would limit WASM_MODULE_ROOT to /home/user/target/machines/*/module-root.txt and drop this entrypoint. If you want the legacy roots anyway: say in the comment and in the error message that they are JIT-only, and add validation.wasm.allowed-wasm-module-roots: ["/home/user/nitro-legacy/machines", "/home/user/target/machines"] to the base node config in scripts/config.ts, so the nodes stop logging that error.
There was a problem hiding this comment.
I think it makes sense to keep the legacy roots to, for example, test upgrades from older chains (I'm currently testing an upgrade from a v32 chain, which is what triggered this PR). I added a comment in 7807d37 to specify that the legacy machines only validate through JIT and can't run challenges, and the base node config in scripts/config.ts sets validation.wasm.allowed-wasm-module-roots to both directories. The --help text of the new --wasm-module-root option also says the nitro-legacy machines can't run challenges. I put that there instead of in the error message, because the error only fires for a root that isn't in the image at all.
| num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l` | ||
| # Env files are skipped since blockscout's is missing when its submodule isn't checked out; compose before v2.35 has | ||
| # no flag for that and needs the submodule. Compose quotes project names that look like numbers or booleans | ||
| project=$({ docker compose config --no-env-resolution 2>/dev/null || docker compose config; } | sed -n 's/^name: //p' | tr -d '"') |
There was a problem hiding this comment.
tr -d '"' strips double quotes only. Compose single-quotes names that are YAML booleans: with a directory or COMPOSE_PROJECT_NAME of yes, no, on, off, y or n, docker compose config prints name: 'yes' (checked on compose v5.7.0). $project then keeps its quotes, every filter matches nothing, the script re-inits on each run and the cleanup misses the real containers. tr -d "\"'" covers both.
| num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l` | ||
| # Env files are skipped since blockscout's is missing when its submodule isn't checked out; compose before v2.35 has | ||
| # no flag for that and needs the submodule. Compose quotes project names that look like numbers or booleans | ||
| project=$({ docker compose config --no-env-resolution 2>/dev/null || docker compose config; } | sed -n 's/^name: //p' | tr -d '"') |
There was a problem hiding this comment.
Without the blockscout submodule the fallback only moves the failure: on compose 2.35 or newer this line passes and the next compose command dies on the required env file, and on older compose the hidden unknown flag error is replaced by the env-file error. Either drop the fallback and let compose's own error explain the missing file, or make the file optional (required: false on the env_file entry, compose 2.24 or newer) and refuse --blockscout when it is missing.
Since this also runs before the option loop, --help now pays the config call and fails in that setup. Moving the project and volume lookup after the loop avoids that.
There was a problem hiding this comment.
On recent compose the commands the script runs next don't fail without the submodule. Since v2.34 compose only loads the env files of the services a command selects, with the same fix for run in v2.35. I checked on v5.7.0 without the submodule: ps, down and a run --no-deps of validation_node work, dry runs of build scripts and up -d redis get past env-file loading, and up -d blockscout reports the missing file. config still needs the file, which is why the lookup passes --no-env-resolution.
I kept the fallback because --no-env-resolution only exists since v2.35, so without it the lookup would fail on older compose even with the submodule checked out. If the submodule is missing on those versions, compose's own env-file error is printed, and those versions need the submodule anyway.
Besides that, the empty-project check is moved below the option loop, so --help prints even when the lookup fails. The lookup itself stays above the loop because the --init prompt reads force_init, which comes from the volume count. Done in 7807d37
| sequenceraddress=`run_script print-address --account sequencer | tail -n 1 | tr -d '\r\n'` | ||
| l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'` | ||
| wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"` | ||
| # WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine |
There was a problem hiding this comment.
Worth saying here: the machine must be at least as new as the chain's starting ArbOS version, which is 40 by default and 60 with --l2-tx-filtering (scripts/config.ts:505). An older replay cannot run a newer ArbOS, so a v30 root, or a v50 root with filtering on, passes the check and then diverges on the first validated block, and with failure-is-fatal the validator restarts in a loop.
There was a problem hiding this comment.
Agreed on the requirement. That comment went away with the env var, so in 7807d37 it's in the --help text of --wasm-module-root: the root needs a machine that supports the ArbOS version the chains start on. I left the numbers out so the text doesn't drift from scripts/config.ts. With the default ArbOS 40 that rules out validating with the v30, v31 and v32 machines. We can also have a new parameter to specify the arbOS version to start with, but that should be done in a separate PR.
One correction: the nitro services have no restart policy in the compose file, so a validator that hits a fatal validation error stops instead of restarting in a loop.
| l2ownerKey=`run_script print-private-key --account l2owner | tail -n 1 | tr -d '\r\n'` | ||
| wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"` | ||
| # WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine | ||
| if [ -n "${WASM_MODULE_ROOT:-}" ]; then |
There was a problem hiding this comment.
This only runs under init, so on an existing chain a set WASM_MODULE_ROOT is ignored without a word, and the variable shows up in no --help line or README. A --wasm-module-root <hex> option with a "requires --init" check, like the existing --nowait requires --detach one, covers both. At minimum, warn when it is set and no init runs.
There was a problem hiding this comment.
Done in 7807d37, WASM_MODULE_ROOT is replaced by a --wasm-module-root option that is listed in --help, and the script exits with an error when the option is passed on a run that won't initialize. The check uses force_init, so the first run of a fresh project, which initializes without --init, accepts it too.
| wasmroot=`docker compose run --rm --entrypoint sh sequencer -c "cat /home/user/target/machines/latest/module-root.txt"` | ||
| # WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine | ||
| if [ -n "${WASM_MODULE_ROOT:-}" ]; then | ||
| wasmroot=$WASM_MODULE_ROOT |
There was a problem hiding this comment.
The check comes late. By this line the old data has been removed (line 482), the volumes pruned, geth started and the L1 traffic loop forked into the background (line 549). A typo in WASM_MODULE_ROOT therefore costs the previous chain, leaves geth and the traffic generator running, and since new volumes now exist, the next plain ./test-node.bash skips init and the nodes fail for lack of l2_chain_info.json. The image is ready at line 465, so the check can run right after it and before "Removing old data", with docker run --rm --entrypoint sh nitro-node-dev-testnode -c … so it does not pull in the sequencer's depends_on: geth.
There was a problem hiding this comment.
Done in 7807d37, the check now runs right after the nitro image is tagged and before "Removing old data".
| # WASM_MODULE_ROOT sets the initial root of the L2 and L3 rollups, which otherwise is the image's latest machine | ||
| if [ -n "${WASM_MODULE_ROOT:-}" ]; then | ||
| wasmroot=$WASM_MODULE_ROOT | ||
| docker compose run --rm --entrypoint sh sequencer -c 'grep -qsxiF "$1" /home/user/target/machines/*/module-root.txt /home/user/nitro-legacy/machines/*/module-root.txt' sh "$wasmroot" || { echo "Error: WASM_MODULE_ROOT $wasmroot is not a machine of the nitro image"; exit 1; } |
There was a problem hiding this comment.
Two things here. Any failure of the docker compose run itself, say the image is not built or geth fails to start, also lands in the || branch and prints "is not a machine of the nitro image", hiding the cause. Checking the exit code first (1 is no match, anything else a run failure) keeps them apart.
The check also tests membership, not shape: grep -i lets 0X… through, which ethers rejects later as an invalid hex value, after L1 is already funded, and a value with a newline matches one of its lines and then word-splits at the unquoted -e WASM_MODULE_ROOT=$wasmroot. A ^0x[0-9a-fA-F]{64}$ check in bash before the container run covers both.
There was a problem hiding this comment.
Done in 7807d37. The ^0x[0-9a-fA-F]{64}$ check runs in the option parser, so 0X…, a value with a newline or a missing value is rejected before anything is built, pulled or deleted. For the run failure I went with less code than an exit-code check. The roots are read with a plain assignment from docker run, so if that command fails the script stops under set -e with docker's own error, and "is not a machine of the nitro image" only prints when the read worked and the root isn't in the list.
This PR adds 2 qol improvements:
nitro-testnodedirectory, and using a different directory doesn't properly clean up the containers)