Skip to content

Tutorials tested in CI against a real Gateway, with a Verified-in-CI banner - #148

Merged
melvincarvalho merged 6 commits into
gh-pagesfrom
q/ci-tutorials
Oct 18, 2026
Merged

melvincarvalho merged 6 commits into
gh-pagesfrom
q/ci-tutorials

Conversation

@melvincarvalho

Copy link
Copy Markdown
Contributor

Quarter roadmap, item 4 (audit roadmap): tutorials tested in CI against a real Gateway. Green run: https://github.com/clawdocs/clawdocs.github.io/actions/runs/39126638059

How it works:

  • Tests: scripts/tutorial-tests/*.test.sh call run "<command as written on the page>". The helper finds that command in the page's own shell blocks and runs the page's text, so a test can't drift from its page unnoticed. Config blocks are applied with config patch and then config validate. Steps that need a model key or a channel account are recorded as skipped, with a reason, never faked, and page commands that are neither run nor skipped are reported as "unmapped" (currently 0).
  • Sandbox: each test gets a fresh HOME under env -i. Onboarding is non-interactive with a generated throwaway token, and openclaw gateway run binds to loopback. The Gateway refuses to start outside GitHub Actions, and TT_DRY=1 checks the page mapping locally.
  • Workflow: .github/workflows/tutorials.yml has contents: read and no secrets. It writes a job summary and uploads a JSON report. It's report-only for now; set TUTORIALS_STRICT=1 to make it blocking.
  • Banner: "Verified in CI" is rendered from ci_verified front matter, which is added by hand after a green run; the workflow never commits.

Steps run / skipped: quick-start 24/12, day-1 17/8, day-3 30/5, day-4 20/3, day-7 30/8, heartbeat 39/5.

Bugs found by actually running the pages (all fixed here):

  • day-1: bare openclaw setup opens a chat; it's now setup --baseline. The (pick one) text was saved as the emoji.
  • quick-start: pairing list needs a channel name.
  • doctor --fix fails while the Gateway is running (GatewayStateOwnerContentionError). Fixed on day-7, heartbeat, day-3 and quick-start.
  • heartbeat: Doctor imports HEARTBEAT.md only into empty scratch.

Known, not changed: doctor --fix disables bundled skills whose requirements are missing (28 on the runner).

@melvincarvalho
melvincarvalho merged commit ba91bd8 into gh-pages Oct 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant