Skip to content

autter tests: pull and run the tests Autter wrote for a PR - #109

Merged
sagnik11 merged 1 commit into
mainfrom
claude/focused-dirac-n3uvqa
Nov 17, 2026
Merged

sagnik11 merged 1 commit into
mainfrom
claude/focused-dirac-n3uvqa

Conversation

@sagnik11

@sagnik11 sagnik11 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Why

When a review's generated tests (TEST-P2) catch a bug, the failing test is never committed (it would turn CI red), so there was no way to pull it into a local environment to debug it. Reported by Hrutuja Khodke.

What

autter tests fetches a PR's generated tests from the Autter API and runs them in the checkout:

  • list: what Autter wrote for the PR, which tests failed (where and why), which passed, and what couldn't run in its sandbox.
  • pull: writes the test files into the checkout. By default it writes the failing ones; --all includes the passing ones and --only <path|gap id> picks one. It never overwrites a file it did not write, and it records what it wrote in .git/ai/generated-tests/pr-<n>.json.
  • run: pulls the tests, then runs each file with its framework's command from its project directory, with CI=true. Supported frameworks: vitest, jest, mocha, node --test, pytest, go test, cargo test, rspec and phpunit.
  • clean: removes exactly the files pull wrote. A file you edited stays unless you pass --force.

The PR is --pr, else the one open from the current branch. The API does that lookup, since the CLI has no GitHub client. If the checkout isn't at the commit the tests were checked on, a note says so.

Safety

The tests are written by a model from the PR's code, and here they run with the developer's environment instead of Autter's sandbox.

  • The command is built locally. The CLI builds it from the framework and the path. The command string stored on the server is never executed, and file names with shell metacharacters are refused (Windows runs through cmd /C).
  • run asks first. --yes skips the prompt. Without a terminal, run refuses rather than assuming yes. --dry-run prints the plan.
  • Database tests need an explicit scratch database. A test that ran against Autter's temporary database runs only with --database-url. It gets the same env names the sandbox set (URL aliases, PG*/POSTGRES_* parts and AUTTER_TEST_DATABASE_URL). It never inherits DATABASE_URL, because the generation prompt tells the model it owns an empty database.
  • --json output stays clean. stdout carries one JSON document, and the test runner's own output goes to stderr.
  • Sharing settings are respected. The command follows the same local-mode and excluded-repository rules as autter memory.

Other changes

  • Help entry, a README.md paragraph, and changelog/run-generated-tests-locally.md.
  • memory.rs: the generic HTTP-error fallback now says "The Autter request failed" instead of "Repository memory request failed", because autter tests shares that helper.

Depends on

This needs the API endpoint GET /api/cli/orgs/:slug/generated-tests, which is added in the companion autter-monorepo PR on the same branch name. Until that endpoint is deployed, the command prints "This Autter deployment does not serve generated tests to the CLI yet."

Testing

  • Unit tests (9, in generated_tests.rs): option validation, which tests are selected, path safety, the write and clean round trip, the manifest, the command for each framework (including refusing metacharacters), the database env mapping, the skip rules, and the list output (including stripping terminal escapes).
  • Integration tests (6, tests/integration/tests_command.rs): these use mockito, a real git repo and a real node --test run.
    • The PR is found from the branch.
    • list writes nothing.
    • run refuses without a terminal.
    • run --yes runs the failing test, which fails; after the fix it passes, and --json output is a single document.
    • The stored command is never executed: it would have created a PWNED file.
    • The database test is skipped and not written.
    • pull never overwrites a file Autter did not write, even with --force.
    • --database-url sets the recorded names.
    • The server's 404 message is shown to the user.
  • Pre-existing failures: 14 integration tests in json_contract, diff, show_prompt and similar, plus upgrade::tests::test_update_prompt_skipped_in_non_interactive_contexts, also fail on main in this environment. The container runs as root, so the CLI prints the superuser warning.
  • Lint and format: task isn't installed here, so I ran cargo clippy --all-targets -D warnings and rustfmt directly. The local toolchain is Rust 1.97, not the pinned 1.93. My files are clippy- and fmt-clean; pre-existing code has new-toolchain lints. CI's task lint / task fmt is the real check.

Generated by Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

When a review's generated tests (TEST-P2) find a bug, the failing test is
never committed, so there was no way to debug it locally. `autter tests`
fetches a PR's generated tests from the API and runs them here:

- list: what Autter wrote, which failed, where and why, what could not run
- pull: write the files into the checkout (failing ones by default; --all,
  --only); never overwrites a file it did not write, and records what it
  wrote in .git/ai/generated-tests/pr-<n>.json
- run: pull, then run each file with its framework's command from its
  project directory (CI=true). The command is built from the framework and
  path, never from the stored command string; names with shell
  metacharacters are refused. Asks before running (--yes; refuses without
  a terminal), --dry-run prints the plan. A test that used Autter's
  temporary database runs only with an explicit --database-url and gets the
  same env names the sandbox set; it never inherits DATABASE_URL.
- clean: remove exactly what pull wrote; edited files stay unless --force

The PR is --pr, else the one open from the current branch (resolved by the
API). A note says when the checkout is not at the commit the tests were
checked on. --json keeps stdout to one document (runner output goes to
stderr). Needs GET /api/cli/orgs/:slug/generated-tests (autter-monorepo).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtUpr4DEgxboQnDVB19Sp7
@sagnik11
sagnik11 merged commit 5aef173 into main Nov 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants