Repository navigation
autter tests: pull and run the tests Autter wrote for a PR - #109
Merged
Merged
Conversation
When a review's generated tests (TEST-P2) find a bug, the failing test is never committed, so there was no way to debug it locally. `autter tests` fetches a PR's generated tests from the API and runs them here: - list: what Autter wrote, which failed, where and why, what could not run - pull: write the files into the checkout (failing ones by default; --all, --only); never overwrites a file it did not write, and records what it wrote in .git/ai/generated-tests/pr-<n>.json - run: pull, then run each file with its framework's command from its project directory (CI=true). The command is built from the framework and path, never from the stored command string; names with shell metacharacters are refused. Asks before running (--yes; refuses without a terminal), --dry-run prints the plan. A test that used Autter's temporary database runs only with an explicit --database-url and gets the same env names the sandbox set; it never inherits DATABASE_URL. - clean: remove exactly what pull wrote; edited files stay unless --force The PR is --pr, else the one open from the current branch (resolved by the API). A note says when the checkout is not at the commit the tests were checked on. --json keeps stdout to one document (runner output goes to stderr). Needs GET /api/cli/orgs/:slug/generated-tests (autter-monorepo). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TtUpr4DEgxboQnDVB19Sp7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When a review's generated tests (TEST-P2) catch a bug, the failing test is never committed (it would turn CI red), so there was no way to pull it into a local environment to debug it. Reported by Hrutuja Khodke.
What
autter testsfetches a PR's generated tests from the Autter API and runs them in the checkout:list: what Autter wrote for the PR, which tests failed (where and why), which passed, and what couldn't run in its sandbox.pull: writes the test files into the checkout. By default it writes the failing ones;--allincludes the passing ones and--only <path|gap id>picks one. It never overwrites a file it did not write, and it records what it wrote in.git/ai/generated-tests/pr-<n>.json.run: pulls the tests, then runs each file with its framework's command from its project directory, withCI=true. Supported frameworks: vitest, jest, mocha, node --test, pytest, go test, cargo test, rspec and phpunit.clean: removes exactly the filespullwrote. A file you edited stays unless you pass--force.The PR is
--pr, else the one open from the current branch. The API does that lookup, since the CLI has no GitHub client. If the checkout isn't at the commit the tests were checked on, a note says so.Safety
The tests are written by a model from the PR's code, and here they run with the developer's environment instead of Autter's sandbox.
cmd /C).runasks first.--yesskips the prompt. Without a terminal,runrefuses rather than assuming yes.--dry-runprints the plan.--database-url. It gets the same env names the sandbox set (URL aliases,PG*/POSTGRES_*parts andAUTTER_TEST_DATABASE_URL). It never inheritsDATABASE_URL, because the generation prompt tells the model it owns an empty database.--jsonoutput stays clean. stdout carries one JSON document, and the test runner's own output goes to stderr.autter memory.Other changes
README.mdparagraph, andchangelog/run-generated-tests-locally.md.memory.rs: the generic HTTP-error fallback now says "The Autter request failed" instead of "Repository memory request failed", becauseautter testsshares that helper.Depends on
This needs the API endpoint
GET /api/cli/orgs/:slug/generated-tests, which is added in the companionautter-monorepoPR on the same branch name. Until that endpoint is deployed, the command prints "This Autter deployment does not serve generated tests to the CLI yet."Testing
generated_tests.rs): option validation, which tests are selected, path safety, the write and clean round trip, the manifest, the command for each framework (including refusing metacharacters), the database env mapping, the skip rules, and the list output (including stripping terminal escapes).tests/integration/tests_command.rs): these use mockito, a real git repo and a realnode --testrun.listwrites nothing.runrefuses without a terminal.run --yesruns the failing test, which fails; after the fix it passes, and--jsonoutput is a single document.PWNEDfile.pullnever overwrites a file Autter did not write, even with--force.--database-urlsets the recorded names.json_contract,diff,show_promptand similar, plusupgrade::tests::test_update_prompt_skipped_in_non_interactive_contexts, also fail onmainin this environment. The container runs as root, so the CLI prints the superuser warning.taskisn't installed here, so I rancargo clippy --all-targets -D warningsandrustfmtdirectly. The local toolchain is Rust 1.97, not the pinned 1.93. My files are clippy- and fmt-clean; pre-existing code has new-toolchain lints. CI'stask lint/task fmtis the real check.Generated by Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.