diff --git a/README.md b/README.md index 5598a94..88fb8b7 100644 --- a/README.md +++ b/README.md @@ -418,6 +418,8 @@ Use `autter memory search ""` to rank what Autter knows about the repo For coding agents, `autter context ""` is the one call to start with: the memory facts (with `rmf_…` ids to cite), wiki pages and code structure that matter for the task, ranked and cut to a token budget, as Markdown. It is the CLI form of the `get_context` MCP tool, and the `repo-memory` skill tells agents to use it when the Autter MCP server isn't connected. `autter memory skills install` writes the repository's memory skill into the checkout (`.agents/skills/autter-repo-memory/SKILL.md` and a Cursor rule) so teammates' agents get it from git. +When a review's generated tests catch a bug, the failing test is not committed to the PR. On the PR's branch, `autter tests run` writes the failing tests Autter kept into your checkout and runs them with their framework's command (it asks first: the tests are model-written and run with your environment). `autter tests list` shows what Autter wrote and why each failed, and `autter tests clean` removes the files again. A test that used Autter's temporary database runs only with `--database-url` pointing at a scratch database. + ### Connect your agents to Autter's MCP server `autter mcp connect` adds Autter's MCP server to every installed agent's user-level config (Claude Code, Codex, Cursor, VS Code, Gemini CLI, OpenCode). Only the URL is written; each agent signs you in with your Autter account the first time it uses it. `--dry-run` shows the changes, `--agent ` picks agents, and `autter mcp disconnect` removes the entry again. diff --git a/changelog/run-generated-tests-locally.md b/changelog/run-generated-tests-locally.md new file mode 100644 index 0000000..476735b --- /dev/null +++ b/changelog/run-generated-tests-locally.md @@ -0,0 +1,10 @@ +# Run the tests Autter wrote for a PR on your machine + +- When a review's generated tests find a bug, the failing test is never committed (it would turn CI red). `autter tests run` now brings it to you: on the PR's branch, it writes the failing tests Autter kept into your checkout and runs each one with its framework's command (vitest, jest, mocha, `node --test`, pytest, `go test`, `cargo test`, rspec, phpunit) from its project directory. Fix the bug and run it again to see it pass. +- `autter tests list` shows what Autter wrote for the PR: which tests failed, where and why, which passed, and what could not run in its sandbox. `autter tests pull` only writes the files, and `autter tests clean` removes exactly what `pull` or `run` wrote (a file you edited stays unless `--force`). +- The PR is the one open from your current branch, or `--pr `. `--all` includes the passing tests and `--only ` picks one. If your checkout is not at the commit the tests were checked on, a note says so. +- These tests are model-written from the PR's code and run here with your environment, not in Autter's sandbox, so `run` asks before it runs anything (`--yes` skips the question; `--dry-run` prints what would run). The command is built by the CLI from the test's framework and path; the command string stored on the server is never executed. `pull` never overwrites a file it did not write. +- A test that ran against Autter's temporary database runs only with `--database-url `, pointing at a scratch database it may write to. It never inherits your `DATABASE_URL`. +- Add `--json` for scripts and coding agents. The command uses your existing `autter login`; in local mode, or for repositories excluded from sharing, it sends nothing. + +This command needs the updated Autter API. diff --git a/src/commands/arg_parser.rs b/src/commands/arg_parser.rs index 06b7894..0537ca7 100644 --- a/src/commands/arg_parser.rs +++ b/src/commands/arg_parser.rs @@ -973,6 +973,42 @@ autter memory skills preview [--repo ] [--org ] [--json The repository is this checkout's default remote unless --repo is given. Uses your autter login. Nothing is sent in local mode or for repositories excluded from sharing. Use --json in scripts and coding agents.", + }, + HelpEntry { + name: "tests", + aliases: &[], + summary: "Pull and run the tests Autter wrote for a pull request", + body: "autter tests list [--pr ] [--repo ] [--org ] [--json] +autter tests pull [--pr ] [--all] [--only ]... [--force] [--json] +autter tests run [--pr ] [--all] [--only ]... [--database-url ] + [--yes] [--dry-run] [--force] [--json] +autter tests clean [--pr ] [--force] [--json] + + On a review Autter writes tests for changed code that had none and runs them + on the PR's base and head. A test that fails on the head reproduces a bug the + PR introduces; those are never committed, so this command brings them here. + + list What Autter wrote for the PR: which tests failed, where and why, which + passed, and what could not run in its sandbox. + pull Write the test files into this checkout (default: the failing ones; + --all for every kept test; --only for one file or gap id). Never + overwrites a file Autter did not write. + run Pull, then run each file with its framework's command (vitest, jest, + mocha, node --test, pytest, go test, cargo test, rspec, phpunit) from + its project directory, with CI=true. Asks first: the files are + model-written from the PR's code and run with your environment here. + --yes skips the question; --dry-run prints what would run. + A test that used Autter's temporary database runs only with + --database-url, pointing at a scratch Postgres database it may write + to; it never inherits your DATABASE_URL. + clean Remove the files pull wrote (all PRs, or --pr). A file you edited + stays unless --force. + + The PR is --pr, else the one open from this checkout's branch. The + repository is this checkout's default remote unless --repo is given. If + this checkout is not at the commit the tests were checked on, a note says + so. Exit status 1 when a test fails, is skipped or cannot be written. + Uses your autter login.", }, HelpEntry { name: "context", @@ -1060,6 +1096,7 @@ const HANDLES_OWN_HELP: &[&str] = &[ "logs", "threads", "memory", + "tests", "context", "mcp", "ci", diff --git a/src/commands/autter_handlers.rs b/src/commands/autter_handlers.rs index 5707bcb..8f2d3f7 100644 --- a/src/commands/autter_handlers.rs +++ b/src/commands/autter_handlers.rs @@ -151,6 +151,7 @@ pub fn handle_autter(args: &[String]) { | "threads" | "badge" | "memory" + | "tests" | "context" | "mcp" ) || (args[0] == "lineage" @@ -219,6 +220,7 @@ pub fn handle_autter(args: &[String]) { None => commands::lineage::handle(&args[1..]), }, "memory" => commands::memory::handle(&args[1..]), + "tests" => commands::generated_tests::handle(&args[1..]), "context" => commands::context::handle(&args[1..]), "mcp" => commands::mcp::handle(&args[1..]), "help" | "--help" | "-h" => { @@ -522,6 +524,7 @@ const AUTTER_COMMANDS: &[&str] = &[ "status", "sync", "telemetry", + "tests", "threads", "uninstall", "uninstall-hooks", diff --git a/src/commands/generated_tests.rs b/src/commands/generated_tests.rs new file mode 100644 index 0000000..48d77dd --- /dev/null +++ b/src/commands/generated_tests.rs @@ -0,0 +1,1686 @@ +//! `autter tests`: the tests Autter wrote for a pull request, on your machine. +//! +//! On a review, Autter writes tests for the changed code it found untested, +//! runs them on the PR's base and head in its sandbox, and keeps the ones that +//! say something: passing tests that pin behaviour, and failing tests that +//! reproduce a bug the PR introduces. Failing ones are never committed (they +//! would turn CI red), so this command brings them to the developer: +//! +//! * `list` — what Autter wrote for the PR, which failed and why. +//! * `pull` — write the test files into this checkout. +//! * `run` — pull, then run each file with its framework's command. +//! * `clean` — remove the files `pull` wrote (and only those). +//! +//! Safety rules, because the files are model-written from the PR's code and +//! run here with your environment instead of Autter's sandbox: +//! +//! * The command that runs a file is built here from its framework and path; +//! the command string the server stored is never executed. +//! * `run` asks before running (`--yes` skips the prompt; without a terminal +//! it refuses instead of assuming yes). +//! * A test that ran against Autter's temporary database runs here only with +//! an explicit `--database-url`: it was told it owns an empty database, so it +//! must never inherit a `DATABASE_URL` that points at a real one. +//! * `pull` never overwrites a file it did not write. What it wrote is +//! recorded under `.git/ai/generated-tests/`, so `clean` removes exactly that. +use crate::commands::arg_parser::{self, ScanMode}; +use crate::commands::ask::{component, org_api_context}; +use crate::commands::memory::{response_json_or, sharing_refusal, target_repository}; +use crate::config::Config; +use crate::error::AutterError; +use crate::git::repository::{Repository, discover_repository_in_path_no_git_exec}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use sha2::{Digest, Sha256}; +use std::collections::BTreeMap; +use std::path::{Component, Path, PathBuf}; +use std::process::Command; + +const MISSING_ROUTE: &str = "This Autter deployment does not serve generated tests to the CLI yet."; +const SUBCOMMANDS: &str = "list, pull, run or clean"; +/// Env names a test that used Autter's temporary database gets when the +/// stored result does not list them (results stored before they were recorded). +const DEFAULT_DATABASE_NAMES: &[&str] = &["AUTTER_TEST_DATABASE_URL", "DATABASE_URL"]; + +#[derive(Debug, Default, PartialEq)] +struct Options { + subcommand: String, + pr: Option, + repository: Option, + organization: Option, + /// Passing tests too (default: only the failing ones). + all: bool, + /// Only these tests (a test file path or a gap id such as `src/svc.ts#discount`). + only: Vec, + database_url: Option, + yes: bool, + dry_run: bool, + /// Replace or remove a file `pull` wrote earlier even though it was edited since. + force: bool, +} + +fn parse_options(args: &[String]) -> Result { + let mut options = Options::default(); + let mut index = 0; + while index < args.len() { + let arg = args[index].as_str(); + if !arg.starts_with("--") { + if options.subcommand.is_empty() { + options.subcommand = arg.to_owned(); + } else if options.pr.is_none() && arg.trim_start_matches('#').parse::().is_ok() { + options.pr = Some(parse_pr(arg)?); + } else { + return Err(format!("unexpected argument: {arg}")); + } + index += 1; + continue; + } + match arg { + "--all" => options.all = true, + "--yes" | "-y" => options.yes = true, + "--dry-run" => options.dry_run = true, + "--force" => options.force = true, + _ => { + let value = args + .get(index + 1) + .filter(|value| !value.starts_with("--")) + .ok_or_else(|| format!("{arg} requires a value"))?; + match arg { + "--pr" => options.pr = Some(parse_pr(value)?), + "--repo" => options.repository = Some(value.clone()), + "--org" => options.organization = Some(value.clone()), + "--only" => options.only.push(value.clone()), + "--database-url" => options.database_url = Some(value.clone()), + _ => return Err(format!("unknown option for tests: {arg}")), + } + index += 1; + } + } + index += 1; + } + let sub = options.subcommand.as_str(); + if sub.is_empty() { + return Err(format!("tests needs a subcommand: {SUBCOMMANDS}")); + } + if !matches!(sub, "list" | "pull" | "run" | "clean") { + return Err(format!("unknown tests subcommand: {sub}")); + } + if options.database_url.is_some() && sub != "run" { + return Err("--database-url only applies to tests run".into()); + } + if (options.yes || options.dry_run) && sub != "run" { + return Err(format!( + "{} only applies to tests run", + if options.yes { "--yes" } else { "--dry-run" } + )); + } + if sub == "clean" && (options.all || !options.only.is_empty()) { + return Err( + "tests clean removes every file pull wrote; it takes no --all or --only".into(), + ); + } + if sub == "list" && options.force { + return Err("tests list writes nothing, so it takes no --force".into()); + } + if let Some(url) = &options.database_url { + database_url_parts(url)?; + } + Ok(options) +} + +fn parse_pr(value: &str) -> Result { + value + .trim_start_matches('#') + .parse::() + .ok() + .filter(|n| *n > 0) + .ok_or_else(|| format!("--pr must be a pull request number, not {value}")) +} + +// ── What the server sends ─────────────────────────────────────────────────── + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct GeneratedTest { + gap_id: String, + path: String, + #[serde(default)] + dir: String, + framework: String, + /// The command Autter's sandbox used. Shown, never executed. + #[serde(default)] + command: String, + source: String, + #[serde(default)] + cases: Vec, + status: String, + #[serde(default)] + behaviour: String, + /// None on results stored before it was recorded. + #[serde(default)] + uses_database: Option, + #[serde(default)] + database_env_names: Vec, + #[serde(default)] + env_names: Vec, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct Failure { + #[serde(default)] + gap_id: String, + test_path: String, + #[serde(default)] + test_name: String, + #[serde(default)] + explanation: String, + #[serde(default)] + location: String, + #[serde(default)] + output: String, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct CouldNotRun { + #[serde(default)] + gap_id: Option, + reason: String, + #[serde(default)] + needs: Vec, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct Dropped { + #[serde(default)] + gap_id: String, + path: String, + #[serde(default)] + reason: String, + #[serde(default)] + detail: String, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct GeneratedTests { + head_sha: String, + #[serde(default)] + base_sha: String, + #[serde(default)] + is_stale: bool, + tests: Vec, + #[serde(default)] + failures: Vec, + #[serde(default)] + could_not_run: Vec, + #[serde(default)] + dropped: Vec, +} + +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(rename_all = "camelCase")] +struct PullRequest { + number: u64, + #[serde(default)] + title: Option, + #[serde(default)] + state: String, + #[serde(default)] + head_ref: String, + #[serde(default)] + head_sha: String, + #[serde(default)] + html_url: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +struct ApiResponse { + repository: String, + pull_request: PullRequest, + #[serde(default)] + generated_tests: Option, + #[serde(default)] + message: Option, +} + +// ── Choosing tests ────────────────────────────────────────────────────────── + +/// The tests a command acts on: `--only` matches, else all with `--all`, else +/// the failing ones. +fn select<'a>(tests: &'a [GeneratedTest], all: bool, only: &[String]) -> Vec<&'a GeneratedTest> { + tests + .iter() + .filter(|t| { + if !only.is_empty() { + only.iter().any(|o| o == &t.path || o == &t.gap_id) + } else { + all || t.status == "failed" + } + }) + .collect() +} + +// ── Paths and the pull manifest ───────────────────────────────────────────── + +/// A path the server says to write: relative, `/`-separated, inside the tree, +/// not under `.git/` or `.github/`. Returns it as a native relative path. +fn safe_relative_path(path: &str) -> Option { + if path.is_empty() || path.contains('\\') || path.contains('\0') { + return None; + } + let p = Path::new(path); + let mut out = PathBuf::new(); + for component in p.components() { + match component { + Component::Normal(part) => out.push(part), + _ => return None, + } + } + let first = path.split('/').next().unwrap_or(""); + if first == ".git" || first == ".github" || out.as_os_str().is_empty() { + return None; + } + Some(out) +} + +/// A test's run directory (`""` = repository root), checked like a path. +fn safe_dir(dir: &str) -> Option { + if dir.is_empty() { + Some(PathBuf::new()) + } else { + safe_relative_path(dir) + } +} + +fn sha256_hex(bytes: &[u8]) -> String { + let digest = Sha256::digest(bytes); + digest.iter().map(|b| format!("{b:02x}")).collect() +} + +#[derive(Debug, Clone, Default, Deserialize, Serialize, PartialEq)] +#[serde(rename_all = "camelCase")] +struct Manifest { + pr: u64, + head_sha: String, + /// Path → sha256 of the content `pull` wrote. + files: BTreeMap, + /// Directories `pull` created (removed by `clean` when empty). + #[serde(default)] + dirs: Vec, +} + +fn manifest_dir(git_dir: &Path) -> PathBuf { + git_dir.join("ai").join("generated-tests") +} + +fn manifest_path(git_dir: &Path, pr: u64) -> PathBuf { + manifest_dir(git_dir).join(format!("pr-{pr}.json")) +} + +fn load_manifest(git_dir: &Path, pr: u64) -> Manifest { + std::fs::read_to_string(manifest_path(git_dir, pr)) + .ok() + .and_then(|text| serde_json::from_str(&text).ok()) + .unwrap_or(Manifest { + pr, + ..Manifest::default() + }) +} + +fn save_manifest(git_dir: &Path, manifest: &Manifest) -> Result<(), AutterError> { + let path = manifest_path(git_dir, manifest.pr); + if manifest.files.is_empty() && manifest.dirs.is_empty() { + if path.exists() { + std::fs::remove_file(&path)?; + } + return Ok(()); + } + std::fs::create_dir_all(manifest_dir(git_dir))?; + std::fs::write(&path, serde_json::to_string_pretty(manifest)?)?; + Ok(()) +} + +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(tag = "outcome", rename_all = "snake_case")] +enum WriteOutcome { + Written, + Unchanged, + Refused { reason: String }, +} + +/// Write `test` into the checkout at `root`, recording it in `manifest`. +/// Never overwrites a file `pull` did not write, nor (without `force`) one it +/// wrote that was edited since. +fn write_test( + root: &Path, + test: &GeneratedTest, + manifest: &mut Manifest, + force: bool, +) -> WriteOutcome { + let refuse = |reason: &str| WriteOutcome::Refused { + reason: reason.to_owned(), + }; + let Some(rel) = safe_relative_path(&test.path) else { + return refuse("not a path inside the repository"); + }; + let target = root.join(&rel); + let wanted = sha256_hex(test.source.as_bytes()); + if let Ok(existing) = std::fs::read(&target) { + let current = sha256_hex(&existing); + if current == wanted { + manifest.files.entry(test.path.clone()).or_insert(wanted); + return WriteOutcome::Unchanged; + } + match manifest.files.get(&test.path) { + None => return refuse("a file Autter did not write is already there"), + Some(written) if *written != current && !force => { + return refuse("you edited it since it was pulled (--force replaces it)"); + } + Some(_) => {} + } + } + // Create missing parent directories one by one, so `clean` can remove them. + let mut missing = Vec::new(); + let mut parent = rel.parent(); + while let Some(dir) = parent.filter(|d| !d.as_os_str().is_empty()) { + if root.join(dir).exists() { + break; + } + missing.push(dir.to_path_buf()); + parent = dir.parent(); + } + for dir in missing.iter().rev() { + if let Err(error) = std::fs::create_dir(root.join(dir)) { + return refuse(&format!("could not create {}: {error}", dir.display())); + } + let posix = posix_path(dir); + if !manifest.dirs.contains(&posix) { + manifest.dirs.push(posix); + } + } + if let Err(error) = std::fs::write(&target, &test.source) { + return refuse(&format!("could not write it: {error}")); + } + manifest.files.insert(test.path.clone(), wanted); + WriteOutcome::Written +} + +fn posix_path(path: &Path) -> String { + path.components() + .filter_map(|c| match c { + Component::Normal(part) => Some(part.to_string_lossy().into_owned()), + _ => None, + }) + .collect::>() + .join("/") +} + +#[derive(Debug, Default, PartialEq, Serialize)] +struct CleanReport { + removed: Vec, + /// Files edited since `pull` wrote them (kept unless `--force`). + kept: Vec, +} + +/// Remove what `pull` wrote for one PR. Edited files stay (and stay recorded) +/// unless `force`. +fn clean_manifest(root: &Path, manifest: &mut Manifest, force: bool) -> CleanReport { + let mut report = CleanReport::default(); + let mut remaining = BTreeMap::new(); + for (path, written) in std::mem::take(&mut manifest.files) { + let Some(rel) = safe_relative_path(&path) else { + continue; + }; + let target = root.join(rel); + match std::fs::read(&target) { + Err(_) => {} + Ok(content) if force || sha256_hex(&content) == written => { + if std::fs::remove_file(&target).is_ok() { + report.removed.push(path); + } else { + remaining.insert(path.clone(), written); + report.kept.push(path); + } + } + Ok(_) => { + remaining.insert(path.clone(), written); + report.kept.push(path); + } + } + } + manifest.files = remaining; + // Deepest first; a directory with anything else in it stays. + let mut dirs = std::mem::take(&mut manifest.dirs); + dirs.sort_by_key(|d| std::cmp::Reverse(d.matches('/').count())); + for dir in dirs { + let Some(rel) = safe_relative_path(&dir) else { + continue; + }; + if std::fs::remove_dir(root.join(rel)).is_err() && root.join(&dir).exists() { + manifest.dirs.push(dir); + } + } + report +} + +// ── Running ───────────────────────────────────────────────────────────────── + +/// The program and arguments that run exactly `test`, from its `dir`. Built +/// from the framework and path (mirroring Autter's sandbox), never from the +/// command string the server stored. None for an unknown framework. +fn run_argv(test: &GeneratedTest, python: &str) -> Option> { + let dir = test.dir.trim_end_matches('/'); + let rel = if dir.is_empty() { + test.path.as_str() + } else { + test.path.strip_prefix(&format!("{dir}/"))? + }; + // Arguments are passed without a shell on Unix, but Windows runs them + // through `cmd /C`; a generated file name never needs these characters. + if rel.chars().any(|c| "&|<>^%\"'`$;!()\n\r".contains(c)) { + return None; + } + let is_ts = Path::new(rel) + .extension() + .and_then(|e| e.to_str()) + .is_some_and(|e| matches!(e, "ts" | "tsx" | "mts" | "cts")); + let argv: Vec<&str> = match test.framework.as_str() { + "vitest" => vec!["npx", "vitest", "run", "--reporter=verbose", rel], + "jest" => vec!["npx", "jest", "--ci", "--verbose", "--runTestsByPath", rel], + "mocha" if is_ts => vec!["npx", "mocha", "--require", "tsx", rel], + "mocha" => vec!["npx", "mocha", rel], + "node-test" if is_ts => vec!["npx", "tsx", "--test", rel], + "node-test" => vec!["node", "--test", rel], + "pytest" => vec![ + python, + "-m", + "pytest", + "-q", + "-rf", + "-p", + "no:cacheprovider", + rel, + ], + "go-test" => { + let pkg = Path::new(rel) + .parent() + .map(posix_path) + .filter(|p| !p.is_empty()) + .map(|p| format!("./{p}")) + .unwrap_or_else(|| ".".into()); + return Some( + [ + "go", + "test", + "-count=1", + "-run", + "^TestAutter", + pkg.as_str(), + ] + .iter() + .map(|s| s.to_string()) + .collect(), + ); + } + "cargo-test" => { + let name = Path::new(rel).file_stem()?.to_str()?; + vec!["cargo", "test", "--quiet", "--test", name] + } + "rspec" => vec!["bundle", "exec", "rspec", rel], + "phpunit" => vec!["vendor/bin/phpunit", rel], + _ => return None, + }; + Some(argv.into_iter().map(str::to_owned).collect()) +} + +/// Shell-style rendering of an argv, for display only. +fn display_argv(argv: &[String]) -> String { + argv.iter() + .map(|a| { + if a.chars() + .all(|c| c.is_ascii_alphanumeric() || "-_./=:@".contains(c)) + { + a.clone() + } else { + format!("'{}'", a.replace('\'', "'\\''")) + } + }) + .collect::>() + .join(" ") +} + +fn on_path(program: &str) -> bool { + let Some(paths) = std::env::var_os("PATH") else { + return false; + }; + std::env::split_paths(&paths).any(|dir| { + let candidate = dir.join(program); + candidate.is_file() || (cfg!(windows) && dir.join(format!("{program}.exe")).is_file()) + }) +} + +fn python_program() -> &'static str { + if !on_path("python") && on_path("python3") { + "python3" + } else { + "python" + } +} + +const TEST_DB_PATTERN: &str = r"\b(?:AUTTER_TEST_DATABASE_URL|DATABASE_URL|POSTGRES_URL|PG(?:HOST|DATABASE|USER|PASSWORD|PORT))\b|\b(?:drizzle|prisma|knex|sequelize|typeorm|psycopg2?|asyncpg|sqlalchemy|sqlx::|diesel::|gorm\.Open)\b|new Pool\("; + +/// Whether a test that predates the recorded flag talks to a database (its +/// source names a database variable or client). +fn source_reads_database(source: &str) -> bool { + regex::Regex::new(TEST_DB_PATTERN) + .map(|re| re.is_match(source)) + .unwrap_or(true) +} + +fn uses_database(test: &GeneratedTest) -> bool { + test.uses_database + .unwrap_or_else(|| source_reads_database(&test.source)) +} + +/// Host, port, user, password and database name of a Postgres URL. +fn database_url_parts(url: &str) -> Result<[(String, String); 5], String> { + let parsed = url::Url::parse(url).map_err(|_| { + "--database-url must be a URL such as postgres://user:pass@localhost:5432/scratch" + .to_owned() + })?; + if !matches!(parsed.scheme(), "postgres" | "postgresql") { + return Err("--database-url must be a postgres:// URL".into()); + } + let decode = |s: &str| { + url::form_urlencoded::parse(format!("x={s}").as_bytes()) + .next() + .map(|(_, v)| v.into_owned()) + .unwrap_or_default() + }; + Ok([ + ("HOST".into(), parsed.host_str().unwrap_or("").to_owned()), + ( + "PORT".into(), + parsed + .port() + .map(|p| p.to_string()) + .unwrap_or("5432".into()), + ), + ("USER".into(), decode(parsed.username())), + ("PASSWORD".into(), decode(parsed.password().unwrap_or(""))), + ( + "DATABASE".into(), + parsed.path().trim_start_matches('/').to_owned(), + ), + ]) +} + +/// The database variables a test gets from `--database-url`: every name the +/// sandbox set, mapped the same way (URL aliases get the URL, PG* and +/// POSTGRES_* parts their piece). +fn database_env(test: &GeneratedTest, url: &str) -> Vec<(String, String)> { + let parts = database_url_parts(url).unwrap_or_default(); + let part = |key: &str| { + parts + .iter() + .find(|(k, _)| k == key) + .map(|(_, v)| v.clone()) + .unwrap_or_default() + }; + let mut names: Vec = test.database_env_names.clone(); + for name in DEFAULT_DATABASE_NAMES { + if !names.iter().any(|n| n == name) { + names.push((*name).to_owned()); + } + } + names + .into_iter() + .map(|name| { + let value = match name.as_str() { + "PGHOST" | "POSTGRES_HOST" => part("HOST"), + "PGPORT" | "POSTGRES_PORT" => part("PORT"), + "PGUSER" | "POSTGRES_USER" => part("USER"), + "PGPASSWORD" | "POSTGRES_PASSWORD" => part("PASSWORD"), + "PGDATABASE" | "POSTGRES_DB" | "POSTGRES_DATABASE" => part("DATABASE"), + _ => url.to_owned(), + }; + (name, value) + }) + .collect() +} + +#[derive(Debug, Clone, PartialEq, Serialize)] +#[serde(tag = "result", rename_all = "snake_case")] +enum RunResult { + Passed, + Failed { exit_code: Option }, + Skipped { reason: String }, + NotStarted { reason: String }, +} + +/// Why `test` must not run here, if it must not. +fn skip_reason(test: &GeneratedTest, database_url: Option<&str>) -> Option { + if uses_database(test) && database_url.is_none() { + return Some( + "it ran against Autter's temporary database; pass --database-url with a scratch Postgres database it may write to (never a real one)".into(), + ); + } + if safe_dir(&test.dir).is_none() { + return Some("its run directory is not inside the repository".into()); + } + None +} + +fn run_one( + root: &Path, + test: &GeneratedTest, + argv: &[String], + database_url: Option<&str>, +) -> RunResult { + let Some(dir) = safe_dir(&test.dir) else { + return RunResult::Skipped { + reason: "its run directory is not inside the repository".into(), + }; + }; + let mut command = if cfg!(windows) { + let mut c = Command::new("cmd"); + c.arg("/C").args(argv); + c + } else { + let mut c = Command::new(&argv[0]); + c.args(&argv[1..]); + c + }; + command.current_dir(root.join(dir)).env("CI", "true"); + if arg_parser::json() { + // stdout carries exactly one JSON document; the runner's output goes to stderr. + command.stdout(std::process::Stdio::from(std::io::stderr())); + } + if let Some(url) = database_url.filter(|_| uses_database(test)) { + command.envs(database_env(test, url)); + } + match command.status() { + Ok(status) if status.success() => RunResult::Passed, + Ok(status) => RunResult::Failed { + exit_code: status.code(), + }, + Err(error) => RunResult::NotStarted { + reason: format!("could not start `{}`: {error}", argv[0]), + }, + } +} + +// ── Output ────────────────────────────────────────────────────────────────── + +fn clean_text(text: &str) -> String { + crate::commands::memory::clean(text) +} + +fn plural(n: usize, one: &str, many: &str) -> String { + format!("{n} {}", if n == 1 { one } else { many }) +} + +fn render_header(response: &ApiResponse, tests: &GeneratedTests) -> String { + let pr = &response.pull_request; + let failing = tests.tests.iter().filter(|t| t.status == "failed").count(); + let mut out = format!( + "{} #{}{} ({})\n", + clean_text(&response.repository), + pr.number, + pr.title + .as_deref() + .map(|t| format!(" — {}", clean_text(t))) + .unwrap_or_default(), + clean_text(&pr.head_ref), + ); + out.push_str(&format!( + "Autter wrote {} for this change on {}: {} passed, {} found a bug.\n", + plural(tests.tests.len(), "test file", "test files"), + short(&tests.head_sha), + tests.tests.len() - failing, + failing, + )); + if tests.is_stale { + out.push_str(&format!( + "The PR has moved on to {} since; the next review checks these tests again.\n", + short(&pr.head_sha) + )); + } + out +} + +fn short(sha: &str) -> &str { + &sha[..sha.len().min(7)] +} + +fn render_list(response: &ApiResponse, tests: &GeneratedTests) -> String { + let mut out = render_header(response, tests); + for test in &tests.tests { + let mark = if test.status == "failed" { + "✗" + } else { + "✓" + }; + out.push_str(&format!( + "\n{mark} {} [{}]{}\n", + clean_text(&test.path), + clean_text(&test.framework), + if uses_database(test) { + " (needs a database)" + } else { + "" + } + )); + for failure in tests.failures.iter().filter(|f| f.test_path == test.path) { + out.push_str(&format!( + " {} — {}\n at {}\n", + clean_text(&failure.test_name), + clean_text(&failure.explanation), + clean_text(&failure.location) + )); + } + if test.status != "failed" { + let what = if test.behaviour == "existing" { + "pins behaviour that worked before this PR" + } else { + "covers what this PR adds" + }; + out.push_str(&format!(" passes; {what}\n")); + } + } + if !tests.could_not_run.is_empty() { + out.push_str("\nCould not run in Autter's sandbox:\n"); + for c in &tests.could_not_run { + out.push_str(&format!(" - {}\n", clean_text(&c.reason))); + } + } + if !tests.dropped.is_empty() { + out.push_str(&format!( + "\n{} not kept (failed on the base too, flaky, or did not compile).\n", + plural(tests.dropped.len(), "test was", "tests were") + )); + } + let failing = tests.tests.iter().any(|t| t.status == "failed"); + out.push_str(&format!( + "\nRun {} here with: autter tests run --pr {}{}\n", + if failing { "the failing ones" } else { "them" }, + response.pull_request.number, + if failing { "" } else { " --all" } + )); + out +} + +// ── Commands ──────────────────────────────────────────────────────────────── + +/// A chosen test, the command that runs it (None when none can be built), +/// and why it must not run here (None when it may). +type Plan<'a> = (&'a GeneratedTest, Option>, Option); + +struct Checkout { + root: PathBuf, + git_dir: PathBuf, + head: Option, + branch: Option, + repo: Repository, +} + +fn checkout(local: Option) -> Result { + let repo = local + .ok_or_else(|| AutterError::Generic("Run this inside the repository's checkout.".into()))?; + let head_ref = repo.head().ok(); + let head = head_ref.as_ref().and_then(|h| h.target().ok()); + let branch = head_ref + .as_ref() + .and_then(|h| h.shorthand().ok()) + .filter(|b| !b.is_empty() && b != "HEAD"); + Ok(Checkout { + root: repo.workdir()?, + git_dir: repo.path().to_path_buf(), + head, + branch, + repo, + }) +} + +fn fetch( + options: &Options, + repository: &str, + branch: Option<&str>, +) -> Result { + let (context, slug) = org_api_context(options.organization.as_deref(), 60)?; + let mut params = url::form_urlencoded::Serializer::new(String::new()); + params.append_pair("repository", repository); + match (options.pr, branch) { + (Some(pr), _) => { + params.append_pair("pr", &pr.to_string()); + } + (None, Some(branch)) => { + params.append_pair("branch", branch); + } + (None, None) => { + return Err(AutterError::Generic(if options.repository.is_some() { + "With --repo, name the pull request with --pr .".into() + } else { + "This checkout is not on a branch. Pass --pr .".into() + })); + } + } + let value = response_json_or( + context.get(&format!( + "/api/cli/orgs/{}/generated-tests?{}", + component(&slug), + params.finish() + ))?, + MISSING_ROUTE, + )?; + serde_json::from_value(value).map_err(|error| { + AutterError::Generic(format!( + "Autter sent generated tests this CLI cannot read ({error}). Run `autter upgrade`." + )) + }) +} + +/// Warn when this checkout is not at the commit the tests were checked on. +fn head_notice(local: &Checkout, tests: &GeneratedTests, pr: &PullRequest) -> Option { + let head = local.head.as_deref()?; + if head == tests.head_sha { + return None; + } + let known = local.repo.find_commit(tests.head_sha.clone()).is_ok(); + Some(if known { + format!( + "note: these tests were checked on {}; this checkout is at {}. Results can differ.", + short(&tests.head_sha), + short(head) + ) + } else { + format!( + "note: these tests were checked on {}, which this checkout does not have (git fetch origin {} to get it). Running them on {}.", + short(&tests.head_sha), + clean_text(&pr.head_ref), + short(head) + ) + }) +} + +fn run_clean(options: &Options, local: &Checkout) -> Result { + let mut prs: Vec = match options.pr { + Some(pr) => vec![pr], + None => std::fs::read_dir(manifest_dir(&local.git_dir)) + .map(|entries| { + entries + .filter_map(|e| e.ok()) + .filter_map(|e| { + let name = e.file_name().to_string_lossy().into_owned(); + name.strip_prefix("pr-")? + .strip_suffix(".json")? + .parse() + .ok() + }) + .collect() + }) + .unwrap_or_default(), + }; + prs.sort_unstable(); + let mut reports = Vec::new(); + for pr in prs { + let mut manifest = load_manifest(&local.git_dir, pr); + let report = clean_manifest(&local.root, &mut manifest, options.force); + save_manifest(&local.git_dir, &manifest)?; + reports.push((pr, report)); + } + let kept = reports.iter().any(|(_, r)| !r.kept.is_empty()); + if arg_parser::json() { + let out: Vec = reports + .iter() + .map(|(pr, r)| json!({ "pr": pr, "removed": r.removed, "kept": r.kept })) + .collect(); + println!( + "{}", + serde_json::to_string_pretty(&json!({ "cleaned": out }))? + ); + } else if reports + .iter() + .all(|(_, r)| r.removed.is_empty() && r.kept.is_empty()) + { + println!("Nothing to clean: no generated tests were pulled into this checkout."); + } else { + for (pr, r) in &reports { + for path in &r.removed { + println!("removed {path} (#{pr})"); + } + for path in &r.kept { + println!("kept {path} (#{pr}): you edited it; --force removes it"); + } + } + } + Ok(if kept { 1 } else { 0 }) +} + +fn run(options: Options) -> Result { + let config = Config::get(); + let local = std::env::current_dir() + .ok() + .and_then(|cwd| discover_repository_in_path_no_git_exec(&cwd).ok()); + if options.subcommand == "clean" { + return run_clean(&options, &checkout(local)?); + } + let (repository, remotes) = target_repository(options.repository.as_deref(), local.as_ref())?; + if let Some(reason) = sharing_refusal( + config.platform_sync_enabled(), + config.effective_prompt_storage_with_remotes(remotes.as_ref()), + ) { + return Err(AutterError::Generic(reason.into())); + } + // list works from anywhere with --repo and --pr; pull and run write into a checkout. + let local = if options.subcommand == "list" { + local.and_then(|repo| checkout(Some(repo)).ok()) + } else { + Some(checkout(local)?) + }; + let branch = if options.repository.is_some() { + None + } else { + local.as_ref().and_then(|c| c.branch.clone()) + }; + let response = fetch(&options, &repository, branch.as_deref())?; + let Some(tests) = response.generated_tests.clone() else { + let message = response.message.clone().unwrap_or_else(|| { + format!( + "Autter has not written tests for #{} yet.", + response.pull_request.number + ) + }); + if options.subcommand == "list" { + if arg_parser::json() { + println!( + "{}", + serde_json::to_string_pretty(&json!({ + "repository": response.repository, + "pullRequest": response.pull_request, + "generatedTests": Value::Null, + "message": message, + }))? + ); + } else { + println!("{}", clean_text(&message)); + } + return Ok(0); + } + return Err(AutterError::Generic(message)); + }; + + if options.subcommand == "list" { + if arg_parser::json() { + println!( + "{}", + serde_json::to_string_pretty(&json!({ + "repository": response.repository, + "pullRequest": response.pull_request, + "generatedTests": tests, + }))? + ); + } else { + print!("{}", render_list(&response, &tests)); + } + return Ok(0); + } + + let local = local.expect("pull and run have a checkout"); + let chosen = select(&tests.tests, options.all, &options.only); + if chosen.is_empty() { + let message = if !options.only.is_empty() { + format!( + "No generated test matches --only {}. See them with: autter tests list --pr {}", + options.only.join(", "), + response.pull_request.number + ) + } else if tests.tests.is_empty() { + format!( + "Autter kept no tests for #{}.", + response.pull_request.number + ) + } else { + format!( + "None of the tests Autter wrote for #{} fails. Pass --all to {} the passing ones.", + response.pull_request.number, options.subcommand + ) + }; + return Err(AutterError::Generic(message)); + } + let human = !arg_parser::json(); + if human { + eprint!("{}", render_header(&response, &tests)); + if let Some(notice) = head_notice(&local, &tests, &response.pull_request) { + eprintln!("{notice}"); + } + } + + let python = python_program(); + let plans: Vec = chosen + .iter() + .map(|t| { + let argv = run_argv(t, python); + let skip = if argv.is_none() { + Some(format!( + "autter cannot build a command to run {} with {}", + t.path, t.framework + )) + } else { + skip_reason(t, options.database_url.as_deref()) + }; + (*t, argv, skip) + }) + .collect(); + + if options.subcommand == "run" && options.dry_run { + let out: Vec = plans + .iter() + .map(|(t, argv, skip)| { + json!({ + "path": t.path, + "dir": t.dir, + "command": argv.as_ref().map(|a| display_argv(a)), + "skip": skip, + }) + }) + .collect(); + if human { + for (t, argv, skip) in &plans { + match (argv, skip) { + (_, Some(reason)) => println!("skip {}: {reason}", t.path), + (Some(argv), None) => println!( + "would write {} and run in {}: {}", + t.path, + if t.dir.is_empty() { "." } else { &t.dir }, + display_argv(argv) + ), + (None, None) => {} + } + } + } else { + println!( + "{}", + serde_json::to_string_pretty(&json!({ "dryRun": out }))? + ); + } + return Ok(0); + } + + if options.subcommand == "run" && !options.yes { + let runnable = plans.iter().filter(|(_, _, s)| s.is_none()).count(); + if runnable > 0 { + eprintln!( + "\nAutter's model wrote these tests from this PR's code. They ran in Autter's sandbox; here they run on your machine, with your environment:" + ); + for (t, argv, skip) in &plans { + if let (Some(argv), None) = (argv, skip) { + eprintln!(" {} → {}", t.path, display_argv(argv)); + } + } + eprintln!( + "Read them first with `autter tests pull` if you do not trust the PR's author." + ); + let question = format!("Run {}?", plural(runnable, "test file", "test files")); + if !crate::ui::is_interactive() { + return Err(AutterError::Generic( + "Not a terminal, so nothing was run. Pass --yes to run without asking.".into(), + )); + } + if !crate::ui::confirm::confirm(&question, false) { + eprintln!("Nothing was run."); + return Ok(1); + } + } + } + + // Write the files. + let mut manifest = load_manifest(&local.git_dir, response.pull_request.number); + manifest.head_sha = tests.head_sha.clone(); + let mut results: Vec = Vec::new(); + let mut ok = true; + let mut written: Vec<(&GeneratedTest, &Vec)> = Vec::new(); + for (test, argv, skip) in &plans { + // `run` writes only what it runs; `pull` writes every chosen file, so + // a test that cannot run here can still be read. + if options.subcommand == "run" + && let Some(reason) = skip + { + if human { + eprintln!("skipped: {}: {reason}", test.path); + } + ok = false; + results.push( + json!({ "path": test.path, "run": RunResult::Skipped { reason: reason.clone() } }), + ); + continue; + } + let outcome = write_test(&local.root, test, &mut manifest, options.force); + if let WriteOutcome::Refused { reason } = &outcome { + ok = false; + if human { + eprintln!("not written: {}: {reason}", test.path); + } + results.push(json!({ "path": test.path, "write": outcome })); + continue; + } + if options.subcommand == "run" { + if let Some(argv) = argv { + written.push((*test, argv)); + } + continue; + } + if human { + println!( + "{} {}{}", + if outcome == WriteOutcome::Written { + "wrote " + } else { + "unchanged" + }, + test.path, + skip.as_ref() + .map(|r| format!(" (won't run here: {r})")) + .unwrap_or_default() + ); + } + results.push(json!({ + "path": test.path, + "write": outcome, + "command": argv.as_ref().map(|a| display_argv(a)), + "dir": test.dir, + "skip": skip, + })); + } + save_manifest(&local.git_dir, &manifest)?; + + if options.subcommand == "pull" { + if human { + println!( + "\nRun one with `autter tests run --pr {} --only `, and remove them with `autter tests clean`.", + response.pull_request.number + ); + } else { + println!( + "{}", + serde_json::to_string_pretty(&json!({ "pulled": results }))? + ); + } + return Ok(if ok { 0 } else { 1 }); + } + + // Run them. + for (test, argv) in &written { + for name in &test.env_names { + if std::env::var_os(name).is_none() { + eprintln!( + "warning: {} reads {name}, which is not set here (Autter's run had it from the repository's environment settings).", + test.path + ); + } + } + if human { + eprintln!( + "\n▶ {} ({}: {})", + test.path, + if test.dir.is_empty() { "." } else { &test.dir }, + display_argv(argv) + ); + } + let result = run_one(&local.root, test, argv, options.database_url.as_deref()); + if result != RunResult::Passed { + ok = false; + } + results.push(json!({ "path": test.path, "run": result })); + } + if human { + eprintln!(); + for r in &results { + let path = r["path"].as_str().unwrap_or(""); + let line = match r["run"]["result"].as_str() { + Some("passed") => format!("✓ {path} passed"), + Some("failed") => format!("✗ {path} failed"), + Some("skipped") => format!( + "- {path} skipped: {}", + r["run"]["reason"].as_str().unwrap_or("") + ), + Some("not_started") => { + format!("! {path}: {}", r["run"]["reason"].as_str().unwrap_or("")) + } + _ => format!( + "! {path} not written: {}", + r["write"]["reason"].as_str().unwrap_or("") + ), + }; + eprintln!("{line}"); + } + eprintln!("\nThe files stay in your checkout; remove them with `autter tests clean`."); + } else { + println!( + "{}", + serde_json::to_string_pretty(&json!({ + "pullRequest": response.pull_request.number, + "headSha": tests.head_sha, + "results": results, + }))? + ); + } + Ok(if ok { 0 } else { 1 }) +} + +pub fn handle(args: &[String]) { + let parsed = arg_parser::pre_parse(args, ScanMode::Full, true).unwrap_or_else(|error| { + eprintln!("error: {error}"); + std::process::exit(2); + }); + arg_parser::merge_global_flags(&parsed.flags); + if parsed.flags.help { + arg_parser::print_command_help("tests"); + return; + } + if let Some(directory) = parsed.flags.change_dir + && let Err(error) = std::env::set_current_dir(directory) + { + eprintln!("error: {error}"); + std::process::exit(1); + } + let options = parse_options(&parsed.rest).unwrap_or_else(|error| { + eprintln!("error: {error}"); + arg_parser::print_command_help("tests"); + std::process::exit(2); + }); + match run(options) { + Ok(code) => std::process::exit(code), + Err(AutterError::Generic(message)) => { + eprintln!("error: {message}"); + std::process::exit(1); + } + Err(other) => { + eprintln!("error: {other}"); + std::process::exit(1); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn args(values: &[&str]) -> Vec { + values.iter().map(|value| value.to_string()).collect() + } + + fn test_file(path: &str, framework: &str, status: &str) -> GeneratedTest { + GeneratedTest { + gap_id: format!("{path}#unit"), + path: path.into(), + dir: String::new(), + framework: framework.into(), + command: "rm -rf / # never executed".into(), + source: format!("// {path}\n"), + cases: vec![], + status: status.into(), + behaviour: "existing".into(), + uses_database: Some(false), + database_env_names: vec![], + env_names: vec![], + } + } + + #[test] + fn options_take_a_subcommand_a_pr_and_filters() { + let o = parse_options(&args(&[ + "run", + "--pr", + "42", + "--only", + "src/a.test.ts", + "--only", + "b#c", + "--yes", + "--database-url", + "postgres://u:p@localhost/scratch", + ])) + .unwrap(); + assert_eq!(o.subcommand, "run"); + assert_eq!(o.pr, Some(42)); + assert_eq!(o.only, vec!["src/a.test.ts", "b#c"]); + assert!(o.yes); + assert_eq!( + o.database_url.as_deref(), + Some("postgres://u:p@localhost/scratch") + ); + // A bare number (or #number) after the subcommand is the PR. + assert_eq!(parse_options(&args(&["list", "#7"])).unwrap().pr, Some(7)); + assert_eq!( + parse_options(&args(&["pull", "7", "--all"])).unwrap().pr, + Some(7) + ); + } + + #[test] + fn options_are_validated_before_anything_is_sent() { + assert!(parse_options(&args(&[])).is_err()); + assert!(parse_options(&args(&["fix"])).is_err()); + assert!(parse_options(&args(&["run", "--pr", "0"])).is_err()); + assert!(parse_options(&args(&["run", "--pr", "x"])).is_err()); + assert!(parse_options(&args(&["run", "--pr"])).is_err()); + assert!(parse_options(&args(&["pull", "--yes"])).is_err()); + assert!(parse_options(&args(&["list", "--dry-run"])).is_err()); + assert!(parse_options(&args(&["list", "--database-url", "postgres://x/y"])).is_err()); + assert!(parse_options(&args(&["run", "--database-url", "mysql://x/y"])).is_err()); + assert!(parse_options(&args(&["run", "--database-url", "not a url"])).is_err()); + assert!(parse_options(&args(&["clean", "--all"])).is_err()); + assert!(parse_options(&args(&["list", "--force"])).is_err()); + assert!(parse_options(&args(&["list", "1", "2"])).is_err()); + assert!(parse_options(&args(&["run", "--bogus", "1"])).is_err()); + } + + #[test] + fn the_failing_tests_are_the_default_selection() { + let tests = vec![ + test_file("a.test.js", "node-test", "failed"), + test_file("b.test.js", "node-test", "passed"), + ]; + let paths = |v: Vec<&GeneratedTest>| v.iter().map(|t| t.path.clone()).collect::>(); + assert_eq!(paths(select(&tests, false, &[])), vec!["a.test.js"]); + assert_eq!( + paths(select(&tests, true, &[])), + vec!["a.test.js", "b.test.js"] + ); + assert_eq!( + paths(select(&tests, false, &["b.test.js".into()])), + vec!["b.test.js"] + ); + assert_eq!( + paths(select(&tests, false, &["a.test.js#unit".into()])), + vec!["a.test.js"] + ); + } + + #[test] + fn only_paths_inside_the_tree_are_written() { + assert!(safe_relative_path("src/a.test.ts").is_some()); + assert!(safe_relative_path("tests/x_test.py").is_some()); + for bad in [ + "", + "/etc/passwd", + "../x.test.js", + "src/../../x.js", + ".git/hooks/pre-commit", + ".github/workflows/x.yml", + "src\\a.js", + "./", + ] { + assert!(safe_relative_path(bad).is_none(), "{bad} accepted"); + } + assert_eq!(safe_dir(""), Some(PathBuf::new())); + assert!(safe_dir("../web").is_none()); + } + + #[test] + fn pull_never_overwrites_a_file_it_did_not_write_and_clean_removes_only_its_own() { + let root = tempfile::tempdir().unwrap(); + let root = root.path(); + std::fs::create_dir_all(root.join("src")).unwrap(); + std::fs::write(root.join("src/existing.test.js"), "mine").unwrap(); + let mut manifest = Manifest { + pr: 3, + ..Manifest::default() + }; + + let new = test_file("src/__tests__/deep/a.test.js", "node-test", "failed"); + assert_eq!( + write_test(root, &new, &mut manifest, false), + WriteOutcome::Written + ); + assert_eq!( + std::fs::read_to_string(root.join(&new.path)).unwrap(), + new.source + ); + assert_eq!(manifest.dirs, vec!["src/__tests__", "src/__tests__/deep"]); + // Pulling again is a no-op. + assert_eq!( + write_test(root, &new, &mut manifest, false), + WriteOutcome::Unchanged + ); + + let clash = test_file("src/existing.test.js", "node-test", "failed"); + assert!(matches!( + write_test(root, &clash, &mut manifest, false), + WriteOutcome::Refused { .. } + )); + assert!( + matches!( + write_test(root, &clash, &mut manifest, true), + WriteOutcome::Refused { .. } + ), + "--force never replaces a file Autter did not write" + ); + assert_eq!( + std::fs::read_to_string(root.join("src/existing.test.js")).unwrap(), + "mine" + ); + + // A regenerated test replaces the copy pull wrote, unless it was edited. + let mut regenerated = new.clone(); + regenerated.source = "// v2\n".into(); + assert_eq!( + write_test(root, ®enerated, &mut manifest, false), + WriteOutcome::Written + ); + std::fs::write(root.join(&new.path), "// my debugging\n").unwrap(); + assert!(matches!( + write_test(root, &new, &mut manifest, false), + WriteOutcome::Refused { .. } + )); + + // clean keeps the edited file, then --force removes it and the directories pull made. + let other = test_file("src/b.test.js", "node-test", "failed"); + write_test(root, &other, &mut manifest, false); + let report = clean_manifest(root, &mut manifest, false); + assert_eq!(report.removed, vec!["src/b.test.js"]); + assert_eq!(report.kept, vec!["src/__tests__/deep/a.test.js"]); + assert!(root.join("src/__tests__/deep/a.test.js").exists()); + let report = clean_manifest(root, &mut manifest, true); + assert_eq!(report.removed, vec!["src/__tests__/deep/a.test.js"]); + assert!(!root.join("src/__tests__").exists()); + assert!(root.join("src/existing.test.js").exists()); + assert!(manifest.files.is_empty() && manifest.dirs.is_empty()); + } + + #[test] + fn the_manifest_lives_in_the_git_dir_and_disappears_when_empty() { + let git_dir = tempfile::tempdir().unwrap(); + let mut manifest = Manifest { + pr: 9, + head_sha: "abc".into(), + ..Manifest::default() + }; + manifest.files.insert("a.test.js".into(), "h".into()); + save_manifest(git_dir.path(), &manifest).unwrap(); + assert_eq!(load_manifest(git_dir.path(), 9), manifest); + manifest.files.clear(); + save_manifest(git_dir.path(), &manifest).unwrap(); + assert!(!manifest_path(git_dir.path(), 9).exists()); + assert_eq!(load_manifest(git_dir.path(), 9).files.len(), 0); + } + + #[test] + fn commands_are_built_from_the_framework_and_path_never_the_stored_string() { + let argv = |framework: &str, path: &str, dir: &str| { + let mut t = test_file(path, framework, "failed"); + t.dir = dir.into(); + run_argv(&t, "python3").map(|a| display_argv(&a)) + }; + assert_eq!( + argv("vitest", "web/src/a.test.ts", "web").unwrap(), + "npx vitest run --reporter=verbose src/a.test.ts" + ); + assert_eq!( + argv("jest", "src/a.test.js", "").unwrap(), + "npx jest --ci --verbose --runTestsByPath src/a.test.js" + ); + assert_eq!( + argv("mocha", "test/a.spec.ts", "").unwrap(), + "npx mocha --require tsx test/a.spec.ts" + ); + assert_eq!( + argv("node-test", "src/a.test.mjs", "").unwrap(), + "node --test src/a.test.mjs" + ); + assert_eq!( + argv("node-test", "src/a.test.ts", "").unwrap(), + "npx tsx --test src/a.test.ts" + ); + assert_eq!( + argv("pytest", "api/tests/test_x.py", "api").unwrap(), + "python3 -m pytest -q -rf -p no:cacheprovider tests/test_x.py" + ); + assert_eq!( + argv("go-test", "svc/pkg/x_autter_test.go", "svc").unwrap(), + "go test -count=1 -run '^TestAutter' ./pkg" + ); + assert_eq!( + argv("go-test", "x_autter_test.go", "").unwrap(), + "go test -count=1 -run '^TestAutter' ." + ); + assert_eq!( + argv("cargo-test", "tests/lib_autter.rs", "").unwrap(), + "cargo test --quiet --test lib_autter" + ); + assert_eq!( + argv("rspec", "spec/a_spec.rb", "").unwrap(), + "bundle exec rspec spec/a_spec.rb" + ); + assert_eq!( + argv("phpunit", "tests/ATest.php", "").unwrap(), + "vendor/bin/phpunit tests/ATest.php" + ); + assert_eq!(argv("unknown", "a.test.js", ""), None); + // A test outside its run directory is not run. + assert_eq!(argv("jest", "src/a.test.js", "web"), None); + // A name with shell metacharacters is not run (Windows runs through cmd /C). + for odd in [ + "src/a; rm -rf ~.test.js", + "src/a&calc.test.js", + "src/$(id).test.js", + "src/a|b.test.js", + ] { + assert_eq!( + run_argv(&test_file(odd, "jest", "failed"), "python"), + None, + "{odd}" + ); + } + assert!( + run_argv( + &test_file("src/@scope/a b+c.test.js", "jest", "failed"), + "python" + ) + .is_some() + ); + } + + #[test] + fn a_database_test_runs_only_with_an_explicit_scratch_url() { + let mut t = test_file("src/repo.test.js", "node-test", "failed"); + t.uses_database = Some(true); + t.database_env_names = vec![ + "DATABASE_URL".into(), + "PGHOST".into(), + "POSTGRES_DB".into(), + "DIRECT_URL".into(), + ]; + assert!(skip_reason(&t, None).unwrap().contains("--database-url")); + assert_eq!( + skip_reason(&t, Some("postgres://u:p%40ss@db.local:6543/scratch")), + None + ); + let env: BTreeMap = + database_env(&t, "postgres://u:p%40ss@db.local:6543/scratch") + .into_iter() + .collect(); + assert_eq!( + env["DATABASE_URL"], + "postgres://u:p%40ss@db.local:6543/scratch" + ); + assert_eq!( + env["DIRECT_URL"], + "postgres://u:p%40ss@db.local:6543/scratch" + ); + assert_eq!( + env["AUTTER_TEST_DATABASE_URL"], + "postgres://u:p%40ss@db.local:6543/scratch" + ); + assert_eq!(env["PGHOST"], "db.local"); + assert_eq!(env["POSTGRES_DB"], "scratch"); + + // Results stored before the flag: the source decides. + let mut old = test_file("src/old.test.js", "node-test", "failed"); + old.uses_database = None; + assert_eq!(skip_reason(&old, None), None); + old.source = "const db = postgres(process.env.DATABASE_URL)".into(); + assert!(skip_reason(&old, None).is_some()); + assert!(source_reads_database( + "import { PrismaClient } from '@prisma/client'; new PrismaClient() // prisma" + )); + assert!(!source_reads_database("assert.equal(add(1, 2), 3)")); + } + + #[test] + fn the_list_names_failures_and_how_to_run_them() { + let mut failing = test_file("src/a.test.js", "node-test", "failed"); + failing.uses_database = Some(true); + let response = ApiResponse { + repository: "acme/api".into(), + pull_request: PullRequest { + number: 12, + title: Some("Discounts".into()), + state: "open".into(), + head_ref: "feat/discount".into(), + head_sha: "bbbbbbbbbb".into(), + html_url: None, + }, + generated_tests: None, + message: None, + }; + let tests = GeneratedTests { + head_sha: "aaaaaaaaaa".into(), + base_sha: String::new(), + is_stale: true, + tests: vec![failing, test_file("src/b.test.js", "node-test", "passed")], + failures: vec![Failure { + gap_id: "src/a.test.js#unit".into(), + test_path: "src/a.test.js".into(), + test_name: "takes 10% off".into(), + explanation: "The discount is added\u{1b}[31m instead of subtracted.".into(), + location: "src/svc.js:3".into(), + output: String::new(), + }], + could_not_run: vec![CouldNotRun { + gap_id: None, + reason: "pay: needs STRIPE_SECRET_KEY".into(), + needs: vec![], + }], + dropped: vec![], + }; + let out = render_list(&response, &tests); + assert!(out.contains("acme/api #12 — Discounts (feat/discount)")); + assert!(out.contains("1 passed, 1 found a bug")); + assert!(out.contains("moved on to bbbbbbb")); + assert!(out.contains("✗ src/a.test.js [node-test] (needs a database)")); + assert!(out.contains("takes 10% off — The discount is added[31m instead of subtracted.")); + assert!( + !out.contains('\u{1b}'), + "no terminal escapes from the server" + ); + assert!(out.contains("at src/svc.js:3")); + assert!(out.contains("pay: needs STRIPE_SECRET_KEY")); + assert!(out.contains("autter tests run --pr 12\n")); + } +} diff --git a/src/commands/memory.rs b/src/commands/memory.rs index 85e5287..9be3c84 100644 --- a/src/commands/memory.rs +++ b/src/commands/memory.rs @@ -327,7 +327,7 @@ pub(crate) fn response_json_or( format!("{message} Connected: {}.", repos.join(", ")) } } - (_, None) => format!("Repository memory request failed (HTTP {code})."), + (_, None) => format!("The Autter request failed (HTTP {code})."), }; Err(AutterError::Generic(message)) } diff --git a/src/commands/mod.rs b/src/commands/mod.rs index 39af809..23104b2 100644 --- a/src/commands/mod.rs +++ b/src/commands/mod.rs @@ -17,6 +17,7 @@ pub mod exchange_nonce; pub mod fetch_notes; pub mod file_changes; pub mod flush_metrics_db; +pub mod generated_tests; pub mod git_handlers; pub mod git_hook_handlers; pub mod home; diff --git a/tests/integration/main.rs b/tests/integration/main.rs index e70896c..36ba9a5 100644 --- a/tests/integration/main.rs +++ b/tests/integration/main.rs @@ -161,6 +161,7 @@ mod superuser_guard; mod sweep_e2e; mod sync_authorship_types; mod test_utils_unit; +mod tests_command; mod tls_native_certs; mod unsupported_agents; mod upload_hold; diff --git a/tests/integration/tests_command.rs b/tests/integration/tests_command.rs new file mode 100644 index 0000000..b6173a6 --- /dev/null +++ b/tests/integration/tests_command.rs @@ -0,0 +1,339 @@ +//! `autter tests` against a mock Autter API: it finds the PR from the branch, +//! writes the tests Autter kept into the checkout, runs them with a command it +//! builds itself (never the stored string), refuses database tests without +//! `--database-url`, never overwrites a file it did not write, and `clean` +//! removes exactly what it wrote. + +use crate::repos::test_repo::{TestRepo, connected_mode}; +use base64::Engine; +use base64::engine::general_purpose::URL_SAFE_NO_PAD; +use mockito::Matcher; +use serde_json::{Value, json}; +use std::fs; +use std::process::{Output, Stdio}; + +const FAILING: &str = "src/sum.autter.test.mjs"; +const PASSING: &str = "src/sum2.autter.test.mjs"; +const DATABASE: &str = "src/store.autter.test.mjs"; + +fn node_available() -> bool { + std::process::Command::new("node") + .arg("--version") + .output() + .is_ok_and(|o| o.status.success()) +} + +/// A signed-in account whose access token names the org `acme`. +fn sign_in(repo: &TestRepo) { + let claims = json!({ + "sub": "user-1", + "email": "dev@example.com", + "org_id": "org-1", + "orgs": [{ "org_id": "org-1", "org_slug": "acme", "org_name": "Acme" }], + }); + let token = format!( + "eyJhbGciOiJub25lIn0.{}.sig", + URL_SAFE_NO_PAD.encode(claims.to_string()) + ); + let path = repo + .test_home_path() + .join(".autter") + .join("internal") + .join("credentials"); + fs::create_dir_all(path.parent().unwrap()).unwrap(); + let far_future = 4_102_444_800i64; // 2100-01-01 + fs::write( + &path, + json!({ + "access_token": token, + "refresh_token": "test-refresh-token", + "access_token_expires_at": far_future, + "refresh_token_expires_at": far_future, + }) + .to_string(), + ) + .unwrap(); +} + +/// A checkout on `feat/sum` whose `sum` subtracts: the bug Autter's test caught. +fn fixture() -> (TestRepo, String) { + let mut repo = TestRepo::new(); + repo.patch_autter_config(connected_mode); + repo.git_og(&[ + "remote", + "add", + "origin", + "https://github.com/acme/calc.git", + ]) + .unwrap(); + fs::create_dir_all(repo.path().join("src")).unwrap(); + fs::write( + repo.path().join("package.json"), + "{\"name\":\"calc\",\"type\":\"module\"}\n", + ) + .unwrap(); + fs::write( + repo.path().join("src/sum.js"), + "export function sum(a, b) {\n return a - b;\n}\n", + ) + .unwrap(); + repo.stage_all_and_commit("sum").unwrap(); + repo.git_og(&["checkout", "-b", "feat/sum"]).unwrap(); + let head = repo + .git_og(&["rev-parse", "HEAD"]) + .unwrap() + .trim() + .to_owned(); + sign_in(&repo); + (repo, head) +} + +fn test_source(expect: i32) -> String { + format!( + "import {{ test }} from 'node:test';\nimport assert from 'node:assert/strict';\nimport {{ sum }} from './sum.js';\ntest('adds', () => assert.equal(sum(1, 2), {expect}));\n" + ) +} + +fn payload(head: &str) -> Value { + let test = |path: &str, status: &str, source: String, db: bool| { + json!({ + "gapId": format!("src/sum.js#{path}"), + "path": path, + "dir": "", + "framework": "node-test", + // Never executed: the CLI builds its own command. + "command": "touch PWNED && node --test", + "source": source, + "cases": ["adds"], + "status": status, + "behaviour": "existing", + "usesDatabase": db, + "databaseEnvNames": if db { json!(["AUTTER_TEST_DATABASE_URL", "DATABASE_URL"]) } else { json!([]) }, + "envNames": [], + }) + }; + json!({ + "repository": "acme/calc", + "pullRequest": { "number": 12, "title": "Sum", "state": "open", "headRef": "feat/sum", "headSha": head, "htmlUrl": null }, + "generatedTests": { + "headSha": head, + "baseSha": "base", + "isStale": false, + "tests": [ + test(FAILING, "failed", test_source(3), false), + test(PASSING, "passed", test_source(-1), false), + test(DATABASE, "failed", "process.env.DATABASE_URL".into(), true), + ], + "failures": [{ "gapId": "src/sum.js#sum", "testPath": FAILING, "testName": "adds", "explanation": "sum subtracts instead of adding.", "location": "src/sum.js:2", "output": "" }], + "couldNotRun": [], + "dropped": [], + }, + }) +} + +fn serve(server: &mut mockito::Server, head: &str) -> mockito::Mock { + server + .mock("GET", "/api/cli/orgs/acme/generated-tests") + .match_query(Matcher::AllOf(vec![ + Matcher::UrlEncoded("repository".into(), "https://github.com/acme/calc".into()), + Matcher::UrlEncoded("branch".into(), "feat/sum".into()), + ])) + .match_header("authorization", Matcher::Regex("^Bearer ".into())) + .with_status(200) + .with_header("content-type", "application/json") + .with_body(payload(head).to_string()) + .create() +} + +fn autter(repo: &TestRepo, server: &mockito::Server, args: &[&str]) -> Output { + repo.autter_command(args) + .env("AUTTER_API_BASE_URL", server.url()) + .env_remove("DATABASE_URL") + .stdin(Stdio::null()) + .output() + .expect("autter tests should run") +} + +fn text(output: &Output) -> String { + format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ) +} + +#[test] +fn tests_list_shows_what_autter_wrote_for_the_branchs_pr() { + let (repo, head) = fixture(); + let mut server = mockito::Server::new(); + let mock = serve(&mut server, &head); + let output = autter(&repo, &server, &["tests", "list"]); + mock.assert(); + let out = text(&output); + assert!(output.status.success(), "{out}"); + assert!(out.contains("acme/calc #12 — Sum (feat/sum)"), "{out}"); + assert!(out.contains("✗ src/sum.autter.test.mjs"), "{out}"); + assert!(out.contains("sum subtracts instead of adding."), "{out}"); + assert!(out.contains("(needs a database)"), "{out}"); + assert!(!repo.path().join(FAILING).exists(), "list writes nothing"); +} + +#[test] +fn tests_run_asks_first_and_refuses_without_a_terminal() { + let (repo, head) = fixture(); + let mut server = mockito::Server::new(); + serve(&mut server, &head); + let output = autter(&repo, &server, &["tests", "run"]); + let out = text(&output); + assert_eq!(output.status.code(), Some(1), "{out}"); + assert!(out.contains("Pass --yes"), "{out}"); + assert!( + !repo.path().join(FAILING).exists(), + "nothing written before the answer" + ); +} + +#[test] +fn tests_run_writes_and_runs_the_failing_tests_then_clean_removes_them() { + if !node_available() { + eprintln!("skipping: node is not installed"); + return; + } + let (repo, head) = fixture(); + let mut server = mockito::Server::new(); + serve(&mut server, &head); + + let output = autter(&repo, &server, &["tests", "run", "--yes"]); + let out = text(&output); + // The failing test reproduces the bug, and the database test is skipped: exit 1. + assert_eq!(output.status.code(), Some(1), "{out}"); + assert!(out.contains("node --test src/sum.autter.test.mjs"), "{out}"); + assert!(out.contains("✗ src/sum.autter.test.mjs failed"), "{out}"); + assert!(out.contains("src/store.autter.test.mjs skipped"), "{out}"); + assert!(out.contains("--database-url"), "{out}"); + assert!(repo.path().join(FAILING).exists()); + assert!( + !repo.path().join(DATABASE).exists(), + "a skipped test is not written" + ); + assert!( + !repo.path().join(PASSING).exists(), + "passing tests only with --all" + ); + assert!( + !repo.path().join("PWNED").exists(), + "the stored command never runs" + ); + + // Fix the bug: the same test passes. + fs::write( + repo.path().join("src/sum.js"), + "export function sum(a, b) {\n return a + b;\n}\n", + ) + .unwrap(); + let output = autter( + &repo, + &server, + &["tests", "run", "--yes", "--only", FAILING, "--json"], + ); + let out = text(&output); + assert!(output.status.success(), "{out}"); + let doc: Value = serde_json::from_slice(&output.stdout).expect("one JSON document"); + assert_eq!(doc["results"][0]["run"]["result"], "passed", "{out}"); + + let output = autter(&repo, &server, &["tests", "clean"]); + let out = text(&output); + assert!(output.status.success(), "{out}"); + assert!(out.contains("removed src/sum.autter.test.mjs"), "{out}"); + assert!(!repo.path().join(FAILING).exists()); + assert!( + repo.path().join("src/sum.js").exists(), + "clean removes only what pull wrote" + ); +} + +#[test] +fn tests_pull_never_overwrites_a_file_autter_did_not_write() { + let (repo, head) = fixture(); + let mut server = mockito::Server::new(); + serve(&mut server, &head); + fs::write(repo.path().join(FAILING), "// my own file\n").unwrap(); + let output = autter(&repo, &server, &["tests", "pull", "--all", "--force"]); + let out = text(&output); + assert_eq!(output.status.code(), Some(1), "{out}"); + assert!( + out.contains("a file Autter did not write is already there"), + "{out}" + ); + assert_eq!( + fs::read_to_string(repo.path().join(FAILING)).unwrap(), + "// my own file\n" + ); + assert!( + repo.path().join(PASSING).exists(), + "the other files are still written" + ); + + let output = autter(&repo, &server, &["tests", "clean"]); + assert!(output.status.success(), "{}", text(&output)); + assert!(!repo.path().join(PASSING).exists()); + assert!(repo.path().join(FAILING).exists(), "the user's file stays"); +} + +#[test] +fn tests_run_with_a_database_url_sets_the_recorded_names() { + if !node_available() { + eprintln!("skipping: node is not installed"); + return; + } + let (repo, head) = fixture(); + let mut server = mockito::Server::new(); + // The database test prints and checks the URL it was given. + let mut body = payload(&head); + body["generatedTests"]["tests"][2]["source"] = json!( + "import { test } from 'node:test';\nimport assert from 'node:assert/strict';\ntest('db', () => { assert.equal(process.env.DATABASE_URL, 'postgres://u:p@localhost:5432/scratch'); assert.equal(process.env.AUTTER_TEST_DATABASE_URL, process.env.DATABASE_URL); assert.equal(process.env.CI, 'true'); });\n" + ); + server + .mock("GET", "/api/cli/orgs/acme/generated-tests") + .match_query(Matcher::UrlEncoded("pr".into(), "12".into())) + .with_status(200) + .with_body(body.to_string()) + .create(); + let output = autter( + &repo, + &server, + &[ + "tests", + "run", + "--pr", + "12", + "--yes", + "--only", + DATABASE, + "--database-url", + "postgres://u:p@localhost:5432/scratch", + ], + ); + let out = text(&output); + assert!(output.status.success(), "{out}"); + assert!(out.contains("✓ src/store.autter.test.mjs passed"), "{out}"); +} + +#[test] +fn tests_reports_the_servers_message_when_there_are_no_tests() { + let (repo, _) = fixture(); + let mut server = mockito::Server::new(); + server + .mock("GET", "/api/cli/orgs/acme/generated-tests") + .match_query(Matcher::Any) + .with_status(404) + .with_body(json!({ "error": "pr_not_found", "message": "Autter has no pull request from `feat/sum` in acme/calc." }).to_string()) + .create(); + let output = autter(&repo, &server, &["tests", "list"]); + let out = text(&output); + assert_eq!(output.status.code(), Some(1), "{out}"); + assert!( + out.contains("Autter has no pull request from `feat/sum`"), + "{out}" + ); +}