From 737055551c1b931903b40a9448b6f8b08c45821e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:37:54 +0000 Subject: [PATCH 01/12] fix(glob): bound expansion while it runs; stop ** following symlinks (#191) ClusterFuzzLite's pr_fuzzing (address) job on PR #230 (check run 134118693669) hit libFuzzer's out-of-memory limit in fuzz_glob_expansion with the input `/**/*`: 2629 MB used against 2560 MB, live allocations in the glob crate's fill_todo, reached from expand_glob. The fault is on main. Cause, reproduced locally before any change (Linux 6.18, release build): - The glob crate follows symbolic links to directories under `**`, so a link back to a parent never ends the walk. `**/*` over a 7-entry tree with `d/up -> ..` and `self -> .` passed 2.5 GB in 15 s; `**/no-such-name` over the same tree never returned, at 2 MB, so a cap on matches alone cannot bound it. - expand_glob collected every path and only then compared the count with MAX_GLOB_RESULTS. `/**/*` from the fuzz target's empty base dir walked the runner's whole file system, including the loops under /proc and /sys, and grew to the limit in 16 s. The real ASan fuzz target replays the CI input to the same out-of-memory report. - When the cap did fire, external-command expansion treated the error as "no match" and ran the command with the literal pattern. Fix: - glob.rs walks the tree itself. Each component is still matched by glob::Pattern; the walk processes each (directory, pattern component) pair once, in name order. `**` lists a symbolic link to a directory but does not descend into it and skips hidden directories; other components still go through links, so `link/*` works. - New src/grade.rs: a Cost grade (sequence adds) and an Occupancy grade (high-water mark), and a Meter that refuses the charge over a limit. The walk meters file-system reads (10,000,000), matches held (100,000) and bytes of matches held (6 MiB, the most Linux execve accepts for argument strings at any stack limit, measured here: 6,279,840 bytes). The grade laws are property-tested, not proved. - Reaching a limit returns GlobLimitExceeded: the limit, the usage so far and the directory being read. Nothing partial is returned, and the shell no longer substitutes the literal word for a limit error. Comparison with bash 5.2.21 (globstar on, dotglob off) on the same trees: - 7-entry looped tree: bash gives 6 paths for `**/*` in under 1 ms; vsh now gives the same 6 (was: out of memory). - 328,420-entry tree: bash expands `**/*` to 328,420 paths in 1.7 s at 63 MB; vsh stops at the 100,000-match limit after 0.46 s at 11 MB. `**/*/**/*`: bash lists 976,400 paths (328,400 distinct) in 5.3 s at 182 MB; vsh lists each path once and stops at the limit in 0.33 s. - `/**/*`: bash expands it to 643,683 paths in 4.6 s at 205 MB; vsh stops at the 6 MiB byte limit at 20 MB. bash has no limits; these are deliberate departures. - `/usr/**/*`: vsh 0.48 s, bash 0.57 s. bash finds 2,956 more paths, all below symlinked directories such as /usr/bin/X11 -> . : when `**` is not the first component, bash enters such a link for one more component. vsh does not; for `**/...` patterns the two agree. Documented in glob.rs. Also: - `.*` and `.h*` now find hidden files as in bash (before: only `.`/`..`, or nothing), and a file name that is not UTF-8 no longer panics. - The fuzz target expands over a small fixed tree (two symlink loops, a hidden directory, a non-UTF-8 name), re-roots absolute inputs into it, skips `..`, runs under small limits and checks every result against the limits and the file system. 382,036 runs in 10 minutes with -rss_limit_mb=2560: no crash. - fuzz/Cargo.lock regenerated: it lacked hmac and subtle. Tests: tests/glob_bounds_tests.rs (14). The two symlink-loop tests fail on main (timeout after 20 s) and pass here. Limit, residue and determinism tests use injected limits; the read counts are checked against counts made directly from the fixtures. A differential property test checks the walker against the glob crate where the rules did not change; three planted bugs were each caught by it. security_recursive_glob_bounded now asserts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- CHANGELOG.adoc | 29 + EXPLAINME.adoc | 3 +- docs/POSIX_COMPLIANCE.adoc | 15 +- docs/PROOF-OPEN-FRONTIER.adoc | 1 + impl/rust-cli/fuzz/Cargo.lock | 20 + .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 140 ++-- impl/rust-cli/src/external.rs | 77 ++- impl/rust-cli/src/glob.rs | 632 ++++++++++++++++-- impl/rust-cli/src/grade.rs | 493 ++++++++++++++ impl/rust-cli/src/lib.rs | 1 + impl/rust-cli/tests/glob_bounds_tests.rs | 395 +++++++++++ impl/rust-cli/tests/security_tests.rs | 29 +- 12 files changed, 1713 insertions(+), 122 deletions(-) create mode 100644 impl/rust-cli/src/grade.rs create mode 100644 impl/rust-cli/tests/glob_bounds_tests.rs diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index de48c3d5..de152145 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -11,6 +11,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 == [Unreleased] +=== Fixed — 2026-11-18 (glob expansion out of memory, #191) + +- `**` no longer follows symbolic links to directories, so a link back to a + parent directory can no longer make an expansion grow without end. On the old + code `**/*` over a 7-entry tree with two such links passed 2.5 GB in 15 s, + and `**/no-such-name` over it never returned. ClusterFuzzLite hit the same + fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run + 134118693669). The rule now matches bash 5.2 with `globstar` for patterns + that start with `**`. +- Glob expansion is metered while it runs: at most 10,000,000 file-system + reads, 100,000 matches and 6 MiB of matches per pattern. Reaching a limit is + a `GlobLimitExceeded` error that names the limit, the usage so far and the + directory being read; the command is not run. Before, the match limit was + checked only after every path had been collected, and an external command + then received the literal pattern instead of an error. +- `.*` and `.h*` find hidden files, as bash does. +- A file name that is not valid UTF-8 no longer panics the expansion. +- The `fuzz_glob_expansion` target expands over a small fixed tree instead of + the runner's file system, and checks the limits on every input. +- `impl/rust-cli/fuzz/Cargo.lock` regenerated; it was missing `hmac` and + `subtle`. + +=== Added — 2026-11-18 + +- `src/grade.rs`: resource grades. A cost adds up in sequence; an occupancy + keeps a high-water mark. A `Meter` refuses the charge that would pass its + limit. Glob expansion is the first user; operation-history retention (#190) + is the intended second. The grade laws are property-tested, not proved. + === Changed — 2026-11-01 (assurance-boundary re-audit) - Corrected prose that had drifted across the assurance boundary. The audit diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 7f0fc4d8..11279c02 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -212,7 +212,8 @@ All share: *Multiple proof systems for cross-validation, honest status reporting | `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine) -| `impl/rust-cli/src/glob.rs` | POSIX-compliant glob expansion: *, ?, [...], {...}. Hidden file handling (requires explicit leading dot). +| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash globstar rules), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). +| `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved. | `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT). diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index cc34cfb5..e8202f26 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -152,7 +152,8 @@ creating empty exported var [width="100%",cols="29%,25%,46%",options="header",] |=== |Feature |Status |Implementation -|`+*+` (match any) |Done |`+glob+` crate +|`+*+` (match any) |Done |`+glob+` crate `+Pattern+` per component; +directory walk in `+glob.rs+` (2026-11-18) |`+?+` (single char) |Done | @@ -161,7 +162,17 @@ creating empty exported var |`+[!...]+` negation |Done | |Hidden file exclusion |Done |`+require_literal_leading_dot: true+` -(fixed 2026-02-12) +(fixed 2026-02-12); `+.*+` and `+.h*+` now find hidden files (fixed +2026-11-18; they previously gave only `+.+`/`+..+` or nothing) + +|Recursive `+**+` (bash globstar) |Done |Does not descend through symbolic +links or into hidden directories. One known difference from bash 5.2: bash +enters a symlinked directory for one more component when `+**+` is not the +first component (2026-11-18) + +|Expansion limits |Done |10,000,000 file-system reads, 100,000 matches, +6 MiB of matches; exceeding one is an error that names it, never a truncated +or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported diff --git a/docs/PROOF-OPEN-FRONTIER.adoc b/docs/PROOF-OPEN-FRONTIER.adoc index 0372b66f..418417b0 100644 --- a/docs/PROOF-OPEN-FRONTIER.adoc +++ b/docs/PROOF-OPEN-FRONTIER.adoc @@ -119,6 +119,7 @@ Pure induction on list length. | File | `impl/rust-cli/src/glob.rs` — no proof today. | Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property. | Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4. +| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Lean bound and no Rust contract. |=== [[A-7]] diff --git a/impl/rust-cli/fuzz/Cargo.lock b/impl/rust-cli/fuzz/Cargo.lock index ec6f8271..7d73278a 100644 --- a/impl/rust-cli/fuzz/Cargo.lock +++ b/impl/rust-cli/fuzz/Cargo.lock @@ -469,6 +469,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", "crypto-common", + "subtle", ] [[package]] @@ -724,6 +725,15 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1706,6 +1716,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + [[package]] name = "syn" version = "1.0.109" @@ -1984,13 +2000,17 @@ dependencies = [ "dirs", "gag", "glob", + "hex", + "hmac", "libc", + "nix 0.31.2", "nu-ansi-term", "ratatui", "reedline", "regex", "serde", "serde_json", + "sha2", "thiserror 2.0.18", "tokio", "uuid", diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index beecbd9d..0c8c4882 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -2,56 +2,114 @@ // Copyright (c) Jonathan D.A. Jewell //! Fuzz Target: Glob Expansion //! -//! Tests glob pattern matching for: -//! - Malicious patterns (DoS via backtracking) -//! - Deep recursion (**/**/...) -//! - Edge cases (empty patterns, special chars) -//! - Resource exhaustion +//! Expands arbitrary patterns over one small, fixed directory tree, so the +//! target exercises the shell's own walker and limits rather than the size of +//! the machine it runs on. (Until 2026-11-17 an absolute input such as `/**/*` +//! walked the runner's whole file system, following symbolic-link loops under +//! /proc and /sys, and hit libFuzzer's 2560 MB limit.) +//! +//! - Absolute patterns are re-rooted under the fixture. +//! - Patterns with a `..` component are skipped: they would leave the fixture. +//! - Expansion runs under small limits so that inputs reach them, and every +//! result is checked against the limits and against the file system. +//! +//! The fixture holds the cases that matter for termination and matching: +//! two symbolic-link loops, a dangling link, a hidden directory and a name +//! that is not valid UTF-8. #![no_main] use libfuzzer_sys::fuzz_target; +use std::fs; +use std::os::unix::ffi::OsStrExt; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::OnceLock; use tempfile::TempDir; -use vsh::glob::expand_glob; +use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; + +const LIMITS: GlobLimits = GlobLimits { + max_reads: 20_000, + max_matches: 2_000, + max_match_bytes: 256 * 1024, +}; + +fn fixture() -> &'static Path { + static FIXTURE: OnceLock = OnceLock::new(); + FIXTURE + .get_or_init(|| { + let t = TempDir::new().expect("create fixture"); + let root = t.path(); + for dir in ["d/sub", ".hd/x", "e"] { + fs::create_dir_all(root.join(dir)).unwrap(); + } + for file in [ + "a.txt", "b.txt", "c.log", ".hidden", "d/f.rs", "d/sub/g.rs", ".hd/x/y", "e/z", + ] { + fs::write(root.join(file), b"").unwrap(); + } + let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); + symlink("..", root.join("d/up")).unwrap(); // loop back to the root + symlink(".", root.join("self")).unwrap(); // loop to itself + symlink("d", root.join("link")).unwrap(); + symlink("missing", root.join("dangling")).unwrap(); + t + }) + .path() +} fuzz_target!(|data: &[u8]| { - // Convert to string - if let Ok(pattern) = std::str::from_utf8(data) { - // Limit length to prevent DoS - if pattern.len() > 200 { - return; - } + let Ok(input) = std::str::from_utf8(data) else { + return; + }; + if input.len() > 200 || input.split('/').any(|part| part == "..") { + return; + } - // Skip patterns with too many wildcards (prevent combinatorial explosion) - let wildcard_count = pattern.matches('*').count() + pattern.matches('?').count(); - if wildcard_count > 10 { - return; - } + let base = fixture(); + let pattern = if input.starts_with('/') { + format!("{}{}", base.display(), input) + } else { + input.to_string() + }; - // Skip patterns with excessive recursion - let recursive_count = pattern.matches("**").count(); - if recursive_count > 3 { - return; - } + match expand_glob_with_limits(&pattern, base, &LIMITS) { + Ok(expansion) => { + let usage = expansion.usage; + assert!(usage.reads.total() <= LIMITS.max_reads); + assert!(usage.matches.peak() <= LIMITS.max_matches); + assert!(usage.match_bytes.peak() <= LIMITS.max_match_bytes); + assert_eq!(expansion.paths.len() as u64, usage.matches.held()); + + let bytes: u64 = expansion + .paths + .iter() + .map(|p| p.as_os_str().len() as u64 + 1) + .sum(); + assert_eq!(bytes, usage.match_bytes.held()); - // Create temporary directory for glob expansion - let temp = match TempDir::new() { - Ok(t) => t, - Err(_) => return, - }; - - // Expand glob (should complete quickly or return error) - // Should NEVER: - // - Hang (exponential backtracking) - // - Panic - // - Use excessive memory - let _ = expand_glob(pattern, temp.path()); - - // Test cases include: - // - Simple wildcards: *.txt, file?.rs - // - Recursive: **/*.rs - // - Character classes: [abc], [!xyz] - // - Braces: {a,b,c} - // - Malicious: *, **, *{,/*}{,/*}{,/*}... (DoS) + // Sorted, each path once, and every path exists. + assert!(expansion.paths.windows(2).all(|w| w[0] < w[1])); + for path in &expansion.paths { + let real: PathBuf = if path.is_absolute() { + path.clone() + } else { + base.join(path) + }; + assert!( + fs::symlink_metadata(&real).is_ok(), + "{} does not exist", + real.display() + ); + } + } + Err(err) => { + if let Some(stopped) = err.downcast_ref::() { + assert!(stopped.exceeded.level > stopped.exceeded.limit); + assert!(stopped.usage.reads.total() <= LIMITS.max_reads); + assert!(stopped.usage.matches.peak() <= LIMITS.max_matches); + assert!(stopped.usage.match_bytes.peak() <= LIMITS.max_match_bytes); + } + } } }); diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 4dbc9f94..60f345a3 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -479,8 +479,11 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { /// /// For each argument: /// - If it contains glob metacharacters (*, ?, [, {), expand it -/// - If expansion succeeds, use matching paths -/// - If expansion fails or returns empty, use literal pattern (POSIX behavior) +/// - If expansion finds matches, use them +/// - If expansion finds nothing, or the word is not a valid pattern, use the +/// word literally (POSIX behavior) +/// - If expansion stops at a limit, fail: the command is not run with a +/// literal or partial argument list in place of the real expansion /// - If no metacharacters, use argument as-is /// /// # POSIX Behavior @@ -495,10 +498,22 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { /// expand_glob_args(&["rm", "*.xyz"], state) // -> ["rm", "*.xyz"] (no .xyz files) /// ``` fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> { - let mut expanded: Vec = Vec::new(); - // Get current working directory for glob expansion let cwd = std::env::current_dir().context("Failed to get current working directory")?; + expand_glob_args_in(args, &cwd, &glob::GlobLimits::DEFAULT) +} + +/// [`expand_glob_args`] with an explicit directory and limits. +/// +/// # Errors +/// Returns the [`glob::GlobLimitExceeded`] error of the first argument whose +/// expansion reaches a limit. +fn expand_glob_args_in( + args: &[String], + cwd: &std::path::Path, + limits: &glob::GlobLimits, +) -> Result> { + let mut expanded: Vec = Vec::new(); for arg in args { // Check if argument contains glob metacharacters @@ -507,25 +522,24 @@ fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> let brace_expanded = glob::expand_braces(arg); // Then expand each brace result for glob patterns - let mut glob_matches: Vec = Vec::new(); for pattern in &brace_expanded { // Expand glob pattern relative to current directory - match glob::expand_glob(pattern, &cwd) { - Ok(matches) if !matches.is_empty() => { - // Found matches - add them as strings - for path in matches { - glob_matches.push(path.to_string_lossy().to_string()); - } - } - _ => { - // No matches or error - use literal pattern (POSIX behavior) - glob_matches.push(pattern.clone()); + match glob::expand_glob_with_limits(pattern, cwd, limits) { + Ok(expansion) if !expansion.paths.is_empty() => { + expanded.extend( + expansion + .paths + .iter() + .map(|path| path.to_string_lossy().to_string()), + ); } + // A limit is not "no match": substituting the literal word + // would run the command on the wrong arguments. + Err(e) if e.is::() => return Err(e), + // No matches or not a valid pattern - use literal (POSIX behavior) + _ => expanded.push(pattern.clone()), } } - - // Add all matches (or literal if no matches) - expanded.extend(glob_matches); } else { // No glob pattern - use argument as-is expanded.push(arg.clone()); @@ -813,4 +827,31 @@ mod tests { let exit_code = execute_pipeline(&stages, &[], &mut state).unwrap(); assert_eq!(exit_code, 0, "Single-stage pipeline should work"); } + #[test] + fn glob_args_use_the_literal_word_when_nothing_matches() { + let dir = tempfile::TempDir::new().unwrap(); + std::fs::write(dir.path().join("a.txt"), b"").unwrap(); + let args = vec!["*.txt".to_string(), "*.none".to_string(), "[z-".to_string()]; + let out = expand_glob_args_in(&args, dir.path(), &glob::GlobLimits::DEFAULT).unwrap(); + assert_eq!(out, ["a.txt", "*.none", "[z-"]); + } + + #[test] + fn glob_args_fail_instead_of_substituting_the_literal_at_a_limit() { + let dir = tempfile::TempDir::new().unwrap(); + for i in 0..5 { + std::fs::write(dir.path().join(format!("f{i}.txt")), b"").unwrap(); + } + let limits = glob::GlobLimits { + max_matches: 4, + ..glob::GlobLimits::DEFAULT + }; + let args = vec!["-l".to_string(), "*.txt".to_string()]; + let err = expand_glob_args_in(&args, dir.path(), &limits).unwrap_err(); + let stopped = err + .downcast_ref::() + .expect("the limit error must reach the caller"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert!(err.to_string().contains("over the limit of 4"), "{err}"); + } } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 0a5ba18d..ae5919f8 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -2,8 +2,72 @@ // Copyright (c) Jonathan D.A. Jewell //! Glob pattern expansion (filename wildcards) //! -//! Implements POSIX-compliant glob expansion with *, ?, [...], and {...}. -//! Glob patterns are expanded to matching file paths at command execution time. +//! Expands `*`, `?`, `[...]` and the recursive `**` against the file system, +//! with fixed limits on how much work and memory one expansion may use. +//! Brace expansion (`{a,b}`) is a separate, purely textual step: see +//! [`expand_braces`]. +//! +//! # Matching rules +//! +//! The rules follow bash 5.2 with `shopt -s globstar`, `dotglob` off and +//! `globskipdots` on (the default): +//! +//! - `*`, `?` and `[...]` match within one path component. Single components +//! are matched by the `glob` crate's [`glob::Pattern`]. +//! - A name that starts with `.` is matched only by a component that itself +//! starts with a literal `.`. So `*` skips hidden files, `.*` and `.h*` find +//! them, and `.` and `..` are never produced by a wildcard. +//! - A component that is exactly `**` matches zero or more directories. It +//! lists a symbolic link to a directory but does not descend into it, and it +//! does not enter hidden directories. Other components do go through +//! symbolic links, so `link/*` lists the target directory. +//! One measured difference: when `**` is *not* the first component (an +//! absolute pattern, `./**/*`, `d/**/*`), bash 5.2 enters a symbolic link to +//! a directory found by `**` for one more component, and vsh does not. On +//! one machine `/usr/**/*` gave bash 2,956 more paths than vsh, every one of +//! them below a link such as `/usr/bin/X11 -> .`. +//! - A pattern that ends in `**` lists everything below its directory, and the +//! directory itself when it is named (`d/**` gives `d/`, `d/e`, ...). +//! - A pattern that ends in `/` matches directories only, and each result ends +//! in `/`. +//! - Results are sorted and each path appears once. (bash lists a path again +//! for every way the pattern matches it, so `**/*/**/*` can repeat a path +//! many times.) +//! - A pattern that matches nothing gives an empty list; the caller decides +//! whether to keep the word literally. +//! +//! # Limits +//! +//! One expansion is metered on three axes ([`GlobLimits`]): +//! +//! | Axis | Kind | Default | +//! |------|------|---------| +//! | file-system reads (directories opened, entries listed, names looked up) | cost: adds up | 10,000,000 | +//! | matches held | occupancy: high-water mark | 100,000 | +//! | bytes of matches held (path length + 1 for each match) | occupancy | 6 MiB | +//! +//! The meters are charged *while* the walk runs, so an expansion stops at the +//! first unit over a limit instead of collecting everything first. Stopping +//! returns a [`GlobLimitExceeded`] error that names the limit and carries the +//! usage so far and the directory being read; no partial result is returned. +//! Because entries are visited in sorted order, the same pattern over the same +//! tree always stops at the same place with the same report. +//! +//! The default byte limit is 6 MiB because Linux refuses any `execve` whose +//! argument strings take more than that, whatever the stack limit (measured on +//! Linux 6.18: 6,279,840 bytes accepted at both a 64 MiB and an unlimited +//! stack), so a larger expansion could never reach an external command. +//! +//! bash has no such limits: it expanded `/**/*` on a test machine to 643,683 +//! paths. The limits here are deliberate departures, made so that a single +//! word cannot exhaust the shell's memory. +//! +//! # Work bound +//! +//! Each pair (directory, pattern component) is processed at most once, so the +//! walk reads each directory at most once per component of the pattern. That +//! is the `O(|pattern| x |entries|)` shape stated as frontier item A-6 in +//! `docs/PROOF-OPEN-FRONTIER.adoc`; here it is tested, not proved. //! //! # Examples //! ```no_run @@ -14,12 +78,115 @@ //! // Returns: ["file1.txt", "file2.txt", ...] //! ``` -use anyhow::{bail, Context, Result}; +use crate::grade::{Cost, Exceeded, Meter, Occupancy}; +use anyhow::{Context, Result}; +use std::collections::{BTreeSet, HashSet}; +use std::ffi::OsString; +use std::fmt; +use std::fs; use std::path::{Path, PathBuf}; -/// Maximum number of results from glob expansion (prevents resource exhaustion from -/// patterns like `/**/*` on deep directory trees). -const MAX_GLOB_RESULTS: usize = 100_000; +/// Default limit on file-system reads in one expansion. +pub const MAX_GLOB_READS: u64 = 10_000_000; + +/// Default limit on the number of matches one expansion may hold. +pub const MAX_GLOB_RESULTS: u64 = 100_000; + +/// Default limit on the bytes of matches one expansion may hold: 6 MiB, the +/// most Linux `execve` accepts for argument strings. +pub const MAX_GLOB_MATCH_BYTES: u64 = 6 * 1024 * 1024; + +/// The limits one glob expansion runs under. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GlobLimits { + /// Most file-system reads: directories opened, entries listed, names looked up. + pub max_reads: u64, + /// Most matches held at once. + pub max_matches: u64, + /// Most bytes of matches held at once (path length + 1 per match). + pub max_match_bytes: u64, +} + +impl GlobLimits { + /// The limits [`expand_glob`] uses. + pub const DEFAULT: GlobLimits = GlobLimits { + max_reads: MAX_GLOB_READS, + max_matches: MAX_GLOB_RESULTS, + max_match_bytes: MAX_GLOB_MATCH_BYTES, + }; +} + +impl Default for GlobLimits { + fn default() -> Self { + GlobLimits::DEFAULT + } +} + +/// What one expansion used, on each metered axis. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct GlobUsage { + /// File-system reads: directories opened, entries listed, names looked up. + pub reads: Cost, + /// Matches held. + pub matches: Occupancy, + /// Bytes of matches held (path length + 1 per match). + pub match_bytes: Occupancy, +} + +impl fmt::Display for GlobUsage { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} file-system reads, {} matches, {} bytes of matches", + self.reads.total(), + self.matches.peak(), + self.match_bytes.peak() + ) + } +} + +/// A successful expansion: the matches and what finding them cost. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Expansion { + /// Matching paths, sorted, each once. + pub paths: Vec, + /// The resources the expansion used. + pub usage: GlobUsage, +} + +/// The error returned when an expansion reaches one of its [`GlobLimits`]. +/// +/// It is a record of where the walk stopped, not a partial result: it says +/// that more exists and where the walk was, without listing what was found. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GlobLimitExceeded { + /// The pattern as given. + pub pattern: String, + /// The limit that was reached: which axis, its value, and the level the + /// refused step would have reached. + pub exceeded: Exceeded, + /// What had been used when the walk stopped. + pub usage: GlobUsage, + /// The directory being read when the walk stopped, written the way the + /// matches would be (relative to the base directory unless the pattern is + /// absolute). + pub at: PathBuf, +} + +impl fmt::Display for GlobLimitExceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "glob pattern '{}' was not expanded: {} (stopped while reading '{}' after {})", + self.pattern, + self.exceeded, + self.at.display(), + self.usage + ) + } +} + +impl std::error::Error for GlobLimitExceeded {} /// Check if a string contains glob metacharacters /// @@ -58,16 +225,11 @@ pub fn contains_glob_pattern(s: &str) -> bool { false } -/// Expand a glob pattern to matching file paths +/// Expand a glob pattern to matching file paths, under [`GlobLimits::DEFAULT`]. /// -/// Uses the `glob` crate for POSIX-compliant pattern matching. -/// Returns paths sorted alphabetically (POSIX requirement). -/// -/// # POSIX Behavior -/// - Patterns that match nothing return empty vec (caller handles literal fallback) -/// - Hidden files (starting with .) require explicit match: .* or .config -/// - Results are sorted alphabetically -/// - * and ? do not match directory separators (/) +/// Relative patterns are matched below `base_dir` and give paths relative to +/// it; patterns that start with `/` give absolute paths. See the +/// [module documentation](self) for the matching rules and the limits. /// /// # Examples /// ```no_run @@ -85,56 +247,418 @@ pub fn contains_glob_pattern(s: &str) -> bool { /// ``` /// /// # Errors -/// Returns error if: -/// - Pattern is invalid -/// - Base directory cannot be accessed +/// Returns an error if the pattern is invalid, or a [`GlobLimitExceeded`] +/// (reachable with `anyhow::Error::downcast_ref`) if the expansion reaches a +/// limit. Directories that cannot be read are skipped, as bash does. pub fn expand_glob(pattern: &str, base_dir: &Path) -> Result> { - // Build full glob pattern (base_dir + pattern) - let full_pattern = if pattern.starts_with('/') { - // Absolute path - pattern.to_string() - } else { - // Relative to base_dir - base_dir.join(pattern).to_string_lossy().to_string() - }; + expand_glob_with_limits(pattern, base_dir, &GlobLimits::DEFAULT).map(|e| e.paths) +} - // Use glob crate for pattern matching with POSIX options - let options = glob::MatchOptions { - // POSIX: * and ? do not match leading dot (hidden files) - require_literal_leading_dot: true, - // Case-sensitive matching (POSIX default) - case_sensitive: true, - // Require literal separator - require_literal_separator: false, - }; +/// Expand a glob pattern under the given limits, reporting what it used. +/// +/// # Examples +/// ``` +/// use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; +/// +/// let dir = tempfile::tempdir().unwrap(); +/// for name in ["a.txt", "b.txt", "c.txt"] { +/// std::fs::write(dir.path().join(name), b"").unwrap(); +/// } +/// +/// let limits = GlobLimits { max_matches: 2, ..GlobLimits::DEFAULT }; +/// let err = expand_glob_with_limits("*.txt", dir.path(), &limits).unwrap_err(); +/// let stopped = err.downcast_ref::().unwrap(); +/// assert_eq!(stopped.exceeded.axis, "matches held"); +/// assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (2, 3)); +/// ``` +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_with_limits( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, +) -> Result { + let compiled = + Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; + let mut walker = Walker::new(pattern, base_dir, &compiled, limits); + walker.run().map_err(anyhow::Error::new)?; + Ok(Expansion { + usage: walker.usage(), + paths: walker.found.into_iter().collect(), + }) +} - let mut matches: Vec = glob::glob_with(&full_pattern, options) - .with_context(|| format!("Invalid glob pattern: {}", pattern))? - .filter_map(|entry| entry.ok()) - .collect(); +/// One path component of a compiled pattern. +#[derive(Debug)] +enum Component { + /// No metacharacters: looked up by name, without listing the directory. + Literal(String), + /// Matched against each name in the directory. + Wild(glob::Pattern), + /// `**`: zero or more directories. + Recursive, +} - // Sort results (POSIX requirement) - matches.sort(); +#[derive(Debug)] +struct Compiled { + components: Vec, + absolute: bool, + dirs_only: bool, +} - // Guard against resource exhaustion from overly broad patterns - if matches.len() > MAX_GLOB_RESULTS { - bail!( - "Glob pattern '{}' matched {} files (limit: {})", +impl Compiled { + fn new(pattern: &str) -> std::result::Result { + let mut components: Vec = Vec::new(); + for part in pattern.split('/').filter(|part| !part.is_empty()) { + let component = if part == "**" { + // `**/**` matches exactly what `**` matches. + if matches!(components.last(), Some(Component::Recursive)) { + continue; + } + Component::Recursive + } else if part.contains(['*', '?', '[']) { + Component::Wild(glob::Pattern::new(part)?) + } else { + Component::Literal(part.to_string()) + }; + components.push(component); + } + Ok(Compiled { + components, + absolute: pattern.starts_with('/'), + dirs_only: pattern.len() > 1 && pattern.ends_with('/'), + }) + } +} + +const MATCH_OPTIONS: glob::MatchOptions = glob::MatchOptions { + case_sensitive: true, + require_literal_separator: true, + require_literal_leading_dot: true, +}; + +/// The walk: a depth-first search over states (directory, component index), +/// each processed at most once. +struct Walker<'a> { + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + reads: Meter, + matches: Meter, + match_bytes: Meter, + found: BTreeSet, +} + +impl<'a> Walker<'a> { + fn new(pattern: &'a str, base: &'a Path, compiled: &'a Compiled, limits: &GlobLimits) -> Self { + Walker { pattern, - matches.len(), - MAX_GLOB_RESULTS, - ); + base, + compiled, + reads: Meter::new("file-system reads", limits.max_reads), + matches: Meter::new("matches held", limits.max_matches), + match_bytes: Meter::new("bytes of matches held", limits.max_match_bytes), + found: BTreeSet::new(), + } + } + + fn usage(&self) -> GlobUsage { + GlobUsage { + reads: self.reads.grade(), + matches: self.matches.grade(), + match_bytes: self.match_bytes.grade(), + } + } + + /// Where `path` (as it would be reported) is on the file system. + fn real(&self, path: &Path) -> PathBuf { + if self.compiled.absolute { + path.to_path_buf() + } else if path.as_os_str().is_empty() { + self.base.to_path_buf() + } else { + self.base.join(path) + } + } + + fn stopped(&self, exceeded: Exceeded, at: &Path) -> GlobLimitExceeded { + GlobLimitExceeded { + pattern: self.pattern.to_string(), + exceeded, + usage: self.usage(), + at: if at.as_os_str().is_empty() { + PathBuf::from(".") + } else { + at.to_path_buf() + }, + } + } + + fn charge_read(&mut self, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + self.reads.charge(1).map_err(|e| self.stopped(e, at)) + } + + /// Record a match, once. + fn emit(&mut self, path: PathBuf, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + let path = if self.compiled.dirs_only { + path.join("") + } else { + path + }; + if self.found.contains(&path) { + return Ok(()); + } + let bytes = path.as_os_str().len() as u64 + 1; + self.matches.charge(1).map_err(|e| self.stopped(e, at))?; + if let Err(e) = self.match_bytes.charge(bytes) { + self.matches.release(1); + return Err(self.stopped(e, at)); + } + self.found.insert(path); + Ok(()) + } + + /// List a directory in name order, charging the open and every entry. + /// An unreadable directory gives no entries. + fn list( + &mut self, + dir: &Path, + ) -> std::result::Result, GlobLimitExceeded> { + self.charge_read(dir)?; + let mut entries = Vec::new(); + if let Ok(read) = fs::read_dir(self.real(dir)) { + for entry in read { + self.charge_read(dir)?; + if let Ok(entry) = entry { + if let Ok(kind) = entry.file_type() { + entries.push((entry.file_name(), kind)); + } + } + } + } + entries.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(entries) + } + + /// Whether `path` is a directory, going through a symbolic link. + fn is_dir(&self, path: &Path, kind: Option) -> bool { + match kind { + Some(kind) if kind.is_dir() => true, + Some(kind) if !kind.is_symlink() => false, + _ => fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir()), + } + } + + fn run(&mut self) -> std::result::Result<(), GlobLimitExceeded> { + let count = self.compiled.components.len(); + if count == 0 { + return Ok(()); + } + let start = if self.compiled.absolute { + PathBuf::from("/") + } else { + PathBuf::new() + }; + let mut stack: Vec<(PathBuf, usize)> = vec![(start, 0)]; + let mut seen: HashSet<(PathBuf, usize)> = HashSet::new(); + + while let Some((dir, index)) = stack.pop() { + if !seen.insert((dir.clone(), index)) { + continue; + } + let last = index + 1 == count; + let compiled = self.compiled; + // Successor states, pushed in reverse so they are visited in name order. + let mut next: Vec<(PathBuf, usize)> = Vec::new(); + + match &compiled.components[index] { + Component::Literal(name) => { + self.charge_read(&dir)?; + let child = dir.join(name); + if last { + let exists = fs::symlink_metadata(self.real(&child)).is_ok(); + if exists && (!compiled.dirs_only || self.is_dir(&child, None)) { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, None) { + next.push((child, index + 1)); + } + } + Component::Wild(pattern) => { + for (name, kind) in self.list(&dir)? { + if !pattern.matches_with(&name.to_string_lossy(), MATCH_OPTIONS) { + continue; + } + let child = dir.join(&name); + if last { + if !compiled.dirs_only || self.is_dir(&child, Some(kind)) { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, Some(kind)) { + next.push((child, index + 1)); + } + } + } + Component::Recursive => { + // Zero directories: the pattern continues here. + if !last { + next.push((dir.clone(), index + 1)); + } else if !dir.as_os_str().is_empty() { + self.emit(dir.join(""), &dir)?; + } + // One more directory: descend into real, visible directories. + for (name, kind) in self.list(&dir)? { + if name.as_encoded_bytes().starts_with(b".") { + continue; + } + let child = dir.join(&name); + if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind))) { + self.emit(child.clone(), &dir)?; + } + if kind.is_dir() { + next.push((child, index)); + } + } + } + } + stack.extend(next.into_iter().rev()); + } + Ok(()) } +} - // Convert absolute paths back to relative if needed - if !pattern.starts_with('/') { - matches = matches +#[cfg(test)] +mod walk_tests { + use super::*; + use proptest::prelude::*; + use tempfile::TempDir; + + fn expand(pattern: &str, base: &Path) -> Vec { + expand_glob(pattern, base) + .unwrap() .into_iter() - .map(|p| p.strip_prefix(base_dir).unwrap_or(&p).to_path_buf()) - .collect(); + .map(|p| p.to_string_lossy().into_owned()) + .collect() } - Ok(matches) + #[test] + fn literal_and_wild_components() { + let t = TempDir::new().unwrap(); + fs::create_dir_all(t.path().join("src/sub")).unwrap(); + for f in ["src/a.rs", "src/b.txt", "src/sub/c.rs", "top.rs"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!(expand("src/*.rs", t.path()), ["src/a.rs"]); + assert_eq!(expand("*/*/*.rs", t.path()), ["src/sub/c.rs"]); + assert_eq!(expand("./*.rs", t.path()), ["./top.rs"]); + assert_eq!( + expand("**/*.rs", t.path()), + ["src/a.rs", "src/sub/c.rs", "top.rs"] + ); + assert_eq!(expand("src/**/c.rs", t.path()), ["src/sub/c.rs"]); + assert!(expand("*.none", t.path()).is_empty()); + } + + #[test] + fn absolute_patterns_give_absolute_paths() { + let t = TempDir::new().unwrap(); + fs::write(t.path().join("x.txt"), b"").unwrap(); + let pattern = format!("{}/*.txt", t.path().display()); + assert_eq!( + expand_glob(&pattern, Path::new("/nonexistent")).unwrap(), + [t.path().join("x.txt")] + ); + } + + #[test] + fn invalid_patterns_are_errors_not_limits() { + let t = TempDir::new().unwrap(); + for bad in ["a**", "***", "[z-"] { + let err = expand_glob(bad, t.path()).unwrap_err(); + assert!(err.downcast_ref::().is_none(), "{bad}"); + } + } + + #[test] + fn a_base_directory_with_metacharacters_is_taken_literally() { + let t = TempDir::new().unwrap(); + let base = t.path().join("we[ir]d*"); + fs::create_dir(&base).unwrap(); + fs::write(base.join("f.txt"), b"").unwrap(); + assert_eq!(expand("*.txt", &base), ["f.txt"]); + } + + #[test] + fn empty_pattern_matches_nothing() { + let t = TempDir::new().unwrap(); + assert!(expand("", t.path()).is_empty()); + } + + // ------------------------------------------------------------------ + // Differential check against the `glob` crate, which this walker + // replaces. On trees without symbolic links and for patterns that do not + // end in `**` and have no component starting with `.`, the two must find + // the same set of paths (the crate can list a path twice; this walker + // lists it once). Those exclusions are exactly where the rules changed. + // ------------------------------------------------------------------ + + fn crate_glob(pattern: &str, base: &Path) -> BTreeSet { + let options = glob::MatchOptions { + require_literal_leading_dot: true, + case_sensitive: true, + require_literal_separator: false, + }; + glob::glob_with(&base.join(pattern).to_string_lossy(), options) + .unwrap() + .filter_map(|p| p.ok()) + .map(|p| p.strip_prefix(base).unwrap().to_path_buf()) + .collect() + } + + fn tree() -> impl Strategy, bool)>> { + let name = prop::sample::select(vec!["a", "b", "ab", "ba", ".h"]); + prop::collection::vec((prop::collection::vec(name, 1..4), any::()), 0..25) + } + + fn pattern() -> impl Strategy { + let component = prop::sample::select(vec![ + "*", "?", "a*", "*b", "[ab]", "[!a]*", "a", "b", "ab", "**", + ]); + (prop::collection::vec(component, 1..5), any::()).prop_filter_map( + "pattern ends in **", + |(parts, slash)| { + if parts.last() == Some(&"**") { + return None; + } + let mut p = parts.join("/"); + if slash { + p.push('/'); + } + Some(p) + }, + ) + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(96))] + #[test] + fn agrees_with_the_glob_crate(entries in tree(), pattern in pattern()) { + let t = TempDir::new().unwrap(); + for (parts, is_dir) in &entries { + let path = parts.iter().collect::(); + let full = t.path().join(&path); + if let Some(parent) = full.parent() { + if fs::create_dir_all(parent).is_err() { + continue; + } + } + let _ = if *is_dir { fs::create_dir_all(&full) } else { fs::write(&full, b"") }; + } + let ours: BTreeSet = expand_glob(&pattern, t.path()).unwrap().into_iter().collect(); + prop_assert_eq!(ours, crate_glob(&pattern, t.path()), "pattern {}", pattern); + } + } } /// Expand brace patterns: a{b,c}d -> [abd, acd] diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs new file mode 100644 index 00000000..8f5143ff --- /dev/null +++ b/impl/rust-cli/src/grade.rs @@ -0,0 +1,493 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Resource grades: deterministic accounting for bounded work and bounded holdings. +//! +//! A shell operation can use a resource in two different ways, and the two +//! compose differently: +//! +//! - **Cost** is consumed and never given back: directory entries read, +//! comparisons made. Doing one thing and then another costs the *sum*. +//! - **Occupancy** is held for a while and may be released: paths kept in +//! memory, history entries retained. What matters is the *high-water mark*, +//! the most that was held at one time. +//! +//! A [`Grade`] is a value of one of those two kinds. A [`Meter`] measures a +//! grade while the work runs and refuses a charge that would pass its limit, +//! so a limit is enforced where the resource is used and not after the fact. +//! +//! Both kinds are plain counts. They depend only on the input and on the state +//! of the file system, never on the clock or on the machine, so a test can +//! compute the expected grade independently and compare it for equality. +//! +//! # Laws +//! +//! For both kinds, [`Grade::then`] is associative with [`Grade::ZERO`] as its +//! identity, [`Grade::either`] is associative, commutative and idempotent, and +//! `then` distributes over `either`. Metering a sequence of charges gives +//! exactly the grade obtained by composing those charges with `then`. The tests +//! at the end of this file check these laws on random inputs; they are tests, +//! not proofs. +//! +//! # Relation to the estate's type theories +//! +//! The split follows the vocabulary of `tropical-types` (cost: sequence adds, +//! choice takes the maximum) and of occupancy grades (high-water marks) as +//! summarised in `docs/THEORY-FEED.adoc`. Nothing is imported from those +//! repositories and no theorem of theirs is relied on here. +//! +//! # Examples +//! +//! ``` +//! use vsh::grade::{Cost, Grade, Meter, Occupancy}; +//! +//! // Cost adds up. +//! assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); +//! +//! // Occupancy keeps what is still held and remembers the peak. +//! let held = Occupancy::of(5).then(Occupancy::of(2)); +//! assert_eq!((held.held(), held.peak()), (7, 7)); +//! assert_eq!(held.released(6).peak(), 7); +//! +//! // A meter refuses the charge that would pass its limit. +//! let mut entries: Meter = Meter::new("directory entries read", 2); +//! assert!(entries.charge(2).is_ok()); +//! let refused = entries.charge(1).unwrap_err(); +//! assert_eq!((refused.limit, refused.level), (2, 3)); +//! assert_eq!(entries.level(), 2); // the refused charge was not recorded +//! ``` + +use std::fmt; + +/// A summary of how much of one resource a piece of work used. +/// +/// See the [module documentation](self) for the two kinds and their laws. +pub trait Grade: Copy + Eq + fmt::Debug { + /// The grade of doing nothing. + const ZERO: Self; + + /// The grade of using `amount` units in a single step. + fn of(amount: u64) -> Self; + + /// Sequential composition: this work, and then `next`. + /// + /// `next` starts from whatever this work still holds. + #[must_use] + fn then(self, next: Self) -> Self; + + /// The worst case of two alternatives, of which only one happens. + #[must_use] + fn either(self, other: Self) -> Self; + + /// The single number a limit is compared against. + fn level(self) -> u64; +} + +/// A cumulative cost: a resource that is consumed and never given back. +/// +/// Sequential composition adds; the worst case of two alternatives is the +/// larger. Addition saturates at `u64::MAX` instead of wrapping. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Cost(u64); + +impl Cost { + /// The total consumed. + pub const fn total(self) -> u64 { + self.0 + } +} + +impl Grade for Cost { + const ZERO: Self = Cost(0); + + fn of(amount: u64) -> Self { + Cost(amount) + } + + fn then(self, next: Self) -> Self { + Cost(self.0.saturating_add(next.0)) + } + + fn either(self, other: Self) -> Self { + Cost(self.0.max(other.0)) + } + + fn level(self) -> u64 { + self.0 + } +} + +impl fmt::Display for Cost { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +/// An occupancy: how much of a resource is held now, and the most that was +/// ever held at one time. +/// +/// Invariant: `peak >= held`. Every constructor and operation preserves it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Occupancy { + held: u64, + peak: u64, +} + +impl Occupancy { + /// How much is held at the end of the work. + pub const fn held(self) -> u64 { + self.held + } + + /// The high-water mark: the most that was held at one time. + pub const fn peak(self) -> u64 { + self.peak + } + + /// The same work after giving back `amount` units of what it holds. + /// + /// Releasing more than is held releases everything. The peak is unchanged: + /// a release cannot undo a high-water mark. + #[must_use] + pub fn released(self, amount: u64) -> Self { + Occupancy { + held: self.held.saturating_sub(amount), + peak: self.peak, + } + } + + /// The same work after giving back everything it holds. + #[must_use] + pub fn settled(self) -> Self { + self.released(self.held) + } +} + +impl Grade for Occupancy { + const ZERO: Self = Occupancy { held: 0, peak: 0 }; + + fn of(amount: u64) -> Self { + Occupancy { + held: amount, + peak: amount, + } + } + + /// `next` runs on top of what this work still holds, so its peak is + /// measured from there. + fn then(self, next: Self) -> Self { + Occupancy { + held: self.held.saturating_add(next.held), + peak: self.peak.max(self.held.saturating_add(next.peak)), + } + } + + fn either(self, other: Self) -> Self { + Occupancy { + held: self.held.max(other.held), + peak: self.peak.max(other.peak), + } + } + + fn level(self) -> u64 { + self.peak + } +} + +impl fmt::Display for Occupancy { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} (peak {})", self.held, self.peak) + } +} + +/// A charge that a [`Meter`] refused because it would pass the limit. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Exceeded { + /// What was being counted, for example `"directory entries read"`. + pub axis: &'static str, + /// The limit that was configured. + pub limit: u64, + /// The level the refused charge would have reached. + pub level: u64, +} + +impl fmt::Display for Exceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} would reach {}, over the limit of {}", + self.axis, self.level, self.limit + ) + } +} + +impl std::error::Error for Exceeded {} + +/// Measures one grade against a limit while the work runs. +/// +/// Invariant: `level() <= limit()`. A charge that would break it is refused +/// and leaves the meter unchanged. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Meter { + axis: &'static str, + limit: u64, + grade: G, +} + +impl Meter { + /// A meter at zero for the resource named `axis`, allowing at most `limit`. + pub const fn new(axis: &'static str, limit: u64) -> Self { + Meter { + axis, + limit, + grade: G::ZERO, + } + } + + /// Record the use of `amount` more units, or refuse if that passes the limit. + /// + /// # Errors + /// Returns [`Exceeded`] and records nothing when the charge would take the + /// level over the limit. + pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + let next = self.grade.then(G::of(amount)); + if next.level() > self.limit { + return Err(Exceeded { + axis: self.axis, + limit: self.limit, + level: next.level(), + }); + } + self.grade = next; + Ok(()) + } + + /// The grade measured so far. + pub fn grade(&self) -> G { + self.grade + } + + /// The level measured so far; never more than [`Meter::limit`]. + pub fn level(&self) -> u64 { + self.grade.level() + } + + /// The configured limit. + pub fn limit(&self) -> u64 { + self.limit + } + + /// What this meter counts. + pub fn axis(&self) -> &'static str { + self.axis + } +} + +impl Meter { + /// Record that `amount` units are no longer held. + /// + /// Only an occupancy can be released. There is deliberately no such method + /// on `Meter`: a cost that has been paid cannot be taken back. + pub fn release(&mut self, amount: u64) { + self.grade = self.grade.released(amount); + } + + /// How much is held right now. + pub fn held(&self) -> u64 { + self.grade.held() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use proptest::prelude::*; + + #[test] + fn cost_adds_in_sequence_and_takes_the_larger_alternative() { + assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); + assert_eq!(Cost::of(3).either(Cost::of(4)), Cost::of(4)); + assert_eq!(Cost::ZERO.total(), 0); + } + + #[test] + fn cost_saturates_instead_of_wrapping() { + assert_eq!(Cost::of(u64::MAX).then(Cost::of(1)), Cost::of(u64::MAX)); + } + + #[test] + fn occupancy_remembers_the_peak_after_a_release() { + let work = Occupancy::of(10).released(8).then(Occupancy::of(3)); + assert_eq!(work.held(), 5); + assert_eq!(work.peak(), 10); + } + + #[test] + fn occupancy_of_two_settled_phases_is_the_larger_peak() { + let first = Occupancy::of(10).settled(); + let second = Occupancy::of(4).settled(); + let both = first.then(second); + assert_eq!((both.held(), both.peak()), (0, 10)); + } + + #[test] + fn occupancy_of_two_retained_phases_adds() { + let both = Occupancy::of(10).then(Occupancy::of(4)); + assert_eq!((both.held(), both.peak()), (14, 14)); + } + + #[test] + fn meter_accepts_up_to_the_limit_and_refuses_the_next_unit() { + let mut meter: Meter = Meter::new("steps", 3); + assert!(meter.charge(3).is_ok()); + assert_eq!(meter.level(), 3); + + let refused = meter.charge(1).unwrap_err(); + assert_eq!( + refused, + Exceeded { + axis: "steps", + limit: 3, + level: 4 + } + ); + assert_eq!(meter.level(), 3, "a refused charge must not be recorded"); + assert_eq!( + refused.to_string(), + "steps would reach 4, over the limit of 3" + ); + } + + #[test] + fn occupancy_meter_can_reuse_what_was_released() { + let mut meter: Meter = Meter::new("items held", 10); + meter.charge(10).unwrap(); + assert!(meter.charge(1).is_err()); + + meter.release(4); + assert_eq!(meter.held(), 6); + assert!(meter.charge(4).is_ok(), "released room can be used again"); + assert!(meter.charge(1).is_err()); + assert_eq!(meter.grade().peak(), 10); + } + + #[test] + fn a_meter_with_a_zero_limit_refuses_any_use() { + let mut meter: Meter = Meter::new("items held", 0); + assert!(meter.charge(0).is_ok()); + assert!(meter.charge(1).is_err()); + } + + /// A well-formed occupancy: `peak >= held`. + fn occupancy() -> impl Strategy { + (any::(), any::()).prop_map(|(held, extra)| Occupancy { + held: u64::from(held), + peak: u64::from(held) + u64::from(extra), + }) + } + + fn cost() -> impl Strategy { + any::().prop_map(Cost::of) + } + + /// The laws stated in the module documentation, for any grade. + fn check_laws(a: G, b: G, c: G) -> Result<(), TestCaseError> { + // `then` is a monoid. + prop_assert_eq!(a.then(b).then(c), a.then(b.then(c))); + prop_assert_eq!(G::ZERO.then(a), a); + prop_assert_eq!(a.then(G::ZERO), a); + // `either` is a semilattice. + prop_assert_eq!(a.either(b).either(c), a.either(b.either(c))); + prop_assert_eq!(a.either(b), b.either(a)); + prop_assert_eq!(a.either(a), a); + // `then` distributes over `either` on both sides. + prop_assert_eq!(a.then(b.either(c)), a.then(b).either(a.then(c))); + prop_assert_eq!(a.either(b).then(c), a.then(c).either(b.then(c))); + // Doing more never lowers the level. + prop_assert!(a.then(b).level() >= a.level()); + prop_assert!(a.either(b).level() >= a.level()); + Ok(()) + } + + proptest! { + #[test] + fn cost_obeys_the_grade_laws(a in cost(), b in cost(), c in cost()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_obeys_the_grade_laws(a in occupancy(), b in occupancy(), c in occupancy()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_operations_keep_peak_at_least_held( + a in occupancy(), + b in occupancy(), + amount in any::(), + ) { + for grade in [a.then(b), a.either(b), a.released(amount), a.settled()] { + prop_assert!(grade.peak() >= grade.held()); + } + prop_assert_eq!(a.released(amount).peak(), a.peak()); + prop_assert_eq!(a.settled().held(), 0); + } + + /// Metering charges one by one gives exactly the grade obtained by + /// composing them, and never passes the limit. + #[test] + fn a_cost_meter_agrees_with_composition( + charges in prop::collection::vec(0u64..1_000, 0..40), + limit in 0u64..20_000, + ) { + let mut meter: Meter = Meter::new("steps", limit); + let mut composed = Cost::ZERO; + for amount in charges { + let wanted = composed.then(Cost::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.limit, limit); + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + } + } + + /// The same for an occupancy meter, with releases mixed in. + #[test] + fn an_occupancy_meter_agrees_with_composition( + steps in prop::collection::vec((any::(), 0u64..1_000), 0..40), + limit in 0u64..5_000, + ) { + let mut meter: Meter = Meter::new("items held", limit); + let mut composed = Occupancy::ZERO; + for (is_release, amount) in steps { + if is_release { + meter.release(amount); + composed = composed.released(amount); + } else { + let wanted = composed.then(Occupancy::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, wanted.level()); + } + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + prop_assert!(meter.held() <= meter.level()); + } + } + } +} diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index 2323f8db..fd5fa34a 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -21,6 +21,7 @@ pub mod friendly_errors; pub mod fs_pure; pub mod functions; pub mod glob; +pub mod grade; pub mod help; pub mod highlighter; pub mod history; diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs new file mode 100644 index 00000000..bd613cef --- /dev/null +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -0,0 +1,395 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Glob expansion: termination, limits and resource accounting (issue #191). +//! +//! The first two tests reproduce the out-of-memory failure that +//! ClusterFuzzLite found on 2026-11-17 (`fuzz_glob_expansion`, input `/**/*`): +//! the previous expansion followed symbolic links under `**`, so a single link +//! back to a parent directory made `**/*` grow without end. Each runs the +//! expansion on a worker thread and fails if it has not finished in time, +//! instead of hanging the test run. +//! +//! The other tests use small injected limits so they are exact and fast, and +//! check the reported usage against counts computed here directly from the +//! fixture. +//! +//! Run with: `cargo test --test glob_bounds_tests` + +#![cfg(unix)] + +use std::collections::BTreeSet; +use std::fs; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::mpsc; +use std::time::{Duration, Instant}; +use tempfile::TempDir; +use vsh::glob::{expand_glob, expand_glob_with_limits, Expansion, GlobLimitExceeded, GlobLimits}; + +const TIME_ALLOWED: Duration = Duration::from_secs(20); + +/// Run `expand_glob` on a worker thread; panic if it does not finish in time. +fn expand_within_time(pattern: &'static str, base: PathBuf) -> Vec { + let (send, receive) = mpsc::channel(); + std::thread::spawn(move || { + let _ = send.send(expand_glob(pattern, &base).map_err(|e| e.to_string())); + }); + match receive.recv_timeout(TIME_ALLOWED) { + Ok(result) => result + .expect("expansion failed") + .into_iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect(), + Err(_) => panic!("expand_glob({pattern:?}) did not finish within {TIME_ALLOWED:?}"), + } +} + +fn strings(paths: &[PathBuf]) -> Vec { + paths + .iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect() +} + +fn stopped(result: anyhow::Result) -> GlobLimitExceeded { + let err = result.expect_err("the expansion must stop at a limit"); + err.downcast_ref::() + .unwrap_or_else(|| panic!("expected a limit error, got: {err}")) + .clone() +} + +/// The 7-entry tree with two symbolic-link loops used in the reproduction: +/// `d/{f1,f2}`, `d/up -> ..`, `self -> .`, `top`. +fn looped_tree() -> TempDir { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("d")).unwrap(); + for f in ["d/f1", "d/f2", "top"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("..", t.path().join("d/up")).unwrap(); + symlink(".", t.path().join("self")).unwrap(); + t +} + +/// `dirs` directories with `files` files each, plus `files` files at the top. +fn broad_tree(dirs: usize, files: usize) -> TempDir { + let t = TempDir::new().unwrap(); + for d in 0..dirs { + let dir = t.path().join(format!("d{d:03}")); + fs::create_dir(&dir).unwrap(); + for f in 0..files { + fs::write(dir.join(format!("f{f:03}")), b"").unwrap(); + } + } + for f in 0..files { + fs::write(t.path().join(format!("f{f:03}")), b"").unwrap(); + } + t +} + +// --------------------------------------------------------------------------- +// Termination over symbolic-link loops (fails on the previous implementation) +// --------------------------------------------------------------------------- + +#[test] +fn recursive_glob_over_symlink_loops_terminates_with_the_bash_result() { + let t = looped_tree(); + // bash 5.2 with globstar gives exactly these six paths on this tree. + assert_eq!( + expand_within_time("**/*", t.path().to_path_buf()), + ["d", "d/f1", "d/f2", "d/up", "self", "top"] + ); +} + +#[test] +fn recursive_glob_with_no_match_over_symlink_loops_terminates() { + let t = looped_tree(); + assert!(expand_within_time("**/no-such-name", t.path().to_path_buf()).is_empty()); +} + +#[test] +fn non_recursive_components_still_go_through_symlinks() { + let t = looped_tree(); + assert_eq!( + expand_within_time("d/up/*", t.path().to_path_buf()), + ["d/up/d", "d/up/self", "d/up/top"] + ); + assert_eq!( + expand_within_time("self/self/top", t.path().to_path_buf()), + ["self/self/top"] + ); +} + +// --------------------------------------------------------------------------- +// Each limit stops the walk, names itself, and returns nothing partial +// --------------------------------------------------------------------------- + +#[test] +fn the_match_limit_stops_the_walk_and_names_itself() { + let t = broad_tree(30, 40); // 1,230 entries + let limits = GlobLimits { + max_matches: 1_000, + ..GlobLimits::DEFAULT + }; + let started = Instant::now(); + let s = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + assert!(started.elapsed() < Duration::from_secs(5)); + + assert_eq!(s.exceeded.axis, "matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (1_000, 1_001)); + assert_eq!(s.usage.matches.peak(), 1_000); + assert_eq!(s.pattern, "**/*"); + let message = s.to_string(); + assert!(message.contains("matches held would reach 1001, over the limit of 1000")); + assert!(message.contains("was not expanded"), "{message}"); +} + +#[test] +fn the_read_limit_stops_a_pattern_that_matches_nothing() { + // A cap on matches alone cannot stop this one: nothing ever matches. + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_reads: 500, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits( + "**/no-such-name", + t.path(), + &limits, + )); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (500, 501)); + assert_eq!(s.usage.matches.peak(), 0); +} + +#[test] +fn the_byte_limit_counts_path_length_plus_one() { + let t = TempDir::new().unwrap(); + for name in ["aaaa", "bbbb", "cccc"] { + fs::write(t.path().join(name), b"").unwrap(); + } + // Each match costs 4 + 1 = 5 bytes: two fit in 10, the third does not. + let limits = GlobLimits { + max_match_bytes: 10, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (10, 15)); + assert_eq!( + s.usage.matches.held(), + 2, + "the refused match is not counted" + ); + + let limits = GlobLimits { + max_match_bytes: 15, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("*", t.path(), &limits).unwrap(); + assert_eq!(ok.usage.match_bytes.peak(), 15); +} + +#[test] +fn the_report_is_the_same_on_every_run() { + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_matches: 777, + ..GlobLimits::DEFAULT + }; + let first = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + for _ in 0..3 { + assert_eq!( + stopped(expand_glob_with_limits("**/*", t.path(), &limits)), + first + ); + } + // Entries are visited in name order. The top level gives 30 directories + // and 40 files (70 matches), then d000..d016 give 40 each (750), so the + // 778th match, the first one refused, is the 28th file of d017. + assert_eq!(first.at, PathBuf::from("d017")); + assert_eq!(first.usage.matches.peak(), 777); +} + +#[test] +fn default_limits_are_the_documented_values() { + assert_eq!( + GlobLimits::default(), + GlobLimits { + max_reads: 10_000_000, + max_matches: 100_000, + max_match_bytes: 6 * 1024 * 1024, + } + ); +} + +// --------------------------------------------------------------------------- +// Accounting: the reported usage equals a count made directly from the tree +// --------------------------------------------------------------------------- + +/// Visible, real (not symbolic-link) directories at or below `dir`, with the +/// number of entries in each. +fn visible_dirs(dir: &Path, out: &mut Vec) { + let entries: Vec<_> = fs::read_dir(dir).unwrap().map(|e| e.unwrap()).collect(); + out.push(entries.len() as u64); + for e in entries { + let hidden = e.file_name().to_string_lossy().starts_with('.'); + if e.file_type().unwrap().is_dir() && !hidden { + visible_dirs(&e.path(), out); + } + } +} + +#[test] +fn reads_for_a_two_level_wildcard_match_a_direct_count() { + let t = broad_tree(7, 5); + let e = expand_glob_with_limits("*/*", t.path(), &GlobLimits::DEFAULT).unwrap(); + // Open the top (1) and list its 7 + 5 entries, then open and list each of + // the 7 directories (1 + 5 each). + assert_eq!(e.usage.reads.total(), 1 + (7 + 5) + 7 * (1 + 5)); + assert_eq!(e.paths.len(), 35); + assert_eq!(e.usage.matches.peak(), 35); +} + +#[test] +fn reads_for_a_recursive_pattern_match_a_direct_count() { + let t = broad_tree(4, 3); + fs::create_dir_all(t.path().join("d001/deep/er")).unwrap(); + fs::create_dir(t.path().join(".hidden")).unwrap(); + fs::write(t.path().join(".hidden/x.rs"), b"").unwrap(); + symlink("d000", t.path().join("link")).unwrap(); + + let e = expand_glob_with_limits("**/*.rs", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert!(e.paths.is_empty()); + + // `**` lists every visible real directory once, and `*.rs` lists each of + // them once more: 2 x (open + entries) per directory. + let mut dirs = Vec::new(); + visible_dirs(t.path(), &mut dirs); + let expected: u64 = dirs.iter().map(|entries| 2 * (1 + entries)).sum(); + assert_eq!(e.usage.reads.total(), expected); +} + +#[test] +fn repeated_recursive_components_read_each_directory_a_bounded_number_of_times() { + // A chain of 30 nested directories. bash lists a path once per way the + // pattern can match it (736,281 results for six `**`); each (directory, + // component) pair is processed once here, so the work stays linear. + let t = TempDir::new().unwrap(); + let mut p = t.path().to_path_buf(); + for _ in 0..30 { + p.push("a"); + } + fs::create_dir_all(&p).unwrap(); + fs::write(p.join("leaf"), b"").unwrap(); + + let pattern = "**/*/**/*/**/*/**/*/**/*/**/*"; + let e = expand_glob_with_limits(pattern, t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(e.paths.len(), 26); // the paths at depth 6 or more, once each + let entries_in_tree = 31; + let components = 12; + assert!( + e.usage.reads.total() <= 2 * components * entries_in_tree, + "{} reads", + e.usage.reads.total() + ); +} + +// --------------------------------------------------------------------------- +// Matching rules shared with bash 5.2 (globstar on, dotglob off) +// --------------------------------------------------------------------------- + +/// The tree on which bash 5.2 results were recorded: +/// `d/e/g d/f d/up -> .. top .hid .hd/x/y ld/z link -> d`. +fn bash_tree() -> TempDir { + let t = TempDir::new().unwrap(); + for d in ["d/e", ".hd/x", "ld"] { + fs::create_dir_all(t.path().join(d)).unwrap(); + } + for f in ["d/f", "d/e/g", "top", ".hid", ".hd/x/y", "ld/z"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("d", t.path().join("link")).unwrap(); + symlink("..", t.path().join("d/up")).unwrap(); + t +} + +#[test] +fn results_match_bash_on_the_recorded_tree() { + let t = bash_tree(); + let cases: &[(&str, &[&str])] = &[ + ( + "**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/", &["d/", "d/e/", "d/up/", "ld/", "link/"]), + ("d/**", &["d/", "d/e", "d/e/g", "d/f", "d/up"]), + ("d/**/", &["d/", "d/e/", "d/up/"]), + ( + "**/*", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/e", &["d/e"]), + ("*/", &["d/", "ld/", "link/"]), + (".*", &[".hd", ".hid"]), + (".h*", &[".hd", ".hid"]), + ("link/*", &["link/e", "link/f", "link/up"]), + ("**/g", &["d/e/g"]), + ("d/**/g", &["d/e/g"]), + ( + "**/**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("./*", &["./d", "./ld", "./link", "./top"]), + (".hd/**", &[".hd/", ".hd/x", ".hd/x/y"]), + ("nomatch*", &[]), + ]; + for (pattern, bash) in cases { + let ours: BTreeSet = strings(&expand_glob(pattern, t.path()).unwrap()) + .into_iter() + .collect(); + let bash: BTreeSet = bash.iter().map(|s| s.to_string()).collect(); + assert_eq!(ours, bash, "pattern {pattern}"); + } +} + +#[test] +fn hidden_files_need_a_leading_dot_in_the_pattern() { + let t = TempDir::new().unwrap(); + for f in [".hid1", ".hid2", "vis"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!( + strings(&expand_glob(".*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!( + strings(&expand_glob(".hid*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!(strings(&expand_glob("*", t.path()).unwrap()), ["vis"]); + assert_eq!( + strings(&expand_glob("?hid1", t.path()).unwrap()), + Vec::::new() + ); +} + +#[test] +fn a_name_that_is_not_utf8_is_matched_without_a_panic() { + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let odd = std::ffi::OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(odd), b"").is_err() { + return; // the file system refuses such names; nothing to test + } + fs::write(t.path().join("ok.txt"), b"").unwrap(); + let found = expand_glob("*.txt", t.path()).unwrap(); + assert_eq!(found.len(), 2); + assert!(found.contains(&PathBuf::from(odd))); +} diff --git a/impl/rust-cli/tests/security_tests.rs b/impl/rust-cli/tests/security_tests.rs index 5cee2c91..c9b1b59d 100644 --- a/impl/rust-cli/tests/security_tests.rs +++ b/impl/rust-cli/tests/security_tests.rs @@ -430,12 +430,29 @@ fn security_recursive_glob_bounded() { } // Total directories: 3^1 + 3^2 + ... + 3^5 = 363 - // Glob should have depth limit or timeout for larger structures - - // This would be a recursive glob like: **/* - // Should either limit depth or timeout gracefully - - // A production glob-depth limit remains tracked by #191. + let all = vsh::glob::expand_glob("**/*", temp.path()).unwrap(); + assert_eq!(all.len(), 363); + + // Under a smaller limit the same walk stops with a typed error that names + // the limit, instead of returning a truncated list (issue #191). + let limits = vsh::glob::GlobLimits { + max_matches: 100, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/*", temp.path(), &limits).unwrap_err(); + let stopped = err + .downcast_ref::() + .expect("a limit error"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert_eq!(stopped.exceeded.limit, 100); + + // Walk budget: a pattern that matches nothing is still bounded. + let limits = vsh::glob::GlobLimits { + max_reads: 50, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/none", temp.path(), &limits).unwrap_err(); + assert!(err.is::()); } // ============================================================ From e93624abf392a0969673c75ba57cba77bfcf2234 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:37:55 +0000 Subject: [PATCH 02/12] fix(confirmation): count files without following symlinks count_files_in_tree followed every symbolic link to a directory, and its only test walked the real /tmp. With a link back to a parent directory anywhere in /tmp the walk branches at every level and does not finish: the test ran for over 10 minutes on this machine, where it had passed before, because /tmp now held the glob reproduction trees. The glob fuzz target leaves its fixture (with two such links) in the temp directory on every run, because libFuzzer exits without running destructors, so any machine that ran the fuzzer would hang this test. Count links as entries and do not follow them, as `rm -r` does; the test uses its own fixture, and a second test covers a looped tree. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- impl/rust-cli/src/confirmation.rs | 32 +++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 57f6ec4f..3d1d5c07 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -209,6 +209,11 @@ fn prompt_yes_no(prompt: &str) -> Result { Ok(input == "y" || input == "yes") } +/// Count the files a recursive removal of `path` would delete. +/// +/// Symbolic links are counted as entries and never followed, as `rm -r` does: +/// following them would count files outside the tree and, with a link back to +/// a parent directory, never finish. fn count_files_in_tree(path: &str) -> Result { let mut count = 0; @@ -216,9 +221,8 @@ fn count_files_in_tree(path: &str) -> Result { if path.is_dir() { for entry in std::fs::read_dir(path)? { let entry = entry?; - let path = entry.path(); - if path.is_dir() { - visit_dirs(&path, count)?; + if entry.file_type()?.is_dir() { + visit_dirs(&entry.path(), count)?; } else { *count += 1; } @@ -338,6 +342,26 @@ mod tests { #[test] fn test_count_files() { - let _ = count_files_in_tree("/tmp").unwrap_or(0); + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::fs::write(t.path().join("top"), b"").unwrap(); + let root = t.path().to_str().unwrap(); + assert_eq!(count_files_in_tree(root).unwrap(), 2); + } + + /// A link back to a parent directory used to make the count recurse until + /// the path was too long, branching at every level. + #[cfg(unix)] + #[test] + fn test_count_files_does_not_follow_symlink_loops() { + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::os::unix::fs::symlink("..", t.path().join("d/up")).unwrap(); + std::os::unix::fs::symlink(".", t.path().join("self")).unwrap(); + let root = t.path().to_str().unwrap(); + // d/f, d/up and self: the two links are counted, not followed. + assert_eq!(count_files_in_tree(root).unwrap(), 3); } } From 4323498ce0e1b4fd144e07ac9ea135e2221d0a66 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:41:43 +0000 Subject: [PATCH 03/12] fix(fuzz): build the glob fixture without unwrap or expect The Hypatia code_safety scan on PR #242 flagged expect() (line 41) and six unwrap() calls (lines 44-55) in the fixture setup of fuzz_glob_expansion. Setup errors are now returned with `?`; if the fixture cannot be built, the target skips its inputs instead of panicking, so a temp-directory failure is never reported as a crash. Rebuilt under ASan and fuzzed for 60 s (31,108 runs) with -rss_limit_mb=2560: no crash. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 59 ++++++++++++------- 1 file changed, 37 insertions(+), 22 deletions(-) diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index 0c8c4882..6f778d58 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -34,28 +34,41 @@ const LIMITS: GlobLimits = GlobLimits { max_match_bytes: 256 * 1024, }; -fn fixture() -> &'static Path { - static FIXTURE: OnceLock = OnceLock::new(); +/// Build the fixture tree. Errors are returned, not unwrapped: if the temp +/// directory cannot be set up, the target skips its inputs instead of panicking. +fn build_fixture() -> std::io::Result { + let t = TempDir::new()?; + let root = t.path(); + for dir in ["d/sub", ".hd/x", "e"] { + fs::create_dir_all(root.join(dir))?; + } + for file in [ + "a.txt", + "b.txt", + "c.log", + ".hidden", + "d/f.rs", + "d/sub/g.rs", + ".hd/x/y", + "e/z", + ] { + fs::write(root.join(file), b"")?; + } + // Some file systems refuse names that are not UTF-8; the rest still works. + let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); + symlink("..", root.join("d/up"))?; // loop back to the root + symlink(".", root.join("self"))?; // loop to itself + symlink("d", root.join("link"))?; + symlink("missing", root.join("dangling"))?; + Ok(t) +} + +fn fixture() -> Option<&'static Path> { + static FIXTURE: OnceLock> = OnceLock::new(); FIXTURE - .get_or_init(|| { - let t = TempDir::new().expect("create fixture"); - let root = t.path(); - for dir in ["d/sub", ".hd/x", "e"] { - fs::create_dir_all(root.join(dir)).unwrap(); - } - for file in [ - "a.txt", "b.txt", "c.log", ".hidden", "d/f.rs", "d/sub/g.rs", ".hd/x/y", "e/z", - ] { - fs::write(root.join(file), b"").unwrap(); - } - let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); - symlink("..", root.join("d/up")).unwrap(); // loop back to the root - symlink(".", root.join("self")).unwrap(); // loop to itself - symlink("d", root.join("link")).unwrap(); - symlink("missing", root.join("dangling")).unwrap(); - t - }) - .path() + .get_or_init(|| build_fixture().ok()) + .as_ref() + .map(TempDir::path) } fuzz_target!(|data: &[u8]| { @@ -66,7 +79,9 @@ fuzz_target!(|data: &[u8]| { return; } - let base = fixture(); + let Some(base) = fixture() else { + return; + }; let pattern = if input.starts_with('/') { format!("{}{}", base.display(), input) } else { From 5038be1c088169b2a5d3c2afa8a147f727048202 Mon Sep 17 00:00:00 2001 From: "Jonathan D.A. Jewell" <6759885+hyperpolymath@users.noreply.github.com> Date: Wed, 18 Nov 2026 15:04:00 +0000 Subject: [PATCH 04/12] fix(word): expand globs for every simple command via typed words (#244) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stacked on hyperpolymath/valence-shell#242 (`claude/gifted-planck-mha9c9`), which adds `glob::expand_glob_with_limits` and `GlobLimitExceeded`. Once #242 merges, retarget this PR to `main`. ## Problem (measured on the debug binary, directory with `a.txt b.txt c.log`) | Command | Before | Now (same as bash) | |---|---|---| | `wc -c *.txt` | `wc: '*.txt': No such file or directory` | `0 a.txt` / `0 b.txt` / `0 total` | | `ls *.txt \| wc -l` | `ls: cannot access '*.txt'` | `2` | | `echo *.txt` | `*.txt` | `a.txt b.txt` | | `echo '*.txt'` | `\*.txt` | `*.txt` | | `echo a{1,2}` / `echo $((2+3))` | `a{1,2}` / `$((2+3))` | `a1 a2` / `5` | Cause: argument globbing lived only in `external::expand_glob_args`, which only `execute_external_with_redirects` called. There was also a second bug: quoted metacharacters were marked with a backslash in the token string, and nothing ever removed it. ## Change: typed word IR - New `src/word.rs`. A `Word` records the quoting of each piece (unquoted, `'…'`, `"…"`, `\x`). `expand_words` runs, in order: tilde, parameter, command and arithmetic expansion, process substitution, then brace expansion, glob expansion and quote removal. - The tokenizer builds a `Word` next to the old string form. `Command::External`, `Command::Pipeline` and `Command::Echo` now carry `Vec`. - The new stage is used for external commands (foreground, background, with or without redirections), every pipeline stage, `echo`, function-call arguments, and the `External` branch of command substitution and process substitution. - Kept from #242: - A pattern with no match, or an invalid pattern, stays as the literal word. - A `GlobLimitExceeded` fails the command without running it (`vsh: prog: …`, exit 1). - `external::expand_glob_args` is removed. Its rules and tests moved to `word.rs`. - Two smaller fixes: - `$(…)` is now read to its matching `)`, so `$((1+(2*3)))` is a single substitution. - `$(echo …)` is supported. ### Behaviour changes - An expansion error used to come back from `execute()` as `Err`. In `-c` mode that printed `Error:` plus a backtrace. Now the command exits with status 1 and does not run. - An unquoted `$X` whose value contains glob characters is now globbed, as POSIX requires. A quoted `"$X"` is not. ### Not migrated in this PR These still read the old string encoding, so their behaviour is unchanged and still differs from bash: - builtin operands (`touch '*q'` creates `\*q`; `ls *.txt` as a builtin does not glob) - redirection targets - assignment values (`X='*.txt'` stores `\*.txt`) - `case` patterns, `test`/`[[ ]]` arguments, `for` word lists Field splitting is still not done. All of this is documented in `docs/POSIX_COMPLIANCE.adoc`, Milestones 5 and 6. ## Tests - `tests/word_expansion_e2e_tests.rs` runs the `vsh` binary. It covers: - an external command without a redirect, with a redirect, and in a pipeline - builtin `echo` - quoted patterns `'…'`, `"…"` and `\*`, which stay literal with no backslash - a pattern that matches nothing - a partly quoted word (`'*'x.*`) - braces - function arguments - Unit tests in `word.rs`: the limit error, an invalid pattern, a literal backslash inside a pattern, quoted vs unquoted parameters, and refused process substitution. - A proptest in `parser.rs`: `Display` then `tokenize` gives back the same `Word` (256 cases). The domain excludes backticks inside double quotes, because the tokenizer has no `\`` escape. - Differential run of 30 commands across the old binary, the new binary and bash: every changed result now matches bash, and no builtin case changed. - `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`: clean. - `cargo test`: 887 passed, 0 failed, 14 ignored (the same ignored set as before: stress tests, the model oracle and 2 in the lib). --- _Generated by [Claude Code](https://claude.ai/code/session_01J4QvDfX8Xoq6vedAKdWBif)_ --------- Co-authored-by: Claude --- CHANGELOG.adoc | 29 + docs/POSIX_COMPLIANCE.adoc | 12 + impl/rust-cli/src/commands.rs | 2 +- impl/rust-cli/src/executable.rs | 121 ++-- impl/rust-cli/src/external.rs | 107 +-- impl/rust-cli/src/lib.rs | 1 + impl/rust-cli/src/parser.rs | 210 ++++-- impl/rust-cli/src/process_sub.rs | 14 +- impl/rust-cli/src/word.rs | 667 ++++++++++++++++++ .../tests/word_expansion_e2e_tests.rs | 126 ++++ 10 files changed, 1075 insertions(+), 214 deletions(-) create mode 100644 impl/rust-cli/src/word.rs create mode 100644 impl/rust-cli/tests/word_expansion_e2e_tests.rs diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index de152145..fb3d88be 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -11,6 +11,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 == [Unreleased] +=== Fixed — 2026-11-18 (arguments: globs, quote removal) + +- Glob expansion now applies to the arguments of every external command, + every pipeline stage, `echo` and function calls. Before, only an external + command that also had a redirection expanded its arguments: `wc -c *.txt` + passed `*.txt` to `wc`, `ls *.txt | wc -l` and `echo *.txt` did not expand. +- Quoted arguments no longer reach the program with a backslash in front of + `*`, `?`, `[` or `{`: `echo '*.txt'` printed `\*.txt` and now prints + `*.txt`. A partly quoted word globs only its unquoted part (`'*'x.*`). +- `echo` performs command substitution and arithmetic expansion + (`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works. +- `$((1+(2*3)))` and other substitutions containing `(` are read to their + matching `)`. +- An expansion error (such as a glob limit from #191) fails the command with + exit status 1 and does not run it. With a redirection the status was 127. + +=== Changed — 2026-11-18 (typed words) + +- New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces + (unquoted, single, double, backslash), and `word::expand_words` performs + tilde, parameter, command, arithmetic and process substitution, brace + expansion, glob expansion and quote removal on it. `Command::External`, + `Command::Pipeline` and `Command::Echo` now carry `Vec` arguments. + Other commands still use the older string encoding; see + `docs/POSIX_COMPLIANCE.adoc` (Milestones 5 and 6) for what is not migrated. +- Removed `external::expand_glob_args`; its rules (no match or invalid + pattern keeps the word, a limit is an error) moved to `word.rs` with their + tests. + === Fixed — 2026-11-18 (glob expansion out of memory, #191) - `**` no longer follows symbolic links to directories, so a link back to a diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index e8202f26..175e144c 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -177,6 +177,12 @@ or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported |Sorted results |Done |POSIX requirement + +|Where globs expand |Partial |Arguments of external commands (with or +without redirections), every pipeline stage, `+echo+` and function calls, +through `+word::expand_words+` (2026-11-18; before, only external commands +with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...), +`+for+` word lists, redirection targets |=== ==== Milestone 6: Quote Processing — COMPLETE @@ -194,6 +200,12 @@ quotes |Unclosed quote detection |Done | |Line continuation |Done | + +|Quote removal |Partial |Done for the arguments that go through +`+word::expand_words+` (see Milestone 5), which keeps the quoting of each +piece of a word. Builtin operands, redirection targets, assignment values, +`+case+` patterns and `+test+` arguments still receive quoted `+*?[{~+` with +a backslash in front (`+touch '*q'+` creates `+\*q+`) |=== ==== Milestone 7: Command Substitution — COMPLETE diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 4f91625d..02918fca 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -2856,7 +2856,7 @@ mod tests { let state = ShellState::new(tmp.path().to_str().unwrap()).unwrap(); let cmd = crate::parser::Command::External { program: "git".to_string(), - args: vec!["status".to_string()], + args: vec!["status".into()], redirects: vec![], background: false, }; diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index aeb72a25..7148cec7 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -488,35 +488,36 @@ impl ExecutableCommand for Command { // Expand program name (no process substitutions in program name) let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + // Word expansion: variables, substitutions, braces, globs and + // quote removal. A failed expansion (for example a glob that + // reaches a limit) fails the command without running it. + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + // Check if the command is a shell function before external execution. // `if let Some(...)` performs a single map lookup; the former // double-lookup plus expect shape could panic and duplicated // the lookup. if let Some(func_def) = state.functions.get(&expanded_program).cloned() { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); - return execute_function_call(&func_def, &expanded_args, state); + let result = execute_function_call(&func_def, &expansion.fields, state); + expansion.cleanup(); + return result; } - // Expand arguments with process substitutions - let mut all_process_subs = Vec::new(); - let mut expanded_args = Vec::new(); - - for arg in args { - let (expanded_arg, proc_subs) = - crate::parser::expand_with_process_sub(arg, state)?; - expanded_args.push(expanded_arg); - all_process_subs.extend(proc_subs); - } + let expanded_args = &expansion.fields; // Execute main command if *background { // Background execution let _job_id = external::execute_external_background( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -525,24 +526,19 @@ impl ExecutableCommand for Command { 0 // No job created }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::Success) } else { // Foreground execution let exit_code = if redirects.is_empty() { // No redirections - use simple path - external::execute_external(&expanded_program, &expanded_args) + external::execute_external(&expanded_program, expanded_args) } else { // Has redirections - use redirection-aware execution external::execute_external_with_redirects( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -552,12 +548,7 @@ impl ExecutableCommand for Command { 127 }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -569,33 +560,42 @@ impl ExecutableCommand for Command { redirects, background, } => { - // Expand variables and command substitutions in all pipeline stages - let expanded_stages: Result)>> = stages - .iter() - .map(|(program, args)| { - let expanded_program = - crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - Ok((expanded_program, expanded_args?)) - }) - .collect(); - let expanded_stages = expanded_stages?; - - if *background && !expanded_stages.is_empty() { + if *background && !stages.is_empty() { return Err(anyhow::anyhow!( "Background pipelines are not supported yet; refusing to run them in the foreground" )); } + // Expand every stage before any stage runs, so a failed + // expansion runs no part of the pipeline. + let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); + let mut process_subs = Vec::new(); + for (program, args) in stages { + let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => { + expanded_stages.push((expanded_program, expansion.fields)); + process_subs.extend(expansion.process_subs); + } + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + } + } + let exit_code = external::execute_pipeline(&expanded_stages, redirects, state) .unwrap_or_else(|e| { eprintln!("Pipeline error: {}", e); 127 }); + crate::word::Expansion { + fields: Vec::new(), + process_subs, + } + .cleanup(); + Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -684,10 +684,15 @@ impl ExecutableCommand for Command { interpret_escapes, redirects, } => { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: echo: {}", e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + let expanded_args = &expansion.fields; let output = if *interpret_escapes { expanded_args @@ -719,6 +724,7 @@ impl ExecutableCommand for Command { Ok(()) })?; } + expansion.cleanup(); Ok(ExecutionResult::Success) } @@ -1291,7 +1297,7 @@ impl ExecutableCommand for Command { if args.is_empty() { program.clone() } else { - format!("{} {}", program, args.join(" ")) + format!("{} {}", program, join_words(args)) } } Command::Pipeline { stages, .. } => { @@ -1301,7 +1307,7 @@ impl ExecutableCommand for Command { if args.is_empty() { prog.clone() } else { - format!("{} {}", prog, args.join(" ")) + format!("{} {}", prog, join_words(args)) } }) .collect(); @@ -1377,7 +1383,7 @@ impl ExecutableCommand for Command { format!("{} {} {}", left.description(), op_str, right.description()) } - Command::Echo { args, .. } => format!("echo {}", args.join(" ")), + Command::Echo { args, .. } => format!("echo {}", join_words(args)), Command::True => "true".to_string(), Command::False => "false".to_string(), Command::Read { var_names, .. } => format!("read {}", var_names.join(" ")), @@ -1645,6 +1651,15 @@ fn execute_loop( Ok(last_result) } +/// Words written back as source and separated by spaces. +fn join_words(words: &[crate::word::Word]) -> String { + words + .iter() + .map(|w| w.to_string()) + .collect::>() + .join(" ") +} + /// Match a word against a case pattern (supports glob-style wildcards) fn case_pattern_matches(word: &str, pattern: &str) -> bool { if pattern == "*" { diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 60f345a3..070e5f99 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -11,7 +11,6 @@ use std::path::PathBuf; use std::process::{Command, Stdio}; use std::time::Duration; -use crate::glob; use crate::redirection::{RedirectSetup, Redirection}; use crate::signals; use crate::state::ShellState; @@ -37,8 +36,7 @@ pub fn execute_external_with_redirects( let (stdin_cfg, stdout_cfg, stderr_cfg) = stdio_config_from_redirects(redirects, &redirect_setup, state)?; - // Expand glob patterns in arguments (Phase 6 M12) - let expanded_args = expand_glob_args(args, state)?; + // The arguments arrive already expanded, globs included (`crate::word`). // PATH lookup let executable = find_in_path(program)?; @@ -46,7 +44,7 @@ pub fn execute_external_with_redirects( // Build command with redirected stdio let mut command = Command::new(&executable); command - .args(&expanded_args) + .args(args) .stdin(stdin_cfg) .stdout(stdout_cfg) .stderr(stderr_cfg); @@ -475,80 +473,6 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { Ok(Stdio::from(reader)) } -/// Expand glob patterns in command arguments (Phase 6 M12) -/// -/// For each argument: -/// - If it contains glob metacharacters (*, ?, [, {), expand it -/// - If expansion finds matches, use them -/// - If expansion finds nothing, or the word is not a valid pattern, use the -/// word literally (POSIX behavior) -/// - If expansion stops at a limit, fail: the command is not run with a -/// literal or partial argument list in place of the real expansion -/// - If no metacharacters, use argument as-is -/// -/// # POSIX Behavior -/// - Brace expansion happens first: {a,b} -> [a, b] -/// - Then glob expansion: *.txt -> [file1.txt, file2.txt] -/// - Empty matches return literal: "*.xyz" if no .xyz files -/// -/// # Examples -/// ```ignore -/// expand_glob_args(&["echo", "*.txt"], state) // -> ["echo", "file1.txt", "file2.txt"] -/// expand_glob_args(&["ls", "file?.rs"], state) // -> ["ls", "file1.rs", "file2.rs"] -/// expand_glob_args(&["rm", "*.xyz"], state) // -> ["rm", "*.xyz"] (no .xyz files) -/// ``` -fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> { - // Get current working directory for glob expansion - let cwd = std::env::current_dir().context("Failed to get current working directory")?; - expand_glob_args_in(args, &cwd, &glob::GlobLimits::DEFAULT) -} - -/// [`expand_glob_args`] with an explicit directory and limits. -/// -/// # Errors -/// Returns the [`glob::GlobLimitExceeded`] error of the first argument whose -/// expansion reaches a limit. -fn expand_glob_args_in( - args: &[String], - cwd: &std::path::Path, - limits: &glob::GlobLimits, -) -> Result> { - let mut expanded: Vec = Vec::new(); - - for arg in args { - // Check if argument contains glob metacharacters - if glob::contains_glob_pattern(arg) { - // First, expand braces: file{1,2}.txt -> [file1.txt, file2.txt] - let brace_expanded = glob::expand_braces(arg); - - // Then expand each brace result for glob patterns - for pattern in &brace_expanded { - // Expand glob pattern relative to current directory - match glob::expand_glob_with_limits(pattern, cwd, limits) { - Ok(expansion) if !expansion.paths.is_empty() => { - expanded.extend( - expansion - .paths - .iter() - .map(|path| path.to_string_lossy().to_string()), - ); - } - // A limit is not "no match": substituting the literal word - // would run the command on the wrong arguments. - Err(e) if e.is::() => return Err(e), - // No matches or not a valid pattern - use literal (POSIX behavior) - _ => expanded.push(pattern.clone()), - } - } - } else { - // No glob pattern - use argument as-is - expanded.push(arg.clone()); - } - } - - Ok(expanded) -} - /// Handle command exit status, converting signals to exit codes fn handle_exit_status(status: std::process::ExitStatus) -> Result { #[cfg(unix)] @@ -827,31 +751,4 @@ mod tests { let exit_code = execute_pipeline(&stages, &[], &mut state).unwrap(); assert_eq!(exit_code, 0, "Single-stage pipeline should work"); } - #[test] - fn glob_args_use_the_literal_word_when_nothing_matches() { - let dir = tempfile::TempDir::new().unwrap(); - std::fs::write(dir.path().join("a.txt"), b"").unwrap(); - let args = vec!["*.txt".to_string(), "*.none".to_string(), "[z-".to_string()]; - let out = expand_glob_args_in(&args, dir.path(), &glob::GlobLimits::DEFAULT).unwrap(); - assert_eq!(out, ["a.txt", "*.none", "[z-"]); - } - - #[test] - fn glob_args_fail_instead_of_substituting_the_literal_at_a_limit() { - let dir = tempfile::TempDir::new().unwrap(); - for i in 0..5 { - std::fs::write(dir.path().join(format!("f{i}.txt")), b"").unwrap(); - } - let limits = glob::GlobLimits { - max_matches: 4, - ..glob::GlobLimits::DEFAULT - }; - let args = vec!["-l".to_string(), "*.txt".to_string()]; - let err = expand_glob_args_in(&args, dir.path(), &limits).unwrap_err(); - let stopped = err - .downcast_ref::() - .expect("the limit error must reach the caller"); - assert_eq!(stopped.exceeded.axis, "matches held"); - assert!(err.to_string().contains("over the limit of 4"), "{err}"); - } } diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index fd5fa34a..e74b4b1f 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -40,6 +40,7 @@ pub mod state; pub mod test_command; pub mod verification; +pub mod word; /// Echidna property-based verification integration (optional feature) #[cfg(feature = "echidna")] diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index da677ce3..1b51f2d0 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -9,6 +9,7 @@ use anyhow::{anyhow, Context, Result}; use crate::redirection::Redirection; +use crate::word::{Quote, Word}; /// Quote type for a word or word part #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -82,6 +83,8 @@ struct ParameterExpansion { struct QuotedWord { parts: Vec, quote_type: QuoteType, + /// The same word with the quoting of each piece kept (see `crate::word`). + typed: Word, } impl QuotedWord { @@ -89,6 +92,7 @@ impl QuotedWord { Self { parts: Vec::new(), quote_type: QuoteType::None, + typed: Word::new(), } } @@ -102,19 +106,23 @@ impl QuotedWord { } } - fn push_variable(&mut self, name: String) { + fn push_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), false, quote); self.parts.push(WordPart::Variable(name)); } - fn push_braced_variable(&mut self, name: String) { + fn push_braced_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), true, quote); self.parts.push(WordPart::BracedVariable(name)); } - fn push_command_sub(&mut self, cmd: String) { + fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.typed.push_command_sub(cmd.clone(), quote); self.parts.push(WordPart::CommandSub(cmd)); } fn push_process_sub(&mut self, sub_type: ProcessSubType, cmd: String) { + self.typed.push_process_sub(sub_type, cmd.clone()); self.parts.push(WordPart::ProcessSub(sub_type, cmd)); } } @@ -285,7 +293,8 @@ pub enum Command { // Shell builtins /// Echo: write arguments to stdout Echo { - args: Vec, + /// Arguments after the options, with their quoting. + args: Vec, no_newline: bool, interpret_escapes: bool, redirects: Vec, @@ -335,7 +344,8 @@ pub enum Command { // External command External { program: String, - args: Vec, + /// Arguments with their quoting; expanded by [`crate::word::expand_words`]. + args: Vec, redirects: Vec, background: bool, }, @@ -345,7 +355,7 @@ pub enum Command { /// Each stage is a (program, args) pair. Intermediate stages use piped stdio. /// Final redirections apply to the last stage only. Pipeline { - stages: Vec<(String, Vec)>, + stages: Vec<(String, Vec)>, redirects: Vec, background: bool, }, @@ -599,6 +609,7 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Backslash); quote_state = QuoteState::None; } @@ -609,6 +620,7 @@ fn tokenize(input: &str) -> Result> { quote_state = QuoteState::None; } else { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Single); } } @@ -631,27 +643,31 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(next_ch); + current_word.typed.push_char(next_ch, Quote::Double); } else { // Not a special char, keep backslash current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } else { current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } '$' => { // Variable expansion in double quotes push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::Double)?; } '`' => { // Backtick command substitution in double quotes push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::Double); } _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Double); } } } @@ -678,14 +694,14 @@ fn tokenize(input: &str) -> Result> { // Variable expansion '$' => { push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::None)?; } // Backtick command substitution '`' => { push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::None); } // Whitespace: end current word @@ -766,6 +782,7 @@ fn tokenize(input: &str) -> Result> { } else { // Invalid: 2>&[not 1] current_literal.push_str("2>&"); + current_word.typed.push_str("2>&", Quote::None); } } _ => { @@ -775,6 +792,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '2' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -816,6 +834,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '[' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -828,12 +847,14 @@ fn tokenize(input: &str) -> Result> { } else { // Regular ']' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } // Regular character _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } } @@ -861,12 +882,13 @@ fn tokenize(input: &str) -> Result> { fn parse_variable( chars: &mut std::iter::Peekable, word: &mut QuotedWord, + quote: Quote, ) -> Result<()> { if chars.peek() == Some(&'(') { // Command substitution: $(cmd) chars.next(); // consume '(' let cmd = parse_command_sub_dollar(chars)?; - word.push_command_sub(cmd); + word.push_command_sub(cmd, quote); } else if chars.peek() == Some(&'{') { // Braced form: ${VAR} chars.next(); // consume '{' @@ -888,7 +910,7 @@ fn parse_variable( } } - word.push_braced_variable(var_name); + word.push_braced_variable(var_name, quote); } else if let Some(&next_ch) = chars.peek() { // Simple form: $VAR or special variables. The peek above already // bound next_ch by value; chars.next() is called purely to advance @@ -897,11 +919,11 @@ fn parse_variable( if next_ch == '?' || next_ch == '$' || next_ch == '#' { // Single-character special variable chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_ascii_digit() { // Positional parameter: $0, $1, $2, etc. chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_alphabetic() || next_ch == '_' { // Variable name let mut var_name = String::new(); @@ -915,18 +937,20 @@ fn parse_variable( } } - word.push_variable(var_name); + word.push_variable(var_name, quote); } else if next_ch == '@' || next_ch == '*' { // Special positional parameters: $@ or $* chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else { // $ not followed by variable, treat as literal word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } } else { // $ at end of string word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } Ok(()) @@ -935,15 +959,14 @@ fn parse_variable( /// Parse command substitution in $(cmd) form fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> Result { let mut cmd = String::new(); - let mut depth = 1; // Track nesting depth for nested $() + let mut depth = 1; // Parenthesis depth: nested $(), $((expr)) and (...) for ch in chars.by_ref() { match ch { '(' => { - // Check if it's $( to track nested command substitution - if cmd.ends_with('$') { - depth += 1; - } + // Every `(` nests, so the `)` that closes `$((1+2))` or + // `$(echo $(pwd))` belongs to the substitution. + depth += 1; cmd.push(ch); } ')' => { @@ -1228,28 +1251,28 @@ fn parse_pipeline(tokens: &[Token]) -> Result { stages[last_stage_idx] = last_command_tokens; // Parse each stage as (program, args) - let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); + let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); for stage_tokens in stages { if stage_tokens.is_empty() { return Err(anyhow!("Empty pipeline stage")); } - let words: Vec = stage_tokens + let words: Vec<&QuotedWord> = stage_tokens .iter() .filter_map(|t| match t { - Token::Word(w) => Some(quoted_word_to_string(w)), + Token::Word(w) => Some(w), _ => None, }) .collect(); - if words.is_empty() { + let Some((program, args)) = words.split_first() else { return Err(anyhow!("Pipeline stage has no command")); - } - - let program = words[0].clone(); - let args = words[1..].to_vec(); - parsed_stages.push((program, args)); + }; + parsed_stages.push(( + quoted_word_to_string(program), + args.iter().map(|w| w.typed.clone()).collect(), + )); } Ok(Command::Pipeline { @@ -1719,6 +1742,7 @@ pub fn parse_command(input: &str) -> Result { // Separate command tokens from redirections let mut command_tokens: Vec = Vec::new(); + let mut command_words: Vec = Vec::new(); let mut redirects = Vec::new(); let mut background = false; @@ -1727,6 +1751,7 @@ pub fn parse_command(input: &str) -> Result { match &tokens[i] { Token::Word(w) => { command_tokens.push(quoted_word_to_string(w)); + command_words.push(w.typed.clone()); i += 1; } @@ -1852,9 +1877,10 @@ pub fn parse_command(input: &str) -> Result { // Parse command from tokens let cmd = &command_tokens[0]; let args: Vec = command_tokens[1..].to_vec(); + let words: Vec = command_words[1..].to_vec(); // Parse base command with redirections - parse_base_command(cmd, args, redirects, background) + parse_base_command(cmd, args, words, redirects, background) } /// Check if a string is a valid variable name @@ -2369,7 +2395,7 @@ fn apply_substring(value: &str, offset: i32, length: Option) -> String { /// /// `~user` is not yet supported (requires getpwnam). An escaped `\~` /// (produced by `quoted_word_to_string` for quoted `~`) stays literal. -fn expand_tilde(input: &str) -> String { +pub(crate) fn expand_tilde(input: &str) -> String { // An escaped tilde `\~` means the tilde was inside quotes — leave it. if let Some(stripped) = input.strip_prefix("\\~") { let mut s = String::with_capacity(input.len()); @@ -2628,14 +2654,12 @@ pub fn expand_command_substitution( process_cmd.stdout(Stdio::piped()); process_cmd.stderr(Stdio::null()); - // Expand variables and command substitutions in args before execution - let expanded_args: Result> = args - .iter() - .map(|arg| expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + // Expand the arguments (variables, command substitution, globs) + // before execution. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; - process_cmd.args(&expanded_args); + process_cmd.args(&expansion.fields); let output_result = process_cmd .output() @@ -2651,6 +2675,22 @@ pub fn expand_command_substitution( } } + Command::Echo { + args, + no_newline, + interpret_escapes: false, + redirects, + } if redirects.is_empty() => { + // echo builtin: its output is the joined, expanded arguments. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + let mut out = expansion.fields.join(" "); + if !no_newline { + out.push('\n'); + } + out + } + Command::Pwd { .. } => { // pwd builtin std::env::current_dir() @@ -3253,9 +3293,13 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { block_depth > 0 } +/// `args` are the arguments in the older string encoding, which the builtins +/// still read; `words` are the same arguments with their quoting, used by +/// `echo` and external commands. The two lists have the same length. fn parse_base_command( cmd: &str, args: Vec, + words: Vec, redirects: Vec, background: bool, ) -> Result { @@ -3382,7 +3426,8 @@ fn parse_base_command( // Re-parse the inner command: first arg is the command name, rest are its args let inner_cmd = &args[0]; let inner_args = args[1..].to_vec(); - let inner = parse_base_command(inner_cmd, inner_args, vec![], false)?; + let inner_words = words[1..].to_vec(); + let inner = parse_base_command(inner_cmd, inner_args, inner_words, vec![], false)?; Ok(Command::Explain { inner: Box::new(inner), }) @@ -3541,7 +3586,7 @@ fn parse_base_command( let mut echo_args = Vec::new(); let mut skip_flags = true; - for arg in &args { + for (arg, word) in args.iter().zip(&words) { if skip_flags && arg.starts_with('-') { for ch in arg[1..].chars() { match ch { @@ -3551,14 +3596,14 @@ fn parse_base_command( _ => { // Unknown flag — treat as regular argument skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); break; } } } } else { skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); } } @@ -3729,7 +3774,7 @@ fn parse_base_command( // Everything else: external command _ => Ok(Command::External { program: cmd.to_string(), - args, + args: words, redirects, background, }), @@ -4246,7 +4291,7 @@ mod tests { match cmd { Command::External { program, args, .. } => { assert_eq!(program, "touch"); - let expanded = expand_variables(&args[0], &state); + let expanded = expand_variables(&args[0].to_string(), &state); assert_eq!(expanded, "file_Alice"); } Command::Touch { path, .. } => { @@ -5313,3 +5358,78 @@ mod semicolon_tests { assert_eq!(parts, vec!["", ""]); } } + +#[cfg(test)] +mod typed_word_tests { + use super::*; + use proptest::prelude::*; + + fn typed_words(input: &str) -> Vec { + tokenize(input) + .unwrap() + .into_iter() + .filter_map(|t| match t { + Token::Word(w) => Some(w.typed), + _ => None, + }) + .collect() + } + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + #[test] + fn the_tokenizer_records_the_quoting_of_each_piece() { + assert_eq!( + typed_words("echo '*'x\\?\"a $V\"*"), + vec![Word::literal("echo"), { + let mut w = word(&[("*", Quote::Single), ("x", Quote::None)]); + w.push_char('?', Quote::Backslash); + w.push_str("a ", Quote::Double); + w.push_parameter("V".into(), false, Quote::Double); + w.push_char('*', Quote::None); + w + }] + ); + } + + #[test] + fn arithmetic_expansion_is_one_command_substitution_piece() { + let mut expected = Word::new(); + expected.push_command_sub("(1+(2*3))".into(), Quote::None); + expected.push_char('x', Quote::None); + assert_eq!(typed_words("$((1+(2*3)))x"), vec![expected]); + } + + /// One literal piece: (text, quoting). Characters are limited to those + /// the tokenizer reads literally in each quoting. + fn piece() -> impl Strategy { + prop_oneof![ + "[a-z0-9._/*?-]{1,4}".prop_map(|s| (s, Quote::None)), + "[a-z *?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Single)), + "[a-z '*?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Double)), + "[a-z*?$'\"\\\\ ]".prop_map(|s| (s, Quote::Backslash)), + ] + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// `Display` writes a word as source that tokenizes back to the + /// same word, so the quoting survives a round trip. + #[test] + fn display_then_tokenize_gives_the_same_word(pieces in prop::collection::vec(piece(), 1..6)) { + let mut w = Word::new(); + for (text, quote) in &pieces { + w.push_str(text, *quote); + } + let source = w.to_string(); + prop_assert_eq!(typed_words(&source), vec![w], "source: {}", source); + } + } +} diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index 7387952b..131eee30 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -157,11 +157,8 @@ fn start_command_with_output_redirect( let cmd_string = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + let expanded_args = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; // Build command string: prog arg1 arg2 > fifo let mut cmd = expanded_program; @@ -210,11 +207,8 @@ fn start_command_with_input_redirect( let cmd_string = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + let expanded_args = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; // Build command string let mut cmd = expanded_program; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs new file mode 100644 index 00000000..98fc8fc9 --- /dev/null +++ b/impl/rust-cli/src/word.rs @@ -0,0 +1,667 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Typed shell words and the word-expansion stage for simple-command arguments. +//! +//! The tokenizer used to flatten each word into a `String` in which quoted +//! glob metacharacters were marked with a backslash (`'*'` became `\*`). +//! Nothing removed those backslashes again, so `echo '*.txt'` printed +//! `\*.txt`, and glob expansion could not tell a quoted `*` from an unquoted +//! one without re-parsing that encoding. +//! +//! A [`Word`] keeps the quoting of every piece instead: each [`Segment`] +//! records whether it was written unquoted, in single or double quotes, or +//! after a backslash. [`expand_words`] then runs the POSIX word-expansion +//! steps on that structure, in this order: +//! +//! 1. tilde expansion (an unquoted `~`, `~/`, `~+`, `~-` at the start), +//! parameter expansion, command substitution, arithmetic expansion and +//! process substitution; +//! 2. brace expansion (`{a,b}`) of unquoted literal text; +//! 3. pathname (glob) expansion of the characters that are not quoted; +//! 4. quote removal. +//! +//! # Rules kept from the previous expansion code +//! +//! - A pattern that matches nothing, or is not a valid pattern, stays as the +//! literal word. +//! - A pattern whose expansion reaches a [`GlobLimits`] limit fails the whole +//! expansion with the [`GlobLimitExceeded`] error; the caller must not run +//! the command. +//! +//! # Known differences from POSIX (not changed here) +//! +//! - No field splitting: an unquoted `$X` whose value is `a b` stays one +//! argument. +//! - An empty word (`''`, or an unquoted expansion that is empty) is dropped +//! by the tokenizer or kept as an empty argument, as before. +//! - Brace expansion runs after parameter expansion (bash runs it before); +//! braces that come out of an expansion are not expanded. +//! +//! Only the arguments of external commands, pipeline stages, `echo` and +//! function calls go through this stage so far. Builtin operands, redirection +//! targets, assignments, `case` patterns and `test` arguments still use the +//! older string encoding. + +use std::fmt; +use std::path::Path; + +use anyhow::{Context, Result}; + +use crate::glob::{self, GlobLimitExceeded, GlobLimits}; +use crate::parser::ProcessSubType; +use crate::process_sub::ProcessSubstitution; +use crate::state::ShellState; + +/// How a piece of a word was written in the source. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Quote { + /// Not quoted: subject to tilde, brace and glob expansion. + None, + /// Inside `'...'`. + Single, + /// Inside `"..."`. + Double, + /// A single character after an unquoted `\`. + Backslash, +} + +impl Quote { + fn is_quoted(self) -> bool { + self != Quote::None + } +} + +/// One piece of a [`Word`]. +#[derive(Debug, Clone, PartialEq)] +pub enum Segment { + /// Literal text. + Literal { text: String, quote: Quote }, + /// `$NAME` (`braced: false`) or `${EXPR}` (`braced: true`). + Parameter { + expr: String, + braced: bool, + quote: Quote, + }, + /// `$(CMD)` or `` `CMD` ``; `$((EXPR))` is stored as the command `(EXPR)`. + CommandSub { cmd: String, quote: Quote }, + /// `<(CMD)` or `>(CMD)`. + ProcessSub { kind: ProcessSubType, cmd: String }, +} + +/// A shell word with the quoting of each of its pieces. +#[derive(Debug, Clone, PartialEq, Default)] +pub struct Word { + segments: Vec, +} + +impl Word { + /// An empty word. + pub fn new() -> Self { + Self::default() + } + + /// A word that is a single unquoted literal. + /// + /// # Examples + /// ``` + /// use vsh::word::Word; + /// assert_eq!(Word::literal("a.txt").to_string(), "a.txt"); + /// ``` + pub fn literal(text: &str) -> Self { + let mut word = Self::new(); + for ch in text.chars() { + word.push_char(ch, Quote::None); + } + word + } + + /// The pieces of the word, in order. + pub fn segments(&self) -> &[Segment] { + &self.segments + } + + /// True when the word has no pieces. + pub fn is_empty(&self) -> bool { + self.segments.is_empty() + } + + /// Append one literal character, merging it into the last piece when that + /// piece is a literal with the same quoting. + pub fn push_char(&mut self, ch: char, quote: Quote) { + if quote != Quote::Backslash { + if let Some(Segment::Literal { text, quote: last }) = self.segments.last_mut() { + if *last == quote { + text.push(ch); + return; + } + } + } + self.segments.push(Segment::Literal { + text: ch.to_string(), + quote, + }); + } + + /// Append literal text. + pub fn push_str(&mut self, text: &str, quote: Quote) { + for ch in text.chars() { + self.push_char(ch, quote); + } + } + + /// Append a parameter expansion. + pub fn push_parameter(&mut self, expr: String, braced: bool, quote: Quote) { + self.segments.push(Segment::Parameter { + expr, + braced, + quote, + }); + } + + /// Append a command substitution. + pub fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.segments.push(Segment::CommandSub { cmd, quote }); + } + + /// Append a process substitution. + pub fn push_process_sub(&mut self, kind: ProcessSubType, cmd: String) { + self.segments.push(Segment::ProcessSub { kind, cmd }); + } +} + +impl From<&str> for Word { + fn from(text: &str) -> Self { + Word::literal(text) + } +} + +/// Write one literal piece back as source with its quoting. +fn fmt_literal(f: &mut fmt::Formatter<'_>, text: &str, quote: Quote) -> fmt::Result { + match quote { + Quote::None => f.write_str(text), + Quote::Backslash => text.chars().try_for_each(|ch| write!(f, "\\{ch}")), + // `'` cannot appear inside single quotes. + Quote::Single => write!(f, "'{}'", text.replace('\'', "'\\''")), + Quote::Double => { + let mut escaped = String::with_capacity(text.len() + 2); + for ch in text.chars() { + if matches!(ch, '"' | '$' | '`' | '\\') { + escaped.push('\\'); + } + escaped.push(ch); + } + write!(f, "\"{escaped}\"") + } + } +} + +/// Writes the word back as shell source that parses to the same word. +impl fmt::Display for Word { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for segment in &self.segments { + match segment { + Segment::Literal { text, quote } => fmt_literal(f, text, *quote)?, + Segment::Parameter { + expr, + braced, + quote, + } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + if *braced { + write!(f, "{open}${{{expr}}}{open}")?; + } else { + write!(f, "{open}${expr}{open}")?; + } + } + Segment::CommandSub { cmd, quote } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + write!(f, "{open}$({cmd}){open}")?; + } + Segment::ProcessSub { kind, cmd } => match kind { + ProcessSubType::Input => write!(f, "<({cmd})")?, + ProcessSubType::Output => write!(f, ">({cmd})")?, + }, + } + } + Ok(()) + } +} + +/// Compares the source form ([`fmt::Display`]) with `other`. +impl PartialEq for Word { + fn eq(&self, other: &str) -> bool { + let source = self.to_string(); + source.as_str() == other + } +} + +impl PartialEq<&str> for Word { + fn eq(&self, other: &&str) -> bool { + *self == **other + } +} + +/// Whether process substitutions may be started while expanding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ProcessSubs { + /// Start them; the caller cleans them up after the command finishes. + Allow, + /// Fail on them, as the code paths without FIFO handling always have. + Refuse, +} + +/// The result of expanding a list of words. +pub struct Expansion { + /// The arguments, after quote removal. + pub fields: Vec, + /// Process substitutions started for these arguments. + pub process_subs: Vec, +} + +impl Expansion { + /// Wait for the process substitutions and remove their FIFOs. + pub fn cleanup(self) { + for proc_sub in self.process_subs { + if let Err(e) = proc_sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + } +} + +/// Expand words relative to the current directory under the default limits. +/// +/// # Errors +/// Fails on an expansion error (a failing command substitution, a refused +/// process substitution) or when a glob reaches a limit +/// ([`GlobLimitExceeded`], reachable with `anyhow::Error::downcast_ref`). +pub fn expand_words( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, +) -> Result { + let cwd = std::env::current_dir().context("Failed to get current working directory")?; + expand_words_in(words, state, process_subs, &cwd, &GlobLimits::DEFAULT) +} + +/// [`expand_words`] with an explicit directory for relative patterns and +/// explicit glob limits. +/// +/// # Errors +/// As for [`expand_words`]. +pub fn expand_words_in( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, + cwd: &Path, + limits: &GlobLimits, +) -> Result { + let mut expansion = Expansion { + fields: Vec::new(), + process_subs: Vec::new(), + }; + for word in words { + // On error the substitutions started so far are dropped, and Drop + // stops them and removes their FIFOs. + let chars = expand_segments(word, state, process_subs, &mut expansion.process_subs)?; + glob_fields(&chars, cwd, limits, &mut expansion.fields)?; + } + Ok(expansion) +} + +/// What later steps may do with a character produced by step 1. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Role { + /// Quoted text or a tilde result: taken literally. + Literal, + /// Unquoted source text: brace and glob expansion apply. + Source, + /// Result of an unquoted expansion: glob expansion applies, braces do not. + Expanded, +} + +/// Step 1: tilde, parameter, command, arithmetic and process substitution. +fn expand_segments( + word: &Word, + state: &mut ShellState, + process_subs: ProcessSubs, + started: &mut Vec, +) -> Result> { + let mut out: Vec<(char, Role)> = Vec::new(); + let push = |out: &mut Vec<(char, Role)>, text: &str, role: Role| { + out.extend(text.chars().map(|ch| (ch, role))); + }; + + for (index, segment) in word.segments.iter().enumerate() { + match segment { + Segment::Literal { text, quote } => { + if !quote.is_quoted() { + if index == 0 && text.starts_with('~') { + let (prefix, rest) = text.split_at(text.find('/').unwrap_or(text.len())); + let expanded = crate::parser::expand_tilde(prefix); + if expanded != prefix { + push(&mut out, &expanded, Role::Literal); + push(&mut out, rest, Role::Source); + continue; + } + } + push(&mut out, text, Role::Source); + } else { + push(&mut out, text, Role::Literal); + } + } + Segment::Parameter { + expr, + braced, + quote, + } => { + let source = if *braced { + format!("${{{expr}}}") + } else { + format!("${expr}") + }; + let value = crate::parser::expand_variables(&source, state); + push(&mut out, &value, expansion_role(*quote)); + } + Segment::CommandSub { cmd, quote } => { + let value = crate::parser::expand_with_command_sub(&format!("$({cmd})"), state)?; + push(&mut out, &value, expansion_role(*quote)); + } + Segment::ProcessSub { kind, cmd } => { + if process_subs == ProcessSubs::Refuse { + anyhow::bail!( + "Process substitution not yet implemented: execution requires FIFO support ({})", + Word { + segments: vec![segment.clone()] + } + ); + } + let mut proc_sub = ProcessSubstitution::create(*kind, cmd.clone(), state)?; + proc_sub.start(state)?; + push( + &mut out, + &proc_sub.fifo_path.to_string_lossy(), + Role::Literal, + ); + started.push(proc_sub); + } + } + } + Ok(out) +} + +fn expansion_role(quote: Quote) -> Role { + if quote.is_quoted() { + Role::Literal + } else { + Role::Expanded + } +} + +/// Encode step-1 output for [`glob::expand_braces`] and +/// [`glob::contains_glob_pattern`], which both read `\x` as a literal `x`. +fn encode(chars: &[(char, Role)]) -> String { + let mut out = String::with_capacity(chars.len() * 2); + for &(ch, role) in chars { + let literal = match role { + Role::Literal => true, + Role::Source => ch == '\\', + Role::Expanded => matches!(ch, '\\' | '{' | '}' | ','), + }; + if literal { + out.push('\\'); + } + out.push(ch); + } + out +} + +/// Steps 2 to 4: brace expansion, glob expansion and quote removal. +fn glob_fields( + chars: &[(char, Role)], + cwd: &Path, + limits: &GlobLimits, + fields: &mut Vec, +) -> Result<()> { + let encoded = encode(chars); + for candidate in glob::expand_braces(&encoded) { + match glob_pattern(&candidate) { + None => fields.push(remove_quotes(&candidate)), + Some(pattern) => match glob::expand_glob_with_limits(&pattern, cwd, limits) { + Ok(found) if !found.paths.is_empty() => fields.extend( + found + .paths + .iter() + .map(|path| path.to_string_lossy().into_owned()), + ), + // A limit is not "no match": substituting the literal word + // would run the command on the wrong arguments. + Err(e) if e.is::() => return Err(e), + // No match, or not a valid pattern: keep the word (POSIX). + _ => fields.push(remove_quotes(&candidate)), + }, + } + } + Ok(()) +} + +/// The pattern for the `glob` crate, or `None` when no `*`, `?` or `[` is +/// left unquoted. Quoted metacharacters are written as one-character classes +/// (`[*]`), the only escape that crate has. +fn glob_pattern(encoded: &str) -> Option { + let mut pattern = String::with_capacity(encoded.len()); + let mut active = false; + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + if let Some(next) = chars.next() { + pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + } + } else { + active |= matches!(ch, '*' | '?' | '['); + pattern.push(ch); + } + } + active.then_some(pattern) +} + +/// Quote removal: `\x` becomes `x`. +fn remove_quotes(encoded: &str) -> String { + let mut out = String::with_capacity(encoded.len()); + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + if let Some(next) = chars.next() { + out.push(next); + } + } else { + out.push(ch); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + fn fixture(names: &[&str]) -> tempfile::TempDir { + let dir = tempfile::TempDir::new().unwrap(); + for name in names { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir + } + + fn expand_in(dir: &Path, words: &[Word]) -> Vec { + let mut state = ShellState::new(dir.to_str().unwrap()).unwrap(); + expand_words_in( + words, + &mut state, + ProcessSubs::Refuse, + dir, + &GlobLimits::DEFAULT, + ) + .unwrap() + .fields + } + + #[test] + fn unquoted_patterns_expand_and_quoted_ones_do_not() { + let dir = fixture(&["a.txt", "b.txt", "c.log"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + word(&[("*.txt", Quote::Single)]), + word(&[("*.txt", Quote::Double)]), + word(&[("*", Quote::Backslash), (".txt", Quote::None)]), + ], + ); + assert_eq!(out, ["a.txt", "b.txt", "*.txt", "*.txt", "*.txt"]); + } + + #[test] + fn a_partly_quoted_pattern_matches_its_quoted_part_literally() { + let dir = fixture(&["*x.txt", "ax.txt", "a.txt"]); + // `'*'x*` : a literal star followed by `x` and a wildcard. + let out = expand_in( + dir.path(), + &[word(&[("*", Quote::Single), ("x*", Quote::None)])], + ); + assert_eq!(out, ["*x.txt"]); + } + + #[test] + fn no_match_and_invalid_patterns_stay_literal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + Word::literal("*.none"), + Word::literal("[z-"), + ], + ); + assert_eq!(out, ["a.txt", "*.none", "[z-"]); + } + + #[test] + fn braces_expand_only_when_unquoted() { + let dir = fixture(&[]); + let out = expand_in( + dir.path(), + &[Word::literal("a{1,2}"), word(&[("a{1,2}", Quote::Single)])], + ); + assert_eq!(out, ["a1", "a2", "a{1,2}"]); + } + + #[test] + fn literal_backslashes_survive_quote_removal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + word(&[("a\\*", Quote::Single)]), + word(&[("\\", Quote::Backslash)]), + ], + ); + assert_eq!(out, ["a\\*", "\\"]); + } + + #[test] + fn a_quoted_backslash_is_a_literal_character_in_a_pattern() { + let dir = fixture(&["a\\b.txt", "ab.txt"]); + // `'a\'*` : names that start with `a\`. + let out = expand_in( + dir.path(), + &[word(&[("a\\", Quote::Single), ("*", Quote::None)])], + ); + assert_eq!(out, ["a\\b.txt"]); + } + + #[test] + fn unquoted_parameter_values_are_globbed_and_quoted_ones_are_not() { + let dir = fixture(&["a.txt", "b.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("P", "*.txt"); + let mut unquoted = Word::new(); + unquoted.push_parameter("P".into(), false, Quote::None); + let mut quoted = Word::new(); + quoted.push_parameter("P".into(), true, Quote::Double); + let out = expand_words_in( + &[unquoted, quoted], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap() + .fields; + assert_eq!(out, ["a.txt", "b.txt", "*.txt"]); + } + + #[test] + fn a_glob_limit_fails_the_expansion_instead_of_substituting_the_literal() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let words = [Word::literal("-l"), Word::literal("*.txt")]; + let err = expand_words_in(&words, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the limit must fail the expansion"); + let stopped = err + .downcast_ref::() + .expect("the limit error must reach the caller"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert!(err.to_string().contains("over the limit of 4"), "{err}"); + } + + #[test] + fn refused_process_substitution_is_an_error() { + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let mut w = Word::new(); + w.push_process_sub(ProcessSubType::Input, "true".into()); + let err = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .err() + .expect("refused"); + assert!(err.to_string().contains("<(true)"), "{err}"); + } + + #[test] + fn display_round_trips_quoting() { + let w = word(&[ + ("a b", Quote::Single), + ("*", Quote::Backslash), + ("x\"$", Quote::Double), + ("y", Quote::None), + ]); + assert_eq!(w.to_string(), "'a b'\\*\"x\\\"\\$\"y"); + assert_eq!(Word::literal("plain"), "plain"); + } + + #[test] + fn glob_pattern_escapes_quoted_metacharacters() { + assert_eq!(glob_pattern("\\*a*"), Some("[*]a*".to_string())); + assert_eq!(glob_pattern("\\*a"), None); + assert_eq!(glob_pattern("\\[x?"), Some("[[]x?".to_string())); + } +} diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs new file mode 100644 index 00000000..bcfa282b --- /dev/null +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! End-to-end tests for argument glob expansion, run through the `vsh` binary. +//! +//! Before the word-expansion stage (`vsh::word`), globs were expanded only +//! for external commands that also had a redirection: `wc -c *.txt` passed +//! `*.txt` to `wc`, pipelines and `echo` never expanded, and quoted patterns +//! reached the program with a stray backslash (`echo '*.txt'` printed +//! `\*.txt`). Each test runs `vsh -c` in a directory holding `a.txt`, +//! `b.txt` and `c.log`. + +use std::path::Path; +use std::process::{Command, Output}; + +use tempfile::TempDir; + +fn fixture() -> TempDir { + let dir = TempDir::new().unwrap(); + for name in ["a.txt", "b.txt", "c.log"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir +} + +fn vsh(dir: &Path, script: &str) -> Output { + Command::new(env!("CARGO_BIN_EXE_vsh")) + .arg("-c") + .arg(script) + .current_dir(dir) + .output() + .expect("vsh runs") +} + +fn stdout(dir: &Path, script: &str) -> String { + let out = vsh(dir, script); + assert!( + out.status.success(), + "`{script}` failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8(out.stdout).unwrap() +} + +#[test] +fn external_command_without_redirect_gets_the_matches() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "wc -c *.txt"), + "0 a.txt\n0 b.txt\n0 total\n" + ); +} + +#[test] +fn external_command_with_redirect_gets_the_matches() { + let dir = fixture(); + stdout(dir.path(), "wc -c *.txt > out.wc"); + assert_eq!( + std::fs::read_to_string(dir.path().join("out.wc")).unwrap(), + "0 a.txt\n0 b.txt\n0 total\n" + ); +} + +#[test] +fn every_pipeline_stage_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "ls *.txt | wc -l").trim(), "2"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.log | cat"), "c.log\n"); +} + +#[test] +fn builtin_echo_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.txt"), "a.txt b.txt\n"); + assert_eq!(stdout(dir.path(), "echo -n *.log"), "c.log"); +} + +#[test] +fn quoted_patterns_stay_literal_without_a_backslash() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo '*.txt'"), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \"*.txt\""), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \\*.txt"), "*.txt\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' \"*.txt\" \\*.txt"), + "*.txt\n*.txt\n*.txt\n" + ); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' | cat"), + "*.txt\n" + ); +} + +#[test] +fn a_pattern_that_matches_nothing_is_passed_literally() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.none"), "*.none\n"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.none"), "*.none\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' *.none > out.none; cat out.none"), + "*.none\n" + ); +} + +#[test] +fn a_partly_quoted_word_globs_only_its_unquoted_part() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + assert_eq!(stdout(dir.path(), "echo '*'x.*"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "echo *x.*"), "*x.txt ax.txt\n"); +} + +#[test] +fn braces_expand_unquoted_and_stay_literal_quoted() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo a{1,2} '{a,b}'"), "a1 a2 {a,b}\n"); +} + +#[test] +fn function_arguments_get_the_matches() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "f() { echo \"$#:$1:$2\"; }; f *.txt"), + "2:a.txt:b.txt\n" + ); +} From b600ca33a7963d6011cd30c606086562a5e10ef1 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 15:28:41 +0000 Subject: [PATCH 05/12] fix(glob): one glob budget per command; address CodeRabbit review on #242 Verified findings from the CodeRabbit review of PR #242: - word.rs:430 (major): every brace alternative and every word started with a fresh glob budget. With up to 10,000 brace alternatives, one word could do 10,000 times the intended work, which defeats the bound #191 adds. glob::expand_glob_continuing starts the meters from the usage already spent; word expansion threads that usage through every word and alternative of a command, so the error still reports the configured limit. New test: five one-match patterns, as brace alternatives and as separate words, now exceed a four-match limit. - glob.rs:415: a match refused by the byte limit still raised the match peak by one. Meter::check tests a charge without recording it; both limits are checked before either is charged. The byte-limit test now asserts both peaks. - confirmation.rs:221: a symbolic link given as the root was followed. The root is now read with symlink_metadata and counted as one entry. - fuzz_glob_expansion.rs:69 (major): a fixture that could not be built made the target skip every input, a green run that tested nothing. It now stops with a clear panic message. - glob.rs:612: the differential test oracle now escapes the temporary directory path before using it as a glob prefix. - EXPLAINME.adoc: the bash compatibility claim names its one documented difference. cargo fmt --check, cargo clippy --all-targets --all-features -D warnings: clean. cargo test: 889 passed, 0 failed, 14 ignored. Fuzz target rebuilt under ASan; 23,410 runs in 60 s, no crash. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- EXPLAINME.adoc | 2 +- .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 22 ++++--- impl/rust-cli/src/confirmation.rs | 6 +- impl/rust-cli/src/glob.rs | 54 +++++++++++++--- impl/rust-cli/src/grade.rs | 41 ++++++++++-- impl/rust-cli/src/word.rs | 62 ++++++++++++++++--- impl/rust-cli/tests/glob_bounds_tests.rs | 2 + 7 files changed, 153 insertions(+), 36 deletions(-) diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 11279c02..df0c1b3c 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -212,7 +212,7 @@ All share: *Multiple proof systems for cross-validation, honest status reporting | `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine) -| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash globstar rules), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). +| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash 5.2 globstar rules, one documented difference for symlinked directories), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). | `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved. | `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT). diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index 6f778d58..ade8a883 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -34,8 +34,7 @@ const LIMITS: GlobLimits = GlobLimits { max_match_bytes: 256 * 1024, }; -/// Build the fixture tree. Errors are returned, not unwrapped: if the temp -/// directory cannot be set up, the target skips its inputs instead of panicking. +/// Build the fixture tree. fn build_fixture() -> std::io::Result { let t = TempDir::new()?; let root = t.path(); @@ -63,12 +62,17 @@ fn build_fixture() -> std::io::Result { Ok(t) } -fn fixture() -> Option<&'static Path> { - static FIXTURE: OnceLock> = OnceLock::new(); +/// The fixture, built once. A fixture that cannot be built stops the run +/// with a clear message: skipping every input would report a green run that +/// tested nothing. +fn fixture() -> &'static Path { + static FIXTURE: OnceLock = OnceLock::new(); FIXTURE - .get_or_init(|| build_fixture().ok()) - .as_ref() - .map(TempDir::path) + .get_or_init(|| match build_fixture() { + Ok(fixture) => fixture, + Err(e) => panic!("fuzz_glob_expansion: cannot build the fixture tree: {e}"), + }) + .path() } fuzz_target!(|data: &[u8]| { @@ -79,9 +83,7 @@ fuzz_target!(|data: &[u8]| { return; } - let Some(base) = fixture() else { - return; - }; + let base = fixture(); let pattern = if input.starts_with('/') { format!("{}{}", base.display(), input) } else { diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 3d1d5c07..5433c5db 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -218,7 +218,8 @@ fn count_files_in_tree(path: &str) -> Result { let mut count = 0; fn visit_dirs(path: &std::path::Path, count: &mut usize) -> Result<()> { - if path.is_dir() { + // symlink_metadata: a link given as the root is one entry, not its target. + if std::fs::symlink_metadata(path)?.is_dir() { for entry in std::fs::read_dir(path)? { let entry = entry?; if entry.file_type()?.is_dir() { @@ -363,5 +364,8 @@ mod tests { let root = t.path().to_str().unwrap(); // d/f, d/up and self: the two links are counted, not followed. assert_eq!(count_files_in_tree(root).unwrap(), 3); + // A link given as the root is one entry too. + let link = t.path().join("self"); + assert_eq!(count_files_in_tree(link.to_str().unwrap()).unwrap(), 1); } } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index ae5919f8..a0b63cf7 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -278,10 +278,28 @@ pub fn expand_glob_with_limits( pattern: &str, base_dir: &Path, limits: &GlobLimits, +) -> Result { + expand_glob_continuing(pattern, base_dir, limits, GlobUsage::default()) +} + +/// Expand a glob pattern under limits that it shares with earlier expansions. +/// +/// The meters start from `spent`, the usage an earlier expansion reported, so +/// several patterns (the brace alternatives of one word, or the words of one +/// command) together stay within one set of [`GlobLimits`]. The returned usage +/// includes `spent`, and the matches of earlier expansions count as still held. +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_continuing( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, + spent: GlobUsage, ) -> Result { let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; - let mut walker = Walker::new(pattern, base_dir, &compiled, limits); + let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); walker.run().map_err(anyhow::Error::new)?; Ok(Expansion { usage: walker.usage(), @@ -351,14 +369,24 @@ struct Walker<'a> { } impl<'a> Walker<'a> { - fn new(pattern: &'a str, base: &'a Path, compiled: &'a Compiled, limits: &GlobLimits) -> Self { + fn new( + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + limits: &GlobLimits, + spent: GlobUsage, + ) -> Self { Walker { pattern, base, compiled, - reads: Meter::new("file-system reads", limits.max_reads), - matches: Meter::new("matches held", limits.max_matches), - match_bytes: Meter::new("bytes of matches held", limits.max_match_bytes), + reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads), + matches: Meter::starting_at("matches held", limits.max_matches, spent.matches), + match_bytes: Meter::starting_at( + "bytes of matches held", + limits.max_match_bytes, + spent.match_bytes, + ), found: BTreeSet::new(), } } @@ -410,11 +438,16 @@ impl<'a> Walker<'a> { return Ok(()); } let bytes = path.as_os_str().len() as u64 + 1; + // Check both limits before recording either, so a refused match leaves + // no trace in the reported usage. + self.matches.check(1).map_err(|e| self.stopped(e, at))?; + self.match_bytes + .check(bytes) + .map_err(|e| self.stopped(e, at))?; self.matches.charge(1).map_err(|e| self.stopped(e, at))?; - if let Err(e) = self.match_bytes.charge(bytes) { - self.matches.release(1); - return Err(self.stopped(e, at)); - } + self.match_bytes + .charge(bytes) + .map_err(|e| self.stopped(e, at))?; self.found.insert(path); Ok(()) } @@ -609,7 +642,8 @@ mod walk_tests { case_sensitive: true, require_literal_separator: false, }; - glob::glob_with(&base.join(pattern).to_string_lossy(), options) + let base_pattern = glob::Pattern::escape(&base.to_string_lossy()); + glob::glob_with(&format!("{base_pattern}/{pattern}"), options) .unwrap() .filter_map(|p| p.ok()) .map(|p| p.strip_prefix(base).unwrap().to_path_buf()) diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs index 8f5143ff..ad56a78b 100644 --- a/impl/rust-cli/src/grade.rs +++ b/impl/rust-cli/src/grade.rs @@ -243,12 +243,21 @@ impl Meter { } } - /// Record the use of `amount` more units, or refuse if that passes the limit. + /// A meter for `axis` that starts from work already measured as `spent`, + /// so that several pieces of work can share one limit. + pub const fn starting_at(axis: &'static str, limit: u64, spent: G) -> Self { + Meter { + axis, + limit, + grade: spent, + } + } + + /// Whether [`Meter::charge`] would accept `amount`, without recording it. /// /// # Errors - /// Returns [`Exceeded`] and records nothing when the charge would take the - /// level over the limit. - pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + /// Returns the [`Exceeded`] that the charge would return. + pub fn check(&self, amount: u64) -> Result<(), Exceeded> { let next = self.grade.then(G::of(amount)); if next.level() > self.limit { return Err(Exceeded { @@ -257,7 +266,17 @@ impl Meter { level: next.level(), }); } - self.grade = next; + Ok(()) + } + + /// Record the use of `amount` more units, or refuse if that passes the limit. + /// + /// # Errors + /// Returns [`Exceeded`] and records nothing when the charge would take the + /// level over the limit. + pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + self.check(amount)?; + self.grade = self.grade.then(G::of(amount)); Ok(()) } @@ -370,6 +389,18 @@ mod tests { assert_eq!(meter.grade().peak(), 10); } + #[test] + fn a_meter_can_start_from_work_already_spent() { + let mut meter: Meter = Meter::starting_at("steps", 10, Cost::of(8)); + assert!(meter.check(2).is_ok()); + assert_eq!(meter.charge(3).unwrap_err().level, 11); + assert_eq!( + meter.level(), + 8, + "check and a refused charge record nothing" + ); + } + #[test] fn a_meter_with_a_zero_limit_refuses_any_use() { let mut meter: Meter = Meter::new("items held", 0); diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 98fc8fc9..6809c23d 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -47,7 +47,7 @@ use std::path::Path; use anyhow::{Context, Result}; -use crate::glob::{self, GlobLimitExceeded, GlobLimits}; +use crate::glob::{self, GlobLimitExceeded, GlobLimits, GlobUsage}; use crate::parser::ProcessSubType; use crate::process_sub::ProcessSubstitution; use crate::state::ShellState; @@ -300,11 +300,14 @@ pub fn expand_words_in( fields: Vec::new(), process_subs: Vec::new(), }; + // One glob budget for the whole command: every word and every brace + // alternative draws on the same limits. + let mut spent = GlobUsage::default(); for word in words { // On error the substitutions started so far are dropped, and Drop // stops them and removes their FIFOs. let chars = expand_segments(word, state, process_subs, &mut expansion.process_subs)?; - glob_fields(&chars, cwd, limits, &mut expansion.fields)?; + glob_fields(&chars, cwd, limits, &mut spent, &mut expansion.fields)?; } Ok(expansion) } @@ -421,19 +424,28 @@ fn glob_fields( chars: &[(char, Role)], cwd: &Path, limits: &GlobLimits, + spent: &mut GlobUsage, fields: &mut Vec, ) -> Result<()> { let encoded = encode(chars); for candidate in glob::expand_braces(&encoded) { match glob_pattern(&candidate) { None => fields.push(remove_quotes(&candidate)), - Some(pattern) => match glob::expand_glob_with_limits(&pattern, cwd, limits) { - Ok(found) if !found.paths.is_empty() => fields.extend( - found - .paths - .iter() - .map(|path| path.to_string_lossy().into_owned()), - ), + Some(pattern) => match glob::expand_glob_continuing(&pattern, cwd, limits, *spent) { + Ok(found) if !found.paths.is_empty() => { + *spent = found.usage; + fields.extend( + found + .paths + .iter() + .map(|path| path.to_string_lossy().into_owned()), + ); + } + // No match still spent reads. + Ok(found) => { + *spent = found.usage; + fields.push(remove_quotes(&candidate)); + } // A limit is not "no match": substituting the literal word // would run the command on the wrong arguments. Err(e) if e.is::() => return Err(e), @@ -628,6 +640,38 @@ mod tests { assert!(err.to_string().contains("over the limit of 4"), "{err}"); } + /// The brace alternatives of one word, and the words of one command, share + /// one glob budget: five patterns of one match each pass a limit of four. + #[test] + fn brace_alternatives_and_words_share_one_glob_budget() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let braces = [Word::literal("{f0,f1,f2,f3,f4}*")]; + let words: Vec = (0..5).map(|i| Word::literal(&format!("f{i}*"))).collect(); + for case in [&braces[..], &words[..]] { + let err = expand_words_in(case, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the shared budget must be exceeded"); + let stopped = err.downcast_ref::().unwrap(); + assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (4, 5)); + } + // Within the budget the same words expand normally. + let out = expand_words_in( + &words[..4], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &limits, + ) + .unwrap() + .fields; + assert_eq!(out, ["f0.txt", "f1.txt", "f2.txt", "f3.txt"]); + } + #[test] fn refused_process_substitution_is_an_error() { let dir = fixture(&[]); diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs index bd613cef..ead12100 100644 --- a/impl/rust-cli/tests/glob_bounds_tests.rs +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -181,6 +181,8 @@ fn the_byte_limit_counts_path_length_plus_one() { 2, "the refused match is not counted" ); + assert_eq!(s.usage.matches.peak(), 2, "nor does it raise the peak"); + assert_eq!(s.usage.match_bytes.peak(), 10); let limits = GlobLimits { max_match_bytes: 15, From 093e86aeeadda046ba0a3ec94bb0fc7312f989bc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 15:31:04 +0000 Subject: [PATCH 06/12] fix(fuzz): exit instead of panicking when the glob fixture cannot be built Hypatia's code_safety scan flagged the panic! added in b600ca3 (panic_macro, CWE-754). CodeRabbit had asked that a fixture failure fail the run rather than skip every input. Printing the error and calling std::process::exit(1) satisfies both: libFuzzer reports an exiting target as an error. Checked with TMPDIR pointing at a missing directory: the run prints "cannot build the fixture tree" and "libFuzzer: fuzz target exited" and exits non-zero. A normal 30 s run (10,012 inputs) finds nothing. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhGsL6zrtXcfEnvKbC --- impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index ade8a883..c467597e 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -70,7 +70,12 @@ fn fixture() -> &'static Path { FIXTURE .get_or_init(|| match build_fixture() { Ok(fixture) => fixture, - Err(e) => panic!("fuzz_glob_expansion: cannot build the fixture tree: {e}"), + Err(e) => { + // Exit rather than panic: libFuzzer reports an exiting target + // as an error ("fuzz target exited"), so the run still fails. + eprintln!("fuzz_glob_expansion: cannot build the fixture tree: {e}"); + std::process::exit(1); + } }) .path() } From d9669b2fad3ba8c9558ef30db39cb7f7985e04db Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 23:31:09 +0000 Subject: [PATCH 07/12] fix(word): seven faults from the review of #242, with regression tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - parser: the statement splitters walk bytes and looped for ever on an unquoted non-ASCII letter (`echo é` hung); only an ASCII letter can start a keyword. - glob: brace expansion used character counts as byte offsets and panicked on `'é'{a,b}`. - parser: ${...} parsing sliced at character counts; it now uses byte offsets (${é}, ${é:-x}). - parser: ${X:2:N} with a huge N overflowed; the end is now saturating. - parser: a quoted or escaped parenthesis no longer ends $(...). - word: a quoted hyphen inside [...] is written last in the class, where it cannot form a range; quoted *, ? and [ stay bare inside a class. - word: a tilde whose prefix has a quoted piece is not expanded. - word, parser: an unbraced parameter followed by a literal is written back with braces, so an AND-OR list reads $A''b as ${A}b, not $Ab. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- CHANGELOG.adoc | 17 +++ impl/rust-cli/src/glob.rs | 3 +- impl/rust-cli/src/parser.rs | 97 ++++++++------ impl/rust-cli/src/word.rs | 123 ++++++++++++++++-- .../tests/word_expansion_e2e_tests.rs | 76 +++++++++++ 5 files changed, 264 insertions(+), 52 deletions(-) diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index fb3d88be..ff987c16 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -27,6 +27,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An expansion error (such as a glob limit from #191) fails the command with exit status 1 and does not run it. With a redirection the status was 127. +=== Fixed — 2026-11-18 (review of #242) + +- An unquoted non-ASCII letter no longer hangs the shell: `echo é` did not + return, because the statement splitter did not advance past it. +- `'é'{a,b}` and `${é:-x}` no longer panic or misread the word: brace and + parameter expansion used character counts as byte offsets. +- `${X:2:N}` with a very large `N` gives the rest of the value; it panicked + on overflow. +- A quoted or escaped parenthesis no longer ends a command substitution: + `$(echo ')')`. +- A quoted hyphen in a bracket expression is a member, not a range: + `[a'-'z]*.txt` no longer matches `m.txt`. +- `~'local'/x` keeps its tilde: a quoted piece in the tilde prefix stops + the expansion. +- `$A''suffix && true` expands `A`: in an AND-OR list the word was read + back as `$Asuffix`. + === Changed — 2026-11-18 (typed words) - New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index a0b63cf7..b43dd92c 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -728,7 +728,8 @@ fn expand_braces_limited(pattern: &str, remaining: usize) -> Vec { } // Find first unescaped { - let chars = pattern.chars().enumerate().peekable(); + // Byte offsets, not character counts: the slices below index bytes. + let chars = pattern.char_indices(); let mut brace_start = None; let mut brace_depth = 0; let mut escaped = false; diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index 1b51f2d0..3082b834 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -960,9 +960,31 @@ fn parse_variable( fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> Result { let mut cmd = String::new(); let mut depth = 1; // Parenthesis depth: nested $(), $((expr)) and (...) + // A parenthesis that is quoted or escaped is text, not nesting. + let mut single = false; + let mut double = false; + let mut escaped = false; for ch in chars.by_ref() { + if escaped { + escaped = false; + cmd.push(ch); + continue; + } match ch { + '\\' if !single => { + escaped = true; + cmd.push(ch); + } + '\'' if !double => { + single = !single; + cmd.push(ch); + } + '"' if !single => { + double = !double; + cmd.push(ch); + } + _ if single || double => cmd.push(ch), '(' => { // Every `(` nests, so the `)` that closes `$((1+2))` or // `$(echo $(pwd))` belongs to the substitution. @@ -1556,7 +1578,7 @@ fn split_on_top_level(input: &str, split_on_newline: bool) -> Vec<&str> { } _ if !in_single_quote && !in_double_quote && paren_depth == 0 // Check for keyword at word boundary - && (ch.is_alphabetic() || ch == '_') => + && (ch.is_ascii_alphabetic() || ch == '_') => { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { @@ -2139,12 +2161,12 @@ fn parse_parameter_expansion(content: &str) -> Result = content.chars().collect(); - #[allow(unused_assignments)] - let mut pos = 0; + // Offsets are bytes. Every operator is one ASCII byte, so a `+ 1` step + // stays on a character boundary whatever the name contains. + let bytes = content.as_bytes(); // Check for length operator: ${#VAR} - if chars[0] == '#' { + if bytes[0] == b'#' { let var_name = content[1..].to_string(); return Ok(ParameterExpansion { var_name, @@ -2153,28 +2175,27 @@ fn parse_parameter_expansion(content: &str) -> Result= chars.len() { + if pos >= content.len() { // ${VAR:} with nothing after - try substring with offset 0 return Ok(ParameterExpansion { var_name, @@ -2185,8 +2206,10 @@ fn parse_parameter_expansion(content: &str) -> Result) -> String { match length { Some(n) => { - let end = (start + n).min(chars.len()); + let end = start.saturating_add(n).min(chars.len()); chars[start..end].iter().collect() } None => chars[start..].iter().collect(), @@ -2574,31 +2597,13 @@ fn quoted_word_to_string(word: &QuotedWord) -> String { result.push_str(s); } } - WordPart::Variable(name) => { - // This shouldn't happen in single quotes (tokenizer prevents it) - // but handle it anyway - if word.quote_type != QuoteType::Single { - result.push('$'); - result.push_str(name); - } else { - // In single quotes, escape the $ - result.push('\\'); - result.push('$'); - result.push_str(name); - } - } - WordPart::BracedVariable(name) => { - // This shouldn't happen in single quotes - if word.quote_type != QuoteType::Single { - result.push_str("${"); - result.push_str(name); - result.push('}'); - } else { - // In single quotes, escape the $ - result.push_str("\\${"); - result.push_str(name); - result.push('}'); - } + // The tokenizer makes a variable part only outside single quotes, + // so it is always an expansion, whatever quotes follow in the word. + // Braces keep a following literal out of the name (`$A''b`). + WordPart::Variable(name) | WordPart::BracedVariable(name) => { + result.push_str("${"); + result.push_str(name); + result.push('}'); } WordPart::CommandSub(cmd) => { // Command substitution - keep as-is for now @@ -2856,7 +2861,10 @@ fn split_control_keywords<'a>(input: &'a str, keywords: &[&str]) -> Vec<(&'a str in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; @@ -3260,7 +3268,10 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 6809c23d..08f7573f 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -198,7 +198,7 @@ fn fmt_literal(f: &mut fmt::Formatter<'_>, text: &str, quote: Quote) -> fmt::Res /// Writes the word back as shell source that parses to the same word. impl fmt::Display for Word { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - for segment in &self.segments { + for (index, segment) in self.segments.iter().enumerate() { match segment { Segment::Literal { text, quote } => fmt_literal(f, text, *quote)?, Segment::Parameter { @@ -207,7 +207,13 @@ impl fmt::Display for Word { quote, } => { let open = if quote.is_quoted() { "\"" } else { "" }; - if *braced { + // `$A''b` must not come back as `$Ab`: brace the name when + // the text after it could continue it. + let runs_on = self.segments.get(index + 1).is_some_and(|next| { + matches!(next, Segment::Literal { text, quote: Quote::None } + if text.starts_with(|c: char| c.is_alphanumeric() || c == '_')) + }) && expr.starts_with(|c: char| c.is_alphabetic() || c == '_'); + if *braced || runs_on { write!(f, "{open}${{{expr}}}{open}")?; } else { write!(f, "{open}${expr}{open}")?; @@ -339,7 +345,14 @@ fn expand_segments( match segment { Segment::Literal { text, quote } => { if !quote.is_quoted() { - if index == 0 && text.starts_with('~') { + // The tilde prefix runs to the first unquoted `/`. It ends in this + // segment only if the segment has a `/` or is the whole word; + // otherwise a quoted piece or an expansion is part of the + // prefix, and the tilde stays literal (`~'local'/x`). + if index == 0 + && text.starts_with('~') + && (text.contains('/') || word.segments.len() == 1) + { let (prefix, rest) = text.split_at(text.find('/').unwrap_or(text.len())); let expanded = crate::parser::expand_tilde(prefix); if expanded != prefix { @@ -463,16 +476,46 @@ fn glob_fields( fn glob_pattern(encoded: &str) -> Option { let mut pattern = String::with_capacity(encoded.len()); let mut active = false; + // Inside an unquoted `[...]`: how many members so far, and whether a + // quoted `-` waits to be written last, where it cannot form a range. + let mut class: Option = None; + let mut hyphen = false; let mut chars = encoded.chars(); while let Some(ch) = chars.next() { if ch == '\\' { - if let Some(next) = chars.next() { - pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + let Some(next) = chars.next() else { continue }; + match (class.as_mut(), next) { + (Some(_), '-') => hyphen = true, + // `*`, `?` and `[` are already literal inside a class. + (Some(members), '*' | '?' | '[') => { + *members += 1; + pattern.push(next); + } + (members, _) => { + if let Some(members) = members { + *members += 1; + } + pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + } } - } else { - active |= matches!(ch, '*' | '?' | '['); - pattern.push(ch); + continue; } + active |= matches!(ch, '*' | '?' | '['); + match class.as_mut() { + None if ch == '[' => class = Some(0), + None => {} + // `]` first in a class (after `!` or `^`) is a member, not the end. + Some(members) if ch == ']' && *members > 0 => { + if std::mem::take(&mut hyphen) { + pattern.push('-'); + } + class = None; + } + Some(members) if *members == 0 && matches!(ch, '!' | '^') && pattern.ends_with('[') => { + } + Some(members) => *members += 1, + } + pattern.push(ch); } active.then_some(pattern) } @@ -702,6 +745,70 @@ mod tests { assert_eq!(Word::literal("plain"), "plain"); } + #[test] + fn display_keeps_a_literal_out_of_an_unbraced_name() { + let mut w = Word::new(); + w.push_parameter("A".to_string(), false, Quote::None); + w.push_str("suffix", Quote::None); + assert_eq!(w.to_string(), "${A}suffix"); + + std::env::remove_var("Asuffix"); + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("A", "x"); + let out = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap(); + assert_eq!(out.fields, ["xsuffix"]); + } + + #[test] + fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + // [a'-'z]*.txt + assert_eq!(glob_pattern("[a\\-z]*.txt"), Some("[az-]*.txt".to_string())); + let dir = fixture(&["-.txt", "a.txt", "m.txt", "z.txt"]); + let w = word(&[ + ("[a", Quote::None), + ("-", Quote::Single), + ("z]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt", "a.txt", "z.txt"]); + } + + #[test] + fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(&[]); + let quoted = word(&[ + ("~", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[quoted]), ["~local/x"]); + // An unquoted slash ends the prefix, so this one still expands. + let slash = word(&[ + ("~/", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + let got = expand_in(dir.path(), &[slash]); + assert!( + got[0].ends_with("/local/x") && !got[0].starts_with('~'), + "{got:?}" + ); + } + + #[test] + fn a_non_ascii_character_before_a_brace_does_not_panic() { + let dir = fixture(&[]); + let w = word(&[("é", Quote::Single), ("{a,b}", Quote::None)]); + assert_eq!(expand_in(dir.path(), &[w]), ["éa", "éb"]); + } + #[test] fn glob_pattern_escapes_quoted_metacharacters() { assert_eq!(glob_pattern("\\*a*"), Some("[*]a*".to_string())); diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs index bcfa282b..86deade1 100644 --- a/impl/rust-cli/tests/word_expansion_e2e_tests.rs +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -124,3 +124,79 @@ fn function_arguments_get_the_matches() { "2:a.txt:b.txt\n" ); } + +// Regressions from the review of #242. Each of these once panicked, hung or +// gave the wrong arguments. + +#[test] +fn non_ascii_text_does_not_hang_or_panic() { + let dir = fixture(); + // The statement splitter looped for ever on an unquoted non-ASCII letter. + assert_eq!(stdout(dir.path(), "echo é"), "é\n"); + assert_eq!(stdout(dir.path(), "if true; then echo ünï; fi"), "ünï\n"); + // Brace expansion sliced the word at a character count. + assert_eq!(stdout(dir.path(), "echo 'é'{a,b}"), "éa éb\n"); + // So did `${...}`. + assert_eq!( + stdout(dir.path(), "é=1; echo ${é} ${é:-f} ${u:-é} ${#é}"), + "1 1 é 1\n" + ); +} + +#[test] +fn a_quoted_parenthesis_does_not_end_a_command_substitution() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo $(echo ')')"), ")\n"); + assert_eq!( + stdout(dir.path(), "echo $(echo \"(\") $(echo \\))"), + "( )\n" + ); + assert_eq!( + stdout(dir.path(), "echo $(echo $(echo a)) $((1+(2*3)))"), + "a 7\n" + ); +} + +#[test] +fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + let dir = fixture(); + for name in ["-.txt", "m.txt", "z.txt"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + assert_eq!( + stdout(dir.path(), "echo [a'-'z]*.txt"), + "-.txt a.txt z.txt\n" + ); + assert_eq!( + stdout(dir.path(), "echo [a-z]*.txt"), + "a.txt b.txt m.txt z.txt\n" + ); +} + +#[test] +fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo ~'local'/x"), "~local/x\n"); +} + +#[test] +fn a_parameter_before_a_quoted_literal_keeps_its_name_in_a_list() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "A=x; echo $A''suffix '$A' \"$A\"b && true"), + "xsuffix $A xb\n" + ); + assert_eq!(stdout(dir.path(), "A=x; echo $A''suffix"), "xsuffix\n"); +} + +#[test] +fn a_substring_length_past_the_end_gives_the_rest() { + let dir = fixture(); + assert_eq!( + stdout( + dir.path(), + "X=abcdef; echo ${X:2:18446744073709551615} ${X:1:2}" + ), + "cdef bc\n" + ); +} From ad8f1d399a8664ff6ad08069cce06fa7a2aa96f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 00:12:13 +0000 Subject: [PATCH 08/12] fix(glob,word): the small findings from the review of #242 Code - parser: an AND-OR list writes its operands back from the typed word, so each piece keeps its quoting (`echo '*'x.* && true` lost the glob). - word, external: expanded fields are OsString; a glob match reaches an external command with the exact bytes of its file name. - glob: a name that is not UTF-8 is matched through a private-use mapping, not a lossy copy, so a literal U+FFFD in a pattern no longer matches it. - glob: resolving a symbolic link in is_dir is a charged read. - glob: new limit max_listing_bytes for the names of one directory listing. - glob: the walker's stop error is boxed; glob::escape is new. - grade: Meter::check computes the attempted level in u128 before the grades are composed; Exceeded::level is u128. - confirmation: the no-follow count is documented as holding for a tree that does not change. Fuzz - fuzz_glob_expansion removes its fixture at exit (libc::atexit), escapes the fixture root in absolute patterns and checks that /**/* reaches it. Tests - charged link lookups, the listing limit, exact name matching, the meter overflow, list operands, exact argument bytes, a glob in a later pipeline stage; wc output is compared without column padding. Documents - CHANGELOG: walker read units defined; the exit-status claim is limited to argument expansion. - POSIX_COMPLIANCE: Milestones 5, 6 and 12 are Partial in the overview and the detail; `for` word lists do not expand globs. - PROOF-OPEN-FRONTIER A-6 and the EXPLAINME proof ledger: fixture evidence is tested-only, the bound is open; Lean is model evidence only, the Rust walker needs Creusot or Verus. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- CHANGELOG.adoc | 38 +++++- EXPLAINME.adoc | 12 ++ docs/POSIX_COMPLIANCE.adoc | 30 +++-- docs/PROOF-OPEN-FRONTIER.adoc | 4 +- impl/rust-cli/fuzz/Cargo.lock | 1 + impl/rust-cli/fuzz/Cargo.toml | 1 + .../fuzz/fuzz_targets/fuzz_glob_expansion.rs | 50 +++++++- impl/rust-cli/src/confirmation.rs | 6 + impl/rust-cli/src/executable.rs | 6 +- impl/rust-cli/src/external.rs | 32 +++-- impl/rust-cli/src/glob.rs | 115 ++++++++++++++---- impl/rust-cli/src/grade.rs | 46 +++++-- impl/rust-cli/src/parser.rs | 6 +- impl/rust-cli/src/process_sub.rs | 6 +- impl/rust-cli/src/word.rs | 41 ++++--- impl/rust-cli/tests/glob_bounds_tests.rs | 75 ++++++++++++ .../tests/word_expansion_e2e_tests.rs | 49 +++++++- 17 files changed, 422 insertions(+), 96 deletions(-) diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index ff987c16..4898dcd0 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -24,8 +24,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works. - `$((1+(2*3)))` and other substitutions containing `(` are read to their matching `)`. -- An expansion error (such as a glob limit from #191) fails the command with - exit status 1 and does not run it. With a redirection the status was 127. +- An expansion error in an argument (such as a glob limit from #191) fails + the command with exit status 1 and does not run it. With a redirection the + status was 127. An error while the program name or a pipeline stage name + is expanded is not changed: it is returned to the caller, so `||` sees it. + +=== Fixed — 2026-11-19 (review of #242, small findings) + +- An AND-OR list keeps the quoting of each piece of a word: `echo '*'x.* && true` + printed `*x.*` and now expands the unquoted part. +- A glob match reaches an external command with the exact bytes of its file + name. A name that is not UTF-8 was replaced by a lossy copy that did not + exist. +- A name that is not UTF-8 is no longer matched by a literal U+FFFD in the + pattern. `*` and `?` still match it. +- Resolving a symbolic link to learn whether it is a directory is charged as + a read unit; it was free. +- New limit `max_listing_bytes` (64 MiB): the names held while one directory + is listed. Exceeding it is a `GlobLimitExceeded` that names the directory. +- A `Meter` charge that would pass `u64::MAX` is refused and reported + exactly; `Exceeded::level` is now `u128`. Before, the level saturated and + a limit of `u64::MAX` accepted the charge. +- `Expansion::fields` in `src/word.rs` is `Vec`; the external + command functions take `OsString` arguments. `glob::escape` is new. +- The `fuzz_glob_expansion` target removes its fixture at exit, escapes the + fixture root in absolute patterns, and checks that `/**/*` reaches the + fixture. +- Tests: `wc` output is compared without its column padding; a glob in a + later pipeline stage is covered. === Fixed — 2026-11-18 (review of #242) @@ -66,8 +92,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run 134118693669). The rule now matches bash 5.2 with `globstar` for patterns that start with `**`. -- Glob expansion is metered while it runs: at most 10,000,000 file-system - reads, 100,000 matches and 6 MiB of matches per pattern. Reaching a limit is +- Glob expansion is metered while it runs: at most 10,000,000 walker read + units, 100,000 matches and 6 MiB of matches per command. A read unit is + charged for each directory opened, each entry listed, each name looked up + and each symbolic link resolved. The type of an entry comes with the + listing and is not charged again. The limit is on these units; it is not + a cap on every file-system operation the shell makes. Reaching a limit is a `GlobLimitExceeded` error that names the limit, the usage so far and the directory being read; the command is not run. Before, the match limit was checked only after every path had been collected, and an external command diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index df0c1b3c..a6b4063f 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -126,6 +126,18 @@ reproduce. Absence of a tool is an open obligation, not a passing leg | `proofs/z3/` (`archive/proofs/` for Isabelle and Mizar) | §4 exception notes are now recorded in each `.smt2` header. `filesystem_operations` and `symlink_operations` were rewritten on 2026-11-09 to define the operations as array `store`s and negate each goal; the other three still rest on asserted axioms. Isabelle and Mizar were never CI-checked and are archived, unchecked, with a successor map +| Glob expansion stays within its read bound on the test fixtures (frontier A-6) +| tested-only +| `cargo test --test glob_bounds_tests`; `proptest` in `src/glob.rs` and `src/grade.rs` +| `impl/rust-cli/tests/glob_bounds_tests.rs`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| Fixed fixtures only: the reported read count equals a count made from the fixture, and a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. The grade laws are property-tested, not proved + +| Glob expansion terminates within `O(|pattern| x |entries|)` reads (frontier A-6) +| open +| - +| `impl/rust-cli/src/glob.rs` `expand_glob`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| No Creusot or Verus contract on the walker (PROOF-POLICY §1), and no Lean model bound. A Lean bound would be model evidence only + | Sandbox path normalisation never escapes the root | proven (model) / tested-only (code) | Lean 4 for the model; `proptest` for the code diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index 175e144c..0d382ebc 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -22,14 +22,14 @@ M12-14 not started |2 |Redirections |*Complete* |All 8 types including `+2>&1+` |3 |Pipelines |*Complete* |Multi-stage, exit code from last |4 |Variables |*Complete* |Scalars, arrays, parameter expansion -|5 |Glob Expansion |*Complete* |POSIX-compliant + brace expansion -|6 |Quote Processing |*Complete* |Single, double, backslash +|5 |Glob Expansion |*Partial* |Matching complete; not yet applied to builtin operands, `+for+` word lists, redirection targets +|6 |Quote Processing |*Partial* |Quote removal complete only for arguments that go through `+word::expand_words+` |7 |Command Substitution |*Complete* |`+$(cmd)+` and backticks |8 |Arithmetic |*Complete* |Full operator set including `+**+` |9 |Control Structures |*Complete* |if/elif/else, while, for, case/esac |10 |Job Control |*Partial* |No SIGCHLD, no Ctrl+Z |11 |Shell Builtins |*Partial* |~30 implemented, ~15 missing -|12 |Functions |*Not started* |No `+func() { ... }+` syntax +|12 |Functions |*Partial* |`+name() { ... }+` definitions and calls work; nesting incomplete |13 |Shell Scripts |*Not started* |No `+.sh+` file execution |14 |Advanced Features |*Partial* |Here docs, process sub, arrays done |=== @@ -147,7 +147,7 @@ quoted context) - `+${VAR:=default}+` does not actually assign — same as `+${VAR:-default}+` - `+export VAR+` on unset var errors instead of creating empty exported var -==== Milestone 5: Glob Expansion — COMPLETE +==== Milestone 5: Glob Expansion — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== @@ -170,8 +170,9 @@ links or into hidden directories. One known difference from bash 5.2: bash enters a symlinked directory for one more component when `+**+` is not the first component (2026-11-18) -|Expansion limits |Done |10,000,000 file-system reads, 100,000 matches, -6 MiB of matches; exceeding one is an error that names it, never a truncated +|Expansion limits |Done |10,000,000 walker read units (directories opened, +entries listed, names looked up, links resolved), 100,000 matches, 6 MiB of +matches, 64 MiB of names for one directory listing; exceeding one is an error that names it, never a truncated or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported @@ -185,7 +186,7 @@ with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...), `+for+` word lists, redirection targets |=== -==== Milestone 6: Quote Processing — COMPLETE +==== Milestone 6: Quote Processing — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== @@ -254,8 +255,9 @@ condition chains |`+while+`/`+do+`/`+done+` |Done |`+Command::While+` with break/continue -|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::For+` with glob -expansion in wordlist +|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::ForLoop+`; the word list gets +parameter expansion and `+$IFS+` splitting, but not glob expansion (see +Milestone 5) |`+case+`/`+esac+` |Done |`+Command::Case+` with glob pattern matching @@ -315,9 +317,13 @@ ops, numeric comparison `+hash+`, `+umask+`, `+wait+`, `+getopts+`, `+shift+`, `+break+`, `+continue+`, `+return+`, `+local+` -==== Milestone 12: Functions — NOT STARTED +==== Milestone 12: Functions — PARTIAL -No function definition or calling syntax exists in the parser. +`+name() { ... }+` and `+function name { ... }+` define a function, and a call +runs it with its own positional parameters, `+local+` and `+return+`. An +end-to-end test runs `+f() { echo "$#:$1:$2"; }; f *.txt+` through the built +shell (`+tests/word_expansion_e2e_tests.rs+`). Full nesting of functions is +not complete. ==== Milestone 13: Shell Scripts — NOT STARTED @@ -378,7 +384,7 @@ composition |0 === Critical Missing Features (by impact) [arabic] -. *Functions* — No `+func() { ... }+` — blocks modular scripts +. *Functions* — definitions and calls work; nesting is incomplete . *Shell script execution* — No `+.sh+` file running . *Word splitting (`+$IFS+`)* — Unquoted expansions not split . *Tilde expansion* — Only works for `+~/+` in `+cd+` diff --git a/docs/PROOF-OPEN-FRONTIER.adoc b/docs/PROOF-OPEN-FRONTIER.adoc index 418417b0..828f2075 100644 --- a/docs/PROOF-OPEN-FRONTIER.adoc +++ b/docs/PROOF-OPEN-FRONTIER.adoc @@ -118,8 +118,8 @@ Pure induction on list length. | Statement | `expand_glob(pattern, base_dir)` terminates in time `O(|pattern| × |files_in_base_dir|)`. | File | `impl/rust-cli/src/glob.rs` — no proof today. | Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property. -| Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4. -| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Lean bound and no Rust contract. +| Lift | Property test (`proptest`) for termination + complexity. A bound proved in Lean 4 would be about a Lean model of the walk: optional model evidence, not a statement about the Rust code. The Rust walker (`expand_glob`) needs source-level verification, which the proof policy assigns to Creusot with Pearlite contracts, and Verus. +| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Creusot or Verus contract exists on `expand_glob`, and none has been attempted; no Lean model bound exists either. Ledger rows: `EXPLAINME.adoc#proof-ledger` ("Glob expansion stays within its read bound on the test fixtures" and "Glob expansion terminates within `O(|pattern| x |entries|)` reads"). |=== [[A-7]] diff --git a/impl/rust-cli/fuzz/Cargo.lock b/impl/rust-cli/fuzz/Cargo.lock index 7d73278a..df29e3de 100644 --- a/impl/rust-cli/fuzz/Cargo.lock +++ b/impl/rust-cli/fuzz/Cargo.lock @@ -2022,6 +2022,7 @@ dependencies = [ name = "vsh-fuzz" version = "0.0.0" dependencies = [ + "libc", "libfuzzer-sys", "tempfile", "vsh", diff --git a/impl/rust-cli/fuzz/Cargo.toml b/impl/rust-cli/fuzz/Cargo.toml index 7ea8a865..008f5bb2 100644 --- a/impl/rust-cli/fuzz/Cargo.toml +++ b/impl/rust-cli/fuzz/Cargo.toml @@ -11,6 +11,7 @@ edition = "2021" cargo-fuzz = true [dependencies] +libc = "0.2" libfuzzer-sys = "0.4" tempfile = "3.8" walkdir = "2.4" diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index c467597e..c4f99e02 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -26,12 +26,13 @@ use std::os::unix::fs::symlink; use std::path::{Path, PathBuf}; use std::sync::OnceLock; use tempfile::TempDir; -use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; +use vsh::glob::{escape, expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; const LIMITS: GlobLimits = GlobLimits { max_reads: 20_000, max_matches: 2_000, max_match_bytes: 256 * 1024, + max_listing_bytes: 64 * 1024, }; /// Build the fixture tree. @@ -62,14 +63,40 @@ fn build_fixture() -> std::io::Result { Ok(t) } +static FIXTURE: OnceLock = OnceLock::new(); + +/// Remove the fixture when the process exits. A static is never dropped, so +/// without this every fuzz worker left its tree in the temporary directory. +/// A crash ends the process without running this; that tree stays, with the +/// crashing input, for the person who looks at it. +extern "C" fn remove_fixture() { + if let Some(root) = FIXTURE.get() { + let _ = fs::remove_dir_all(root); + } +} + /// The fixture, built once. A fixture that cannot be built stops the run /// with a clear message: skipping every input would report a green run that /// tested nothing. fn fixture() -> &'static Path { - static FIXTURE: OnceLock = OnceLock::new(); FIXTURE .get_or_init(|| match build_fixture() { - Ok(fixture) => fixture, + Ok(fixture) => { + let root = fixture.keep(); + // The pattern for an absolute input is built from this path; + // it is escaped below, and it must stay valid UTF-8 for that. + if root.to_str().is_none() { + eprintln!("fuzz_glob_expansion: the fixture path is not UTF-8"); + let _ = fs::remove_dir_all(&root); + std::process::exit(1); + } + // SAFETY: `remove_fixture` is a plain `extern "C"` function + // that takes no arguments and does not unwind. + unsafe { + libc::atexit(remove_fixture); + } + root + } Err(e) => { // Exit rather than panic: libFuzzer reports an exiting target // as an error ("fuzz target exited"), so the run still fails. @@ -77,7 +104,7 @@ fn fixture() -> &'static Path { std::process::exit(1); } }) - .path() + .as_path() } fuzz_target!(|data: &[u8]| { @@ -90,7 +117,9 @@ fuzz_target!(|data: &[u8]| { let base = fixture(); let pattern = if input.starts_with('/') { - format!("{}{}", base.display(), input) + // The fixture root is a path, not a pattern: escape it, so a `[` or + // `*` in the temporary directory name is taken literally. + format!("{}{}", escape(&base.to_string_lossy()), input) } else { input.to_string() }; @@ -110,6 +139,15 @@ fuzz_target!(|data: &[u8]| { .sum(); assert_eq!(bytes, usage.match_bytes.held()); + // The input that found the out-of-memory fault must reach the + // fixture, not pass with nothing matched. + if input == "/**/*" { + assert!( + expansion.paths.contains(&base.join("d/sub/g.rs")), + "/**/* did not reach the fixture" + ); + } + // Sorted, each path once, and every path exists. assert!(expansion.paths.windows(2).all(|w| w[0] < w[1])); for path in &expansion.paths { @@ -127,7 +165,7 @@ fuzz_target!(|data: &[u8]| { } Err(err) => { if let Some(stopped) = err.downcast_ref::() { - assert!(stopped.exceeded.level > stopped.exceeded.limit); + assert!(stopped.exceeded.level > u128::from(stopped.exceeded.limit)); assert!(stopped.usage.reads.total() <= LIMITS.max_reads); assert!(stopped.usage.matches.peak() <= LIMITS.max_matches); assert!(stopped.usage.match_bytes.peak() <= LIMITS.max_match_bytes); diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 5433c5db..ba45a541 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -214,6 +214,12 @@ fn prompt_yes_no(prompt: &str) -> Result { /// Symbolic links are counted as entries and never followed, as `rm -r` does: /// following them would count files outside the tree and, with a link back to /// a parent directory, never finish. +/// +/// This holds for a tree that does not change during the count. Each child +/// directory is opened again by its path, so a directory that another process +/// replaces with a link between the type check and the open is followed. The +/// count is only shown in the confirmation prompt; the removal does its own +/// checks. fn count_files_in_tree(path: &str) -> Result { let mut count = 0; diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index 7148cec7..80756c3a 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -505,7 +505,7 @@ impl ExecutableCommand for Command { // double-lookup plus expect shape could panic and duplicated // the lookup. if let Some(func_def) = state.functions.get(&expanded_program).cloned() { - let result = execute_function_call(&func_def, &expansion.fields, state); + let result = execute_function_call(&func_def, &expansion.lossy_fields(), state); expansion.cleanup(); return result; } @@ -568,7 +568,7 @@ impl ExecutableCommand for Command { // Expand every stage before any stage runs, so a failed // expansion runs no part of the pipeline. - let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); + let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); let mut process_subs = Vec::new(); for (program, args) in stages { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; @@ -692,7 +692,7 @@ impl ExecutableCommand for Command { return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); } }; - let expanded_args = &expansion.fields; + let expanded_args = &expansion.lossy_fields(); let output = if *interpret_escapes { expanded_args diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 070e5f99..07de86a2 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -6,6 +6,7 @@ //! and manages stdio. use anyhow::{Context, Result}; +use std::ffi::OsString; use std::fs::{File, OpenOptions}; use std::path::PathBuf; use std::process::{Command, Stdio}; @@ -25,7 +26,7 @@ use crate::state::ShellState; /// - File modification tracking for undo pub fn execute_external_with_redirects( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -116,15 +117,15 @@ pub fn execute_external_with_redirects( /// use vsh::state::ShellState; /// /// let stages = vec![ -/// ("ls".to_string(), vec!["-la".to_string()]), -/// ("grep".to_string(), vec!["test".to_string()]), +/// ("ls".to_string(), vec!["-la".into()]), +/// ("grep".to_string(), vec!["test".into()]), /// ]; /// let mut state = ShellState::new("/tmp")?; /// let exit_code = external::execute_pipeline(&stages, &[], &mut state)?; /// # Ok::<(), anyhow::Error>(()) /// ``` pub fn execute_pipeline( - stages: &[(String, Vec)], + stages: &[(String, Vec)], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -283,11 +284,11 @@ pub fn execute_pipeline( /// ```no_run /// use vsh::external; /// -/// let exit_code = external::execute_external("ls", &["-la".to_string()])?; +/// let exit_code = external::execute_external("ls", &["-la".into()])?; /// assert_eq!(exit_code, 0); /// # Ok::<(), anyhow::Error>(()) /// ``` -pub fn execute_external(program: &str, args: &[String]) -> Result { +pub fn execute_external(program: &str, args: &[OsString]) -> Result { // PATH lookup let executable = find_in_path(program)?; @@ -551,7 +552,7 @@ fn is_executable(path: &PathBuf) -> bool { /// Job ID for the background job pub fn execute_external_background( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -569,7 +570,14 @@ pub fn execute_external_background( let command_str = if args.is_empty() { format!("{} &", program) } else { - format!("{} {} &", program, args.join(" ")) + format!( + "{} {} &", + program, + args.iter() + .map(|arg| arg.to_string_lossy()) + .collect::>() + .join(" ") + ) }; // Build command with redirected stdio @@ -658,7 +666,7 @@ mod tests { #[test] fn test_external_command_with_args() { // Test command with arguments - let exit_code = execute_external("echo", &["hello".to_string()]).unwrap(); + let exit_code = execute_external("echo", &["hello".into()]).unwrap(); assert_eq!(exit_code, 0, "echo should return 0"); } @@ -694,7 +702,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["hello".to_string()]), + ("echo".to_string(), vec!["hello".into()]), ("cat".to_string(), vec![]), ("cat".to_string(), vec![]), ]; @@ -711,7 +719,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["test".to_string()]), + ("echo".to_string(), vec!["test".into()]), ("cat".to_string(), vec![]), ]; let redirects = vec![crate::redirection::Redirection::Output { @@ -735,7 +743,7 @@ mod tests { let temp = TempDir::new().unwrap(); let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); - let stages: Vec<(String, Vec)> = vec![]; + let stages: Vec<(String, Vec)> = vec![]; let result = execute_pipeline(&stages, &[], &mut state); assert!(result.is_err(), "Empty pipeline should return error"); } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index b43dd92c..158f65d2 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -38,11 +38,12 @@ //! //! # Limits //! -//! One expansion is metered on three axes ([`GlobLimits`]): +//! One expansion is metered on these axes ([`GlobLimits`]): //! //! | Axis | Kind | Default | //! |------|------|---------| -//! | file-system reads (directories opened, entries listed, names looked up) | cost: adds up | 10,000,000 | +//! | walker read units (one for each directory opened, entry listed, name looked up and symbolic link resolved; an entry type that the listing gave is not charged) | cost: adds up | 10,000,000 | +//! | bytes of one directory listing held (name length + 1 for each entry) | occupancy, per listing | 64 MiB | //! | matches held | occupancy: high-water mark | 100,000 | //! | bytes of matches held (path length + 1 for each match) | occupancy | 6 MiB | //! @@ -96,6 +97,10 @@ pub const MAX_GLOB_RESULTS: u64 = 100_000; /// most Linux `execve` accepts for argument strings. pub const MAX_GLOB_MATCH_BYTES: u64 = 6 * 1024 * 1024; +/// Default limit on the bytes of names held while one directory is listed +/// (name length + 1 for each entry): 64 MiB. +pub const MAX_GLOB_LISTING_BYTES: u64 = 64 * 1024 * 1024; + /// The limits one glob expansion runs under. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct GlobLimits { @@ -105,6 +110,9 @@ pub struct GlobLimits { pub max_matches: u64, /// Most bytes of matches held at once (path length + 1 per match). pub max_match_bytes: u64, + /// Most bytes of names held while one directory is listed (name length + 1 + /// per entry). The listing is sorted before use, so it is held whole. + pub max_listing_bytes: u64, } impl GlobLimits { @@ -113,6 +121,7 @@ impl GlobLimits { max_reads: MAX_GLOB_READS, max_matches: MAX_GLOB_RESULTS, max_match_bytes: MAX_GLOB_MATCH_BYTES, + max_listing_bytes: MAX_GLOB_LISTING_BYTES, }; } @@ -300,7 +309,9 @@ pub fn expand_glob_continuing( let compiled = Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); - walker.run().map_err(anyhow::Error::new)?; + walker + .run() + .map_err(|stopped| anyhow::Error::new(*stopped))?; Ok(Expansion { usage: walker.usage(), paths: walker.found.into_iter().collect(), @@ -350,12 +361,52 @@ impl Compiled { } } +/// The text a name is matched as. +/// +/// The `glob` crate matches `str` only. A byte that is not valid UTF-8 becomes +/// one character from U+F780..U+F7FF (private use), so `?` and `*` still +/// match it, and it does not look like a literal U+FFFD in the name or the +/// pattern, as a lossy conversion would make it. +// snip: a pattern that contains one of those 128 private-use characters +// matches the invalid byte too; an exact matcher needs a byte-level `Pattern`. +fn match_text(name: &std::ffi::OsStr) -> std::borrow::Cow<'_, str> { + if let Some(text) = name.to_str() { + return std::borrow::Cow::Borrowed(text); + } + let mut out = String::new(); + for chunk in name.as_encoded_bytes().utf8_chunks() { + out.push_str(chunk.valid()); + for &byte in chunk.invalid() { + // Invalid bytes are 0x80..=0xFF, so this stays in U+F780..U+F7FF. + out.push( + char::from_u32(0xF700 + u32::from(byte)).unwrap_or(char::REPLACEMENT_CHARACTER), + ); + } + } + std::borrow::Cow::Owned(out) +} + +/// `text` as a pattern that matches exactly `text`: `*`, `?` and `[` in it +/// are taken literally. Use it for a directory prefix that is not a pattern. +/// +/// # Examples +/// ``` +/// assert_eq!(vsh::glob::escape("/tmp/a*b"), "/tmp/a[*]b"); +/// ``` +pub fn escape(text: &str) -> String { + glob::Pattern::escape(text) +} + const MATCH_OPTIONS: glob::MatchOptions = glob::MatchOptions { case_sensitive: true, require_literal_separator: true, require_literal_leading_dot: true, }; +/// The result of a step of the walk. The error is boxed: it is built once, on +/// the path that stops the walk, and is larger than the values the steps return. +type Walk = std::result::Result>; + /// The walk: a depth-first search over states (directory, component index), /// each processed at most once. struct Walker<'a> { @@ -365,6 +416,7 @@ struct Walker<'a> { reads: Meter, matches: Meter, match_bytes: Meter, + max_listing_bytes: u64, found: BTreeSet, } @@ -387,6 +439,7 @@ impl<'a> Walker<'a> { limits.max_match_bytes, spent.match_bytes, ), + max_listing_bytes: limits.max_listing_bytes, found: BTreeSet::new(), } } @@ -410,8 +463,8 @@ impl<'a> Walker<'a> { } } - fn stopped(&self, exceeded: Exceeded, at: &Path) -> GlobLimitExceeded { - GlobLimitExceeded { + fn stopped(&self, exceeded: Exceeded, at: &Path) -> Box { + Box::new(GlobLimitExceeded { pattern: self.pattern.to_string(), exceeded, usage: self.usage(), @@ -420,15 +473,15 @@ impl<'a> Walker<'a> { } else { at.to_path_buf() }, - } + }) } - fn charge_read(&mut self, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + fn charge_read(&mut self, at: &Path) -> Walk<()> { self.reads.charge(1).map_err(|e| self.stopped(e, at)) } /// Record a match, once. - fn emit(&mut self, path: PathBuf, at: &Path) -> std::result::Result<(), GlobLimitExceeded> { + fn emit(&mut self, path: PathBuf, at: &Path) -> Walk<()> { let path = if self.compiled.dirs_only { path.join("") } else { @@ -454,18 +507,23 @@ impl<'a> Walker<'a> { /// List a directory in name order, charging the open and every entry. /// An unreadable directory gives no entries. - fn list( - &mut self, - dir: &Path, - ) -> std::result::Result, GlobLimitExceeded> { + fn list(&mut self, dir: &Path) -> Walk> { self.charge_read(dir)?; let mut entries = Vec::new(); + // The listing is held whole to sort it; meter the names it holds. + let mut held = Meter::::new( + "bytes of one directory listing held", + self.max_listing_bytes, + ); if let Ok(read) = fs::read_dir(self.real(dir)) { for entry in read { self.charge_read(dir)?; if let Ok(entry) = entry { if let Ok(kind) = entry.file_type() { - entries.push((entry.file_name(), kind)); + let name = entry.file_name(); + held.charge(name.len() as u64 + 1) + .map_err(|e| self.stopped(e, dir))?; + entries.push((name, kind)); } } } @@ -475,15 +533,22 @@ impl<'a> Walker<'a> { } /// Whether `path` is a directory, going through a symbolic link. - fn is_dir(&self, path: &Path, kind: Option) -> bool { + /// + /// A type that the directory listing already gave costs nothing. Looking + /// the path up (no type known, or a symbolic link to resolve) is one read, + /// charged before the lookup. + fn is_dir(&mut self, path: &Path, kind: Option, at: &Path) -> Walk { match kind { - Some(kind) if kind.is_dir() => true, - Some(kind) if !kind.is_symlink() => false, - _ => fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir()), + Some(kind) if kind.is_dir() => Ok(true), + Some(kind) if !kind.is_symlink() => Ok(false), + _ => { + self.charge_read(at)?; + Ok(fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir())) + } } } - fn run(&mut self) -> std::result::Result<(), GlobLimitExceeded> { + fn run(&mut self) -> Walk<()> { let count = self.compiled.components.len(); if count == 0 { return Ok(()); @@ -511,24 +576,24 @@ impl<'a> Walker<'a> { let child = dir.join(name); if last { let exists = fs::symlink_metadata(self.real(&child)).is_ok(); - if exists && (!compiled.dirs_only || self.is_dir(&child, None)) { + if exists && (!compiled.dirs_only || self.is_dir(&child, None, &dir)?) { self.emit(child, &dir)?; } - } else if self.is_dir(&child, None) { + } else if self.is_dir(&child, None, &dir)? { next.push((child, index + 1)); } } Component::Wild(pattern) => { for (name, kind) in self.list(&dir)? { - if !pattern.matches_with(&name.to_string_lossy(), MATCH_OPTIONS) { + if !pattern.matches_with(&match_text(&name), MATCH_OPTIONS) { continue; } let child = dir.join(&name); if last { - if !compiled.dirs_only || self.is_dir(&child, Some(kind)) { + if !compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)? { self.emit(child, &dir)?; } - } else if self.is_dir(&child, Some(kind)) { + } else if self.is_dir(&child, Some(kind), &dir)? { next.push((child, index + 1)); } } @@ -546,7 +611,7 @@ impl<'a> Walker<'a> { continue; } let child = dir.join(&name); - if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind))) { + if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)?) { self.emit(child.clone(), &dir)?; } if kind.is_dir() { @@ -597,7 +662,7 @@ mod walk_tests { fn absolute_patterns_give_absolute_paths() { let t = TempDir::new().unwrap(); fs::write(t.path().join("x.txt"), b"").unwrap(); - let pattern = format!("{}/*.txt", t.path().display()); + let pattern = format!("{}/*.txt", escape(&t.path().to_string_lossy())); assert_eq!( expand_glob(&pattern, Path::new("/nonexistent")).unwrap(), [t.path().join("x.txt")] diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs index ad56a78b..66fc01ec 100644 --- a/impl/rust-cli/src/grade.rs +++ b/impl/rust-cli/src/grade.rs @@ -80,6 +80,10 @@ pub trait Grade: Copy + Eq + fmt::Debug { /// The single number a limit is compared against. fn level(self) -> u64; + + /// What a further `of(amount)` is added to: everything spent for a cost, + /// what is still held for an occupancy. + fn base(self) -> u64; } /// A cumulative cost: a resource that is consumed and never given back. @@ -114,6 +118,10 @@ impl Grade for Cost { fn level(self) -> u64 { self.0 } + + fn base(self) -> u64 { + self.0 + } } impl fmt::Display for Cost { @@ -191,6 +199,10 @@ impl Grade for Occupancy { fn level(self) -> u64 { self.peak } + + fn base(self) -> u64 { + self.held + } } impl fmt::Display for Occupancy { @@ -206,8 +218,9 @@ pub struct Exceeded { pub axis: &'static str, /// The limit that was configured. pub limit: u64, - /// The level the refused charge would have reached. - pub level: u64, + /// The level the refused charge would have reached. Wider than a limit, + /// so a charge that passes `u64::MAX` is reported exactly, not saturated. + pub level: u128, } impl fmt::Display for Exceeded { @@ -258,12 +271,15 @@ impl Meter { /// # Errors /// Returns the [`Exceeded`] that the charge would return. pub fn check(&self, amount: u64) -> Result<(), Exceeded> { - let next = self.grade.then(G::of(amount)); - if next.level() > self.limit { + // Computed in u128 before the grades are composed: composition + // saturates, and a saturated level would pass a limit of `u64::MAX`. + let level = + u128::from(self.grade.level()).max(u128::from(self.grade.base()) + u128::from(amount)); + if level > u128::from(self.limit) { return Err(Exceeded { axis: self.axis, limit: self.limit, - level: next.level(), + level, }); } Ok(()) @@ -401,6 +417,22 @@ mod tests { ); } + #[test] + fn a_charge_past_u64_max_is_refused_and_reported_exactly() { + // A saturated level would equal the limit and be accepted. + let mut cost = Meter::::new("units", u64::MAX); + cost.charge(u64::MAX - 1).unwrap(); + cost.charge(1).unwrap(); // reaches the limit exactly + let refused = cost.charge(5).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 5); + assert_eq!(cost.level(), u64::MAX); + + let mut held = Meter::::new("held", u64::MAX); + held.charge(u64::MAX).unwrap(); + let refused = held.charge(1).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 1); + } + #[test] fn a_meter_with_a_zero_limit_refuses_any_use() { let mut meter: Meter = Meter::new("items held", 0); @@ -481,7 +513,7 @@ mod tests { } Err(refused) => { prop_assert!(wanted.level() > limit); - prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.level, u128::from(wanted.level())); prop_assert_eq!(refused.limit, limit); } } @@ -511,7 +543,7 @@ mod tests { } Err(refused) => { prop_assert!(wanted.level() > limit); - prop_assert_eq!(refused.level, wanted.level()); + prop_assert_eq!(refused.level, u128::from(wanted.level())); } } } diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index 3082b834..b111f0fe 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -1203,7 +1203,9 @@ fn tokens_to_string(tokens: &[Token]) -> Result { } match token { - Token::Word(w) => result.push_str("ed_word_to_string(w)), + // The typed word keeps the quoting of each piece, so `'*'x.*` + // comes back with only its unquoted part active. + Token::Word(w) => result.push_str(&w.typed.to_string()), Token::OutputRedirect => result.push('>'), Token::AppendRedirect => result.push_str(">>"), Token::InputRedirect => result.push('<'), @@ -2689,7 +2691,7 @@ pub fn expand_command_substitution( // echo builtin: its output is the joined, expanded arguments. let expansion = crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; - let mut out = expansion.fields.join(" "); + let mut out = expansion.lossy_fields().join(" "); if !no_newline { out.push('\n'); } diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index 131eee30..d20b400c 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -158,7 +158,8 @@ fn start_command_with_output_redirect( Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? + .lossy_fields(); // Build command string: prog arg1 arg2 > fifo let mut cmd = expanded_program; @@ -208,7 +209,8 @@ fn start_command_with_input_redirect( Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?.fields; + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? + .lossy_fields(); // Build command string let mut cmd = expanded_program; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index 08f7573f..b2b4c60e 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -42,8 +42,9 @@ //! targets, assignments, `case` patterns and `test` arguments still use the //! older string encoding. +use std::ffi::OsString; use std::fmt; -use std::path::Path; +use std::path::{Path, PathBuf}; use anyhow::{Context, Result}; @@ -258,13 +259,24 @@ pub enum ProcessSubs { /// The result of expanding a list of words. pub struct Expansion { - /// The arguments, after quote removal. - pub fields: Vec, + /// The arguments, after quote removal. A glob match keeps the exact bytes + /// of its file name, which need not be UTF-8. + pub fields: Vec, /// Process substitutions started for these arguments. pub process_subs: Vec, } impl Expansion { + /// The arguments as text, for a builtin or a function that works on + /// strings. Bytes that are not UTF-8 are replaced; external commands get + /// [`Expansion::fields`] unchanged. + pub fn lossy_fields(&self) -> Vec { + self.fields + .iter() + .map(|field| field.to_string_lossy().into_owned()) + .collect() + } + /// Wait for the process substitutions and remove their FIFOs. pub fn cleanup(self) { for proc_sub in self.process_subs { @@ -438,32 +450,27 @@ fn glob_fields( cwd: &Path, limits: &GlobLimits, spent: &mut GlobUsage, - fields: &mut Vec, + fields: &mut Vec, ) -> Result<()> { let encoded = encode(chars); for candidate in glob::expand_braces(&encoded) { match glob_pattern(&candidate) { - None => fields.push(remove_quotes(&candidate)), + None => fields.push(remove_quotes(&candidate).into()), Some(pattern) => match glob::expand_glob_continuing(&pattern, cwd, limits, *spent) { Ok(found) if !found.paths.is_empty() => { *spent = found.usage; - fields.extend( - found - .paths - .iter() - .map(|path| path.to_string_lossy().into_owned()), - ); + fields.extend(found.paths.into_iter().map(PathBuf::into_os_string)); } // No match still spent reads. Ok(found) => { *spent = found.usage; - fields.push(remove_quotes(&candidate)); + fields.push(remove_quotes(&candidate).into()); } // A limit is not "no match": substituting the literal word // would run the command on the wrong arguments. Err(e) if e.is::() => return Err(e), // No match, or not a valid pattern: keep the word (POSIX). - _ => fields.push(remove_quotes(&candidate)), + _ => fields.push(remove_quotes(&candidate).into()), }, } } @@ -566,7 +573,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap() - .fields + .lossy_fields() } #[test] @@ -660,7 +667,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap() - .fields; + .lossy_fields(); assert_eq!(out, ["a.txt", "b.txt", "*.txt"]); } @@ -711,7 +718,7 @@ mod tests { &limits, ) .unwrap() - .fields; + .lossy_fields(); assert_eq!(out, ["f0.txt", "f1.txt", "f2.txt", "f3.txt"]); } @@ -764,7 +771,7 @@ mod tests { &GlobLimits::DEFAULT, ) .unwrap(); - assert_eq!(out.fields, ["xsuffix"]); + assert_eq!(out.lossy_fields(), ["xsuffix"]); } #[test] diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs index ead12100..aff5dfa3 100644 --- a/impl/rust-cli/tests/glob_bounds_tests.rs +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -221,6 +221,7 @@ fn default_limits_are_the_documented_values() { max_reads: 10_000_000, max_matches: 100_000, max_match_bytes: 6 * 1024 * 1024, + max_listing_bytes: 64 * 1024 * 1024, } ); } @@ -395,3 +396,77 @@ fn a_name_that_is_not_utf8_is_matched_without_a_panic() { assert_eq!(found.len(), 2); assert!(found.contains(&PathBuf::from(odd))); } + +// --------------------------------------------------------------------------- +// Review of #242: charged lookups, the listing budget, exact name matching +// --------------------------------------------------------------------------- + +#[test] +fn resolving_a_symbolic_link_is_a_charged_read() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + for i in 0..5 { + symlink("real", t.path().join(format!("link{i}"))).unwrap(); + } + // `*/` lists 6 entries (1 open + 6 entries = 7 reads) and then resolves + // each of the 5 links to learn whether it is a directory: 12 reads. + let all = expand_glob_with_limits("*/", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(all.paths.len(), 6); + assert_eq!(all.usage.reads.total(), 12); + + // With room for the listing only, the first lookup is refused. Before, + // the lookups were free and the expansion finished. + let limits = GlobLimits { + max_reads: 7, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*/", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (7, 8)); +} + +#[test] +fn the_listing_limit_stops_a_directory_that_is_too_large_to_hold() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("big")).unwrap(); + for i in 0..10 { + fs::write(t.path().join(format!("big/f{i}")), b"").unwrap(); + } + // Each name is 2 bytes + 1, so 10 names need 30 bytes. + let limits = GlobLimits { + max_listing_bytes: 29, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("big/*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of one directory listing held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (29, 30)); + assert_eq!(s.at, PathBuf::from("big")); + + let limits = GlobLimits { + max_listing_bytes: 30, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("big/*", t.path(), &limits).unwrap(); + assert_eq!(ok.paths.len(), 10); +} + +#[test] +fn an_invalid_byte_is_not_matched_as_a_replacement_character() { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let invalid = OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(invalid), b"").is_err() { + return; // this file system refuses names that are not UTF-8 + } + fs::write(t.path().join("bad\u{fffd}.txt"), b"").unwrap(); + + // Broad patterns match both names. + for pattern in ["*.txt", "bad?.txt"] { + let found = expand_glob(pattern, t.path()).unwrap(); + assert_eq!(found.len(), 2, "{pattern}"); + } + // A literal U+FFFD matches only the name that really contains it. + let found = expand_glob("bad\u{fffd}*", t.path()).unwrap(); + assert_eq!(found, [PathBuf::from("bad\u{fffd}.txt")]); +} diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs index 86deade1..888adee8 100644 --- a/impl/rust-cli/tests/word_expansion_e2e_tests.rs +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -41,12 +41,20 @@ fn stdout(dir: &Path, script: &str) -> String { String::from_utf8(out.stdout).unwrap() } +/// The lines of `text` with runs of white space made one space, so the column +/// padding of `wc` does not matter. +fn rows(text: &str) -> Vec { + text.lines() + .map(|line| line.split_whitespace().collect::>().join(" ")) + .collect() +} + #[test] fn external_command_without_redirect_gets_the_matches() { let dir = fixture(); assert_eq!( - stdout(dir.path(), "wc -c *.txt"), - "0 a.txt\n0 b.txt\n0 total\n" + rows(&stdout(dir.path(), "wc -c *.txt")), + ["0 a.txt", "0 b.txt", "0 total"] ); } @@ -55,8 +63,8 @@ fn external_command_with_redirect_gets_the_matches() { let dir = fixture(); stdout(dir.path(), "wc -c *.txt > out.wc"); assert_eq!( - std::fs::read_to_string(dir.path().join("out.wc")).unwrap(), - "0 a.txt\n0 b.txt\n0 total\n" + rows(&std::fs::read_to_string(dir.path().join("out.wc")).unwrap()), + ["0 a.txt", "0 b.txt", "0 total"] ); } @@ -200,3 +208,36 @@ fn a_substring_length_past_the_end_gives_the_rest() { "cdef bc\n" ); } + +#[test] +fn a_mixed_quote_word_keeps_its_quoting_in_a_list() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + // The operands of && were written back with one quoting for the whole + // word, so the unquoted `.*` stopped being a pattern. + assert_eq!(stdout(dir.path(), "echo '*'x.* && true"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "false || echo *x.*"), "*x.txt ax.txt\n"); + assert_eq!(stdout(dir.path(), "echo '*.txt' && true"), "*.txt\n"); +} + +#[cfg(unix)] +#[test] +fn an_external_command_gets_the_exact_bytes_of_a_matched_name() { + use std::os::unix::ffi::OsStrExt; + let dir = fixture(); + let name = std::ffi::OsStr::from_bytes(b"bad\xff.bin"); + if std::fs::write(dir.path().join(name), b"12345").is_err() { + return; // this file system refuses names that are not UTF-8 + } + // A lossy name would not exist, and `cat` would fail. + assert_eq!(stdout(dir.path(), "cat *.bin"), "12345"); + assert_eq!(stdout(dir.path(), "cat *.bin | cat"), "12345"); +} + +#[test] +fn a_later_pipeline_stage_gets_the_matches() { + let dir = fixture(); + let out = stdout(dir.path(), "printf '' | wc -c *.txt"); + assert_eq!(rows(&out), ["0 a.txt", "0 b.txt", "0 total"]); +} From 217f593a2417cb9e4e347c628c06cb767cebff09 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 19 Nov 2026 01:16:53 +0000 Subject: [PATCH 09/12] test(parser): expand a double-quoted parameter through the typed external path test_double_quotes_with_expansion parsed `touch`, which is a builtin, so its Command::External arm never ran, and that arm used the Display form of the word as if it were the expanded argument. The test now parses an external command, expands its typed words with word::expand_words, and checks parameter expansion and quote removal. The builtin form is checked for the exact value. Co-Authored-By: Claude Fable 5.2 Claude-Session: https://claude.ai/code/session_0136P7ZnLF9EM2PnNxFaCFov --- impl/rust-cli/src/parser.rs | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index b111f0fe..f61888bd 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -4299,20 +4299,26 @@ mod tests { let mut state = ShellState::new("/tmp/vsh_test").unwrap(); state.set_variable("NAME", "Alice"); - // Test external command with variable expansion - let cmd = parse_command("touch \"file_$NAME\"").unwrap(); - match cmd { - Command::External { program, args, .. } => { - assert_eq!(program, "touch"); - let expanded = expand_variables(&args[0].to_string(), &state); - assert_eq!(expanded, "file_Alice"); - } + // An external command: its argument is a typed word, expanded by + // `word::expand_words`. `touch` is a builtin and would not take this + // path, so the old form of this test never ran its external arm. + let cmd = parse_command("wc \"file_$NAME\" \"a $NAME\"x").unwrap(); + let Command::External { program, args, .. } = cmd else { + panic!("Expected an external command"); + }; + assert_eq!(program, "wc"); + let expansion = + crate::word::expand_words(&args, &mut state, crate::word::ProcessSubs::Refuse).unwrap(); + // The parameter is expanded, the quotes are removed, and the spaces + // inside the quotes stay in one argument. + assert_eq!(expansion.lossy_fields(), ["file_Alice", "a Alicex"]); + + // The builtin still takes the older string form. + match parse_command("touch \"file_$NAME\"").unwrap() { Command::Touch { path, .. } => { - // Could also be parsed as Touch command - let expanded = expand_variables(&path, &state); - assert!(expanded.contains("file_")); + assert_eq!(expand_variables(&path, &state), "file_Alice"); } - _ => panic!("Expected External or Touch command"), + other => panic!("Expected Touch, got {other:?}"), } } From bb40fc5354ff945a75130313889116e3000deddf Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:28:51 +0000 Subject: [PATCH 10/12] fix(glob): charge an intermediate literal lookup once A non-final literal component was charged in run() and again in is_dir(), so real/file cost three reads for two lookups and a budget of two failed early. The literal is now charged only in the final-component branch; is_dir charges the intermediate lookup. Adds a regression test. Answers the CodeRabbit finding on glob.rs line 582 of #242. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0136P7ZpKjBuoCRahUbNPywX --- impl/rust-cli/src/glob.rs | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 158f65d2..f1b6bfa0 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -572,9 +572,11 @@ impl<'a> Walker<'a> { match &compiled.components[index] { Component::Literal(name) => { - self.charge_read(&dir)?; let child = dir.join(name); if last { + // One lookup. An intermediate component is charged by + // `is_dir` below, so it is not charged here as well. + self.charge_read(&dir)?; let exists = fs::symlink_metadata(self.real(&child)).is_ok(); if exists && (!compiled.dirs_only || self.is_dir(&child, None, &dir)?) { self.emit(child, &dir)?; @@ -658,6 +660,29 @@ mod walk_tests { assert!(expand("*.none", t.path()).is_empty()); } + #[test] + fn intermediate_literal_is_charged_once() { + // `real/file` needs two lookups: `real`, then `file`. A budget of two + // reads must be enough, and a budget of one must stop at the second. + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + fs::write(t.path().join("real/file"), b"").unwrap(); + + let two = GlobLimits { + max_reads: 2, + ..GlobLimits::DEFAULT + }; + let found = expand_glob_with_limits("real/file", t.path(), &two).unwrap(); + assert_eq!(found.paths, [PathBuf::from("real/file")]); + + let one = GlobLimits { + max_reads: 1, + ..GlobLimits::DEFAULT + }; + let err = expand_glob_with_limits("real/file", t.path(), &one).unwrap_err(); + assert!(err.downcast_ref::().is_some()); + } + #[test] fn absolute_patterns_give_absolute_paths() { let t = TempDir::new().unwrap(); From 6ff543cc5f839182914bf3de5262d819fbbcce79 Mon Sep 17 00:00:00 2001 From: CLAUDE <355856132+HoTT-stuff@users.noreply.github.com> Date: Thu, 19 Nov 2026 09:29:54 +0000 Subject: [PATCH 11/12] fix(process_sub): keep non-UTF-8 argument bytes Both helpers turned the expanded arguments into text with lossy_fields(), so a name such as bad\xff.bin became bad.bin. The arguments now stay OsString values and reach sh as positional parameters, forwarded with "$@". Adds a regression test for each helper. Answers the CodeRabbit finding on process_sub.rs lines 162 and 213 of #242. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_0136P7ZpKjBuoCRahUbNPywX --- impl/rust-cli/src/process_sub.rs | 105 +++++++++++++++++++------------ 1 file changed, 66 insertions(+), 39 deletions(-) diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index d20b400c..fe3bf2ae 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -15,6 +15,7 @@ //! ``` use anyhow::{anyhow, Result}; +use std::ffi::OsString; use std::path::{Path, PathBuf}; use std::process::{Child, Command as ProcessCommand}; @@ -154,30 +155,24 @@ fn start_command_with_output_redirect( // For FIFOs, we use shell redirection (sh -c "cmd > fifo") to avoid blocking // The shell handles the FIFO opening in the background properly - let cmd_string = match cmd { + // The arguments stay OsString values and reach the shell as positional + // parameters, forwarded with "$@". Quoting them into the script text would + // need UTF-8 and would lose the bytes of a name that is not UTF-8. + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? - .lossy_fields(); - - // Build command string: prog arg1 arg2 > fifo - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - // Simple shell escaping (single quotes) - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } - Command::Pwd { .. } => "pwd".to_string(), + Command::Pwd { .. } => ("pwd".to_string(), Vec::new()), Command::Ls { path, .. } => { let mut cmd = "ls".to_string(); if let Some(p) = path { let expanded = crate::parser::expand_with_command_sub(p, state)?; cmd.push_str(&format!(" '{}'", expanded.replace('\'', "'\\''"))); } - cmd + (cmd, Vec::new()) } _ => { return Err(anyhow!( @@ -188,13 +183,7 @@ fn start_command_with_output_redirect( // Use sh -c to run: cmd > fifo_path let shell_cmd = format!("{} > {}", cmd_string, output_path.display()); - - let child = ProcessCommand::new("sh") - .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + spawn_sh(&shell_cmd, &args) } /// Start command with stdin redirected from file (for >(cmd)) @@ -205,20 +194,13 @@ fn start_command_with_input_redirect( ) -> Result { // For FIFOs, use shell redirection: sh -c "cmd < fifo" - let cmd_string = match cmd { + // As above: OsString positional parameters forwarded with "$@". + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args = - crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)? - .lossy_fields(); - - // Build command string - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } _ => { return Err(anyhow!( @@ -229,13 +211,18 @@ fn start_command_with_input_redirect( // Use sh -c to run: cmd < fifo_path let shell_cmd = format!("{} < {}", cmd_string, input_path.display()); + spawn_sh(&shell_cmd, &args) +} - let child = ProcessCommand::new("sh") +/// Run `sh -c script sh args...`. The arguments become `$1`, `$2`, ... and are +/// passed as raw OS strings, so their bytes and their boundaries survive. +fn spawn_sh(script: &str, args: &[OsString]) -> Result { + Ok(ProcessCommand::new("sh") .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + .arg(script) + .arg("sh") + .args(args) + .spawn()?) } #[cfg(test)] @@ -289,4 +276,44 @@ mod tests { proc_sub1.cleanup().unwrap(); proc_sub2.cleanup().unwrap(); } + + /// A file named `bad\xff.bin` (not UTF-8) and a pattern that matches it. + #[cfg(unix)] + fn non_utf8_fixture(dir: &Path) -> String { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let name = OsStr::from_bytes(b"bad\xff.bin"); + std::fs::write(dir.join(name), "12345").unwrap(); + format!("{}/bad*.bin", crate::glob::escape(&dir.to_string_lossy())) + } + + #[cfg(unix)] + #[test] + fn output_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let cmd = parse_command(&format!("cat {pattern}")).unwrap(); + let out = temp_dir.path().join("out.txt"); + + let mut child = start_command_with_output_redirect(&cmd, &mut state, &out).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(&out).unwrap(), "12345"); + } + + #[cfg(unix)] + #[test] + fn input_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let copy = temp_dir.path().join("copy.txt"); + let cmd = parse_command(&format!("sort -o {} {pattern}", copy.display())).unwrap(); + let input = temp_dir.path().join("in.txt"); + std::fs::write(&input, "").unwrap(); + + let mut child = start_command_with_input_redirect(&cmd, &mut state, &input).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(©).unwrap(), "12345\n"); + } } From 281b135ac76bb84bc3b9e8244f64b859093a5a16 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Thu, 19 Nov 2026 12:10:04 +0000 Subject: [PATCH 12/12] fix(shell): Correct quoting and glob edge cases --- impl/rust-cli/src/parser.rs | 63 ++++++++++++++++++++++++++++++++++--- impl/rust-cli/src/word.rs | 17 +++++++++- 2 files changed, 74 insertions(+), 6 deletions(-) diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index f61888bd..2b054384 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -965,7 +965,9 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> let mut double = false; let mut escaped = false; - for ch in chars.by_ref() { + // A nested $() has its own quotes, even inside outer double quotes. + let mut contexts = Vec::new(); + while let Some(ch) = chars.next() { if escaped { escaped = false; cmd.push(ch); @@ -984,6 +986,12 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> double = !double; cmd.push(ch); } + '$' if !single && chars.peek() == Some(&'(') => { + chars.next(); + cmd.push_str("$("); + contexts.push((depth, single, double)); + (depth, single, double) = (1, false, false); + } _ if single || double => cmd.push(ch), '(' => { // Every `(` nests, so the `)` that closes `$((1+2))` or @@ -994,7 +1002,11 @@ fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> ')' => { depth -= 1; if depth == 0 { - return Ok(cmd); + if let Some(context) = contexts.pop() { + (depth, single, double) = context; + } else { + return Ok(cmd); + } } cmd.push(ch); } @@ -2603,9 +2615,14 @@ fn quoted_word_to_string(word: &QuotedWord) -> String { // so it is always an expansion, whatever quotes follow in the word. // Braces keep a following literal out of the name (`$A''b`). WordPart::Variable(name) | WordPart::BracedVariable(name) => { - result.push_str("${"); - result.push_str(name); - result.push('}'); + if matches!(name.as_str(), "?" | "$" | "#" | "@" | "*") { + result.push('$'); + result.push_str(name); + } else { + result.push_str("${"); + result.push_str(name); + result.push('}'); + } } WordPart::CommandSub(cmd) => { // Command substitution - keep as-is for now @@ -4275,6 +4292,42 @@ mod tests { // Quote Parsing Tests (Phase 6 M5) // ========================================================================= + #[test] + fn command_substitution_quotes_have_separate_nested_contexts() { + for source in [ + r#"echo "$(echo ")")" tail"#, + r#"echo "$(echo '"')" tail"#, + r#"echo "$(echo "$(echo ')')")" tail"#, + r#"echo ')' "(" \) (echo nested) tail"#, + r#"echo $(echo "(") tail"#, + ] { + let input = format!("{source})suffix"); + let mut chars = input.chars().peekable(); + assert_eq!(parse_command_sub_dollar(&mut chars).unwrap(), source); + assert_eq!(chars.collect::(), "suffix"); + } + let mut chars = r#"echo "$(echo ")")""#.chars().peekable(); + assert!(parse_command_sub_dollar(&mut chars).is_err()); + } + + #[test] + fn special_parameters_render_unbraced_and_names_keep_their_boundary() { + for name in ["?", "$", "#", "@", "*"] { + for source in [format!("${name}"), format!("${{{name}}}")] { + let tokens = tokenize(&source).unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), format!("${name}")); + } + } + let tokens = tokenize("$A''suffix").unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), "${A}suffix"); + } + #[test] fn test_single_quotes_no_expansion() { use crate::state::ShellState; diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs index b2b4c60e..55550aa1 100644 --- a/impl/rust-cli/src/word.rs +++ b/impl/rust-cli/src/word.rs @@ -492,7 +492,10 @@ fn glob_pattern(encoded: &str) -> Option { if ch == '\\' { let Some(next) = chars.next() else { continue }; match (class.as_mut(), next) { - (Some(_), '-') => hyphen = true, + (Some(members), '-') => { + *members += 1; + hyphen = true; + } // `*`, `?` and `[` are already literal inside a class. (Some(members), '*' | '?' | '[') => { *members += 1; @@ -787,6 +790,18 @@ mod tests { assert_eq!(expand_in(dir.path(), &[w]), ["-.txt", "a.txt", "z.txt"]); } + #[test] + fn a_sole_quoted_hyphen_closes_a_bracket_class() { + // ['-']*.txt + let dir = fixture(&["-.txt", "a.txt"]); + let w = word(&[ + ("[", Quote::None), + ("-", Quote::Single), + ("]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt"]); + } + #[test] fn a_tilde_with_a_quoted_prefix_is_not_expanded() { let dir = fixture(&[]);