diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index de48c3d..4898dcd 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -11,6 +11,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 == [Unreleased] +=== Fixed — 2026-11-18 (arguments: globs, quote removal) + +- Glob expansion now applies to the arguments of every external command, + every pipeline stage, `echo` and function calls. Before, only an external + command that also had a redirection expanded its arguments: `wc -c *.txt` + passed `*.txt` to `wc`, `ls *.txt | wc -l` and `echo *.txt` did not expand. +- Quoted arguments no longer reach the program with a backslash in front of + `*`, `?`, `[` or `{`: `echo '*.txt'` printed `\*.txt` and now prints + `*.txt`. A partly quoted word globs only its unquoted part (`'*'x.*`). +- `echo` performs command substitution and arithmetic expansion + (`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works. +- `$((1+(2*3)))` and other substitutions containing `(` are read to their + matching `)`. +- An expansion error in an argument (such as a glob limit from #191) fails + the command with exit status 1 and does not run it. With a redirection the + status was 127. An error while the program name or a pipeline stage name + is expanded is not changed: it is returned to the caller, so `||` sees it. + +=== Fixed — 2026-11-19 (review of #242, small findings) + +- An AND-OR list keeps the quoting of each piece of a word: `echo '*'x.* && true` + printed `*x.*` and now expands the unquoted part. +- A glob match reaches an external command with the exact bytes of its file + name. A name that is not UTF-8 was replaced by a lossy copy that did not + exist. +- A name that is not UTF-8 is no longer matched by a literal U+FFFD in the + pattern. `*` and `?` still match it. +- Resolving a symbolic link to learn whether it is a directory is charged as + a read unit; it was free. +- New limit `max_listing_bytes` (64 MiB): the names held while one directory + is listed. Exceeding it is a `GlobLimitExceeded` that names the directory. +- A `Meter` charge that would pass `u64::MAX` is refused and reported + exactly; `Exceeded::level` is now `u128`. Before, the level saturated and + a limit of `u64::MAX` accepted the charge. +- `Expansion::fields` in `src/word.rs` is `Vec`; the external + command functions take `OsString` arguments. `glob::escape` is new. +- The `fuzz_glob_expansion` target removes its fixture at exit, escapes the + fixture root in absolute patterns, and checks that `/**/*` reaches the + fixture. +- Tests: `wc` output is compared without its column padding; a glob in a + later pipeline stage is covered. + +=== Fixed — 2026-11-18 (review of #242) + +- An unquoted non-ASCII letter no longer hangs the shell: `echo é` did not + return, because the statement splitter did not advance past it. +- `'é'{a,b}` and `${é:-x}` no longer panic or misread the word: brace and + parameter expansion used character counts as byte offsets. +- `${X:2:N}` with a very large `N` gives the rest of the value; it panicked + on overflow. +- A quoted or escaped parenthesis no longer ends a command substitution: + `$(echo ')')`. +- A quoted hyphen in a bracket expression is a member, not a range: + `[a'-'z]*.txt` no longer matches `m.txt`. +- `~'local'/x` keeps its tilde: a quoted piece in the tilde prefix stops + the expansion. +- `$A''suffix && true` expands `A`: in an AND-OR list the word was read + back as `$Asuffix`. + +=== Changed — 2026-11-18 (typed words) + +- New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces + (unquoted, single, double, backslash), and `word::expand_words` performs + tilde, parameter, command, arithmetic and process substitution, brace + expansion, glob expansion and quote removal on it. `Command::External`, + `Command::Pipeline` and `Command::Echo` now carry `Vec` arguments. + Other commands still use the older string encoding; see + `docs/POSIX_COMPLIANCE.adoc` (Milestones 5 and 6) for what is not migrated. +- Removed `external::expand_glob_args`; its rules (no match or invalid + pattern keeps the word, a limit is an error) moved to `word.rs` with their + tests. + +=== Fixed — 2026-11-18 (glob expansion out of memory, #191) + +- `**` no longer follows symbolic links to directories, so a link back to a + parent directory can no longer make an expansion grow without end. On the old + code `**/*` over a 7-entry tree with two such links passed 2.5 GB in 15 s, + and `**/no-such-name` over it never returned. ClusterFuzzLite hit the same + fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run + 134118693669). The rule now matches bash 5.2 with `globstar` for patterns + that start with `**`. +- Glob expansion is metered while it runs: at most 10,000,000 walker read + units, 100,000 matches and 6 MiB of matches per command. A read unit is + charged for each directory opened, each entry listed, each name looked up + and each symbolic link resolved. The type of an entry comes with the + listing and is not charged again. The limit is on these units; it is not + a cap on every file-system operation the shell makes. Reaching a limit is + a `GlobLimitExceeded` error that names the limit, the usage so far and the + directory being read; the command is not run. Before, the match limit was + checked only after every path had been collected, and an external command + then received the literal pattern instead of an error. +- `.*` and `.h*` find hidden files, as bash does. +- A file name that is not valid UTF-8 no longer panics the expansion. +- The `fuzz_glob_expansion` target expands over a small fixed tree instead of + the runner's file system, and checks the limits on every input. +- `impl/rust-cli/fuzz/Cargo.lock` regenerated; it was missing `hmac` and + `subtle`. + +=== Added — 2026-11-18 + +- `src/grade.rs`: resource grades. A cost adds up in sequence; an occupancy + keeps a high-water mark. A `Meter` refuses the charge that would pass its + limit. Glob expansion is the first user; operation-history retention (#190) + is the intended second. The grade laws are property-tested, not proved. + === Changed — 2026-11-01 (assurance-boundary re-audit) - Corrected prose that had drifted across the assurance boundary. The audit diff --git a/EXPLAINME.adoc b/EXPLAINME.adoc index 7f0fc4d..a6b4063 100644 --- a/EXPLAINME.adoc +++ b/EXPLAINME.adoc @@ -126,6 +126,18 @@ reproduce. Absence of a tool is an open obligation, not a passing leg | `proofs/z3/` (`archive/proofs/` for Isabelle and Mizar) | §4 exception notes are now recorded in each `.smt2` header. `filesystem_operations` and `symlink_operations` were rewritten on 2026-11-09 to define the operations as array `store`s and negate each goal; the other three still rest on asserted axioms. Isabelle and Mizar were never CI-checked and are archived, unchecked, with a successor map +| Glob expansion stays within its read bound on the test fixtures (frontier A-6) +| tested-only +| `cargo test --test glob_bounds_tests`; `proptest` in `src/glob.rs` and `src/grade.rs` +| `impl/rust-cli/tests/glob_bounds_tests.rs`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| Fixed fixtures only: the reported read count equals a count made from the fixture, and a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. The grade laws are property-tested, not proved + +| Glob expansion terminates within `O(|pattern| x |entries|)` reads (frontier A-6) +| open +| - +| `impl/rust-cli/src/glob.rs` `expand_glob`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6 +| No Creusot or Verus contract on the walker (PROOF-POLICY §1), and no Lean model bound. A Lean bound would be model evidence only + | Sandbox path normalisation never escapes the root | proven (model) / tested-only (code) | Lean 4 for the model; `proptest` for the code @@ -212,7 +224,8 @@ All share: *Multiple proof systems for cross-validation, honest status reporting | `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine) -| `impl/rust-cli/src/glob.rs` | POSIX-compliant glob expansion: *, ?, [...], {...}. Hidden file handling (requires explicit leading dot). +| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash 5.2 globstar rules, one documented difference for symlinked directories), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191). +| `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved. | `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT). diff --git a/docs/POSIX_COMPLIANCE.adoc b/docs/POSIX_COMPLIANCE.adoc index cc34cfb..0d382eb 100644 --- a/docs/POSIX_COMPLIANCE.adoc +++ b/docs/POSIX_COMPLIANCE.adoc @@ -22,14 +22,14 @@ M12-14 not started |2 |Redirections |*Complete* |All 8 types including `+2>&1+` |3 |Pipelines |*Complete* |Multi-stage, exit code from last |4 |Variables |*Complete* |Scalars, arrays, parameter expansion -|5 |Glob Expansion |*Complete* |POSIX-compliant + brace expansion -|6 |Quote Processing |*Complete* |Single, double, backslash +|5 |Glob Expansion |*Partial* |Matching complete; not yet applied to builtin operands, `+for+` word lists, redirection targets +|6 |Quote Processing |*Partial* |Quote removal complete only for arguments that go through `+word::expand_words+` |7 |Command Substitution |*Complete* |`+$(cmd)+` and backticks |8 |Arithmetic |*Complete* |Full operator set including `+**+` |9 |Control Structures |*Complete* |if/elif/else, while, for, case/esac |10 |Job Control |*Partial* |No SIGCHLD, no Ctrl+Z |11 |Shell Builtins |*Partial* |~30 implemented, ~15 missing -|12 |Functions |*Not started* |No `+func() { ... }+` syntax +|12 |Functions |*Partial* |`+name() { ... }+` definitions and calls work; nesting incomplete |13 |Shell Scripts |*Not started* |No `+.sh+` file execution |14 |Advanced Features |*Partial* |Here docs, process sub, arrays done |=== @@ -147,12 +147,13 @@ quoted context) - `+${VAR:=default}+` does not actually assign — same as `+${VAR:-default}+` - `+export VAR+` on unset var errors instead of creating empty exported var -==== Milestone 5: Glob Expansion — COMPLETE +==== Milestone 5: Glob Expansion — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== |Feature |Status |Implementation -|`+*+` (match any) |Done |`+glob+` crate +|`+*+` (match any) |Done |`+glob+` crate `+Pattern+` per component; +directory walk in `+glob.rs+` (2026-11-18) |`+?+` (single char) |Done | @@ -161,14 +162,31 @@ creating empty exported var |`+[!...]+` negation |Done | |Hidden file exclusion |Done |`+require_literal_leading_dot: true+` -(fixed 2026-02-12) +(fixed 2026-02-12); `+.*+` and `+.h*+` now find hidden files (fixed +2026-11-18; they previously gave only `+.+`/`+..+` or nothing) + +|Recursive `+**+` (bash globstar) |Done |Does not descend through symbolic +links or into hidden directories. One known difference from bash 5.2: bash +enters a symlinked directory for one more component when `+**+` is not the +first component (2026-11-18) + +|Expansion limits |Done |10,000,000 walker read units (directories opened, +entries listed, names looked up, links resolved), 100,000 matches, 6 MiB of +matches, 64 MiB of names for one directory listing; exceeding one is an error that names it, never a truncated +or literal result (#191, 2026-11-18) |Brace expansion `+{a,b,c}+` |Done |Nested braces supported |Sorted results |Done |POSIX requirement + +|Where globs expand |Partial |Arguments of external commands (with or +without redirections), every pipeline stage, `+echo+` and function calls, +through `+word::expand_words+` (2026-11-18; before, only external commands +with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...), +`+for+` word lists, redirection targets |=== -==== Milestone 6: Quote Processing — COMPLETE +==== Milestone 6: Quote Processing — PARTIAL [width="100%",cols="29%,25%,46%",options="header",] |=== @@ -183,6 +201,12 @@ quotes |Unclosed quote detection |Done | |Line continuation |Done | + +|Quote removal |Partial |Done for the arguments that go through +`+word::expand_words+` (see Milestone 5), which keeps the quoting of each +piece of a word. Builtin operands, redirection targets, assignment values, +`+case+` patterns and `+test+` arguments still receive quoted `+*?[{~+` with +a backslash in front (`+touch '*q'+` creates `+\*q+`) |=== ==== Milestone 7: Command Substitution — COMPLETE @@ -231,8 +255,9 @@ condition chains |`+while+`/`+do+`/`+done+` |Done |`+Command::While+` with break/continue -|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::For+` with glob -expansion in wordlist +|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::ForLoop+`; the word list gets +parameter expansion and `+$IFS+` splitting, but not glob expansion (see +Milestone 5) |`+case+`/`+esac+` |Done |`+Command::Case+` with glob pattern matching @@ -292,9 +317,13 @@ ops, numeric comparison `+hash+`, `+umask+`, `+wait+`, `+getopts+`, `+shift+`, `+break+`, `+continue+`, `+return+`, `+local+` -==== Milestone 12: Functions — NOT STARTED +==== Milestone 12: Functions — PARTIAL -No function definition or calling syntax exists in the parser. +`+name() { ... }+` and `+function name { ... }+` define a function, and a call +runs it with its own positional parameters, `+local+` and `+return+`. An +end-to-end test runs `+f() { echo "$#:$1:$2"; }; f *.txt+` through the built +shell (`+tests/word_expansion_e2e_tests.rs+`). Full nesting of functions is +not complete. ==== Milestone 13: Shell Scripts — NOT STARTED @@ -355,7 +384,7 @@ composition |0 === Critical Missing Features (by impact) [arabic] -. *Functions* — No `+func() { ... }+` — blocks modular scripts +. *Functions* — definitions and calls work; nesting is incomplete . *Shell script execution* — No `+.sh+` file running . *Word splitting (`+$IFS+`)* — Unquoted expansions not split . *Tilde expansion* — Only works for `+~/+` in `+cd+` diff --git a/docs/PROOF-OPEN-FRONTIER.adoc b/docs/PROOF-OPEN-FRONTIER.adoc index 0372b66..828f207 100644 --- a/docs/PROOF-OPEN-FRONTIER.adoc +++ b/docs/PROOF-OPEN-FRONTIER.adoc @@ -118,7 +118,8 @@ Pure induction on list length. | Statement | `expand_glob(pattern, base_dir)` terminates in time `O(|pattern| × |files_in_base_dir|)`. | File | `impl/rust-cli/src/glob.rs` — no proof today. | Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property. -| Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4. +| Lift | Property test (`proptest`) for termination + complexity. A bound proved in Lean 4 would be about a Lean model of the walk: optional model evidence, not a statement about the Rust code. The Rust walker (`expand_glob`) needs source-level verification, which the proof policy assigns to Creusot with Pearlite contracts, and Verus. +| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Creusot or Verus contract exists on `expand_glob`, and none has been attempted; no Lean model bound exists either. Ledger rows: `EXPLAINME.adoc#proof-ledger` ("Glob expansion stays within its read bound on the test fixtures" and "Glob expansion terminates within `O(|pattern| x |entries|)` reads"). |=== [[A-7]] diff --git a/impl/rust-cli/fuzz/Cargo.lock b/impl/rust-cli/fuzz/Cargo.lock index ec6f827..df29e3d 100644 --- a/impl/rust-cli/fuzz/Cargo.lock +++ b/impl/rust-cli/fuzz/Cargo.lock @@ -469,6 +469,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", "crypto-common", + "subtle", ] [[package]] @@ -724,6 +725,15 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" +[[package]] +name = "hmac" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +dependencies = [ + "digest", +] + [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1706,6 +1716,12 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "subtle" +version = "2.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" + [[package]] name = "syn" version = "1.0.109" @@ -1984,13 +2000,17 @@ dependencies = [ "dirs", "gag", "glob", + "hex", + "hmac", "libc", + "nix 0.31.2", "nu-ansi-term", "ratatui", "reedline", "regex", "serde", "serde_json", + "sha2", "thiserror 2.0.18", "tokio", "uuid", @@ -2002,6 +2022,7 @@ dependencies = [ name = "vsh-fuzz" version = "0.0.0" dependencies = [ + "libc", "libfuzzer-sys", "tempfile", "vsh", diff --git a/impl/rust-cli/fuzz/Cargo.toml b/impl/rust-cli/fuzz/Cargo.toml index 7ea8a86..008f5bb 100644 --- a/impl/rust-cli/fuzz/Cargo.toml +++ b/impl/rust-cli/fuzz/Cargo.toml @@ -11,6 +11,7 @@ edition = "2021" cargo-fuzz = true [dependencies] +libc = "0.2" libfuzzer-sys = "0.4" tempfile = "3.8" walkdir = "2.4" diff --git a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs index beecbd9..c4f99e0 100644 --- a/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs +++ b/impl/rust-cli/fuzz/fuzz_targets/fuzz_glob_expansion.rs @@ -2,56 +2,174 @@ // Copyright (c) Jonathan D.A. Jewell //! Fuzz Target: Glob Expansion //! -//! Tests glob pattern matching for: -//! - Malicious patterns (DoS via backtracking) -//! - Deep recursion (**/**/...) -//! - Edge cases (empty patterns, special chars) -//! - Resource exhaustion +//! Expands arbitrary patterns over one small, fixed directory tree, so the +//! target exercises the shell's own walker and limits rather than the size of +//! the machine it runs on. (Until 2026-11-17 an absolute input such as `/**/*` +//! walked the runner's whole file system, following symbolic-link loops under +//! /proc and /sys, and hit libFuzzer's 2560 MB limit.) +//! +//! - Absolute patterns are re-rooted under the fixture. +//! - Patterns with a `..` component are skipped: they would leave the fixture. +//! - Expansion runs under small limits so that inputs reach them, and every +//! result is checked against the limits and against the file system. +//! +//! The fixture holds the cases that matter for termination and matching: +//! two symbolic-link loops, a dangling link, a hidden directory and a name +//! that is not valid UTF-8. #![no_main] use libfuzzer_sys::fuzz_target; +use std::fs; +use std::os::unix::ffi::OsStrExt; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::OnceLock; use tempfile::TempDir; -use vsh::glob::expand_glob; +use vsh::glob::{escape, expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; + +const LIMITS: GlobLimits = GlobLimits { + max_reads: 20_000, + max_matches: 2_000, + max_match_bytes: 256 * 1024, + max_listing_bytes: 64 * 1024, +}; + +/// Build the fixture tree. +fn build_fixture() -> std::io::Result { + let t = TempDir::new()?; + let root = t.path(); + for dir in ["d/sub", ".hd/x", "e"] { + fs::create_dir_all(root.join(dir))?; + } + for file in [ + "a.txt", + "b.txt", + "c.log", + ".hidden", + "d/f.rs", + "d/sub/g.rs", + ".hd/x/y", + "e/z", + ] { + fs::write(root.join(file), b"")?; + } + // Some file systems refuse names that are not UTF-8; the rest still works. + let _ = fs::write(root.join(std::ffi::OsStr::from_bytes(b"bad\xff.txt")), b""); + symlink("..", root.join("d/up"))?; // loop back to the root + symlink(".", root.join("self"))?; // loop to itself + symlink("d", root.join("link"))?; + symlink("missing", root.join("dangling"))?; + Ok(t) +} + +static FIXTURE: OnceLock = OnceLock::new(); + +/// Remove the fixture when the process exits. A static is never dropped, so +/// without this every fuzz worker left its tree in the temporary directory. +/// A crash ends the process without running this; that tree stays, with the +/// crashing input, for the person who looks at it. +extern "C" fn remove_fixture() { + if let Some(root) = FIXTURE.get() { + let _ = fs::remove_dir_all(root); + } +} + +/// The fixture, built once. A fixture that cannot be built stops the run +/// with a clear message: skipping every input would report a green run that +/// tested nothing. +fn fixture() -> &'static Path { + FIXTURE + .get_or_init(|| match build_fixture() { + Ok(fixture) => { + let root = fixture.keep(); + // The pattern for an absolute input is built from this path; + // it is escaped below, and it must stay valid UTF-8 for that. + if root.to_str().is_none() { + eprintln!("fuzz_glob_expansion: the fixture path is not UTF-8"); + let _ = fs::remove_dir_all(&root); + std::process::exit(1); + } + // SAFETY: `remove_fixture` is a plain `extern "C"` function + // that takes no arguments and does not unwind. + unsafe { + libc::atexit(remove_fixture); + } + root + } + Err(e) => { + // Exit rather than panic: libFuzzer reports an exiting target + // as an error ("fuzz target exited"), so the run still fails. + eprintln!("fuzz_glob_expansion: cannot build the fixture tree: {e}"); + std::process::exit(1); + } + }) + .as_path() +} fuzz_target!(|data: &[u8]| { - // Convert to string - if let Ok(pattern) = std::str::from_utf8(data) { - // Limit length to prevent DoS - if pattern.len() > 200 { - return; - } + let Ok(input) = std::str::from_utf8(data) else { + return; + }; + if input.len() > 200 || input.split('/').any(|part| part == "..") { + return; + } - // Skip patterns with too many wildcards (prevent combinatorial explosion) - let wildcard_count = pattern.matches('*').count() + pattern.matches('?').count(); - if wildcard_count > 10 { - return; - } + let base = fixture(); + let pattern = if input.starts_with('/') { + // The fixture root is a path, not a pattern: escape it, so a `[` or + // `*` in the temporary directory name is taken literally. + format!("{}{}", escape(&base.to_string_lossy()), input) + } else { + input.to_string() + }; - // Skip patterns with excessive recursion - let recursive_count = pattern.matches("**").count(); - if recursive_count > 3 { - return; - } + match expand_glob_with_limits(&pattern, base, &LIMITS) { + Ok(expansion) => { + let usage = expansion.usage; + assert!(usage.reads.total() <= LIMITS.max_reads); + assert!(usage.matches.peak() <= LIMITS.max_matches); + assert!(usage.match_bytes.peak() <= LIMITS.max_match_bytes); + assert_eq!(expansion.paths.len() as u64, usage.matches.held()); - // Create temporary directory for glob expansion - let temp = match TempDir::new() { - Ok(t) => t, - Err(_) => return, - }; - - // Expand glob (should complete quickly or return error) - // Should NEVER: - // - Hang (exponential backtracking) - // - Panic - // - Use excessive memory - let _ = expand_glob(pattern, temp.path()); - - // Test cases include: - // - Simple wildcards: *.txt, file?.rs - // - Recursive: **/*.rs - // - Character classes: [abc], [!xyz] - // - Braces: {a,b,c} - // - Malicious: *, **, *{,/*}{,/*}{,/*}... (DoS) + let bytes: u64 = expansion + .paths + .iter() + .map(|p| p.as_os_str().len() as u64 + 1) + .sum(); + assert_eq!(bytes, usage.match_bytes.held()); + + // The input that found the out-of-memory fault must reach the + // fixture, not pass with nothing matched. + if input == "/**/*" { + assert!( + expansion.paths.contains(&base.join("d/sub/g.rs")), + "/**/* did not reach the fixture" + ); + } + + // Sorted, each path once, and every path exists. + assert!(expansion.paths.windows(2).all(|w| w[0] < w[1])); + for path in &expansion.paths { + let real: PathBuf = if path.is_absolute() { + path.clone() + } else { + base.join(path) + }; + assert!( + fs::symlink_metadata(&real).is_ok(), + "{} does not exist", + real.display() + ); + } + } + Err(err) => { + if let Some(stopped) = err.downcast_ref::() { + assert!(stopped.exceeded.level > u128::from(stopped.exceeded.limit)); + assert!(stopped.usage.reads.total() <= LIMITS.max_reads); + assert!(stopped.usage.matches.peak() <= LIMITS.max_matches); + assert!(stopped.usage.match_bytes.peak() <= LIMITS.max_match_bytes); + } + } } }); diff --git a/impl/rust-cli/src/commands.rs b/impl/rust-cli/src/commands.rs index 4f91625..02918fc 100644 --- a/impl/rust-cli/src/commands.rs +++ b/impl/rust-cli/src/commands.rs @@ -2856,7 +2856,7 @@ mod tests { let state = ShellState::new(tmp.path().to_str().unwrap()).unwrap(); let cmd = crate::parser::Command::External { program: "git".to_string(), - args: vec!["status".to_string()], + args: vec!["status".into()], redirects: vec![], background: false, }; diff --git a/impl/rust-cli/src/confirmation.rs b/impl/rust-cli/src/confirmation.rs index 57f6ec4..ba45a54 100644 --- a/impl/rust-cli/src/confirmation.rs +++ b/impl/rust-cli/src/confirmation.rs @@ -209,16 +209,27 @@ fn prompt_yes_no(prompt: &str) -> Result { Ok(input == "y" || input == "yes") } +/// Count the files a recursive removal of `path` would delete. +/// +/// Symbolic links are counted as entries and never followed, as `rm -r` does: +/// following them would count files outside the tree and, with a link back to +/// a parent directory, never finish. +/// +/// This holds for a tree that does not change during the count. Each child +/// directory is opened again by its path, so a directory that another process +/// replaces with a link between the type check and the open is followed. The +/// count is only shown in the confirmation prompt; the removal does its own +/// checks. fn count_files_in_tree(path: &str) -> Result { let mut count = 0; fn visit_dirs(path: &std::path::Path, count: &mut usize) -> Result<()> { - if path.is_dir() { + // symlink_metadata: a link given as the root is one entry, not its target. + if std::fs::symlink_metadata(path)?.is_dir() { for entry in std::fs::read_dir(path)? { let entry = entry?; - let path = entry.path(); - if path.is_dir() { - visit_dirs(&path, count)?; + if entry.file_type()?.is_dir() { + visit_dirs(&entry.path(), count)?; } else { *count += 1; } @@ -338,6 +349,29 @@ mod tests { #[test] fn test_count_files() { - let _ = count_files_in_tree("/tmp").unwrap_or(0); + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::fs::write(t.path().join("top"), b"").unwrap(); + let root = t.path().to_str().unwrap(); + assert_eq!(count_files_in_tree(root).unwrap(), 2); + } + + /// A link back to a parent directory used to make the count recurse until + /// the path was too long, branching at every level. + #[cfg(unix)] + #[test] + fn test_count_files_does_not_follow_symlink_loops() { + let t = tempfile::TempDir::new().unwrap(); + std::fs::create_dir(t.path().join("d")).unwrap(); + std::fs::write(t.path().join("d/f"), b"").unwrap(); + std::os::unix::fs::symlink("..", t.path().join("d/up")).unwrap(); + std::os::unix::fs::symlink(".", t.path().join("self")).unwrap(); + let root = t.path().to_str().unwrap(); + // d/f, d/up and self: the two links are counted, not followed. + assert_eq!(count_files_in_tree(root).unwrap(), 3); + // A link given as the root is one entry too. + let link = t.path().join("self"); + assert_eq!(count_files_in_tree(link.to_str().unwrap()).unwrap(), 1); } } diff --git a/impl/rust-cli/src/executable.rs b/impl/rust-cli/src/executable.rs index aeb72a2..80756c3 100644 --- a/impl/rust-cli/src/executable.rs +++ b/impl/rust-cli/src/executable.rs @@ -488,35 +488,36 @@ impl ExecutableCommand for Command { // Expand program name (no process substitutions in program name) let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + // Word expansion: variables, substitutions, braces, globs and + // quote removal. A failed expansion (for example a glob that + // reaches a limit) fails the command without running it. + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + // Check if the command is a shell function before external execution. // `if let Some(...)` performs a single map lookup; the former // double-lookup plus expect shape could panic and duplicated // the lookup. if let Some(func_def) = state.functions.get(&expanded_program).cloned() { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); - return execute_function_call(&func_def, &expanded_args, state); + let result = execute_function_call(&func_def, &expansion.lossy_fields(), state); + expansion.cleanup(); + return result; } - // Expand arguments with process substitutions - let mut all_process_subs = Vec::new(); - let mut expanded_args = Vec::new(); - - for arg in args { - let (expanded_arg, proc_subs) = - crate::parser::expand_with_process_sub(arg, state)?; - expanded_args.push(expanded_arg); - all_process_subs.extend(proc_subs); - } + let expanded_args = &expansion.fields; // Execute main command if *background { // Background execution let _job_id = external::execute_external_background( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -525,24 +526,19 @@ impl ExecutableCommand for Command { 0 // No job created }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::Success) } else { // Foreground execution let exit_code = if redirects.is_empty() { // No redirections - use simple path - external::execute_external(&expanded_program, &expanded_args) + external::execute_external(&expanded_program, expanded_args) } else { // Has redirections - use redirection-aware execution external::execute_external_with_redirects( &expanded_program, - &expanded_args, + expanded_args, redirects, state, ) @@ -552,12 +548,7 @@ impl ExecutableCommand for Command { 127 }); - // Cleanup all process substitutions - for proc_sub in all_process_subs { - if let Err(e) = proc_sub.cleanup() { - eprintln!("Warning: Failed to cleanup process substitution: {}", e); - } - } + expansion.cleanup(); Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -569,33 +560,42 @@ impl ExecutableCommand for Command { redirects, background, } => { - // Expand variables and command substitutions in all pipeline stages - let expanded_stages: Result)>> = stages - .iter() - .map(|(program, args)| { - let expanded_program = - crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - Ok((expanded_program, expanded_args?)) - }) - .collect(); - let expanded_stages = expanded_stages?; - - if *background && !expanded_stages.is_empty() { + if *background && !stages.is_empty() { return Err(anyhow::anyhow!( "Background pipelines are not supported yet; refusing to run them in the foreground" )); } + // Expand every stage before any stage runs, so a failed + // expansion runs no part of the pipeline. + let mut expanded_stages: Vec<(String, Vec)> = Vec::new(); + let mut process_subs = Vec::new(); + for (program, args) in stages { + let expanded_program = crate::parser::expand_with_command_sub(program, state)?; + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => { + expanded_stages.push((expanded_program, expansion.fields)); + process_subs.extend(expansion.process_subs); + } + Err(e) => { + eprintln!("vsh: {}: {}", expanded_program, e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + } + } + let exit_code = external::execute_pipeline(&expanded_stages, redirects, state) .unwrap_or_else(|e| { eprintln!("Pipeline error: {}", e); 127 }); + crate::word::Expansion { + fields: Vec::new(), + process_subs, + } + .cleanup(); + Ok(ExecutionResult::ExternalCommand { exit_code }) } @@ -684,10 +684,15 @@ impl ExecutableCommand for Command { interpret_escapes, redirects, } => { - let expanded_args: Vec = args - .iter() - .map(|a| crate::parser::expand_variables(a, state)) - .collect(); + let expansion = + match crate::word::expand_words(args, state, crate::word::ProcessSubs::Allow) { + Ok(expansion) => expansion, + Err(e) => { + eprintln!("vsh: echo: {}", e); + return Ok(ExecutionResult::ExternalCommand { exit_code: 1 }); + } + }; + let expanded_args = &expansion.lossy_fields(); let output = if *interpret_escapes { expanded_args @@ -719,6 +724,7 @@ impl ExecutableCommand for Command { Ok(()) })?; } + expansion.cleanup(); Ok(ExecutionResult::Success) } @@ -1291,7 +1297,7 @@ impl ExecutableCommand for Command { if args.is_empty() { program.clone() } else { - format!("{} {}", program, args.join(" ")) + format!("{} {}", program, join_words(args)) } } Command::Pipeline { stages, .. } => { @@ -1301,7 +1307,7 @@ impl ExecutableCommand for Command { if args.is_empty() { prog.clone() } else { - format!("{} {}", prog, args.join(" ")) + format!("{} {}", prog, join_words(args)) } }) .collect(); @@ -1377,7 +1383,7 @@ impl ExecutableCommand for Command { format!("{} {} {}", left.description(), op_str, right.description()) } - Command::Echo { args, .. } => format!("echo {}", args.join(" ")), + Command::Echo { args, .. } => format!("echo {}", join_words(args)), Command::True => "true".to_string(), Command::False => "false".to_string(), Command::Read { var_names, .. } => format!("read {}", var_names.join(" ")), @@ -1645,6 +1651,15 @@ fn execute_loop( Ok(last_result) } +/// Words written back as source and separated by spaces. +fn join_words(words: &[crate::word::Word]) -> String { + words + .iter() + .map(|w| w.to_string()) + .collect::>() + .join(" ") +} + /// Match a word against a case pattern (supports glob-style wildcards) fn case_pattern_matches(word: &str, pattern: &str) -> bool { if pattern == "*" { diff --git a/impl/rust-cli/src/external.rs b/impl/rust-cli/src/external.rs index 4dbc9f9..07de86a 100644 --- a/impl/rust-cli/src/external.rs +++ b/impl/rust-cli/src/external.rs @@ -6,12 +6,12 @@ //! and manages stdio. use anyhow::{Context, Result}; +use std::ffi::OsString; use std::fs::{File, OpenOptions}; use std::path::PathBuf; use std::process::{Command, Stdio}; use std::time::Duration; -use crate::glob; use crate::redirection::{RedirectSetup, Redirection}; use crate::signals; use crate::state::ShellState; @@ -26,7 +26,7 @@ use crate::state::ShellState; /// - File modification tracking for undo pub fn execute_external_with_redirects( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -37,8 +37,7 @@ pub fn execute_external_with_redirects( let (stdin_cfg, stdout_cfg, stderr_cfg) = stdio_config_from_redirects(redirects, &redirect_setup, state)?; - // Expand glob patterns in arguments (Phase 6 M12) - let expanded_args = expand_glob_args(args, state)?; + // The arguments arrive already expanded, globs included (`crate::word`). // PATH lookup let executable = find_in_path(program)?; @@ -46,7 +45,7 @@ pub fn execute_external_with_redirects( // Build command with redirected stdio let mut command = Command::new(&executable); command - .args(&expanded_args) + .args(args) .stdin(stdin_cfg) .stdout(stdout_cfg) .stderr(stderr_cfg); @@ -118,15 +117,15 @@ pub fn execute_external_with_redirects( /// use vsh::state::ShellState; /// /// let stages = vec![ -/// ("ls".to_string(), vec!["-la".to_string()]), -/// ("grep".to_string(), vec!["test".to_string()]), +/// ("ls".to_string(), vec!["-la".into()]), +/// ("grep".to_string(), vec!["test".into()]), /// ]; /// let mut state = ShellState::new("/tmp")?; /// let exit_code = external::execute_pipeline(&stages, &[], &mut state)?; /// # Ok::<(), anyhow::Error>(()) /// ``` pub fn execute_pipeline( - stages: &[(String, Vec)], + stages: &[(String, Vec)], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -285,11 +284,11 @@ pub fn execute_pipeline( /// ```no_run /// use vsh::external; /// -/// let exit_code = external::execute_external("ls", &["-la".to_string()])?; +/// let exit_code = external::execute_external("ls", &["-la".into()])?; /// assert_eq!(exit_code, 0); /// # Ok::<(), anyhow::Error>(()) /// ``` -pub fn execute_external(program: &str, args: &[String]) -> Result { +pub fn execute_external(program: &str, args: &[OsString]) -> Result { // PATH lookup let executable = find_in_path(program)?; @@ -475,66 +474,6 @@ fn stdin_from_bytes(content: &[u8], what: &str) -> Result { Ok(Stdio::from(reader)) } -/// Expand glob patterns in command arguments (Phase 6 M12) -/// -/// For each argument: -/// - If it contains glob metacharacters (*, ?, [, {), expand it -/// - If expansion succeeds, use matching paths -/// - If expansion fails or returns empty, use literal pattern (POSIX behavior) -/// - If no metacharacters, use argument as-is -/// -/// # POSIX Behavior -/// - Brace expansion happens first: {a,b} -> [a, b] -/// - Then glob expansion: *.txt -> [file1.txt, file2.txt] -/// - Empty matches return literal: "*.xyz" if no .xyz files -/// -/// # Examples -/// ```ignore -/// expand_glob_args(&["echo", "*.txt"], state) // -> ["echo", "file1.txt", "file2.txt"] -/// expand_glob_args(&["ls", "file?.rs"], state) // -> ["ls", "file1.rs", "file2.rs"] -/// expand_glob_args(&["rm", "*.xyz"], state) // -> ["rm", "*.xyz"] (no .xyz files) -/// ``` -fn expand_glob_args(args: &[String], _state: &ShellState) -> Result> { - let mut expanded: Vec = Vec::new(); - - // Get current working directory for glob expansion - let cwd = std::env::current_dir().context("Failed to get current working directory")?; - - for arg in args { - // Check if argument contains glob metacharacters - if glob::contains_glob_pattern(arg) { - // First, expand braces: file{1,2}.txt -> [file1.txt, file2.txt] - let brace_expanded = glob::expand_braces(arg); - - // Then expand each brace result for glob patterns - let mut glob_matches: Vec = Vec::new(); - for pattern in &brace_expanded { - // Expand glob pattern relative to current directory - match glob::expand_glob(pattern, &cwd) { - Ok(matches) if !matches.is_empty() => { - // Found matches - add them as strings - for path in matches { - glob_matches.push(path.to_string_lossy().to_string()); - } - } - _ => { - // No matches or error - use literal pattern (POSIX behavior) - glob_matches.push(pattern.clone()); - } - } - } - - // Add all matches (or literal if no matches) - expanded.extend(glob_matches); - } else { - // No glob pattern - use argument as-is - expanded.push(arg.clone()); - } - } - - Ok(expanded) -} - /// Handle command exit status, converting signals to exit codes fn handle_exit_status(status: std::process::ExitStatus) -> Result { #[cfg(unix)] @@ -613,7 +552,7 @@ fn is_executable(path: &PathBuf) -> bool { /// Job ID for the background job pub fn execute_external_background( program: &str, - args: &[String], + args: &[OsString], redirects: &[Redirection], state: &mut ShellState, ) -> Result { @@ -631,7 +570,14 @@ pub fn execute_external_background( let command_str = if args.is_empty() { format!("{} &", program) } else { - format!("{} {} &", program, args.join(" ")) + format!( + "{} {} &", + program, + args.iter() + .map(|arg| arg.to_string_lossy()) + .collect::>() + .join(" ") + ) }; // Build command with redirected stdio @@ -720,7 +666,7 @@ mod tests { #[test] fn test_external_command_with_args() { // Test command with arguments - let exit_code = execute_external("echo", &["hello".to_string()]).unwrap(); + let exit_code = execute_external("echo", &["hello".into()]).unwrap(); assert_eq!(exit_code, 0, "echo should return 0"); } @@ -756,7 +702,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["hello".to_string()]), + ("echo".to_string(), vec!["hello".into()]), ("cat".to_string(), vec![]), ("cat".to_string(), vec![]), ]; @@ -773,7 +719,7 @@ mod tests { let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); let stages = vec![ - ("echo".to_string(), vec!["test".to_string()]), + ("echo".to_string(), vec!["test".into()]), ("cat".to_string(), vec![]), ]; let redirects = vec![crate::redirection::Redirection::Output { @@ -797,7 +743,7 @@ mod tests { let temp = TempDir::new().unwrap(); let mut state = crate::state::ShellState::new(temp.path().to_str().unwrap()).unwrap(); - let stages: Vec<(String, Vec)> = vec![]; + let stages: Vec<(String, Vec)> = vec![]; let result = execute_pipeline(&stages, &[], &mut state); assert!(result.is_err(), "Empty pipeline should return error"); } diff --git a/impl/rust-cli/src/glob.rs b/impl/rust-cli/src/glob.rs index 0a5ba18..f1b6bfa 100644 --- a/impl/rust-cli/src/glob.rs +++ b/impl/rust-cli/src/glob.rs @@ -2,8 +2,73 @@ // Copyright (c) Jonathan D.A. Jewell //! Glob pattern expansion (filename wildcards) //! -//! Implements POSIX-compliant glob expansion with *, ?, [...], and {...}. -//! Glob patterns are expanded to matching file paths at command execution time. +//! Expands `*`, `?`, `[...]` and the recursive `**` against the file system, +//! with fixed limits on how much work and memory one expansion may use. +//! Brace expansion (`{a,b}`) is a separate, purely textual step: see +//! [`expand_braces`]. +//! +//! # Matching rules +//! +//! The rules follow bash 5.2 with `shopt -s globstar`, `dotglob` off and +//! `globskipdots` on (the default): +//! +//! - `*`, `?` and `[...]` match within one path component. Single components +//! are matched by the `glob` crate's [`glob::Pattern`]. +//! - A name that starts with `.` is matched only by a component that itself +//! starts with a literal `.`. So `*` skips hidden files, `.*` and `.h*` find +//! them, and `.` and `..` are never produced by a wildcard. +//! - A component that is exactly `**` matches zero or more directories. It +//! lists a symbolic link to a directory but does not descend into it, and it +//! does not enter hidden directories. Other components do go through +//! symbolic links, so `link/*` lists the target directory. +//! One measured difference: when `**` is *not* the first component (an +//! absolute pattern, `./**/*`, `d/**/*`), bash 5.2 enters a symbolic link to +//! a directory found by `**` for one more component, and vsh does not. On +//! one machine `/usr/**/*` gave bash 2,956 more paths than vsh, every one of +//! them below a link such as `/usr/bin/X11 -> .`. +//! - A pattern that ends in `**` lists everything below its directory, and the +//! directory itself when it is named (`d/**` gives `d/`, `d/e`, ...). +//! - A pattern that ends in `/` matches directories only, and each result ends +//! in `/`. +//! - Results are sorted and each path appears once. (bash lists a path again +//! for every way the pattern matches it, so `**/*/**/*` can repeat a path +//! many times.) +//! - A pattern that matches nothing gives an empty list; the caller decides +//! whether to keep the word literally. +//! +//! # Limits +//! +//! One expansion is metered on these axes ([`GlobLimits`]): +//! +//! | Axis | Kind | Default | +//! |------|------|---------| +//! | walker read units (one for each directory opened, entry listed, name looked up and symbolic link resolved; an entry type that the listing gave is not charged) | cost: adds up | 10,000,000 | +//! | bytes of one directory listing held (name length + 1 for each entry) | occupancy, per listing | 64 MiB | +//! | matches held | occupancy: high-water mark | 100,000 | +//! | bytes of matches held (path length + 1 for each match) | occupancy | 6 MiB | +//! +//! The meters are charged *while* the walk runs, so an expansion stops at the +//! first unit over a limit instead of collecting everything first. Stopping +//! returns a [`GlobLimitExceeded`] error that names the limit and carries the +//! usage so far and the directory being read; no partial result is returned. +//! Because entries are visited in sorted order, the same pattern over the same +//! tree always stops at the same place with the same report. +//! +//! The default byte limit is 6 MiB because Linux refuses any `execve` whose +//! argument strings take more than that, whatever the stack limit (measured on +//! Linux 6.18: 6,279,840 bytes accepted at both a 64 MiB and an unlimited +//! stack), so a larger expansion could never reach an external command. +//! +//! bash has no such limits: it expanded `/**/*` on a test machine to 643,683 +//! paths. The limits here are deliberate departures, made so that a single +//! word cannot exhaust the shell's memory. +//! +//! # Work bound +//! +//! Each pair (directory, pattern component) is processed at most once, so the +//! walk reads each directory at most once per component of the pattern. That +//! is the `O(|pattern| x |entries|)` shape stated as frontier item A-6 in +//! `docs/PROOF-OPEN-FRONTIER.adoc`; here it is tested, not proved. //! //! # Examples //! ```no_run @@ -14,12 +79,123 @@ //! // Returns: ["file1.txt", "file2.txt", ...] //! ``` -use anyhow::{bail, Context, Result}; +use crate::grade::{Cost, Exceeded, Meter, Occupancy}; +use anyhow::{Context, Result}; +use std::collections::{BTreeSet, HashSet}; +use std::ffi::OsString; +use std::fmt; +use std::fs; use std::path::{Path, PathBuf}; -/// Maximum number of results from glob expansion (prevents resource exhaustion from -/// patterns like `/**/*` on deep directory trees). -const MAX_GLOB_RESULTS: usize = 100_000; +/// Default limit on file-system reads in one expansion. +pub const MAX_GLOB_READS: u64 = 10_000_000; + +/// Default limit on the number of matches one expansion may hold. +pub const MAX_GLOB_RESULTS: u64 = 100_000; + +/// Default limit on the bytes of matches one expansion may hold: 6 MiB, the +/// most Linux `execve` accepts for argument strings. +pub const MAX_GLOB_MATCH_BYTES: u64 = 6 * 1024 * 1024; + +/// Default limit on the bytes of names held while one directory is listed +/// (name length + 1 for each entry): 64 MiB. +pub const MAX_GLOB_LISTING_BYTES: u64 = 64 * 1024 * 1024; + +/// The limits one glob expansion runs under. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GlobLimits { + /// Most file-system reads: directories opened, entries listed, names looked up. + pub max_reads: u64, + /// Most matches held at once. + pub max_matches: u64, + /// Most bytes of matches held at once (path length + 1 per match). + pub max_match_bytes: u64, + /// Most bytes of names held while one directory is listed (name length + 1 + /// per entry). The listing is sorted before use, so it is held whole. + pub max_listing_bytes: u64, +} + +impl GlobLimits { + /// The limits [`expand_glob`] uses. + pub const DEFAULT: GlobLimits = GlobLimits { + max_reads: MAX_GLOB_READS, + max_matches: MAX_GLOB_RESULTS, + max_match_bytes: MAX_GLOB_MATCH_BYTES, + max_listing_bytes: MAX_GLOB_LISTING_BYTES, + }; +} + +impl Default for GlobLimits { + fn default() -> Self { + GlobLimits::DEFAULT + } +} + +/// What one expansion used, on each metered axis. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct GlobUsage { + /// File-system reads: directories opened, entries listed, names looked up. + pub reads: Cost, + /// Matches held. + pub matches: Occupancy, + /// Bytes of matches held (path length + 1 per match). + pub match_bytes: Occupancy, +} + +impl fmt::Display for GlobUsage { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} file-system reads, {} matches, {} bytes of matches", + self.reads.total(), + self.matches.peak(), + self.match_bytes.peak() + ) + } +} + +/// A successful expansion: the matches and what finding them cost. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Expansion { + /// Matching paths, sorted, each once. + pub paths: Vec, + /// The resources the expansion used. + pub usage: GlobUsage, +} + +/// The error returned when an expansion reaches one of its [`GlobLimits`]. +/// +/// It is a record of where the walk stopped, not a partial result: it says +/// that more exists and where the walk was, without listing what was found. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GlobLimitExceeded { + /// The pattern as given. + pub pattern: String, + /// The limit that was reached: which axis, its value, and the level the + /// refused step would have reached. + pub exceeded: Exceeded, + /// What had been used when the walk stopped. + pub usage: GlobUsage, + /// The directory being read when the walk stopped, written the way the + /// matches would be (relative to the base directory unless the pattern is + /// absolute). + pub at: PathBuf, +} + +impl fmt::Display for GlobLimitExceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "glob pattern '{}' was not expanded: {} (stopped while reading '{}' after {})", + self.pattern, + self.exceeded, + self.at.display(), + self.usage + ) + } +} + +impl std::error::Error for GlobLimitExceeded {} /// Check if a string contains glob metacharacters /// @@ -58,16 +234,11 @@ pub fn contains_glob_pattern(s: &str) -> bool { false } -/// Expand a glob pattern to matching file paths +/// Expand a glob pattern to matching file paths, under [`GlobLimits::DEFAULT`]. /// -/// Uses the `glob` crate for POSIX-compliant pattern matching. -/// Returns paths sorted alphabetically (POSIX requirement). -/// -/// # POSIX Behavior -/// - Patterns that match nothing return empty vec (caller handles literal fallback) -/// - Hidden files (starting with .) require explicit match: .* or .config -/// - Results are sorted alphabetically -/// - * and ? do not match directory separators (/) +/// Relative patterns are matched below `base_dir` and give paths relative to +/// it; patterns that start with `/` give absolute paths. See the +/// [module documentation](self) for the matching rules and the limits. /// /// # Examples /// ```no_run @@ -85,56 +256,533 @@ pub fn contains_glob_pattern(s: &str) -> bool { /// ``` /// /// # Errors -/// Returns error if: -/// - Pattern is invalid -/// - Base directory cannot be accessed +/// Returns an error if the pattern is invalid, or a [`GlobLimitExceeded`] +/// (reachable with `anyhow::Error::downcast_ref`) if the expansion reaches a +/// limit. Directories that cannot be read are skipped, as bash does. pub fn expand_glob(pattern: &str, base_dir: &Path) -> Result> { - // Build full glob pattern (base_dir + pattern) - let full_pattern = if pattern.starts_with('/') { - // Absolute path - pattern.to_string() - } else { - // Relative to base_dir - base_dir.join(pattern).to_string_lossy().to_string() - }; + expand_glob_with_limits(pattern, base_dir, &GlobLimits::DEFAULT).map(|e| e.paths) +} - // Use glob crate for pattern matching with POSIX options - let options = glob::MatchOptions { - // POSIX: * and ? do not match leading dot (hidden files) - require_literal_leading_dot: true, - // Case-sensitive matching (POSIX default) - case_sensitive: true, - // Require literal separator - require_literal_separator: false, - }; +/// Expand a glob pattern under the given limits, reporting what it used. +/// +/// # Examples +/// ``` +/// use vsh::glob::{expand_glob_with_limits, GlobLimitExceeded, GlobLimits}; +/// +/// let dir = tempfile::tempdir().unwrap(); +/// for name in ["a.txt", "b.txt", "c.txt"] { +/// std::fs::write(dir.path().join(name), b"").unwrap(); +/// } +/// +/// let limits = GlobLimits { max_matches: 2, ..GlobLimits::DEFAULT }; +/// let err = expand_glob_with_limits("*.txt", dir.path(), &limits).unwrap_err(); +/// let stopped = err.downcast_ref::().unwrap(); +/// assert_eq!(stopped.exceeded.axis, "matches held"); +/// assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (2, 3)); +/// ``` +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_with_limits( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, +) -> Result { + expand_glob_continuing(pattern, base_dir, limits, GlobUsage::default()) +} + +/// Expand a glob pattern under limits that it shares with earlier expansions. +/// +/// The meters start from `spent`, the usage an earlier expansion reported, so +/// several patterns (the brace alternatives of one word, or the words of one +/// command) together stay within one set of [`GlobLimits`]. The returned usage +/// includes `spent`, and the matches of earlier expansions count as still held. +/// +/// # Errors +/// As for [`expand_glob`]. +pub fn expand_glob_continuing( + pattern: &str, + base_dir: &Path, + limits: &GlobLimits, + spent: GlobUsage, +) -> Result { + let compiled = + Compiled::new(pattern).with_context(|| format!("Invalid glob pattern: {}", pattern))?; + let mut walker = Walker::new(pattern, base_dir, &compiled, limits, spent); + walker + .run() + .map_err(|stopped| anyhow::Error::new(*stopped))?; + Ok(Expansion { + usage: walker.usage(), + paths: walker.found.into_iter().collect(), + }) +} + +/// One path component of a compiled pattern. +#[derive(Debug)] +enum Component { + /// No metacharacters: looked up by name, without listing the directory. + Literal(String), + /// Matched against each name in the directory. + Wild(glob::Pattern), + /// `**`: zero or more directories. + Recursive, +} + +#[derive(Debug)] +struct Compiled { + components: Vec, + absolute: bool, + dirs_only: bool, +} + +impl Compiled { + fn new(pattern: &str) -> std::result::Result { + let mut components: Vec = Vec::new(); + for part in pattern.split('/').filter(|part| !part.is_empty()) { + let component = if part == "**" { + // `**/**` matches exactly what `**` matches. + if matches!(components.last(), Some(Component::Recursive)) { + continue; + } + Component::Recursive + } else if part.contains(['*', '?', '[']) { + Component::Wild(glob::Pattern::new(part)?) + } else { + Component::Literal(part.to_string()) + }; + components.push(component); + } + Ok(Compiled { + components, + absolute: pattern.starts_with('/'), + dirs_only: pattern.len() > 1 && pattern.ends_with('/'), + }) + } +} - let mut matches: Vec = glob::glob_with(&full_pattern, options) - .with_context(|| format!("Invalid glob pattern: {}", pattern))? - .filter_map(|entry| entry.ok()) - .collect(); +/// The text a name is matched as. +/// +/// The `glob` crate matches `str` only. A byte that is not valid UTF-8 becomes +/// one character from U+F780..U+F7FF (private use), so `?` and `*` still +/// match it, and it does not look like a literal U+FFFD in the name or the +/// pattern, as a lossy conversion would make it. +// snip: a pattern that contains one of those 128 private-use characters +// matches the invalid byte too; an exact matcher needs a byte-level `Pattern`. +fn match_text(name: &std::ffi::OsStr) -> std::borrow::Cow<'_, str> { + if let Some(text) = name.to_str() { + return std::borrow::Cow::Borrowed(text); + } + let mut out = String::new(); + for chunk in name.as_encoded_bytes().utf8_chunks() { + out.push_str(chunk.valid()); + for &byte in chunk.invalid() { + // Invalid bytes are 0x80..=0xFF, so this stays in U+F780..U+F7FF. + out.push( + char::from_u32(0xF700 + u32::from(byte)).unwrap_or(char::REPLACEMENT_CHARACTER), + ); + } + } + std::borrow::Cow::Owned(out) +} - // Sort results (POSIX requirement) - matches.sort(); +/// `text` as a pattern that matches exactly `text`: `*`, `?` and `[` in it +/// are taken literally. Use it for a directory prefix that is not a pattern. +/// +/// # Examples +/// ``` +/// assert_eq!(vsh::glob::escape("/tmp/a*b"), "/tmp/a[*]b"); +/// ``` +pub fn escape(text: &str) -> String { + glob::Pattern::escape(text) +} - // Guard against resource exhaustion from overly broad patterns - if matches.len() > MAX_GLOB_RESULTS { - bail!( - "Glob pattern '{}' matched {} files (limit: {})", +const MATCH_OPTIONS: glob::MatchOptions = glob::MatchOptions { + case_sensitive: true, + require_literal_separator: true, + require_literal_leading_dot: true, +}; + +/// The result of a step of the walk. The error is boxed: it is built once, on +/// the path that stops the walk, and is larger than the values the steps return. +type Walk = std::result::Result>; + +/// The walk: a depth-first search over states (directory, component index), +/// each processed at most once. +struct Walker<'a> { + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + reads: Meter, + matches: Meter, + match_bytes: Meter, + max_listing_bytes: u64, + found: BTreeSet, +} + +impl<'a> Walker<'a> { + fn new( + pattern: &'a str, + base: &'a Path, + compiled: &'a Compiled, + limits: &GlobLimits, + spent: GlobUsage, + ) -> Self { + Walker { pattern, - matches.len(), - MAX_GLOB_RESULTS, + base, + compiled, + reads: Meter::starting_at("file-system reads", limits.max_reads, spent.reads), + matches: Meter::starting_at("matches held", limits.max_matches, spent.matches), + match_bytes: Meter::starting_at( + "bytes of matches held", + limits.max_match_bytes, + spent.match_bytes, + ), + max_listing_bytes: limits.max_listing_bytes, + found: BTreeSet::new(), + } + } + + fn usage(&self) -> GlobUsage { + GlobUsage { + reads: self.reads.grade(), + matches: self.matches.grade(), + match_bytes: self.match_bytes.grade(), + } + } + + /// Where `path` (as it would be reported) is on the file system. + fn real(&self, path: &Path) -> PathBuf { + if self.compiled.absolute { + path.to_path_buf() + } else if path.as_os_str().is_empty() { + self.base.to_path_buf() + } else { + self.base.join(path) + } + } + + fn stopped(&self, exceeded: Exceeded, at: &Path) -> Box { + Box::new(GlobLimitExceeded { + pattern: self.pattern.to_string(), + exceeded, + usage: self.usage(), + at: if at.as_os_str().is_empty() { + PathBuf::from(".") + } else { + at.to_path_buf() + }, + }) + } + + fn charge_read(&mut self, at: &Path) -> Walk<()> { + self.reads.charge(1).map_err(|e| self.stopped(e, at)) + } + + /// Record a match, once. + fn emit(&mut self, path: PathBuf, at: &Path) -> Walk<()> { + let path = if self.compiled.dirs_only { + path.join("") + } else { + path + }; + if self.found.contains(&path) { + return Ok(()); + } + let bytes = path.as_os_str().len() as u64 + 1; + // Check both limits before recording either, so a refused match leaves + // no trace in the reported usage. + self.matches.check(1).map_err(|e| self.stopped(e, at))?; + self.match_bytes + .check(bytes) + .map_err(|e| self.stopped(e, at))?; + self.matches.charge(1).map_err(|e| self.stopped(e, at))?; + self.match_bytes + .charge(bytes) + .map_err(|e| self.stopped(e, at))?; + self.found.insert(path); + Ok(()) + } + + /// List a directory in name order, charging the open and every entry. + /// An unreadable directory gives no entries. + fn list(&mut self, dir: &Path) -> Walk> { + self.charge_read(dir)?; + let mut entries = Vec::new(); + // The listing is held whole to sort it; meter the names it holds. + let mut held = Meter::::new( + "bytes of one directory listing held", + self.max_listing_bytes, ); + if let Ok(read) = fs::read_dir(self.real(dir)) { + for entry in read { + self.charge_read(dir)?; + if let Ok(entry) = entry { + if let Ok(kind) = entry.file_type() { + let name = entry.file_name(); + held.charge(name.len() as u64 + 1) + .map_err(|e| self.stopped(e, dir))?; + entries.push((name, kind)); + } + } + } + } + entries.sort_by(|a, b| a.0.cmp(&b.0)); + Ok(entries) + } + + /// Whether `path` is a directory, going through a symbolic link. + /// + /// A type that the directory listing already gave costs nothing. Looking + /// the path up (no type known, or a symbolic link to resolve) is one read, + /// charged before the lookup. + fn is_dir(&mut self, path: &Path, kind: Option, at: &Path) -> Walk { + match kind { + Some(kind) if kind.is_dir() => Ok(true), + Some(kind) if !kind.is_symlink() => Ok(false), + _ => { + self.charge_read(at)?; + Ok(fs::metadata(self.real(path)).is_ok_and(|m| m.is_dir())) + } + } } - // Convert absolute paths back to relative if needed - if !pattern.starts_with('/') { - matches = matches + fn run(&mut self) -> Walk<()> { + let count = self.compiled.components.len(); + if count == 0 { + return Ok(()); + } + let start = if self.compiled.absolute { + PathBuf::from("/") + } else { + PathBuf::new() + }; + let mut stack: Vec<(PathBuf, usize)> = vec![(start, 0)]; + let mut seen: HashSet<(PathBuf, usize)> = HashSet::new(); + + while let Some((dir, index)) = stack.pop() { + if !seen.insert((dir.clone(), index)) { + continue; + } + let last = index + 1 == count; + let compiled = self.compiled; + // Successor states, pushed in reverse so they are visited in name order. + let mut next: Vec<(PathBuf, usize)> = Vec::new(); + + match &compiled.components[index] { + Component::Literal(name) => { + let child = dir.join(name); + if last { + // One lookup. An intermediate component is charged by + // `is_dir` below, so it is not charged here as well. + self.charge_read(&dir)?; + let exists = fs::symlink_metadata(self.real(&child)).is_ok(); + if exists && (!compiled.dirs_only || self.is_dir(&child, None, &dir)?) { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, None, &dir)? { + next.push((child, index + 1)); + } + } + Component::Wild(pattern) => { + for (name, kind) in self.list(&dir)? { + if !pattern.matches_with(&match_text(&name), MATCH_OPTIONS) { + continue; + } + let child = dir.join(&name); + if last { + if !compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)? { + self.emit(child, &dir)?; + } + } else if self.is_dir(&child, Some(kind), &dir)? { + next.push((child, index + 1)); + } + } + } + Component::Recursive => { + // Zero directories: the pattern continues here. + if !last { + next.push((dir.clone(), index + 1)); + } else if !dir.as_os_str().is_empty() { + self.emit(dir.join(""), &dir)?; + } + // One more directory: descend into real, visible directories. + for (name, kind) in self.list(&dir)? { + if name.as_encoded_bytes().starts_with(b".") { + continue; + } + let child = dir.join(&name); + if last && (!compiled.dirs_only || self.is_dir(&child, Some(kind), &dir)?) { + self.emit(child.clone(), &dir)?; + } + if kind.is_dir() { + next.push((child, index)); + } + } + } + } + stack.extend(next.into_iter().rev()); + } + Ok(()) + } +} + +#[cfg(test)] +mod walk_tests { + use super::*; + use proptest::prelude::*; + use tempfile::TempDir; + + fn expand(pattern: &str, base: &Path) -> Vec { + expand_glob(pattern, base) + .unwrap() .into_iter() - .map(|p| p.strip_prefix(base_dir).unwrap_or(&p).to_path_buf()) - .collect(); + .map(|p| p.to_string_lossy().into_owned()) + .collect() + } + + #[test] + fn literal_and_wild_components() { + let t = TempDir::new().unwrap(); + fs::create_dir_all(t.path().join("src/sub")).unwrap(); + for f in ["src/a.rs", "src/b.txt", "src/sub/c.rs", "top.rs"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!(expand("src/*.rs", t.path()), ["src/a.rs"]); + assert_eq!(expand("*/*/*.rs", t.path()), ["src/sub/c.rs"]); + assert_eq!(expand("./*.rs", t.path()), ["./top.rs"]); + assert_eq!( + expand("**/*.rs", t.path()), + ["src/a.rs", "src/sub/c.rs", "top.rs"] + ); + assert_eq!(expand("src/**/c.rs", t.path()), ["src/sub/c.rs"]); + assert!(expand("*.none", t.path()).is_empty()); + } + + #[test] + fn intermediate_literal_is_charged_once() { + // `real/file` needs two lookups: `real`, then `file`. A budget of two + // reads must be enough, and a budget of one must stop at the second. + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + fs::write(t.path().join("real/file"), b"").unwrap(); + + let two = GlobLimits { + max_reads: 2, + ..GlobLimits::DEFAULT + }; + let found = expand_glob_with_limits("real/file", t.path(), &two).unwrap(); + assert_eq!(found.paths, [PathBuf::from("real/file")]); + + let one = GlobLimits { + max_reads: 1, + ..GlobLimits::DEFAULT + }; + let err = expand_glob_with_limits("real/file", t.path(), &one).unwrap_err(); + assert!(err.downcast_ref::().is_some()); + } + + #[test] + fn absolute_patterns_give_absolute_paths() { + let t = TempDir::new().unwrap(); + fs::write(t.path().join("x.txt"), b"").unwrap(); + let pattern = format!("{}/*.txt", escape(&t.path().to_string_lossy())); + assert_eq!( + expand_glob(&pattern, Path::new("/nonexistent")).unwrap(), + [t.path().join("x.txt")] + ); + } + + #[test] + fn invalid_patterns_are_errors_not_limits() { + let t = TempDir::new().unwrap(); + for bad in ["a**", "***", "[z-"] { + let err = expand_glob(bad, t.path()).unwrap_err(); + assert!(err.downcast_ref::().is_none(), "{bad}"); + } } - Ok(matches) + #[test] + fn a_base_directory_with_metacharacters_is_taken_literally() { + let t = TempDir::new().unwrap(); + let base = t.path().join("we[ir]d*"); + fs::create_dir(&base).unwrap(); + fs::write(base.join("f.txt"), b"").unwrap(); + assert_eq!(expand("*.txt", &base), ["f.txt"]); + } + + #[test] + fn empty_pattern_matches_nothing() { + let t = TempDir::new().unwrap(); + assert!(expand("", t.path()).is_empty()); + } + + // ------------------------------------------------------------------ + // Differential check against the `glob` crate, which this walker + // replaces. On trees without symbolic links and for patterns that do not + // end in `**` and have no component starting with `.`, the two must find + // the same set of paths (the crate can list a path twice; this walker + // lists it once). Those exclusions are exactly where the rules changed. + // ------------------------------------------------------------------ + + fn crate_glob(pattern: &str, base: &Path) -> BTreeSet { + let options = glob::MatchOptions { + require_literal_leading_dot: true, + case_sensitive: true, + require_literal_separator: false, + }; + let base_pattern = glob::Pattern::escape(&base.to_string_lossy()); + glob::glob_with(&format!("{base_pattern}/{pattern}"), options) + .unwrap() + .filter_map(|p| p.ok()) + .map(|p| p.strip_prefix(base).unwrap().to_path_buf()) + .collect() + } + + fn tree() -> impl Strategy, bool)>> { + let name = prop::sample::select(vec!["a", "b", "ab", "ba", ".h"]); + prop::collection::vec((prop::collection::vec(name, 1..4), any::()), 0..25) + } + + fn pattern() -> impl Strategy { + let component = prop::sample::select(vec![ + "*", "?", "a*", "*b", "[ab]", "[!a]*", "a", "b", "ab", "**", + ]); + (prop::collection::vec(component, 1..5), any::()).prop_filter_map( + "pattern ends in **", + |(parts, slash)| { + if parts.last() == Some(&"**") { + return None; + } + let mut p = parts.join("/"); + if slash { + p.push('/'); + } + Some(p) + }, + ) + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(96))] + #[test] + fn agrees_with_the_glob_crate(entries in tree(), pattern in pattern()) { + let t = TempDir::new().unwrap(); + for (parts, is_dir) in &entries { + let path = parts.iter().collect::(); + let full = t.path().join(&path); + if let Some(parent) = full.parent() { + if fs::create_dir_all(parent).is_err() { + continue; + } + } + let _ = if *is_dir { fs::create_dir_all(&full) } else { fs::write(&full, b"") }; + } + let ours: BTreeSet = expand_glob(&pattern, t.path()).unwrap().into_iter().collect(); + prop_assert_eq!(ours, crate_glob(&pattern, t.path()), "pattern {}", pattern); + } + } } /// Expand brace patterns: a{b,c}d -> [abd, acd] @@ -170,7 +818,8 @@ fn expand_braces_limited(pattern: &str, remaining: usize) -> Vec { } // Find first unescaped { - let chars = pattern.chars().enumerate().peekable(); + // Byte offsets, not character counts: the slices below index bytes. + let chars = pattern.char_indices(); let mut brace_start = None; let mut brace_depth = 0; let mut escaped = false; diff --git a/impl/rust-cli/src/grade.rs b/impl/rust-cli/src/grade.rs new file mode 100644 index 0000000..66fc01e --- /dev/null +++ b/impl/rust-cli/src/grade.rs @@ -0,0 +1,556 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Resource grades: deterministic accounting for bounded work and bounded holdings. +//! +//! A shell operation can use a resource in two different ways, and the two +//! compose differently: +//! +//! - **Cost** is consumed and never given back: directory entries read, +//! comparisons made. Doing one thing and then another costs the *sum*. +//! - **Occupancy** is held for a while and may be released: paths kept in +//! memory, history entries retained. What matters is the *high-water mark*, +//! the most that was held at one time. +//! +//! A [`Grade`] is a value of one of those two kinds. A [`Meter`] measures a +//! grade while the work runs and refuses a charge that would pass its limit, +//! so a limit is enforced where the resource is used and not after the fact. +//! +//! Both kinds are plain counts. They depend only on the input and on the state +//! of the file system, never on the clock or on the machine, so a test can +//! compute the expected grade independently and compare it for equality. +//! +//! # Laws +//! +//! For both kinds, [`Grade::then`] is associative with [`Grade::ZERO`] as its +//! identity, [`Grade::either`] is associative, commutative and idempotent, and +//! `then` distributes over `either`. Metering a sequence of charges gives +//! exactly the grade obtained by composing those charges with `then`. The tests +//! at the end of this file check these laws on random inputs; they are tests, +//! not proofs. +//! +//! # Relation to the estate's type theories +//! +//! The split follows the vocabulary of `tropical-types` (cost: sequence adds, +//! choice takes the maximum) and of occupancy grades (high-water marks) as +//! summarised in `docs/THEORY-FEED.adoc`. Nothing is imported from those +//! repositories and no theorem of theirs is relied on here. +//! +//! # Examples +//! +//! ``` +//! use vsh::grade::{Cost, Grade, Meter, Occupancy}; +//! +//! // Cost adds up. +//! assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); +//! +//! // Occupancy keeps what is still held and remembers the peak. +//! let held = Occupancy::of(5).then(Occupancy::of(2)); +//! assert_eq!((held.held(), held.peak()), (7, 7)); +//! assert_eq!(held.released(6).peak(), 7); +//! +//! // A meter refuses the charge that would pass its limit. +//! let mut entries: Meter = Meter::new("directory entries read", 2); +//! assert!(entries.charge(2).is_ok()); +//! let refused = entries.charge(1).unwrap_err(); +//! assert_eq!((refused.limit, refused.level), (2, 3)); +//! assert_eq!(entries.level(), 2); // the refused charge was not recorded +//! ``` + +use std::fmt; + +/// A summary of how much of one resource a piece of work used. +/// +/// See the [module documentation](self) for the two kinds and their laws. +pub trait Grade: Copy + Eq + fmt::Debug { + /// The grade of doing nothing. + const ZERO: Self; + + /// The grade of using `amount` units in a single step. + fn of(amount: u64) -> Self; + + /// Sequential composition: this work, and then `next`. + /// + /// `next` starts from whatever this work still holds. + #[must_use] + fn then(self, next: Self) -> Self; + + /// The worst case of two alternatives, of which only one happens. + #[must_use] + fn either(self, other: Self) -> Self; + + /// The single number a limit is compared against. + fn level(self) -> u64; + + /// What a further `of(amount)` is added to: everything spent for a cost, + /// what is still held for an occupancy. + fn base(self) -> u64; +} + +/// A cumulative cost: a resource that is consumed and never given back. +/// +/// Sequential composition adds; the worst case of two alternatives is the +/// larger. Addition saturates at `u64::MAX` instead of wrapping. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct Cost(u64); + +impl Cost { + /// The total consumed. + pub const fn total(self) -> u64 { + self.0 + } +} + +impl Grade for Cost { + const ZERO: Self = Cost(0); + + fn of(amount: u64) -> Self { + Cost(amount) + } + + fn then(self, next: Self) -> Self { + Cost(self.0.saturating_add(next.0)) + } + + fn either(self, other: Self) -> Self { + Cost(self.0.max(other.0)) + } + + fn level(self) -> u64 { + self.0 + } + + fn base(self) -> u64 { + self.0 + } +} + +impl fmt::Display for Cost { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.0) + } +} + +/// An occupancy: how much of a resource is held now, and the most that was +/// ever held at one time. +/// +/// Invariant: `peak >= held`. Every constructor and operation preserves it. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct Occupancy { + held: u64, + peak: u64, +} + +impl Occupancy { + /// How much is held at the end of the work. + pub const fn held(self) -> u64 { + self.held + } + + /// The high-water mark: the most that was held at one time. + pub const fn peak(self) -> u64 { + self.peak + } + + /// The same work after giving back `amount` units of what it holds. + /// + /// Releasing more than is held releases everything. The peak is unchanged: + /// a release cannot undo a high-water mark. + #[must_use] + pub fn released(self, amount: u64) -> Self { + Occupancy { + held: self.held.saturating_sub(amount), + peak: self.peak, + } + } + + /// The same work after giving back everything it holds. + #[must_use] + pub fn settled(self) -> Self { + self.released(self.held) + } +} + +impl Grade for Occupancy { + const ZERO: Self = Occupancy { held: 0, peak: 0 }; + + fn of(amount: u64) -> Self { + Occupancy { + held: amount, + peak: amount, + } + } + + /// `next` runs on top of what this work still holds, so its peak is + /// measured from there. + fn then(self, next: Self) -> Self { + Occupancy { + held: self.held.saturating_add(next.held), + peak: self.peak.max(self.held.saturating_add(next.peak)), + } + } + + fn either(self, other: Self) -> Self { + Occupancy { + held: self.held.max(other.held), + peak: self.peak.max(other.peak), + } + } + + fn level(self) -> u64 { + self.peak + } + + fn base(self) -> u64 { + self.held + } +} + +impl fmt::Display for Occupancy { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{} (peak {})", self.held, self.peak) + } +} + +/// A charge that a [`Meter`] refused because it would pass the limit. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Exceeded { + /// What was being counted, for example `"directory entries read"`. + pub axis: &'static str, + /// The limit that was configured. + pub limit: u64, + /// The level the refused charge would have reached. Wider than a limit, + /// so a charge that passes `u64::MAX` is reported exactly, not saturated. + pub level: u128, +} + +impl fmt::Display for Exceeded { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "{} would reach {}, over the limit of {}", + self.axis, self.level, self.limit + ) + } +} + +impl std::error::Error for Exceeded {} + +/// Measures one grade against a limit while the work runs. +/// +/// Invariant: `level() <= limit()`. A charge that would break it is refused +/// and leaves the meter unchanged. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Meter { + axis: &'static str, + limit: u64, + grade: G, +} + +impl Meter { + /// A meter at zero for the resource named `axis`, allowing at most `limit`. + pub const fn new(axis: &'static str, limit: u64) -> Self { + Meter { + axis, + limit, + grade: G::ZERO, + } + } + + /// A meter for `axis` that starts from work already measured as `spent`, + /// so that several pieces of work can share one limit. + pub const fn starting_at(axis: &'static str, limit: u64, spent: G) -> Self { + Meter { + axis, + limit, + grade: spent, + } + } + + /// Whether [`Meter::charge`] would accept `amount`, without recording it. + /// + /// # Errors + /// Returns the [`Exceeded`] that the charge would return. + pub fn check(&self, amount: u64) -> Result<(), Exceeded> { + // Computed in u128 before the grades are composed: composition + // saturates, and a saturated level would pass a limit of `u64::MAX`. + let level = + u128::from(self.grade.level()).max(u128::from(self.grade.base()) + u128::from(amount)); + if level > u128::from(self.limit) { + return Err(Exceeded { + axis: self.axis, + limit: self.limit, + level, + }); + } + Ok(()) + } + + /// Record the use of `amount` more units, or refuse if that passes the limit. + /// + /// # Errors + /// Returns [`Exceeded`] and records nothing when the charge would take the + /// level over the limit. + pub fn charge(&mut self, amount: u64) -> Result<(), Exceeded> { + self.check(amount)?; + self.grade = self.grade.then(G::of(amount)); + Ok(()) + } + + /// The grade measured so far. + pub fn grade(&self) -> G { + self.grade + } + + /// The level measured so far; never more than [`Meter::limit`]. + pub fn level(&self) -> u64 { + self.grade.level() + } + + /// The configured limit. + pub fn limit(&self) -> u64 { + self.limit + } + + /// What this meter counts. + pub fn axis(&self) -> &'static str { + self.axis + } +} + +impl Meter { + /// Record that `amount` units are no longer held. + /// + /// Only an occupancy can be released. There is deliberately no such method + /// on `Meter`: a cost that has been paid cannot be taken back. + pub fn release(&mut self, amount: u64) { + self.grade = self.grade.released(amount); + } + + /// How much is held right now. + pub fn held(&self) -> u64 { + self.grade.held() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use proptest::prelude::*; + + #[test] + fn cost_adds_in_sequence_and_takes_the_larger_alternative() { + assert_eq!(Cost::of(3).then(Cost::of(4)), Cost::of(7)); + assert_eq!(Cost::of(3).either(Cost::of(4)), Cost::of(4)); + assert_eq!(Cost::ZERO.total(), 0); + } + + #[test] + fn cost_saturates_instead_of_wrapping() { + assert_eq!(Cost::of(u64::MAX).then(Cost::of(1)), Cost::of(u64::MAX)); + } + + #[test] + fn occupancy_remembers_the_peak_after_a_release() { + let work = Occupancy::of(10).released(8).then(Occupancy::of(3)); + assert_eq!(work.held(), 5); + assert_eq!(work.peak(), 10); + } + + #[test] + fn occupancy_of_two_settled_phases_is_the_larger_peak() { + let first = Occupancy::of(10).settled(); + let second = Occupancy::of(4).settled(); + let both = first.then(second); + assert_eq!((both.held(), both.peak()), (0, 10)); + } + + #[test] + fn occupancy_of_two_retained_phases_adds() { + let both = Occupancy::of(10).then(Occupancy::of(4)); + assert_eq!((both.held(), both.peak()), (14, 14)); + } + + #[test] + fn meter_accepts_up_to_the_limit_and_refuses_the_next_unit() { + let mut meter: Meter = Meter::new("steps", 3); + assert!(meter.charge(3).is_ok()); + assert_eq!(meter.level(), 3); + + let refused = meter.charge(1).unwrap_err(); + assert_eq!( + refused, + Exceeded { + axis: "steps", + limit: 3, + level: 4 + } + ); + assert_eq!(meter.level(), 3, "a refused charge must not be recorded"); + assert_eq!( + refused.to_string(), + "steps would reach 4, over the limit of 3" + ); + } + + #[test] + fn occupancy_meter_can_reuse_what_was_released() { + let mut meter: Meter = Meter::new("items held", 10); + meter.charge(10).unwrap(); + assert!(meter.charge(1).is_err()); + + meter.release(4); + assert_eq!(meter.held(), 6); + assert!(meter.charge(4).is_ok(), "released room can be used again"); + assert!(meter.charge(1).is_err()); + assert_eq!(meter.grade().peak(), 10); + } + + #[test] + fn a_meter_can_start_from_work_already_spent() { + let mut meter: Meter = Meter::starting_at("steps", 10, Cost::of(8)); + assert!(meter.check(2).is_ok()); + assert_eq!(meter.charge(3).unwrap_err().level, 11); + assert_eq!( + meter.level(), + 8, + "check and a refused charge record nothing" + ); + } + + #[test] + fn a_charge_past_u64_max_is_refused_and_reported_exactly() { + // A saturated level would equal the limit and be accepted. + let mut cost = Meter::::new("units", u64::MAX); + cost.charge(u64::MAX - 1).unwrap(); + cost.charge(1).unwrap(); // reaches the limit exactly + let refused = cost.charge(5).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 5); + assert_eq!(cost.level(), u64::MAX); + + let mut held = Meter::::new("held", u64::MAX); + held.charge(u64::MAX).unwrap(); + let refused = held.charge(1).unwrap_err(); + assert_eq!(refused.level, u128::from(u64::MAX) + 1); + } + + #[test] + fn a_meter_with_a_zero_limit_refuses_any_use() { + let mut meter: Meter = Meter::new("items held", 0); + assert!(meter.charge(0).is_ok()); + assert!(meter.charge(1).is_err()); + } + + /// A well-formed occupancy: `peak >= held`. + fn occupancy() -> impl Strategy { + (any::(), any::()).prop_map(|(held, extra)| Occupancy { + held: u64::from(held), + peak: u64::from(held) + u64::from(extra), + }) + } + + fn cost() -> impl Strategy { + any::().prop_map(Cost::of) + } + + /// The laws stated in the module documentation, for any grade. + fn check_laws(a: G, b: G, c: G) -> Result<(), TestCaseError> { + // `then` is a monoid. + prop_assert_eq!(a.then(b).then(c), a.then(b.then(c))); + prop_assert_eq!(G::ZERO.then(a), a); + prop_assert_eq!(a.then(G::ZERO), a); + // `either` is a semilattice. + prop_assert_eq!(a.either(b).either(c), a.either(b.either(c))); + prop_assert_eq!(a.either(b), b.either(a)); + prop_assert_eq!(a.either(a), a); + // `then` distributes over `either` on both sides. + prop_assert_eq!(a.then(b.either(c)), a.then(b).either(a.then(c))); + prop_assert_eq!(a.either(b).then(c), a.then(c).either(b.then(c))); + // Doing more never lowers the level. + prop_assert!(a.then(b).level() >= a.level()); + prop_assert!(a.either(b).level() >= a.level()); + Ok(()) + } + + proptest! { + #[test] + fn cost_obeys_the_grade_laws(a in cost(), b in cost(), c in cost()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_obeys_the_grade_laws(a in occupancy(), b in occupancy(), c in occupancy()) { + check_laws(a, b, c)?; + } + + #[test] + fn occupancy_operations_keep_peak_at_least_held( + a in occupancy(), + b in occupancy(), + amount in any::(), + ) { + for grade in [a.then(b), a.either(b), a.released(amount), a.settled()] { + prop_assert!(grade.peak() >= grade.held()); + } + prop_assert_eq!(a.released(amount).peak(), a.peak()); + prop_assert_eq!(a.settled().held(), 0); + } + + /// Metering charges one by one gives exactly the grade obtained by + /// composing them, and never passes the limit. + #[test] + fn a_cost_meter_agrees_with_composition( + charges in prop::collection::vec(0u64..1_000, 0..40), + limit in 0u64..20_000, + ) { + let mut meter: Meter = Meter::new("steps", limit); + let mut composed = Cost::ZERO; + for amount in charges { + let wanted = composed.then(Cost::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, u128::from(wanted.level())); + prop_assert_eq!(refused.limit, limit); + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + } + } + + /// The same for an occupancy meter, with releases mixed in. + #[test] + fn an_occupancy_meter_agrees_with_composition( + steps in prop::collection::vec((any::(), 0u64..1_000), 0..40), + limit in 0u64..5_000, + ) { + let mut meter: Meter = Meter::new("items held", limit); + let mut composed = Occupancy::ZERO; + for (is_release, amount) in steps { + if is_release { + meter.release(amount); + composed = composed.released(amount); + } else { + let wanted = composed.then(Occupancy::of(amount)); + match meter.charge(amount) { + Ok(()) => { + prop_assert!(wanted.level() <= limit); + composed = wanted; + } + Err(refused) => { + prop_assert!(wanted.level() > limit); + prop_assert_eq!(refused.level, u128::from(wanted.level())); + } + } + } + prop_assert_eq!(meter.grade(), composed); + prop_assert!(meter.level() <= meter.limit()); + prop_assert!(meter.held() <= meter.level()); + } + } + } +} diff --git a/impl/rust-cli/src/lib.rs b/impl/rust-cli/src/lib.rs index 2323f8d..e74b4b1 100644 --- a/impl/rust-cli/src/lib.rs +++ b/impl/rust-cli/src/lib.rs @@ -21,6 +21,7 @@ pub mod friendly_errors; pub mod fs_pure; pub mod functions; pub mod glob; +pub mod grade; pub mod help; pub mod highlighter; pub mod history; @@ -39,6 +40,7 @@ pub mod state; pub mod test_command; pub mod verification; +pub mod word; /// Echidna property-based verification integration (optional feature) #[cfg(feature = "echidna")] diff --git a/impl/rust-cli/src/parser.rs b/impl/rust-cli/src/parser.rs index da677ce..2b05438 100644 --- a/impl/rust-cli/src/parser.rs +++ b/impl/rust-cli/src/parser.rs @@ -9,6 +9,7 @@ use anyhow::{anyhow, Context, Result}; use crate::redirection::Redirection; +use crate::word::{Quote, Word}; /// Quote type for a word or word part #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -82,6 +83,8 @@ struct ParameterExpansion { struct QuotedWord { parts: Vec, quote_type: QuoteType, + /// The same word with the quoting of each piece kept (see `crate::word`). + typed: Word, } impl QuotedWord { @@ -89,6 +92,7 @@ impl QuotedWord { Self { parts: Vec::new(), quote_type: QuoteType::None, + typed: Word::new(), } } @@ -102,19 +106,23 @@ impl QuotedWord { } } - fn push_variable(&mut self, name: String) { + fn push_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), false, quote); self.parts.push(WordPart::Variable(name)); } - fn push_braced_variable(&mut self, name: String) { + fn push_braced_variable(&mut self, name: String, quote: Quote) { + self.typed.push_parameter(name.clone(), true, quote); self.parts.push(WordPart::BracedVariable(name)); } - fn push_command_sub(&mut self, cmd: String) { + fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.typed.push_command_sub(cmd.clone(), quote); self.parts.push(WordPart::CommandSub(cmd)); } fn push_process_sub(&mut self, sub_type: ProcessSubType, cmd: String) { + self.typed.push_process_sub(sub_type, cmd.clone()); self.parts.push(WordPart::ProcessSub(sub_type, cmd)); } } @@ -285,7 +293,8 @@ pub enum Command { // Shell builtins /// Echo: write arguments to stdout Echo { - args: Vec, + /// Arguments after the options, with their quoting. + args: Vec, no_newline: bool, interpret_escapes: bool, redirects: Vec, @@ -335,7 +344,8 @@ pub enum Command { // External command External { program: String, - args: Vec, + /// Arguments with their quoting; expanded by [`crate::word::expand_words`]. + args: Vec, redirects: Vec, background: bool, }, @@ -345,7 +355,7 @@ pub enum Command { /// Each stage is a (program, args) pair. Intermediate stages use piped stdio. /// Final redirections apply to the last stage only. Pipeline { - stages: Vec<(String, Vec)>, + stages: Vec<(String, Vec)>, redirects: Vec, background: bool, }, @@ -599,6 +609,7 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Backslash); quote_state = QuoteState::None; } @@ -609,6 +620,7 @@ fn tokenize(input: &str) -> Result> { quote_state = QuoteState::None; } else { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Single); } } @@ -631,27 +643,31 @@ fn tokenize(input: &str) -> Result> { current_literal.push('\\'); } current_literal.push(next_ch); + current_word.typed.push_char(next_ch, Quote::Double); } else { // Not a special char, keep backslash current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } else { current_literal.push('\\'); + current_word.typed.push_char('\\', Quote::Double); } } '$' => { // Variable expansion in double quotes push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::Double)?; } '`' => { // Backtick command substitution in double quotes push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::Double); } _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::Double); } } } @@ -678,14 +694,14 @@ fn tokenize(input: &str) -> Result> { // Variable expansion '$' => { push_literal!(); - parse_variable(&mut chars, &mut current_word)?; + parse_variable(&mut chars, &mut current_word, Quote::None)?; } // Backtick command substitution '`' => { push_literal!(); let cmd = parse_command_sub_backtick(&mut chars)?; - current_word.push_command_sub(cmd); + current_word.push_command_sub(cmd, Quote::None); } // Whitespace: end current word @@ -766,6 +782,7 @@ fn tokenize(input: &str) -> Result> { } else { // Invalid: 2>&[not 1] current_literal.push_str("2>&"); + current_word.typed.push_str("2>&", Quote::None); } } _ => { @@ -775,6 +792,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '2' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -816,6 +834,7 @@ fn tokenize(input: &str) -> Result> { } else { // Regular '[' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } @@ -828,12 +847,14 @@ fn tokenize(input: &str) -> Result> { } else { // Regular ']' character, part of word current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } // Regular character _ => { current_literal.push(ch); + current_word.typed.push_char(ch, Quote::None); } } } @@ -861,12 +882,13 @@ fn tokenize(input: &str) -> Result> { fn parse_variable( chars: &mut std::iter::Peekable, word: &mut QuotedWord, + quote: Quote, ) -> Result<()> { if chars.peek() == Some(&'(') { // Command substitution: $(cmd) chars.next(); // consume '(' let cmd = parse_command_sub_dollar(chars)?; - word.push_command_sub(cmd); + word.push_command_sub(cmd, quote); } else if chars.peek() == Some(&'{') { // Braced form: ${VAR} chars.next(); // consume '{' @@ -888,7 +910,7 @@ fn parse_variable( } } - word.push_braced_variable(var_name); + word.push_braced_variable(var_name, quote); } else if let Some(&next_ch) = chars.peek() { // Simple form: $VAR or special variables. The peek above already // bound next_ch by value; chars.next() is called purely to advance @@ -897,11 +919,11 @@ fn parse_variable( if next_ch == '?' || next_ch == '$' || next_ch == '#' { // Single-character special variable chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_ascii_digit() { // Positional parameter: $0, $1, $2, etc. chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else if next_ch.is_alphabetic() || next_ch == '_' { // Variable name let mut var_name = String::new(); @@ -915,18 +937,20 @@ fn parse_variable( } } - word.push_variable(var_name); + word.push_variable(var_name, quote); } else if next_ch == '@' || next_ch == '*' { // Special positional parameters: $@ or $* chars.next(); - word.push_variable(next_ch.to_string()); + word.push_variable(next_ch.to_string(), quote); } else { // $ not followed by variable, treat as literal word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } } else { // $ at end of string word.push_literal("$".to_string()); + word.typed.push_char('$', quote); } Ok(()) @@ -935,21 +959,54 @@ fn parse_variable( /// Parse command substitution in $(cmd) form fn parse_command_sub_dollar(chars: &mut std::iter::Peekable) -> Result { let mut cmd = String::new(); - let mut depth = 1; // Track nesting depth for nested $() + let mut depth = 1; // Parenthesis depth: nested $(), $((expr)) and (...) + // A parenthesis that is quoted or escaped is text, not nesting. + let mut single = false; + let mut double = false; + let mut escaped = false; - for ch in chars.by_ref() { + // A nested $() has its own quotes, even inside outer double quotes. + let mut contexts = Vec::new(); + while let Some(ch) = chars.next() { + if escaped { + escaped = false; + cmd.push(ch); + continue; + } match ch { + '\\' if !single => { + escaped = true; + cmd.push(ch); + } + '\'' if !double => { + single = !single; + cmd.push(ch); + } + '"' if !single => { + double = !double; + cmd.push(ch); + } + '$' if !single && chars.peek() == Some(&'(') => { + chars.next(); + cmd.push_str("$("); + contexts.push((depth, single, double)); + (depth, single, double) = (1, false, false); + } + _ if single || double => cmd.push(ch), '(' => { - // Check if it's $( to track nested command substitution - if cmd.ends_with('$') { - depth += 1; - } + // Every `(` nests, so the `)` that closes `$((1+2))` or + // `$(echo $(pwd))` belongs to the substitution. + depth += 1; cmd.push(ch); } ')' => { depth -= 1; if depth == 0 { - return Ok(cmd); + if let Some(context) = contexts.pop() { + (depth, single, double) = context; + } else { + return Ok(cmd); + } } cmd.push(ch); } @@ -1158,7 +1215,9 @@ fn tokens_to_string(tokens: &[Token]) -> Result { } match token { - Token::Word(w) => result.push_str("ed_word_to_string(w)), + // The typed word keeps the quoting of each piece, so `'*'x.*` + // comes back with only its unquoted part active. + Token::Word(w) => result.push_str(&w.typed.to_string()), Token::OutputRedirect => result.push('>'), Token::AppendRedirect => result.push_str(">>"), Token::InputRedirect => result.push('<'), @@ -1228,28 +1287,28 @@ fn parse_pipeline(tokens: &[Token]) -> Result { stages[last_stage_idx] = last_command_tokens; // Parse each stage as (program, args) - let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); + let mut parsed_stages: Vec<(String, Vec)> = Vec::new(); for stage_tokens in stages { if stage_tokens.is_empty() { return Err(anyhow!("Empty pipeline stage")); } - let words: Vec = stage_tokens + let words: Vec<&QuotedWord> = stage_tokens .iter() .filter_map(|t| match t { - Token::Word(w) => Some(quoted_word_to_string(w)), + Token::Word(w) => Some(w), _ => None, }) .collect(); - if words.is_empty() { + let Some((program, args)) = words.split_first() else { return Err(anyhow!("Pipeline stage has no command")); - } - - let program = words[0].clone(); - let args = words[1..].to_vec(); - parsed_stages.push((program, args)); + }; + parsed_stages.push(( + quoted_word_to_string(program), + args.iter().map(|w| w.typed.clone()).collect(), + )); } Ok(Command::Pipeline { @@ -1533,7 +1592,7 @@ fn split_on_top_level(input: &str, split_on_newline: bool) -> Vec<&str> { } _ if !in_single_quote && !in_double_quote && paren_depth == 0 // Check for keyword at word boundary - && (ch.is_alphabetic() || ch == '_') => + && (ch.is_ascii_alphabetic() || ch == '_') => { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { @@ -1719,6 +1778,7 @@ pub fn parse_command(input: &str) -> Result { // Separate command tokens from redirections let mut command_tokens: Vec = Vec::new(); + let mut command_words: Vec = Vec::new(); let mut redirects = Vec::new(); let mut background = false; @@ -1727,6 +1787,7 @@ pub fn parse_command(input: &str) -> Result { match &tokens[i] { Token::Word(w) => { command_tokens.push(quoted_word_to_string(w)); + command_words.push(w.typed.clone()); i += 1; } @@ -1852,9 +1913,10 @@ pub fn parse_command(input: &str) -> Result { // Parse command from tokens let cmd = &command_tokens[0]; let args: Vec = command_tokens[1..].to_vec(); + let words: Vec = command_words[1..].to_vec(); // Parse base command with redirections - parse_base_command(cmd, args, redirects, background) + parse_base_command(cmd, args, words, redirects, background) } /// Check if a string is a valid variable name @@ -2113,12 +2175,12 @@ fn parse_parameter_expansion(content: &str) -> Result = content.chars().collect(); - #[allow(unused_assignments)] - let mut pos = 0; + // Offsets are bytes. Every operator is one ASCII byte, so a `+ 1` step + // stays on a character boundary whatever the name contains. + let bytes = content.as_bytes(); // Check for length operator: ${#VAR} - if chars[0] == '#' { + if bytes[0] == b'#' { let var_name = content[1..].to_string(); return Ok(ParameterExpansion { var_name, @@ -2127,28 +2189,27 @@ fn parse_parameter_expansion(content: &str) -> Result= chars.len() { + if pos >= content.len() { // ${VAR:} with nothing after - try substring with offset 0 return Ok(ParameterExpansion { var_name, @@ -2159,8 +2220,10 @@ fn parse_parameter_expansion(content: &str) -> Result) -> String { match length { Some(n) => { - let end = (start + n).min(chars.len()); + let end = start.saturating_add(n).min(chars.len()); chars[start..end].iter().collect() } None => chars[start..].iter().collect(), @@ -2369,7 +2432,7 @@ fn apply_substring(value: &str, offset: i32, length: Option) -> String { /// /// `~user` is not yet supported (requires getpwnam). An escaped `\~` /// (produced by `quoted_word_to_string` for quoted `~`) stays literal. -fn expand_tilde(input: &str) -> String { +pub(crate) fn expand_tilde(input: &str) -> String { // An escaped tilde `\~` means the tilde was inside quotes — leave it. if let Some(stripped) = input.strip_prefix("\\~") { let mut s = String::with_capacity(input.len()); @@ -2548,30 +2611,17 @@ fn quoted_word_to_string(word: &QuotedWord) -> String { result.push_str(s); } } - WordPart::Variable(name) => { - // This shouldn't happen in single quotes (tokenizer prevents it) - // but handle it anyway - if word.quote_type != QuoteType::Single { + // The tokenizer makes a variable part only outside single quotes, + // so it is always an expansion, whatever quotes follow in the word. + // Braces keep a following literal out of the name (`$A''b`). + WordPart::Variable(name) | WordPart::BracedVariable(name) => { + if matches!(name.as_str(), "?" | "$" | "#" | "@" | "*") { result.push('$'); result.push_str(name); } else { - // In single quotes, escape the $ - result.push('\\'); - result.push('$'); - result.push_str(name); - } - } - WordPart::BracedVariable(name) => { - // This shouldn't happen in single quotes - if word.quote_type != QuoteType::Single { result.push_str("${"); result.push_str(name); result.push('}'); - } else { - // In single quotes, escape the $ - result.push_str("\\${"); - result.push_str(name); - result.push('}'); } } WordPart::CommandSub(cmd) => { @@ -2628,14 +2678,12 @@ pub fn expand_command_substitution( process_cmd.stdout(Stdio::piped()); process_cmd.stderr(Stdio::null()); - // Expand variables and command substitutions in args before execution - let expanded_args: Result> = args - .iter() - .map(|arg| expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; + // Expand the arguments (variables, command substitution, globs) + // before execution. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; - process_cmd.args(&expanded_args); + process_cmd.args(&expansion.fields); let output_result = process_cmd .output() @@ -2651,6 +2699,22 @@ pub fn expand_command_substitution( } } + Command::Echo { + args, + no_newline, + interpret_escapes: false, + redirects, + } if redirects.is_empty() => { + // echo builtin: its output is the joined, expanded arguments. + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + let mut out = expansion.lossy_fields().join(" "); + if !no_newline { + out.push('\n'); + } + out + } + Command::Pwd { .. } => { // pwd builtin std::env::current_dir() @@ -2816,7 +2880,10 @@ fn split_control_keywords<'a>(input: &'a str, keywords: &[&str]) -> Vec<(&'a str in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; @@ -3220,7 +3287,10 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { in_double_quote = !in_double_quote; i += 1; } - _ if !in_single_quote && !in_double_quote && (ch.is_alphabetic() || ch == '_') => { + _ if !in_single_quote + && !in_double_quote + && (ch.is_ascii_alphabetic() || ch == '_') => + { let word_start = i; while i < bytes.len() && (bytes[i].is_ascii_alphanumeric() || bytes[i] == b'_') { i += 1; @@ -3253,9 +3323,13 @@ pub fn is_incomplete_control_structure(input: &str) -> bool { block_depth > 0 } +/// `args` are the arguments in the older string encoding, which the builtins +/// still read; `words` are the same arguments with their quoting, used by +/// `echo` and external commands. The two lists have the same length. fn parse_base_command( cmd: &str, args: Vec, + words: Vec, redirects: Vec, background: bool, ) -> Result { @@ -3382,7 +3456,8 @@ fn parse_base_command( // Re-parse the inner command: first arg is the command name, rest are its args let inner_cmd = &args[0]; let inner_args = args[1..].to_vec(); - let inner = parse_base_command(inner_cmd, inner_args, vec![], false)?; + let inner_words = words[1..].to_vec(); + let inner = parse_base_command(inner_cmd, inner_args, inner_words, vec![], false)?; Ok(Command::Explain { inner: Box::new(inner), }) @@ -3541,7 +3616,7 @@ fn parse_base_command( let mut echo_args = Vec::new(); let mut skip_flags = true; - for arg in &args { + for (arg, word) in args.iter().zip(&words) { if skip_flags && arg.starts_with('-') { for ch in arg[1..].chars() { match ch { @@ -3551,14 +3626,14 @@ fn parse_base_command( _ => { // Unknown flag — treat as regular argument skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); break; } } } } else { skip_flags = false; - echo_args.push(arg.clone()); + echo_args.push(word.clone()); } } @@ -3729,7 +3804,7 @@ fn parse_base_command( // Everything else: external command _ => Ok(Command::External { program: cmd.to_string(), - args, + args: words, redirects, background, }), @@ -4217,6 +4292,42 @@ mod tests { // Quote Parsing Tests (Phase 6 M5) // ========================================================================= + #[test] + fn command_substitution_quotes_have_separate_nested_contexts() { + for source in [ + r#"echo "$(echo ")")" tail"#, + r#"echo "$(echo '"')" tail"#, + r#"echo "$(echo "$(echo ')')")" tail"#, + r#"echo ')' "(" \) (echo nested) tail"#, + r#"echo $(echo "(") tail"#, + ] { + let input = format!("{source})suffix"); + let mut chars = input.chars().peekable(); + assert_eq!(parse_command_sub_dollar(&mut chars).unwrap(), source); + assert_eq!(chars.collect::(), "suffix"); + } + let mut chars = r#"echo "$(echo ")")""#.chars().peekable(); + assert!(parse_command_sub_dollar(&mut chars).is_err()); + } + + #[test] + fn special_parameters_render_unbraced_and_names_keep_their_boundary() { + for name in ["?", "$", "#", "@", "*"] { + for source in [format!("${name}"), format!("${{{name}}}")] { + let tokens = tokenize(&source).unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), format!("${name}")); + } + } + let tokens = tokenize("$A''suffix").unwrap(); + let Token::Word(w) = &tokens[0] else { + panic!("Expected word") + }; + assert_eq!(quoted_word_to_string(w), "${A}suffix"); + } + #[test] fn test_single_quotes_no_expansion() { use crate::state::ShellState; @@ -4241,20 +4352,26 @@ mod tests { let mut state = ShellState::new("/tmp/vsh_test").unwrap(); state.set_variable("NAME", "Alice"); - // Test external command with variable expansion - let cmd = parse_command("touch \"file_$NAME\"").unwrap(); - match cmd { - Command::External { program, args, .. } => { - assert_eq!(program, "touch"); - let expanded = expand_variables(&args[0], &state); - assert_eq!(expanded, "file_Alice"); - } + // An external command: its argument is a typed word, expanded by + // `word::expand_words`. `touch` is a builtin and would not take this + // path, so the old form of this test never ran its external arm. + let cmd = parse_command("wc \"file_$NAME\" \"a $NAME\"x").unwrap(); + let Command::External { program, args, .. } = cmd else { + panic!("Expected an external command"); + }; + assert_eq!(program, "wc"); + let expansion = + crate::word::expand_words(&args, &mut state, crate::word::ProcessSubs::Refuse).unwrap(); + // The parameter is expanded, the quotes are removed, and the spaces + // inside the quotes stay in one argument. + assert_eq!(expansion.lossy_fields(), ["file_Alice", "a Alicex"]); + + // The builtin still takes the older string form. + match parse_command("touch \"file_$NAME\"").unwrap() { Command::Touch { path, .. } => { - // Could also be parsed as Touch command - let expanded = expand_variables(&path, &state); - assert!(expanded.contains("file_")); + assert_eq!(expand_variables(&path, &state), "file_Alice"); } - _ => panic!("Expected External or Touch command"), + other => panic!("Expected Touch, got {other:?}"), } } @@ -5313,3 +5430,78 @@ mod semicolon_tests { assert_eq!(parts, vec!["", ""]); } } + +#[cfg(test)] +mod typed_word_tests { + use super::*; + use proptest::prelude::*; + + fn typed_words(input: &str) -> Vec { + tokenize(input) + .unwrap() + .into_iter() + .filter_map(|t| match t { + Token::Word(w) => Some(w.typed), + _ => None, + }) + .collect() + } + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + #[test] + fn the_tokenizer_records_the_quoting_of_each_piece() { + assert_eq!( + typed_words("echo '*'x\\?\"a $V\"*"), + vec![Word::literal("echo"), { + let mut w = word(&[("*", Quote::Single), ("x", Quote::None)]); + w.push_char('?', Quote::Backslash); + w.push_str("a ", Quote::Double); + w.push_parameter("V".into(), false, Quote::Double); + w.push_char('*', Quote::None); + w + }] + ); + } + + #[test] + fn arithmetic_expansion_is_one_command_substitution_piece() { + let mut expected = Word::new(); + expected.push_command_sub("(1+(2*3))".into(), Quote::None); + expected.push_char('x', Quote::None); + assert_eq!(typed_words("$((1+(2*3)))x"), vec![expected]); + } + + /// One literal piece: (text, quoting). Characters are limited to those + /// the tokenizer reads literally in each quoting. + fn piece() -> impl Strategy { + prop_oneof![ + "[a-z0-9._/*?-]{1,4}".prop_map(|s| (s, Quote::None)), + "[a-z *?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Single)), + "[a-z '*?$\"\\\\]{1,4}".prop_map(|s| (s, Quote::Double)), + "[a-z*?$'\"\\\\ ]".prop_map(|s| (s, Quote::Backslash)), + ] + } + + proptest! { + #![proptest_config(ProptestConfig::with_cases(256))] + + /// `Display` writes a word as source that tokenizes back to the + /// same word, so the quoting survives a round trip. + #[test] + fn display_then_tokenize_gives_the_same_word(pieces in prop::collection::vec(piece(), 1..6)) { + let mut w = Word::new(); + for (text, quote) in &pieces { + w.push_str(text, *quote); + } + let source = w.to_string(); + prop_assert_eq!(typed_words(&source), vec![w], "source: {}", source); + } + } +} diff --git a/impl/rust-cli/src/process_sub.rs b/impl/rust-cli/src/process_sub.rs index 7387952..fe3bf2a 100644 --- a/impl/rust-cli/src/process_sub.rs +++ b/impl/rust-cli/src/process_sub.rs @@ -15,6 +15,7 @@ //! ``` use anyhow::{anyhow, Result}; +use std::ffi::OsString; use std::path::{Path, PathBuf}; use std::process::{Child, Command as ProcessCommand}; @@ -154,32 +155,24 @@ fn start_command_with_output_redirect( // For FIFOs, we use shell redirection (sh -c "cmd > fifo") to avoid blocking // The shell handles the FIFO opening in the background properly - let cmd_string = match cmd { + // The arguments stay OsString values and reach the shell as positional + // parameters, forwarded with "$@". Quoting them into the script text would + // need UTF-8 and would lose the bytes of a name that is not UTF-8. + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; - - // Build command string: prog arg1 arg2 > fifo - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - // Simple shell escaping (single quotes) - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } - Command::Pwd { .. } => "pwd".to_string(), + Command::Pwd { .. } => ("pwd".to_string(), Vec::new()), Command::Ls { path, .. } => { let mut cmd = "ls".to_string(); if let Some(p) = path { let expanded = crate::parser::expand_with_command_sub(p, state)?; cmd.push_str(&format!(" '{}'", expanded.replace('\'', "'\\''"))); } - cmd + (cmd, Vec::new()) } _ => { return Err(anyhow!( @@ -190,13 +183,7 @@ fn start_command_with_output_redirect( // Use sh -c to run: cmd > fifo_path let shell_cmd = format!("{} > {}", cmd_string, output_path.display()); - - let child = ProcessCommand::new("sh") - .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + spawn_sh(&shell_cmd, &args) } /// Start command with stdin redirected from file (for >(cmd)) @@ -207,22 +194,13 @@ fn start_command_with_input_redirect( ) -> Result { // For FIFOs, use shell redirection: sh -c "cmd < fifo" - let cmd_string = match cmd { + // As above: OsString positional parameters forwarded with "$@". + let (cmd_string, args): (String, Vec) = match cmd { Command::External { program, args, .. } => { let expanded_program = crate::parser::expand_with_command_sub(program, state)?; - let expanded_args: Result> = args - .iter() - .map(|arg| crate::parser::expand_with_command_sub(arg, state)) - .collect(); - let expanded_args = expanded_args?; - - // Build command string - let mut cmd = expanded_program; - for arg in expanded_args { - cmd.push(' '); - cmd.push_str(&format!("'{}'", arg.replace('\'', "'\\''"))); - } - cmd + let expansion = + crate::word::expand_words(args, state, crate::word::ProcessSubs::Refuse)?; + (format!("{expanded_program} \"$@\""), expansion.fields) } _ => { return Err(anyhow!( @@ -233,13 +211,18 @@ fn start_command_with_input_redirect( // Use sh -c to run: cmd < fifo_path let shell_cmd = format!("{} < {}", cmd_string, input_path.display()); + spawn_sh(&shell_cmd, &args) +} - let child = ProcessCommand::new("sh") +/// Run `sh -c script sh args...`. The arguments become `$1`, `$2`, ... and are +/// passed as raw OS strings, so their bytes and their boundaries survive. +fn spawn_sh(script: &str, args: &[OsString]) -> Result { + Ok(ProcessCommand::new("sh") .arg("-c") - .arg(&shell_cmd) - .spawn()?; - - Ok(child) + .arg(script) + .arg("sh") + .args(args) + .spawn()?) } #[cfg(test)] @@ -293,4 +276,44 @@ mod tests { proc_sub1.cleanup().unwrap(); proc_sub2.cleanup().unwrap(); } + + /// A file named `bad\xff.bin` (not UTF-8) and a pattern that matches it. + #[cfg(unix)] + fn non_utf8_fixture(dir: &Path) -> String { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let name = OsStr::from_bytes(b"bad\xff.bin"); + std::fs::write(dir.join(name), "12345").unwrap(); + format!("{}/bad*.bin", crate::glob::escape(&dir.to_string_lossy())) + } + + #[cfg(unix)] + #[test] + fn output_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let cmd = parse_command(&format!("cat {pattern}")).unwrap(); + let out = temp_dir.path().join("out.txt"); + + let mut child = start_command_with_output_redirect(&cmd, &mut state, &out).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(&out).unwrap(), "12345"); + } + + #[cfg(unix)] + #[test] + fn input_redirect_keeps_non_utf8_filename_bytes() { + let temp_dir = TempDir::new().unwrap(); + let mut state = ShellState::new(temp_dir.path().to_str().unwrap()).unwrap(); + let pattern = non_utf8_fixture(temp_dir.path()); + let copy = temp_dir.path().join("copy.txt"); + let cmd = parse_command(&format!("sort -o {} {pattern}", copy.display())).unwrap(); + let input = temp_dir.path().join("in.txt"); + std::fs::write(&input, "").unwrap(); + + let mut child = start_command_with_input_redirect(&cmd, &mut state, &input).unwrap(); + assert!(child.wait().unwrap().success()); + assert_eq!(std::fs::read_to_string(©).unwrap(), "12345\n"); + } } diff --git a/impl/rust-cli/src/word.rs b/impl/rust-cli/src/word.rs new file mode 100644 index 0000000..55550aa --- /dev/null +++ b/impl/rust-cli/src/word.rs @@ -0,0 +1,840 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Typed shell words and the word-expansion stage for simple-command arguments. +//! +//! The tokenizer used to flatten each word into a `String` in which quoted +//! glob metacharacters were marked with a backslash (`'*'` became `\*`). +//! Nothing removed those backslashes again, so `echo '*.txt'` printed +//! `\*.txt`, and glob expansion could not tell a quoted `*` from an unquoted +//! one without re-parsing that encoding. +//! +//! A [`Word`] keeps the quoting of every piece instead: each [`Segment`] +//! records whether it was written unquoted, in single or double quotes, or +//! after a backslash. [`expand_words`] then runs the POSIX word-expansion +//! steps on that structure, in this order: +//! +//! 1. tilde expansion (an unquoted `~`, `~/`, `~+`, `~-` at the start), +//! parameter expansion, command substitution, arithmetic expansion and +//! process substitution; +//! 2. brace expansion (`{a,b}`) of unquoted literal text; +//! 3. pathname (glob) expansion of the characters that are not quoted; +//! 4. quote removal. +//! +//! # Rules kept from the previous expansion code +//! +//! - A pattern that matches nothing, or is not a valid pattern, stays as the +//! literal word. +//! - A pattern whose expansion reaches a [`GlobLimits`] limit fails the whole +//! expansion with the [`GlobLimitExceeded`] error; the caller must not run +//! the command. +//! +//! # Known differences from POSIX (not changed here) +//! +//! - No field splitting: an unquoted `$X` whose value is `a b` stays one +//! argument. +//! - An empty word (`''`, or an unquoted expansion that is empty) is dropped +//! by the tokenizer or kept as an empty argument, as before. +//! - Brace expansion runs after parameter expansion (bash runs it before); +//! braces that come out of an expansion are not expanded. +//! +//! Only the arguments of external commands, pipeline stages, `echo` and +//! function calls go through this stage so far. Builtin operands, redirection +//! targets, assignments, `case` patterns and `test` arguments still use the +//! older string encoding. + +use std::ffi::OsString; +use std::fmt; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result}; + +use crate::glob::{self, GlobLimitExceeded, GlobLimits, GlobUsage}; +use crate::parser::ProcessSubType; +use crate::process_sub::ProcessSubstitution; +use crate::state::ShellState; + +/// How a piece of a word was written in the source. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Quote { + /// Not quoted: subject to tilde, brace and glob expansion. + None, + /// Inside `'...'`. + Single, + /// Inside `"..."`. + Double, + /// A single character after an unquoted `\`. + Backslash, +} + +impl Quote { + fn is_quoted(self) -> bool { + self != Quote::None + } +} + +/// One piece of a [`Word`]. +#[derive(Debug, Clone, PartialEq)] +pub enum Segment { + /// Literal text. + Literal { text: String, quote: Quote }, + /// `$NAME` (`braced: false`) or `${EXPR}` (`braced: true`). + Parameter { + expr: String, + braced: bool, + quote: Quote, + }, + /// `$(CMD)` or `` `CMD` ``; `$((EXPR))` is stored as the command `(EXPR)`. + CommandSub { cmd: String, quote: Quote }, + /// `<(CMD)` or `>(CMD)`. + ProcessSub { kind: ProcessSubType, cmd: String }, +} + +/// A shell word with the quoting of each of its pieces. +#[derive(Debug, Clone, PartialEq, Default)] +pub struct Word { + segments: Vec, +} + +impl Word { + /// An empty word. + pub fn new() -> Self { + Self::default() + } + + /// A word that is a single unquoted literal. + /// + /// # Examples + /// ``` + /// use vsh::word::Word; + /// assert_eq!(Word::literal("a.txt").to_string(), "a.txt"); + /// ``` + pub fn literal(text: &str) -> Self { + let mut word = Self::new(); + for ch in text.chars() { + word.push_char(ch, Quote::None); + } + word + } + + /// The pieces of the word, in order. + pub fn segments(&self) -> &[Segment] { + &self.segments + } + + /// True when the word has no pieces. + pub fn is_empty(&self) -> bool { + self.segments.is_empty() + } + + /// Append one literal character, merging it into the last piece when that + /// piece is a literal with the same quoting. + pub fn push_char(&mut self, ch: char, quote: Quote) { + if quote != Quote::Backslash { + if let Some(Segment::Literal { text, quote: last }) = self.segments.last_mut() { + if *last == quote { + text.push(ch); + return; + } + } + } + self.segments.push(Segment::Literal { + text: ch.to_string(), + quote, + }); + } + + /// Append literal text. + pub fn push_str(&mut self, text: &str, quote: Quote) { + for ch in text.chars() { + self.push_char(ch, quote); + } + } + + /// Append a parameter expansion. + pub fn push_parameter(&mut self, expr: String, braced: bool, quote: Quote) { + self.segments.push(Segment::Parameter { + expr, + braced, + quote, + }); + } + + /// Append a command substitution. + pub fn push_command_sub(&mut self, cmd: String, quote: Quote) { + self.segments.push(Segment::CommandSub { cmd, quote }); + } + + /// Append a process substitution. + pub fn push_process_sub(&mut self, kind: ProcessSubType, cmd: String) { + self.segments.push(Segment::ProcessSub { kind, cmd }); + } +} + +impl From<&str> for Word { + fn from(text: &str) -> Self { + Word::literal(text) + } +} + +/// Write one literal piece back as source with its quoting. +fn fmt_literal(f: &mut fmt::Formatter<'_>, text: &str, quote: Quote) -> fmt::Result { + match quote { + Quote::None => f.write_str(text), + Quote::Backslash => text.chars().try_for_each(|ch| write!(f, "\\{ch}")), + // `'` cannot appear inside single quotes. + Quote::Single => write!(f, "'{}'", text.replace('\'', "'\\''")), + Quote::Double => { + let mut escaped = String::with_capacity(text.len() + 2); + for ch in text.chars() { + if matches!(ch, '"' | '$' | '`' | '\\') { + escaped.push('\\'); + } + escaped.push(ch); + } + write!(f, "\"{escaped}\"") + } + } +} + +/// Writes the word back as shell source that parses to the same word. +impl fmt::Display for Word { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + for (index, segment) in self.segments.iter().enumerate() { + match segment { + Segment::Literal { text, quote } => fmt_literal(f, text, *quote)?, + Segment::Parameter { + expr, + braced, + quote, + } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + // `$A''b` must not come back as `$Ab`: brace the name when + // the text after it could continue it. + let runs_on = self.segments.get(index + 1).is_some_and(|next| { + matches!(next, Segment::Literal { text, quote: Quote::None } + if text.starts_with(|c: char| c.is_alphanumeric() || c == '_')) + }) && expr.starts_with(|c: char| c.is_alphabetic() || c == '_'); + if *braced || runs_on { + write!(f, "{open}${{{expr}}}{open}")?; + } else { + write!(f, "{open}${expr}{open}")?; + } + } + Segment::CommandSub { cmd, quote } => { + let open = if quote.is_quoted() { "\"" } else { "" }; + write!(f, "{open}$({cmd}){open}")?; + } + Segment::ProcessSub { kind, cmd } => match kind { + ProcessSubType::Input => write!(f, "<({cmd})")?, + ProcessSubType::Output => write!(f, ">({cmd})")?, + }, + } + } + Ok(()) + } +} + +/// Compares the source form ([`fmt::Display`]) with `other`. +impl PartialEq for Word { + fn eq(&self, other: &str) -> bool { + let source = self.to_string(); + source.as_str() == other + } +} + +impl PartialEq<&str> for Word { + fn eq(&self, other: &&str) -> bool { + *self == **other + } +} + +/// Whether process substitutions may be started while expanding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ProcessSubs { + /// Start them; the caller cleans them up after the command finishes. + Allow, + /// Fail on them, as the code paths without FIFO handling always have. + Refuse, +} + +/// The result of expanding a list of words. +pub struct Expansion { + /// The arguments, after quote removal. A glob match keeps the exact bytes + /// of its file name, which need not be UTF-8. + pub fields: Vec, + /// Process substitutions started for these arguments. + pub process_subs: Vec, +} + +impl Expansion { + /// The arguments as text, for a builtin or a function that works on + /// strings. Bytes that are not UTF-8 are replaced; external commands get + /// [`Expansion::fields`] unchanged. + pub fn lossy_fields(&self) -> Vec { + self.fields + .iter() + .map(|field| field.to_string_lossy().into_owned()) + .collect() + } + + /// Wait for the process substitutions and remove their FIFOs. + pub fn cleanup(self) { + for proc_sub in self.process_subs { + if let Err(e) = proc_sub.cleanup() { + eprintln!("Warning: Failed to cleanup process substitution: {}", e); + } + } + } +} + +/// Expand words relative to the current directory under the default limits. +/// +/// # Errors +/// Fails on an expansion error (a failing command substitution, a refused +/// process substitution) or when a glob reaches a limit +/// ([`GlobLimitExceeded`], reachable with `anyhow::Error::downcast_ref`). +pub fn expand_words( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, +) -> Result { + let cwd = std::env::current_dir().context("Failed to get current working directory")?; + expand_words_in(words, state, process_subs, &cwd, &GlobLimits::DEFAULT) +} + +/// [`expand_words`] with an explicit directory for relative patterns and +/// explicit glob limits. +/// +/// # Errors +/// As for [`expand_words`]. +pub fn expand_words_in( + words: &[Word], + state: &mut ShellState, + process_subs: ProcessSubs, + cwd: &Path, + limits: &GlobLimits, +) -> Result { + let mut expansion = Expansion { + fields: Vec::new(), + process_subs: Vec::new(), + }; + // One glob budget for the whole command: every word and every brace + // alternative draws on the same limits. + let mut spent = GlobUsage::default(); + for word in words { + // On error the substitutions started so far are dropped, and Drop + // stops them and removes their FIFOs. + let chars = expand_segments(word, state, process_subs, &mut expansion.process_subs)?; + glob_fields(&chars, cwd, limits, &mut spent, &mut expansion.fields)?; + } + Ok(expansion) +} + +/// What later steps may do with a character produced by step 1. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Role { + /// Quoted text or a tilde result: taken literally. + Literal, + /// Unquoted source text: brace and glob expansion apply. + Source, + /// Result of an unquoted expansion: glob expansion applies, braces do not. + Expanded, +} + +/// Step 1: tilde, parameter, command, arithmetic and process substitution. +fn expand_segments( + word: &Word, + state: &mut ShellState, + process_subs: ProcessSubs, + started: &mut Vec, +) -> Result> { + let mut out: Vec<(char, Role)> = Vec::new(); + let push = |out: &mut Vec<(char, Role)>, text: &str, role: Role| { + out.extend(text.chars().map(|ch| (ch, role))); + }; + + for (index, segment) in word.segments.iter().enumerate() { + match segment { + Segment::Literal { text, quote } => { + if !quote.is_quoted() { + // The tilde prefix runs to the first unquoted `/`. It ends in this + // segment only if the segment has a `/` or is the whole word; + // otherwise a quoted piece or an expansion is part of the + // prefix, and the tilde stays literal (`~'local'/x`). + if index == 0 + && text.starts_with('~') + && (text.contains('/') || word.segments.len() == 1) + { + let (prefix, rest) = text.split_at(text.find('/').unwrap_or(text.len())); + let expanded = crate::parser::expand_tilde(prefix); + if expanded != prefix { + push(&mut out, &expanded, Role::Literal); + push(&mut out, rest, Role::Source); + continue; + } + } + push(&mut out, text, Role::Source); + } else { + push(&mut out, text, Role::Literal); + } + } + Segment::Parameter { + expr, + braced, + quote, + } => { + let source = if *braced { + format!("${{{expr}}}") + } else { + format!("${expr}") + }; + let value = crate::parser::expand_variables(&source, state); + push(&mut out, &value, expansion_role(*quote)); + } + Segment::CommandSub { cmd, quote } => { + let value = crate::parser::expand_with_command_sub(&format!("$({cmd})"), state)?; + push(&mut out, &value, expansion_role(*quote)); + } + Segment::ProcessSub { kind, cmd } => { + if process_subs == ProcessSubs::Refuse { + anyhow::bail!( + "Process substitution not yet implemented: execution requires FIFO support ({})", + Word { + segments: vec![segment.clone()] + } + ); + } + let mut proc_sub = ProcessSubstitution::create(*kind, cmd.clone(), state)?; + proc_sub.start(state)?; + push( + &mut out, + &proc_sub.fifo_path.to_string_lossy(), + Role::Literal, + ); + started.push(proc_sub); + } + } + } + Ok(out) +} + +fn expansion_role(quote: Quote) -> Role { + if quote.is_quoted() { + Role::Literal + } else { + Role::Expanded + } +} + +/// Encode step-1 output for [`glob::expand_braces`] and +/// [`glob::contains_glob_pattern`], which both read `\x` as a literal `x`. +fn encode(chars: &[(char, Role)]) -> String { + let mut out = String::with_capacity(chars.len() * 2); + for &(ch, role) in chars { + let literal = match role { + Role::Literal => true, + Role::Source => ch == '\\', + Role::Expanded => matches!(ch, '\\' | '{' | '}' | ','), + }; + if literal { + out.push('\\'); + } + out.push(ch); + } + out +} + +/// Steps 2 to 4: brace expansion, glob expansion and quote removal. +fn glob_fields( + chars: &[(char, Role)], + cwd: &Path, + limits: &GlobLimits, + spent: &mut GlobUsage, + fields: &mut Vec, +) -> Result<()> { + let encoded = encode(chars); + for candidate in glob::expand_braces(&encoded) { + match glob_pattern(&candidate) { + None => fields.push(remove_quotes(&candidate).into()), + Some(pattern) => match glob::expand_glob_continuing(&pattern, cwd, limits, *spent) { + Ok(found) if !found.paths.is_empty() => { + *spent = found.usage; + fields.extend(found.paths.into_iter().map(PathBuf::into_os_string)); + } + // No match still spent reads. + Ok(found) => { + *spent = found.usage; + fields.push(remove_quotes(&candidate).into()); + } + // A limit is not "no match": substituting the literal word + // would run the command on the wrong arguments. + Err(e) if e.is::() => return Err(e), + // No match, or not a valid pattern: keep the word (POSIX). + _ => fields.push(remove_quotes(&candidate).into()), + }, + } + } + Ok(()) +} + +/// The pattern for the `glob` crate, or `None` when no `*`, `?` or `[` is +/// left unquoted. Quoted metacharacters are written as one-character classes +/// (`[*]`), the only escape that crate has. +fn glob_pattern(encoded: &str) -> Option { + let mut pattern = String::with_capacity(encoded.len()); + let mut active = false; + // Inside an unquoted `[...]`: how many members so far, and whether a + // quoted `-` waits to be written last, where it cannot form a range. + let mut class: Option = None; + let mut hyphen = false; + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + let Some(next) = chars.next() else { continue }; + match (class.as_mut(), next) { + (Some(members), '-') => { + *members += 1; + hyphen = true; + } + // `*`, `?` and `[` are already literal inside a class. + (Some(members), '*' | '?' | '[') => { + *members += 1; + pattern.push(next); + } + (members, _) => { + if let Some(members) = members { + *members += 1; + } + pattern.push_str(&::glob::Pattern::escape(&next.to_string())); + } + } + continue; + } + active |= matches!(ch, '*' | '?' | '['); + match class.as_mut() { + None if ch == '[' => class = Some(0), + None => {} + // `]` first in a class (after `!` or `^`) is a member, not the end. + Some(members) if ch == ']' && *members > 0 => { + if std::mem::take(&mut hyphen) { + pattern.push('-'); + } + class = None; + } + Some(members) if *members == 0 && matches!(ch, '!' | '^') && pattern.ends_with('[') => { + } + Some(members) => *members += 1, + } + pattern.push(ch); + } + active.then_some(pattern) +} + +/// Quote removal: `\x` becomes `x`. +fn remove_quotes(encoded: &str) -> String { + let mut out = String::with_capacity(encoded.len()); + let mut chars = encoded.chars(); + while let Some(ch) = chars.next() { + if ch == '\\' { + if let Some(next) = chars.next() { + out.push(next); + } + } else { + out.push(ch); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn word(parts: &[(&str, Quote)]) -> Word { + let mut w = Word::new(); + for (text, quote) in parts { + w.push_str(text, *quote); + } + w + } + + fn fixture(names: &[&str]) -> tempfile::TempDir { + let dir = tempfile::TempDir::new().unwrap(); + for name in names { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir + } + + fn expand_in(dir: &Path, words: &[Word]) -> Vec { + let mut state = ShellState::new(dir.to_str().unwrap()).unwrap(); + expand_words_in( + words, + &mut state, + ProcessSubs::Refuse, + dir, + &GlobLimits::DEFAULT, + ) + .unwrap() + .lossy_fields() + } + + #[test] + fn unquoted_patterns_expand_and_quoted_ones_do_not() { + let dir = fixture(&["a.txt", "b.txt", "c.log"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + word(&[("*.txt", Quote::Single)]), + word(&[("*.txt", Quote::Double)]), + word(&[("*", Quote::Backslash), (".txt", Quote::None)]), + ], + ); + assert_eq!(out, ["a.txt", "b.txt", "*.txt", "*.txt", "*.txt"]); + } + + #[test] + fn a_partly_quoted_pattern_matches_its_quoted_part_literally() { + let dir = fixture(&["*x.txt", "ax.txt", "a.txt"]); + // `'*'x*` : a literal star followed by `x` and a wildcard. + let out = expand_in( + dir.path(), + &[word(&[("*", Quote::Single), ("x*", Quote::None)])], + ); + assert_eq!(out, ["*x.txt"]); + } + + #[test] + fn no_match_and_invalid_patterns_stay_literal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + Word::literal("*.txt"), + Word::literal("*.none"), + Word::literal("[z-"), + ], + ); + assert_eq!(out, ["a.txt", "*.none", "[z-"]); + } + + #[test] + fn braces_expand_only_when_unquoted() { + let dir = fixture(&[]); + let out = expand_in( + dir.path(), + &[Word::literal("a{1,2}"), word(&[("a{1,2}", Quote::Single)])], + ); + assert_eq!(out, ["a1", "a2", "a{1,2}"]); + } + + #[test] + fn literal_backslashes_survive_quote_removal() { + let dir = fixture(&["a.txt"]); + let out = expand_in( + dir.path(), + &[ + word(&[("a\\*", Quote::Single)]), + word(&[("\\", Quote::Backslash)]), + ], + ); + assert_eq!(out, ["a\\*", "\\"]); + } + + #[test] + fn a_quoted_backslash_is_a_literal_character_in_a_pattern() { + let dir = fixture(&["a\\b.txt", "ab.txt"]); + // `'a\'*` : names that start with `a\`. + let out = expand_in( + dir.path(), + &[word(&[("a\\", Quote::Single), ("*", Quote::None)])], + ); + assert_eq!(out, ["a\\b.txt"]); + } + + #[test] + fn unquoted_parameter_values_are_globbed_and_quoted_ones_are_not() { + let dir = fixture(&["a.txt", "b.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("P", "*.txt"); + let mut unquoted = Word::new(); + unquoted.push_parameter("P".into(), false, Quote::None); + let mut quoted = Word::new(); + quoted.push_parameter("P".into(), true, Quote::Double); + let out = expand_words_in( + &[unquoted, quoted], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap() + .lossy_fields(); + assert_eq!(out, ["a.txt", "b.txt", "*.txt"]); + } + + #[test] + fn a_glob_limit_fails_the_expansion_instead_of_substituting_the_literal() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let words = [Word::literal("-l"), Word::literal("*.txt")]; + let err = expand_words_in(&words, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the limit must fail the expansion"); + let stopped = err + .downcast_ref::() + .expect("the limit error must reach the caller"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert!(err.to_string().contains("over the limit of 4"), "{err}"); + } + + /// The brace alternatives of one word, and the words of one command, share + /// one glob budget: five patterns of one match each pass a limit of four. + #[test] + fn brace_alternatives_and_words_share_one_glob_budget() { + let dir = fixture(&["f0.txt", "f1.txt", "f2.txt", "f3.txt", "f4.txt"]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let limits = GlobLimits { + max_matches: 4, + ..GlobLimits::DEFAULT + }; + let braces = [Word::literal("{f0,f1,f2,f3,f4}*")]; + let words: Vec = (0..5).map(|i| Word::literal(&format!("f{i}*"))).collect(); + for case in [&braces[..], &words[..]] { + let err = expand_words_in(case, &mut state, ProcessSubs::Refuse, dir.path(), &limits) + .err() + .expect("the shared budget must be exceeded"); + let stopped = err.downcast_ref::().unwrap(); + assert_eq!((stopped.exceeded.limit, stopped.exceeded.level), (4, 5)); + } + // Within the budget the same words expand normally. + let out = expand_words_in( + &words[..4], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &limits, + ) + .unwrap() + .lossy_fields(); + assert_eq!(out, ["f0.txt", "f1.txt", "f2.txt", "f3.txt"]); + } + + #[test] + fn refused_process_substitution_is_an_error() { + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + let mut w = Word::new(); + w.push_process_sub(ProcessSubType::Input, "true".into()); + let err = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .err() + .expect("refused"); + assert!(err.to_string().contains("<(true)"), "{err}"); + } + + #[test] + fn display_round_trips_quoting() { + let w = word(&[ + ("a b", Quote::Single), + ("*", Quote::Backslash), + ("x\"$", Quote::Double), + ("y", Quote::None), + ]); + assert_eq!(w.to_string(), "'a b'\\*\"x\\\"\\$\"y"); + assert_eq!(Word::literal("plain"), "plain"); + } + + #[test] + fn display_keeps_a_literal_out_of_an_unbraced_name() { + let mut w = Word::new(); + w.push_parameter("A".to_string(), false, Quote::None); + w.push_str("suffix", Quote::None); + assert_eq!(w.to_string(), "${A}suffix"); + + std::env::remove_var("Asuffix"); + let dir = fixture(&[]); + let mut state = ShellState::new(dir.path().to_str().unwrap()).unwrap(); + state.set_variable("A", "x"); + let out = expand_words_in( + &[w], + &mut state, + ProcessSubs::Refuse, + dir.path(), + &GlobLimits::DEFAULT, + ) + .unwrap(); + assert_eq!(out.lossy_fields(), ["xsuffix"]); + } + + #[test] + fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + // [a'-'z]*.txt + assert_eq!(glob_pattern("[a\\-z]*.txt"), Some("[az-]*.txt".to_string())); + let dir = fixture(&["-.txt", "a.txt", "m.txt", "z.txt"]); + let w = word(&[ + ("[a", Quote::None), + ("-", Quote::Single), + ("z]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt", "a.txt", "z.txt"]); + } + + #[test] + fn a_sole_quoted_hyphen_closes_a_bracket_class() { + // ['-']*.txt + let dir = fixture(&["-.txt", "a.txt"]); + let w = word(&[ + ("[", Quote::None), + ("-", Quote::Single), + ("]*.txt", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[w]), ["-.txt"]); + } + + #[test] + fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(&[]); + let quoted = word(&[ + ("~", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + assert_eq!(expand_in(dir.path(), &[quoted]), ["~local/x"]); + // An unquoted slash ends the prefix, so this one still expands. + let slash = word(&[ + ("~/", Quote::None), + ("local", Quote::Single), + ("/x", Quote::None), + ]); + let got = expand_in(dir.path(), &[slash]); + assert!( + got[0].ends_with("/local/x") && !got[0].starts_with('~'), + "{got:?}" + ); + } + + #[test] + fn a_non_ascii_character_before_a_brace_does_not_panic() { + let dir = fixture(&[]); + let w = word(&[("é", Quote::Single), ("{a,b}", Quote::None)]); + assert_eq!(expand_in(dir.path(), &[w]), ["éa", "éb"]); + } + + #[test] + fn glob_pattern_escapes_quoted_metacharacters() { + assert_eq!(glob_pattern("\\*a*"), Some("[*]a*".to_string())); + assert_eq!(glob_pattern("\\*a"), None); + assert_eq!(glob_pattern("\\[x?"), Some("[[]x?".to_string())); + } +} diff --git a/impl/rust-cli/tests/glob_bounds_tests.rs b/impl/rust-cli/tests/glob_bounds_tests.rs new file mode 100644 index 0000000..aff5dfa --- /dev/null +++ b/impl/rust-cli/tests/glob_bounds_tests.rs @@ -0,0 +1,472 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! Glob expansion: termination, limits and resource accounting (issue #191). +//! +//! The first two tests reproduce the out-of-memory failure that +//! ClusterFuzzLite found on 2026-11-17 (`fuzz_glob_expansion`, input `/**/*`): +//! the previous expansion followed symbolic links under `**`, so a single link +//! back to a parent directory made `**/*` grow without end. Each runs the +//! expansion on a worker thread and fails if it has not finished in time, +//! instead of hanging the test run. +//! +//! The other tests use small injected limits so they are exact and fast, and +//! check the reported usage against counts computed here directly from the +//! fixture. +//! +//! Run with: `cargo test --test glob_bounds_tests` + +#![cfg(unix)] + +use std::collections::BTreeSet; +use std::fs; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; +use std::sync::mpsc; +use std::time::{Duration, Instant}; +use tempfile::TempDir; +use vsh::glob::{expand_glob, expand_glob_with_limits, Expansion, GlobLimitExceeded, GlobLimits}; + +const TIME_ALLOWED: Duration = Duration::from_secs(20); + +/// Run `expand_glob` on a worker thread; panic if it does not finish in time. +fn expand_within_time(pattern: &'static str, base: PathBuf) -> Vec { + let (send, receive) = mpsc::channel(); + std::thread::spawn(move || { + let _ = send.send(expand_glob(pattern, &base).map_err(|e| e.to_string())); + }); + match receive.recv_timeout(TIME_ALLOWED) { + Ok(result) => result + .expect("expansion failed") + .into_iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect(), + Err(_) => panic!("expand_glob({pattern:?}) did not finish within {TIME_ALLOWED:?}"), + } +} + +fn strings(paths: &[PathBuf]) -> Vec { + paths + .iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect() +} + +fn stopped(result: anyhow::Result) -> GlobLimitExceeded { + let err = result.expect_err("the expansion must stop at a limit"); + err.downcast_ref::() + .unwrap_or_else(|| panic!("expected a limit error, got: {err}")) + .clone() +} + +/// The 7-entry tree with two symbolic-link loops used in the reproduction: +/// `d/{f1,f2}`, `d/up -> ..`, `self -> .`, `top`. +fn looped_tree() -> TempDir { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("d")).unwrap(); + for f in ["d/f1", "d/f2", "top"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("..", t.path().join("d/up")).unwrap(); + symlink(".", t.path().join("self")).unwrap(); + t +} + +/// `dirs` directories with `files` files each, plus `files` files at the top. +fn broad_tree(dirs: usize, files: usize) -> TempDir { + let t = TempDir::new().unwrap(); + for d in 0..dirs { + let dir = t.path().join(format!("d{d:03}")); + fs::create_dir(&dir).unwrap(); + for f in 0..files { + fs::write(dir.join(format!("f{f:03}")), b"").unwrap(); + } + } + for f in 0..files { + fs::write(t.path().join(format!("f{f:03}")), b"").unwrap(); + } + t +} + +// --------------------------------------------------------------------------- +// Termination over symbolic-link loops (fails on the previous implementation) +// --------------------------------------------------------------------------- + +#[test] +fn recursive_glob_over_symlink_loops_terminates_with_the_bash_result() { + let t = looped_tree(); + // bash 5.2 with globstar gives exactly these six paths on this tree. + assert_eq!( + expand_within_time("**/*", t.path().to_path_buf()), + ["d", "d/f1", "d/f2", "d/up", "self", "top"] + ); +} + +#[test] +fn recursive_glob_with_no_match_over_symlink_loops_terminates() { + let t = looped_tree(); + assert!(expand_within_time("**/no-such-name", t.path().to_path_buf()).is_empty()); +} + +#[test] +fn non_recursive_components_still_go_through_symlinks() { + let t = looped_tree(); + assert_eq!( + expand_within_time("d/up/*", t.path().to_path_buf()), + ["d/up/d", "d/up/self", "d/up/top"] + ); + assert_eq!( + expand_within_time("self/self/top", t.path().to_path_buf()), + ["self/self/top"] + ); +} + +// --------------------------------------------------------------------------- +// Each limit stops the walk, names itself, and returns nothing partial +// --------------------------------------------------------------------------- + +#[test] +fn the_match_limit_stops_the_walk_and_names_itself() { + let t = broad_tree(30, 40); // 1,230 entries + let limits = GlobLimits { + max_matches: 1_000, + ..GlobLimits::DEFAULT + }; + let started = Instant::now(); + let s = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + assert!(started.elapsed() < Duration::from_secs(5)); + + assert_eq!(s.exceeded.axis, "matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (1_000, 1_001)); + assert_eq!(s.usage.matches.peak(), 1_000); + assert_eq!(s.pattern, "**/*"); + let message = s.to_string(); + assert!(message.contains("matches held would reach 1001, over the limit of 1000")); + assert!(message.contains("was not expanded"), "{message}"); +} + +#[test] +fn the_read_limit_stops_a_pattern_that_matches_nothing() { + // A cap on matches alone cannot stop this one: nothing ever matches. + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_reads: 500, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits( + "**/no-such-name", + t.path(), + &limits, + )); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (500, 501)); + assert_eq!(s.usage.matches.peak(), 0); +} + +#[test] +fn the_byte_limit_counts_path_length_plus_one() { + let t = TempDir::new().unwrap(); + for name in ["aaaa", "bbbb", "cccc"] { + fs::write(t.path().join(name), b"").unwrap(); + } + // Each match costs 4 + 1 = 5 bytes: two fit in 10, the third does not. + let limits = GlobLimits { + max_match_bytes: 10, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of matches held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (10, 15)); + assert_eq!( + s.usage.matches.held(), + 2, + "the refused match is not counted" + ); + assert_eq!(s.usage.matches.peak(), 2, "nor does it raise the peak"); + assert_eq!(s.usage.match_bytes.peak(), 10); + + let limits = GlobLimits { + max_match_bytes: 15, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("*", t.path(), &limits).unwrap(); + assert_eq!(ok.usage.match_bytes.peak(), 15); +} + +#[test] +fn the_report_is_the_same_on_every_run() { + let t = broad_tree(30, 40); + let limits = GlobLimits { + max_matches: 777, + ..GlobLimits::DEFAULT + }; + let first = stopped(expand_glob_with_limits("**/*", t.path(), &limits)); + for _ in 0..3 { + assert_eq!( + stopped(expand_glob_with_limits("**/*", t.path(), &limits)), + first + ); + } + // Entries are visited in name order. The top level gives 30 directories + // and 40 files (70 matches), then d000..d016 give 40 each (750), so the + // 778th match, the first one refused, is the 28th file of d017. + assert_eq!(first.at, PathBuf::from("d017")); + assert_eq!(first.usage.matches.peak(), 777); +} + +#[test] +fn default_limits_are_the_documented_values() { + assert_eq!( + GlobLimits::default(), + GlobLimits { + max_reads: 10_000_000, + max_matches: 100_000, + max_match_bytes: 6 * 1024 * 1024, + max_listing_bytes: 64 * 1024 * 1024, + } + ); +} + +// --------------------------------------------------------------------------- +// Accounting: the reported usage equals a count made directly from the tree +// --------------------------------------------------------------------------- + +/// Visible, real (not symbolic-link) directories at or below `dir`, with the +/// number of entries in each. +fn visible_dirs(dir: &Path, out: &mut Vec) { + let entries: Vec<_> = fs::read_dir(dir).unwrap().map(|e| e.unwrap()).collect(); + out.push(entries.len() as u64); + for e in entries { + let hidden = e.file_name().to_string_lossy().starts_with('.'); + if e.file_type().unwrap().is_dir() && !hidden { + visible_dirs(&e.path(), out); + } + } +} + +#[test] +fn reads_for_a_two_level_wildcard_match_a_direct_count() { + let t = broad_tree(7, 5); + let e = expand_glob_with_limits("*/*", t.path(), &GlobLimits::DEFAULT).unwrap(); + // Open the top (1) and list its 7 + 5 entries, then open and list each of + // the 7 directories (1 + 5 each). + assert_eq!(e.usage.reads.total(), 1 + (7 + 5) + 7 * (1 + 5)); + assert_eq!(e.paths.len(), 35); + assert_eq!(e.usage.matches.peak(), 35); +} + +#[test] +fn reads_for_a_recursive_pattern_match_a_direct_count() { + let t = broad_tree(4, 3); + fs::create_dir_all(t.path().join("d001/deep/er")).unwrap(); + fs::create_dir(t.path().join(".hidden")).unwrap(); + fs::write(t.path().join(".hidden/x.rs"), b"").unwrap(); + symlink("d000", t.path().join("link")).unwrap(); + + let e = expand_glob_with_limits("**/*.rs", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert!(e.paths.is_empty()); + + // `**` lists every visible real directory once, and `*.rs` lists each of + // them once more: 2 x (open + entries) per directory. + let mut dirs = Vec::new(); + visible_dirs(t.path(), &mut dirs); + let expected: u64 = dirs.iter().map(|entries| 2 * (1 + entries)).sum(); + assert_eq!(e.usage.reads.total(), expected); +} + +#[test] +fn repeated_recursive_components_read_each_directory_a_bounded_number_of_times() { + // A chain of 30 nested directories. bash lists a path once per way the + // pattern can match it (736,281 results for six `**`); each (directory, + // component) pair is processed once here, so the work stays linear. + let t = TempDir::new().unwrap(); + let mut p = t.path().to_path_buf(); + for _ in 0..30 { + p.push("a"); + } + fs::create_dir_all(&p).unwrap(); + fs::write(p.join("leaf"), b"").unwrap(); + + let pattern = "**/*/**/*/**/*/**/*/**/*/**/*"; + let e = expand_glob_with_limits(pattern, t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(e.paths.len(), 26); // the paths at depth 6 or more, once each + let entries_in_tree = 31; + let components = 12; + assert!( + e.usage.reads.total() <= 2 * components * entries_in_tree, + "{} reads", + e.usage.reads.total() + ); +} + +// --------------------------------------------------------------------------- +// Matching rules shared with bash 5.2 (globstar on, dotglob off) +// --------------------------------------------------------------------------- + +/// The tree on which bash 5.2 results were recorded: +/// `d/e/g d/f d/up -> .. top .hid .hd/x/y ld/z link -> d`. +fn bash_tree() -> TempDir { + let t = TempDir::new().unwrap(); + for d in ["d/e", ".hd/x", "ld"] { + fs::create_dir_all(t.path().join(d)).unwrap(); + } + for f in ["d/f", "d/e/g", "top", ".hid", ".hd/x/y", "ld/z"] { + fs::write(t.path().join(f), b"").unwrap(); + } + symlink("d", t.path().join("link")).unwrap(); + symlink("..", t.path().join("d/up")).unwrap(); + t +} + +#[test] +fn results_match_bash_on_the_recorded_tree() { + let t = bash_tree(); + let cases: &[(&str, &[&str])] = &[ + ( + "**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/", &["d/", "d/e/", "d/up/", "ld/", "link/"]), + ("d/**", &["d/", "d/e", "d/e/g", "d/f", "d/up"]), + ("d/**/", &["d/", "d/e/", "d/up/"]), + ( + "**/*", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("**/e", &["d/e"]), + ("*/", &["d/", "ld/", "link/"]), + (".*", &[".hd", ".hid"]), + (".h*", &[".hd", ".hid"]), + ("link/*", &["link/e", "link/f", "link/up"]), + ("**/g", &["d/e/g"]), + ("d/**/g", &["d/e/g"]), + ( + "**/**", + &[ + "d", "d/e", "d/e/g", "d/f", "d/up", "ld", "ld/z", "link", "top", + ], + ), + ("./*", &["./d", "./ld", "./link", "./top"]), + (".hd/**", &[".hd/", ".hd/x", ".hd/x/y"]), + ("nomatch*", &[]), + ]; + for (pattern, bash) in cases { + let ours: BTreeSet = strings(&expand_glob(pattern, t.path()).unwrap()) + .into_iter() + .collect(); + let bash: BTreeSet = bash.iter().map(|s| s.to_string()).collect(); + assert_eq!(ours, bash, "pattern {pattern}"); + } +} + +#[test] +fn hidden_files_need_a_leading_dot_in_the_pattern() { + let t = TempDir::new().unwrap(); + for f in [".hid1", ".hid2", "vis"] { + fs::write(t.path().join(f), b"").unwrap(); + } + assert_eq!( + strings(&expand_glob(".*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!( + strings(&expand_glob(".hid*", t.path()).unwrap()), + [".hid1", ".hid2"] + ); + assert_eq!(strings(&expand_glob("*", t.path()).unwrap()), ["vis"]); + assert_eq!( + strings(&expand_glob("?hid1", t.path()).unwrap()), + Vec::::new() + ); +} + +#[test] +fn a_name_that_is_not_utf8_is_matched_without_a_panic() { + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let odd = std::ffi::OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(odd), b"").is_err() { + return; // the file system refuses such names; nothing to test + } + fs::write(t.path().join("ok.txt"), b"").unwrap(); + let found = expand_glob("*.txt", t.path()).unwrap(); + assert_eq!(found.len(), 2); + assert!(found.contains(&PathBuf::from(odd))); +} + +// --------------------------------------------------------------------------- +// Review of #242: charged lookups, the listing budget, exact name matching +// --------------------------------------------------------------------------- + +#[test] +fn resolving_a_symbolic_link_is_a_charged_read() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("real")).unwrap(); + for i in 0..5 { + symlink("real", t.path().join(format!("link{i}"))).unwrap(); + } + // `*/` lists 6 entries (1 open + 6 entries = 7 reads) and then resolves + // each of the 5 links to learn whether it is a directory: 12 reads. + let all = expand_glob_with_limits("*/", t.path(), &GlobLimits::DEFAULT).unwrap(); + assert_eq!(all.paths.len(), 6); + assert_eq!(all.usage.reads.total(), 12); + + // With room for the listing only, the first lookup is refused. Before, + // the lookups were free and the expansion finished. + let limits = GlobLimits { + max_reads: 7, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("*/", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "file-system reads"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (7, 8)); +} + +#[test] +fn the_listing_limit_stops_a_directory_that_is_too_large_to_hold() { + let t = TempDir::new().unwrap(); + fs::create_dir(t.path().join("big")).unwrap(); + for i in 0..10 { + fs::write(t.path().join(format!("big/f{i}")), b"").unwrap(); + } + // Each name is 2 bytes + 1, so 10 names need 30 bytes. + let limits = GlobLimits { + max_listing_bytes: 29, + ..GlobLimits::DEFAULT + }; + let s = stopped(expand_glob_with_limits("big/*", t.path(), &limits)); + assert_eq!(s.exceeded.axis, "bytes of one directory listing held"); + assert_eq!((s.exceeded.limit, s.exceeded.level), (29, 30)); + assert_eq!(s.at, PathBuf::from("big")); + + let limits = GlobLimits { + max_listing_bytes: 30, + ..GlobLimits::DEFAULT + }; + let ok = expand_glob_with_limits("big/*", t.path(), &limits).unwrap(); + assert_eq!(ok.paths.len(), 10); +} + +#[test] +fn an_invalid_byte_is_not_matched_as_a_replacement_character() { + use std::ffi::OsStr; + use std::os::unix::ffi::OsStrExt; + let t = TempDir::new().unwrap(); + let invalid = OsStr::from_bytes(b"bad\xff.txt"); + if fs::write(t.path().join(invalid), b"").is_err() { + return; // this file system refuses names that are not UTF-8 + } + fs::write(t.path().join("bad\u{fffd}.txt"), b"").unwrap(); + + // Broad patterns match both names. + for pattern in ["*.txt", "bad?.txt"] { + let found = expand_glob(pattern, t.path()).unwrap(); + assert_eq!(found.len(), 2, "{pattern}"); + } + // A literal U+FFFD matches only the name that really contains it. + let found = expand_glob("bad\u{fffd}*", t.path()).unwrap(); + assert_eq!(found, [PathBuf::from("bad\u{fffd}.txt")]); +} diff --git a/impl/rust-cli/tests/security_tests.rs b/impl/rust-cli/tests/security_tests.rs index 5cee2c9..c9b1b59 100644 --- a/impl/rust-cli/tests/security_tests.rs +++ b/impl/rust-cli/tests/security_tests.rs @@ -430,12 +430,29 @@ fn security_recursive_glob_bounded() { } // Total directories: 3^1 + 3^2 + ... + 3^5 = 363 - // Glob should have depth limit or timeout for larger structures - - // This would be a recursive glob like: **/* - // Should either limit depth or timeout gracefully - - // A production glob-depth limit remains tracked by #191. + let all = vsh::glob::expand_glob("**/*", temp.path()).unwrap(); + assert_eq!(all.len(), 363); + + // Under a smaller limit the same walk stops with a typed error that names + // the limit, instead of returning a truncated list (issue #191). + let limits = vsh::glob::GlobLimits { + max_matches: 100, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/*", temp.path(), &limits).unwrap_err(); + let stopped = err + .downcast_ref::() + .expect("a limit error"); + assert_eq!(stopped.exceeded.axis, "matches held"); + assert_eq!(stopped.exceeded.limit, 100); + + // Walk budget: a pattern that matches nothing is still bounded. + let limits = vsh::glob::GlobLimits { + max_reads: 50, + ..vsh::glob::GlobLimits::DEFAULT + }; + let err = vsh::glob::expand_glob_with_limits("**/none", temp.path(), &limits).unwrap_err(); + assert!(err.is::()); } // ============================================================ diff --git a/impl/rust-cli/tests/word_expansion_e2e_tests.rs b/impl/rust-cli/tests/word_expansion_e2e_tests.rs new file mode 100644 index 0000000..888adee --- /dev/null +++ b/impl/rust-cli/tests/word_expansion_e2e_tests.rs @@ -0,0 +1,243 @@ +// SPDX-License-Identifier: MPL-2.0 +// Copyright (c) Jonathan D.A. Jewell +//! End-to-end tests for argument glob expansion, run through the `vsh` binary. +//! +//! Before the word-expansion stage (`vsh::word`), globs were expanded only +//! for external commands that also had a redirection: `wc -c *.txt` passed +//! `*.txt` to `wc`, pipelines and `echo` never expanded, and quoted patterns +//! reached the program with a stray backslash (`echo '*.txt'` printed +//! `\*.txt`). Each test runs `vsh -c` in a directory holding `a.txt`, +//! `b.txt` and `c.log`. + +use std::path::Path; +use std::process::{Command, Output}; + +use tempfile::TempDir; + +fn fixture() -> TempDir { + let dir = TempDir::new().unwrap(); + for name in ["a.txt", "b.txt", "c.log"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + dir +} + +fn vsh(dir: &Path, script: &str) -> Output { + Command::new(env!("CARGO_BIN_EXE_vsh")) + .arg("-c") + .arg(script) + .current_dir(dir) + .output() + .expect("vsh runs") +} + +fn stdout(dir: &Path, script: &str) -> String { + let out = vsh(dir, script); + assert!( + out.status.success(), + "`{script}` failed: {}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8(out.stdout).unwrap() +} + +/// The lines of `text` with runs of white space made one space, so the column +/// padding of `wc` does not matter. +fn rows(text: &str) -> Vec { + text.lines() + .map(|line| line.split_whitespace().collect::>().join(" ")) + .collect() +} + +#[test] +fn external_command_without_redirect_gets_the_matches() { + let dir = fixture(); + assert_eq!( + rows(&stdout(dir.path(), "wc -c *.txt")), + ["0 a.txt", "0 b.txt", "0 total"] + ); +} + +#[test] +fn external_command_with_redirect_gets_the_matches() { + let dir = fixture(); + stdout(dir.path(), "wc -c *.txt > out.wc"); + assert_eq!( + rows(&std::fs::read_to_string(dir.path().join("out.wc")).unwrap()), + ["0 a.txt", "0 b.txt", "0 total"] + ); +} + +#[test] +fn every_pipeline_stage_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "ls *.txt | wc -l").trim(), "2"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.log | cat"), "c.log\n"); +} + +#[test] +fn builtin_echo_gets_the_matches() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.txt"), "a.txt b.txt\n"); + assert_eq!(stdout(dir.path(), "echo -n *.log"), "c.log"); +} + +#[test] +fn quoted_patterns_stay_literal_without_a_backslash() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo '*.txt'"), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \"*.txt\""), "*.txt\n"); + assert_eq!(stdout(dir.path(), "echo \\*.txt"), "*.txt\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' \"*.txt\" \\*.txt"), + "*.txt\n*.txt\n*.txt\n" + ); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' '*.txt' | cat"), + "*.txt\n" + ); +} + +#[test] +fn a_pattern_that_matches_nothing_is_passed_literally() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo *.none"), "*.none\n"); + assert_eq!(stdout(dir.path(), "printf '%s\\n' *.none"), "*.none\n"); + assert_eq!( + stdout(dir.path(), "printf '%s\\n' *.none > out.none; cat out.none"), + "*.none\n" + ); +} + +#[test] +fn a_partly_quoted_word_globs_only_its_unquoted_part() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + assert_eq!(stdout(dir.path(), "echo '*'x.*"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "echo *x.*"), "*x.txt ax.txt\n"); +} + +#[test] +fn braces_expand_unquoted_and_stay_literal_quoted() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo a{1,2} '{a,b}'"), "a1 a2 {a,b}\n"); +} + +#[test] +fn function_arguments_get_the_matches() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "f() { echo \"$#:$1:$2\"; }; f *.txt"), + "2:a.txt:b.txt\n" + ); +} + +// Regressions from the review of #242. Each of these once panicked, hung or +// gave the wrong arguments. + +#[test] +fn non_ascii_text_does_not_hang_or_panic() { + let dir = fixture(); + // The statement splitter looped for ever on an unquoted non-ASCII letter. + assert_eq!(stdout(dir.path(), "echo é"), "é\n"); + assert_eq!(stdout(dir.path(), "if true; then echo ünï; fi"), "ünï\n"); + // Brace expansion sliced the word at a character count. + assert_eq!(stdout(dir.path(), "echo 'é'{a,b}"), "éa éb\n"); + // So did `${...}`. + assert_eq!( + stdout(dir.path(), "é=1; echo ${é} ${é:-f} ${u:-é} ${#é}"), + "1 1 é 1\n" + ); +} + +#[test] +fn a_quoted_parenthesis_does_not_end_a_command_substitution() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo $(echo ')')"), ")\n"); + assert_eq!( + stdout(dir.path(), "echo $(echo \"(\") $(echo \\))"), + "( )\n" + ); + assert_eq!( + stdout(dir.path(), "echo $(echo $(echo a)) $((1+(2*3)))"), + "a 7\n" + ); +} + +#[test] +fn a_quoted_hyphen_in_a_bracket_is_not_a_range() { + let dir = fixture(); + for name in ["-.txt", "m.txt", "z.txt"] { + std::fs::write(dir.path().join(name), b"").unwrap(); + } + assert_eq!( + stdout(dir.path(), "echo [a'-'z]*.txt"), + "-.txt a.txt z.txt\n" + ); + assert_eq!( + stdout(dir.path(), "echo [a-z]*.txt"), + "a.txt b.txt m.txt z.txt\n" + ); +} + +#[test] +fn a_tilde_with_a_quoted_prefix_is_not_expanded() { + let dir = fixture(); + assert_eq!(stdout(dir.path(), "echo ~'local'/x"), "~local/x\n"); +} + +#[test] +fn a_parameter_before_a_quoted_literal_keeps_its_name_in_a_list() { + let dir = fixture(); + assert_eq!( + stdout(dir.path(), "A=x; echo $A''suffix '$A' \"$A\"b && true"), + "xsuffix $A xb\n" + ); + assert_eq!(stdout(dir.path(), "A=x; echo $A''suffix"), "xsuffix\n"); +} + +#[test] +fn a_substring_length_past_the_end_gives_the_rest() { + let dir = fixture(); + assert_eq!( + stdout( + dir.path(), + "X=abcdef; echo ${X:2:18446744073709551615} ${X:1:2}" + ), + "cdef bc\n" + ); +} + +#[test] +fn a_mixed_quote_word_keeps_its_quoting_in_a_list() { + let dir = fixture(); + std::fs::write(dir.path().join("*x.txt"), b"").unwrap(); + std::fs::write(dir.path().join("ax.txt"), b"").unwrap(); + // The operands of && were written back with one quoting for the whole + // word, so the unquoted `.*` stopped being a pattern. + assert_eq!(stdout(dir.path(), "echo '*'x.* && true"), "*x.txt\n"); + assert_eq!(stdout(dir.path(), "false || echo *x.*"), "*x.txt ax.txt\n"); + assert_eq!(stdout(dir.path(), "echo '*.txt' && true"), "*.txt\n"); +} + +#[cfg(unix)] +#[test] +fn an_external_command_gets_the_exact_bytes_of_a_matched_name() { + use std::os::unix::ffi::OsStrExt; + let dir = fixture(); + let name = std::ffi::OsStr::from_bytes(b"bad\xff.bin"); + if std::fs::write(dir.path().join(name), b"12345").is_err() { + return; // this file system refuses names that are not UTF-8 + } + // A lossy name would not exist, and `cat` would fail. + assert_eq!(stdout(dir.path(), "cat *.bin"), "12345"); + assert_eq!(stdout(dir.path(), "cat *.bin | cat"), "12345"); +} + +#[test] +fn a_later_pipeline_stage_gets_the_matches() { + let dir = fixture(); + let out = stdout(dir.path(), "printf '' | wc -c *.txt"); + assert_eq!(rows(&out), ["0 a.txt", "0 b.txt", "0 total"]); +}