Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,111 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

== [Unreleased]

=== Fixed — 2026-11-18 (arguments: globs, quote removal)

- Glob expansion now applies to the arguments of every external command,
every pipeline stage, `echo` and function calls. Before, only an external
command that also had a redirection expanded its arguments: `wc -c *.txt`
passed `*.txt` to `wc`, `ls *.txt | wc -l` and `echo *.txt` did not expand.
- Quoted arguments no longer reach the program with a backslash in front of
`*`, `?`, `[` or `{`: `echo '*.txt'` printed `\*.txt` and now prints
`*.txt`. A partly quoted word globs only its unquoted part (`'*'x.*`).
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- `echo` performs command substitution and arithmetic expansion
(`echo $((2+3))` printed `$((2+3))`), and `$(echo ...)` works.
- `$((1+(2*3)))` and other substitutions containing `(` are read to their
matching `)`.
- An expansion error in an argument (such as a glob limit from #191) fails
the command with exit status 1 and does not run it. With a redirection the
status was 127. An error while the program name or a pipeline stage name
is expanded is not changed: it is returned to the caller, so `||` sees it.

=== Fixed — 2026-11-19 (review of #242, small findings)

- An AND-OR list keeps the quoting of each piece of a word: `echo '*'x.* && true`
printed `*x.*` and now expands the unquoted part.
- A glob match reaches an external command with the exact bytes of its file
name. A name that is not UTF-8 was replaced by a lossy copy that did not
exist.
- A name that is not UTF-8 is no longer matched by a literal U+FFFD in the
pattern. `*` and `?` still match it.
- Resolving a symbolic link to learn whether it is a directory is charged as
a read unit; it was free.
- New limit `max_listing_bytes` (64 MiB): the names held while one directory
is listed. Exceeding it is a `GlobLimitExceeded` that names the directory.
- A `Meter` charge that would pass `u64::MAX` is refused and reported
exactly; `Exceeded::level` is now `u128`. Before, the level saturated and
a limit of `u64::MAX` accepted the charge.
- `Expansion::fields` in `src/word.rs` is `Vec<OsString>`; the external
command functions take `OsString` arguments. `glob::escape` is new.
- The `fuzz_glob_expansion` target removes its fixture at exit, escapes the
fixture root in absolute patterns, and checks that `/**/*` reaches the
fixture.
- Tests: `wc` output is compared without its column padding; a glob in a
later pipeline stage is covered.

=== Fixed — 2026-11-18 (review of #242)

- An unquoted non-ASCII letter no longer hangs the shell: `echo é` did not
return, because the statement splitter did not advance past it.
- `'é'{a,b}` and `${é:-x}` no longer panic or misread the word: brace and
parameter expansion used character counts as byte offsets.
- `${X:2:N}` with a very large `N` gives the rest of the value; it panicked
on overflow.
- A quoted or escaped parenthesis no longer ends a command substitution:
`$(echo ')')`.
- A quoted hyphen in a bracket expression is a member, not a range:
`[a'-'z]*.txt` no longer matches `m.txt`.
- `~'local'/x` keeps its tilde: a quoted piece in the tilde prefix stops
the expansion.
- `$A''suffix && true` expands `A`: in an AND-OR list the word was read
back as `$Asuffix`.

=== Changed — 2026-11-18 (typed words)

- New module `src/word.rs`: a `Word` keeps the quoting of each of its pieces
(unquoted, single, double, backslash), and `word::expand_words` performs
tilde, parameter, command, arithmetic and process substitution, brace
expansion, glob expansion and quote removal on it. `Command::External`,
`Command::Pipeline` and `Command::Echo` now carry `Vec<Word>` arguments.
Other commands still use the older string encoding; see
`docs/POSIX_COMPLIANCE.adoc` (Milestones 5 and 6) for what is not migrated.
- Removed `external::expand_glob_args`; its rules (no match or invalid
pattern keeps the word, a limit is an error) moved to `word.rs` with their
tests.

=== Fixed — 2026-11-18 (glob expansion out of memory, #191)

- `**` no longer follows symbolic links to directories, so a link back to a
parent directory can no longer make an expansion grow without end. On the old
code `**/*` over a 7-entry tree with two such links passed 2.5 GB in 15 s,
and `**/no-such-name` over it never returned. ClusterFuzzLite hit the same
fault with `/**/*` (libFuzzer out-of-memory at 2560 MB, PR #230, check run
134118693669). The rule now matches bash 5.2 with `globstar` for patterns
that start with `**`.
- Glob expansion is metered while it runs: at most 10,000,000 walker read
units, 100,000 matches and 6 MiB of matches per command. A read unit is
charged for each directory opened, each entry listed, each name looked up
and each symbolic link resolved. The type of an entry comes with the
listing and is not charged again. The limit is on these units; it is not
a cap on every file-system operation the shell makes. Reaching a limit is
a `GlobLimitExceeded` error that names the limit, the usage so far and the
directory being read; the command is not run. Before, the match limit was
checked only after every path had been collected, and an external command
then received the literal pattern instead of an error.
- `.*` and `.h*` find hidden files, as bash does.
- A file name that is not valid UTF-8 no longer panics the expansion.
- The `fuzz_glob_expansion` target expands over a small fixed tree instead of
the runner's file system, and checks the limits on every input.
- `impl/rust-cli/fuzz/Cargo.lock` regenerated; it was missing `hmac` and
`subtle`.

=== Added — 2026-11-18

- `src/grade.rs`: resource grades. A cost adds up in sequence; an occupancy
keeps a high-water mark. A `Meter` refuses the charge that would pass its
limit. Glob expansion is the first user; operation-history retention (#190)
is the intended second. The grade laws are property-tested, not proved.

=== Changed — 2026-11-01 (assurance-boundary re-audit)

- Corrected prose that had drifted across the assurance boundary. The audit
Expand Down
15 changes: 14 additions & 1 deletion EXPLAINME.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,18 @@ reproduce. Absence of a tool is an open obligation, not a passing leg
| `proofs/z3/` (`archive/proofs/` for Isabelle and Mizar)
| §4 exception notes are now recorded in each `.smt2` header. `filesystem_operations` and `symlink_operations` were rewritten on 2026-11-09 to define the operations as array `store`s and negate each goal; the other three still rest on asserted axioms. Isabelle and Mizar were never CI-checked and are archived, unchecked, with a successor map

| Glob expansion stays within its read bound on the test fixtures (frontier A-6)
| tested-only
| `cargo test --test glob_bounds_tests`; `proptest` in `src/glob.rs` and `src/grade.rs`
| `impl/rust-cli/tests/glob_bounds_tests.rs`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6
| Fixed fixtures only: the reported read count equals a count made from the fixture, and a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. The grade laws are property-tested, not proved

| Glob expansion terminates within `O(|pattern| x |entries|)` reads (frontier A-6)
| open
| -
| `impl/rust-cli/src/glob.rs` `expand_glob`; `docs/PROOF-OPEN-FRONTIER.adoc` A-6
| No Creusot or Verus contract on the walker (PROOF-POLICY §1), and no Lean model bound. A Lean bound would be model evidence only

| Sandbox path normalisation never escapes the root
| proven (model) / tested-only (code)
| Lean 4 for the model; `proptest` for the code
Expand Down Expand Up @@ -212,7 +224,8 @@ All share: *Multiple proof systems for cross-validation, honest status reporting

| `impl/rust-cli/src/redirection.rs` | I/O redirection: > (write), >> (append), < (read), 2> (stderr), 2>> (append stderr), &> (both), 2>&1 (combine)

| `impl/rust-cli/src/glob.rs` | POSIX-compliant glob expansion: *, ?, [...], {...}. Hidden file handling (requires explicit leading dot).
| `impl/rust-cli/src/glob.rs` | Glob expansion: *, ?, [...], `**` (bash 5.2 globstar rules, one documented difference for symlinked directories), {...}. Hidden file handling (requires explicit leading dot). Bounded walk: limits on reads, matches and match bytes, with a typed error (#191).
| `impl/rust-cli/src/grade.rs` | Resource grades: cumulative cost and high-water-mark occupancy, and a meter that refuses the charge over a limit. Laws are property-tested, not proved.

| `impl/rust-cli/src/job_control.rs` | Job control: bg, fg, jobs, kill commands. Signal handling for Ctrl+Z (SIGTSTP), Ctrl+C (SIGINT).

Expand Down
53 changes: 41 additions & 12 deletions docs/POSIX_COMPLIANCE.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,14 @@ M12-14 not started
|2 |Redirections |*Complete* |All 8 types including `+2>&1+`
|3 |Pipelines |*Complete* |Multi-stage, exit code from last
|4 |Variables |*Complete* |Scalars, arrays, parameter expansion
|5 |Glob Expansion |*Complete* |POSIX-compliant + brace expansion
|6 |Quote Processing |*Complete* |Single, double, backslash
|5 |Glob Expansion |*Partial* |Matching complete; not yet applied to builtin operands, `+for+` word lists, redirection targets
|6 |Quote Processing |*Partial* |Quote removal complete only for arguments that go through `+word::expand_words+`
|7 |Command Substitution |*Complete* |`+$(cmd)+` and backticks
|8 |Arithmetic |*Complete* |Full operator set including `+**+`
|9 |Control Structures |*Complete* |if/elif/else, while, for, case/esac
|10 |Job Control |*Partial* |No SIGCHLD, no Ctrl+Z
|11 |Shell Builtins |*Partial* |~30 implemented, ~15 missing
|12 |Functions |*Not started* |No `+func() { ... }+` syntax
|12 |Functions |*Partial* |`+name() { ... }+` definitions and calls work; nesting incomplete
|13 |Shell Scripts |*Not started* |No `+.sh+` file execution
|14 |Advanced Features |*Partial* |Here docs, process sub, arrays done
|===
Expand Down Expand Up @@ -147,12 +147,13 @@ quoted context) - `+${VAR:=default}+` does not actually assign — same as
`+${VAR:-default}+` - `+export VAR+` on unset var errors instead of
creating empty exported var

==== Milestone 5: Glob Expansion — COMPLETE
==== Milestone 5: Glob Expansion — PARTIAL

[width="100%",cols="29%,25%,46%",options="header",]
|===
|Feature |Status |Implementation
|`+*+` (match any) |Done |`+glob+` crate
|`+*+` (match any) |Done |`+glob+` crate `+Pattern+` per component;
directory walk in `+glob.rs+` (2026-11-18)

|`+?+` (single char) |Done |

Expand All @@ -161,14 +162,31 @@ creating empty exported var
|`+[!...]+` negation |Done |

|Hidden file exclusion |Done |`+require_literal_leading_dot: true+`
(fixed 2026-02-12)
(fixed 2026-02-12); `+.*+` and `+.h*+` now find hidden files (fixed
2026-11-18; they previously gave only `+.+`/`+..+` or nothing)

|Recursive `+**+` (bash globstar) |Done |Does not descend through symbolic
links or into hidden directories. One known difference from bash 5.2: bash
enters a symlinked directory for one more component when `+**+` is not the
first component (2026-11-18)

|Expansion limits |Done |10,000,000 walker read units (directories opened,
entries listed, names looked up, links resolved), 100,000 matches, 6 MiB of
matches, 64 MiB of names for one directory listing; exceeding one is an error that names it, never a truncated
or literal result (#191, 2026-11-18)

|Brace expansion `+{a,b,c}+` |Done |Nested braces supported

|Sorted results |Done |POSIX requirement

|Where globs expand |Partial |Arguments of external commands (with or
without redirections), every pipeline stage, `+echo+` and function calls,
through `+word::expand_words+` (2026-11-18; before, only external commands
with a redirection). Not yet: builtin operands (`+ls+`, `+rm+`, `+cp+`, ...),
`+for+` word lists, redirection targets
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
|===

==== Milestone 6: Quote Processing — COMPLETE
==== Milestone 6: Quote Processing — PARTIAL

[width="100%",cols="29%,25%,46%",options="header",]
|===
Expand All @@ -183,6 +201,12 @@ quotes
|Unclosed quote detection |Done |

|Line continuation |Done |

|Quote removal |Partial |Done for the arguments that go through
`+word::expand_words+` (see Milestone 5), which keeps the quoting of each
piece of a word. Builtin operands, redirection targets, assignment values,
`+case+` patterns and `+test+` arguments still receive quoted `+*?[{~+` with
a backslash in front (`+touch '*q'+` creates `+\*q+`)
|===

==== Milestone 7: Command Substitution — COMPLETE
Expand Down Expand Up @@ -231,8 +255,9 @@ condition chains

|`+while+`/`+do+`/`+done+` |Done |`+Command::While+` with break/continue

|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::For+` with glob
expansion in wordlist
|`+for+`/`+in+`/`+do+`/`+done+` |Done |`+Command::ForLoop+`; the word list gets
parameter expansion and `+$IFS+` splitting, but not glob expansion (see
Milestone 5)

|`+case+`/`+esac+` |Done |`+Command::Case+` with glob pattern matching

Expand Down Expand Up @@ -292,9 +317,13 @@ ops, numeric comparison
`+hash+`, `+umask+`, `+wait+`, `+getopts+`, `+shift+`, `+break+`,
`+continue+`, `+return+`, `+local+`

==== Milestone 12: Functions — NOT STARTED
==== Milestone 12: Functions — PARTIAL

No function definition or calling syntax exists in the parser.
`+name() { ... }+` and `+function name { ... }+` define a function, and a call
runs it with its own positional parameters, `+local+` and `+return+`. An
end-to-end test runs `+f() { echo "$#:$1:$2"; }; f *.txt+` through the built
shell (`+tests/word_expansion_e2e_tests.rs+`). Full nesting of functions is
not complete.

==== Milestone 13: Shell Scripts — NOT STARTED

Expand Down Expand Up @@ -355,7 +384,7 @@ composition |0
=== Critical Missing Features (by impact)

[arabic]
. *Functions* — No `+func() { ... }+` — blocks modular scripts
. *Functions* — definitions and calls work; nesting is incomplete
. *Shell script execution* — No `+.sh+` file running
. *Word splitting (`+$IFS+`)* — Unquoted expansions not split
. *Tilde expansion* — Only works for `+~/+` in `+cd+`
Expand Down
3 changes: 2 additions & 1 deletion docs/PROOF-OPEN-FRONTIER.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ Pure induction on list length.
| Statement | `expand_glob(pattern, base_dir)` terminates in time `O(|pattern| × |files_in_base_dir|)`.
| File | `impl/rust-cli/src/glob.rs` — no proof today.
| Why | Glob expansion is a frequent attack surface for DoS in shells. A complexity bound is a real security property.
| Lift | Property test (`proptest`) for termination + complexity, then a formal bound in Lean 4.
| Lift | Property test (`proptest`) for termination + complexity. A bound proved in Lean 4 would be about a Lean model of the walk: optional model evidence, not a statement about the Rust code. The Rust walker (`expand_glob`) needs source-level verification, which the proof policy assigns to Creusot with Pearlite contracts, and Verus.
| Progress (2026-11-18) | `tested-only`. The walk now processes each (directory, pattern component) pair at most once and meters reads, matches and match bytes (`src/grade.rs`). `tests/glob_bounds_tests.rs` checks the reported read count against counts made directly from fixtures, and that a 12-component pattern over a 31-entry chain stays within 2 x 12 x 31 reads. Still `open` as a proof: no Creusot or Verus contract exists on `expand_glob`, and none has been attempted; no Lean model bound exists either. Ledger rows: `EXPLAINME.adoc#proof-ledger` ("Glob expansion stays within its read bound on the test fixtures" and "Glob expansion terminates within `O(|pattern| x |entries|)` reads").
|===

[[A-7]]
Expand Down
21 changes: 21 additions & 0 deletions impl/rust-cli/fuzz/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions impl/rust-cli/fuzz/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ edition = "2021"
cargo-fuzz = true

[dependencies]
libc = "0.2"
libfuzzer-sys = "0.4"
tempfile = "3.8"
walkdir = "2.4"
Expand Down
Loading
Loading