Skip to content

fix(word): expand globs for every simple command via typed words - #244

Merged
hyperpolymath merged 2 commits into
claude/gifted-planck-mha9c9from
claude/keen-babbage-qptnu3
Nov 18, 2026
Merged

hyperpolymath merged 2 commits into
claude/gifted-planck-mha9c9from
claude/keen-babbage-qptnu3

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

Stacked on #242 (claude/gifted-planck-mha9c9), which adds glob::expand_glob_with_limits and GlobLimitExceeded. Once #242 merges, retarget this PR to main.

Problem (measured on the debug binary, directory with a.txt b.txt c.log)

Command Before Now (same as bash)
wc -c *.txt wc: '*.txt': No such file or directory 0 a.txt / 0 b.txt / 0 total
ls *.txt | wc -l ls: cannot access '*.txt' 2
echo *.txt *.txt a.txt b.txt
echo '*.txt' \*.txt *.txt
echo a{1,2} / echo $((2+3)) a{1,2} / $((2+3)) a1 a2 / 5

Cause: argument globbing lived only in external::expand_glob_args, which only execute_external_with_redirects called. There was also a second bug: quoted metacharacters were marked with a backslash in the token string, and nothing ever removed it.

Change: typed word IR

  • New src/word.rs. A Word records the quoting of each piece (unquoted, '…', "…", \x). expand_words runs, in order: tilde, parameter, command and arithmetic expansion, process substitution, then brace expansion, glob expansion and quote removal.
  • The tokenizer builds a Word next to the old string form. Command::External, Command::Pipeline and Command::Echo now carry Vec<Word>.
  • The new stage is used for external commands (foreground, background, with or without redirections), every pipeline stage, echo, function-call arguments, and the External branch of command substitution and process substitution.
  • Kept from fix(glob): bound expansion while it runs; stop ** following symlinks #242:
    • A pattern with no match, or an invalid pattern, stays as the literal word.
    • A GlobLimitExceeded fails the command without running it (vsh: prog: …, exit 1).
  • external::expand_glob_args is removed. Its rules and tests moved to word.rs.
  • Two smaller fixes:
    • $(…) is now read to its matching ), so $((1+(2*3))) is a single substitution.
    • $(echo …) is supported.

Behaviour changes

  • An expansion error used to come back from execute() as Err. In -c mode that printed Error: plus a backtrace. Now the command exits with status 1 and does not run.
  • An unquoted $X whose value contains glob characters is now globbed, as POSIX requires. A quoted "$X" is not.

Not migrated in this PR

These still read the old string encoding, so their behaviour is unchanged and still differs from bash:

  • builtin operands (touch '*q' creates \*q; ls *.txt as a builtin does not glob)
  • redirection targets
  • assignment values (X='*.txt' stores \*.txt)
  • case patterns, test/[[ ]] arguments, for word lists

Field splitting is still not done. All of this is documented in docs/POSIX_COMPLIANCE.adoc, Milestones 5 and 6.

Tests

  • tests/word_expansion_e2e_tests.rs runs the vsh binary. It covers:
    • an external command without a redirect, with a redirect, and in a pipeline
    • builtin echo
    • quoted patterns '…', "…" and \*, which stay literal with no backslash
    • a pattern that matches nothing
    • a partly quoted word ('*'x.*)
    • braces
    • function arguments
  • Unit tests in word.rs: the limit error, an invalid pattern, a literal backslash inside a pattern, quoted vs unquoted parameters, and refused process substitution.
  • A proptest in parser.rs: Display then tokenize gives back the same Word (256 cases). The domain excludes backticks inside double quotes, because the tokenizer has no ``` escape.
  • Differential run of 30 commands across the old binary, the new binary and bash: every changed result now matches bash, and no builtin case changed.
  • cargo fmt --check, cargo clippy --all-targets -- -D warnings: clean.
  • cargo test: 887 passed, 0 failed, 14 ignored (the same ignored set as before: stress tests, the model oracle and 2 in the lib).

Generated by Claude Code

Glob expansion lived only in `external::expand_glob_args`, which only
`execute_external_with_redirects` called. `wc -c *.txt` passed `*.txt` to
wc, pipeline stages and `echo` never expanded, and quoted metacharacters
reached programs with the tokenizer's escape backslash (`echo '*.txt'`
printed `\*.txt`).

The tokenizer now also builds a typed `word::Word` that records the
quoting of each piece (unquoted, single, double, backslash).
`word::expand_words` runs tilde, parameter, command, arithmetic and
process substitution, then brace expansion, glob expansion and quote
removal on it. External commands (with or without redirections and in
the background), every pipeline stage, `echo`, function calls,
command substitution and process substitution use it; `Command::External`,
`Command::Pipeline` and `Command::Echo` carry `Vec<Word>` arguments.

Rules kept from `expand_glob_args`: no match or an invalid pattern keeps
the word; a `GlobLimitExceeded` fails the command (exit 1) without
running it. Builtin operands, redirection targets, assignments, `case`
and `test` still use the old string encoding (documented in
POSIX_COMPLIANCE.adoc).

Also: `$( ... )` is read to its matching `)`, so `$((1+(2*3)))` is one
substitution, and `$(echo ...)` is supported.

Tests: unit tests in word.rs, a Display/tokenize round-trip property
test, and end-to-end tests through the vsh binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4QvDfX8Xoq6vedAKdWBif
SonarCloud rust:S3776 flagged the cognitive complexity of Display::fmt
on PR #244. The per-quoting literal case is now fmt_literal; output is
unchanged (round-trip property test and Display unit test pass).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4QvDfX8Xoq6vedAKdWBif
@hyperpolymath
hyperpolymath marked this pull request as ready for review November 18, 2026 15:03
Repository owner deleted a comment from coderabbitai Bot Nov 18, 2026
@hyperpolymath
hyperpolymath merged commit 5038be1 into claude/gifted-planck-mha9c9 Nov 18, 2026
10 checks passed
@hyperpolymath
hyperpolymath deleted the claude/keen-babbage-qptnu3 branch November 18, 2026 15:04
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants