Skip to content

ls: preserve bytes in non-UTF-8 ignore patterns - #15501

Open
limadog9 wants to merge 1 commit into
uutils:mainfrom
limadog9:fix/ls-non-utf8-ignore-patterns-7161
Open

limadog9 wants to merge 1 commit into
uutils:mainfrom
limadog9:fix/ls-non-utf8-ignore-patterns-7161

Conversation

@limadog9

Copy link
Copy Markdown
Contributor

Use native glob matching for non-UTF-8 Unix filenames and ignore/hide patterns. Add regressions distinguishing invalid bytes from literal replacement characters.

Closes #7161.

Comment thread src/uu/ls/src/config.rs Outdated

impl IgnorePattern {
fn new(pattern: &OsStr) -> Result<Self, glob::PatternError> {
// Lossy decoding is sufficient for validating wildcard syntax. Matching

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pattern is only used for validation when non_utf8, do we need both bytes and non_utf8? bytes: Option could carry it

@sylvestre

Copy link
Copy Markdown
Contributor

please make the matching consistent (same matcher for UTF-8 and non-UTF-8 names), then it should be good to go.

Comment thread src/uu/ls/src/config.rs
let Some(pattern) = &self.bytes else {
return false;
};
let Ok(name) = std::ffi::CString::new(name.as_encoded_bytes()) else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this allocates a CString per entry and per pattern, could we avoid it?

Comment thread src/uu/ls/src/config.rs Outdated
case_sensitive: true,
};
#[cfg(all(unix, not(target_os = "redox")))]
if self.non_utf8 || name.to_str().is_none() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same pattern now gives different results depending on the file name encoding: fnmatch handles a\* or [[:digit:]], the glob crate doesn't.
why not always use fnmatch on unix? it would also remove the dual path.

Comment thread tests/by-util/test_ls.rs
let ts = TestScenario::new(util_name!());
for name in [b"bad\xff.txt".as_slice(), "bad�.txt".as_bytes(), b"keep"] {
ts.fixtures.touch(OsStr::from_bytes(name));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also add a case with a UTF-8 pattern using \ or [[:alpha:]] against a non-UTF-8 name

Comment thread tests/by-util/test_ls.rs Outdated
.args(&["--zero", option])
.arg(OsStr::from_bytes(b"bad\xff.*"))
.succeeds();
result.stdout_contains("bad�.txt");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be a simple stdout_only_bytes(b"bad\xef\xbf\xbd.txt\0keep\0"), no?

Comment thread tests/by-util/test_ls.rs
.args(&["--zero", option, "bad*.txt"])
.succeeds()
.stdout_only("keep\0");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the comment says --ignore still applies but it isn't tested, please add it

@limadog9
limadog9 force-pushed the fix/ls-non-utf8-ignore-patterns-7161 branch 2 times, most recently from de5f68e to 5c7fa73 Compare October 19, 2026 02:16
Comment thread src/uu/ls/src/config.rs Outdated
impl IgnorePattern {
fn new(pattern: &OsStr) -> Result<Self, glob::PatternError> {
#[cfg(all(unix, not(target_os = "redox")))]
parse_glob::from_str(&pattern.to_string_lossy())?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we match with fnmatch but still check the pattern with the glob crate, so --ignore='a**b' or --ignore='[' gets a warning and is dropped. could we skip this check on unix?

Comment thread src/uu/ls/src/config.rs
pattern: parse_glob::from_str(&pattern.to_string_lossy())?,
#[cfg(all(unix, not(target_os = "redox")))]
bytes: std::ffi::CString::new(pattern.as_encoded_bytes()).ok(),
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an argument can't contain a NUL, do we need the Option here? the .ok() would silently drop the pattern

@limadog9
limadog9 force-pushed the fix/ls-non-utf8-ignore-patterns-7161 branch from 5c7fa73 to bc04ab3 Compare October 19, 2026 19:32
Use native glob matching for non-UTF-8 Unix filenames and ignore/hide patterns. Add regressions distinguishing invalid bytes from literal replacement characters.

Closes uutils#7161.
@limadog9
limadog9 force-pushed the fix/ls-non-utf8-ignore-patterns-7161 branch from bc04ab3 to 2612968 Compare October 19, 2026 22:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ls: glob support for non-UTF-8 missing

2 participants