stdbuf: shrink libstdbuf by building it without std by charly-mp · Pull Request #15461 · uutils/coreutils

charly-mp · 2026-10-17T19:47:32Z

libstdbuf is preloaded into every process stdbuf runs, and into their children.
Linking std made it much larger than needed, which also slows down the start of each of these processes.
Build it as no_std, which brings its size and load cost in line with GNU's.

no_std applies when panics abort, which stdbuf's build script forces for the embedded copy; other builds keep std, since a no_std cdylib cannot unwind on stable Rust.
A hidden rust_eh_personality stub lets it link without LTO, and libstdbuf's build script links the C library (as std did) and libgcc_s on ARM.
ctor's std feature is off, and the stream getters lose their unsafe marker.
CI now runs test_stdbuf in release, with and without LTO.

Benchmarks

	before	after	GNU 9.12
`libstdbuf.so` size	346 KB	15 KB	14 KB
`coreutils` multicall size	15.29 MB	14.96 MB (−331 KB)
preload overhead per process	52 µs	17 µs	19 µs
`grep -q` started under the library	404 µs	368 µs (−9 %)	368 µs
shell script running 100 commands	20.9 ms	17.2 ms (−18 %)	17.4 ms
`stdbuf -oL true` (embedded library)	987 µs	868 µs (−12 %)

Medians of 1,500 to 30,000 interleaved runs on an idle pinned core (Ryzen 9 9950X, 4.3 GHz locked).

github-actions · 2026-10-17T20:04:14Z

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 155.10 MB (-220 KB, -0.14%)

Significant per-binary changes:
  libstdbuf.so      268 KB ->      12 KB  (-256 KB, -95.52%)

xtqqczze · 2026-10-17T22:13:56Z

+    fn write_str(&mut self, s: &str) -> fmt::Result {
+        let mut bytes = s.as_bytes();
+        while !bytes.is_empty() {
+            let n = unsafe { libc::write(2, bytes.as_ptr().cast(), bytes.len()) };


Suggested change

let n = unsafe { libc::write(2, bytes.as_ptr().cast(), bytes.len()) };

let n = unsafe { libc::write(libc::STDERR_FILENO, bytes.as_ptr().cast(), bytes.len()) };

also, I think we could use the rustix crate here:

https://docs.rs/rustix/latest/rustix/io/fn.write.html

https://docs.rs/rustix/latest/rustix/stdio/fn.take_stdout.html

I replaced 2 by STDERR_FILENO.
For rustix, I tried but uucore's fs feature enables rustix/std and uu_stdbuf depends on libstdbuf so Cargo unifies the features and reintroduces std.

That's not ideal, but not really an issue so long as we use nothing that requires rustix/std?

Does rustix with std increase binary size again? If not, I prefer rustix.

Someone could accidentaly use API requiring rustix/std in the furure, but we can catch such change like #15461 (comment) .

Would moving libstdbuf to a separate workspace avoid the feature unification issue?

It's not about size: rustix/std makes rustix itself a std crate, so std gets linked and the build fails (duplicate panic_impl) whatever API we call.
A separate workspace doesn't help while uu_stdbuf depends on libstdbuf directly (tried it). And without std, getting the stderr fd from rustix is unsafe too, so it wouldn't remove an unsafe here.

In that case, the current approach is reasonable.

xtqqczze · 2026-10-17T22:37:38Z

+fn set_buffer_from_env(name: &CStr, get_stream: impl FnOnce() -> *mut FILE) {
+    // SAFETY: getenv returns null or a NUL-terminated string, which stays
+    // valid as nothing changes the environment while this runs.
+    let value = unsafe {


Each unsafe operation should have it's own unsafe block and SAFETY comment,

xtqqczze · 2026-10-17T22:54:01Z

 required-features = ["uudoc"]

 [profile.release]
+# libstdbuf is no_std in release builds, which needs LTO to link.


Can we still build without LTO now? What happens when building with:

CARGO_PROFILE_RELEASE_LTO=false cargo build --release

Yes, LTO is mandatory for release builds, is that a problem?
I could keep non-LTO builds working with an empty rust_eh_personality stub (never called since panics abort), made hidden with global_asm! so it can't override the real one in programs like rustc, is that ok or too tricky?

What actually happens when LTO is disabled? If it results in a duplicate-symbol linker error, relying on LTO to work around it seems like a hack.

= note: ld.lld: error: undefined symbol: rust_eh_personality >>> referenced by core-5dac9b97830f4695.core.eb549dbce5e65809-cgu.0.rcgu.o:(DW.ref.rust_eh_personality) in archive /usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-5dac9b97830f4695.rlib collect2: error: ld returned 1 exit status = note: ld.lld: error: undefined symbol: rust_eh_personality >>> referenced by core-5dac9b97830f4695.core.eb549dbce5e65809-cgu.0.rcgu.o:(DW.ref.rust_eh_personality) in archive /usr/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-5dac9b97830f4695.rlib

I can build this branch successfully on macOS without LTO:

CARGO_PROFILE_RELEASE_LTO=false cargo build -p uu_stdbuf --release

This suggests that whether LTO is needed to eliminate references to rust_eh_personality depends on the target and linker configuration.

I think we should avoid breaking non-LTO builds if possible. I'm also concerned that LTO may not necessarily eliminate every reference to rust_eh_personality, so relying on it to resolve the issue might not be a robust solution.

Fixed: non-LTO builds now link, through a hidden rust_eh_personality stub (never called since panics abort). CI runs test_stdbuf in release with and without LTO.

sylvestre · 2026-10-18T08:04:56Z

+// library is preloaded into every process stdbuf runs, and embedded in stdbuf
+// itself. A no_std cdylib needs panics to abort and LTO on stable Rust, so
+// other builds keep std.
+#![cfg_attr(all(panic = "abort", not(debug_assertions)), no_std)]


this cfg doesn't check for LTO, so a release build with CARGO_PROFILE_RELEASE_LTO=false (distros do that) would fail to link, no?
could we gate it on an explicit cfg/feature instead?

I've checked locally and a build with CARGO_PROFILE_RELEASE_LTO=off fails to link with undefined reference to rust_eh_personality. Something like this could be added to avoid the issue:

#[cfg(all(panic = "abort", not(debug_assertions)))] #[unsafe(no_mangle)] extern "C" fn rust_eh_personality() {}

Thanks, that's what I added, but with hidden visibility: an exported stub replaces the real routine of programs that export one, e.g. rustc aborts with "failed to initiate panic" when run under stdbuf.

Since the no_std build no longer depends on LTO, I kept the panic = "abort" cfg; happy to add an explicit feature if you still want one.

sylvestre · 2026-10-18T08:05:02Z

+#[cfg(all(panic = "abort", not(debug_assertions)))]
+#[panic_handler]
+fn panic(_: &core::panic::PanicInfo) -> ! {
+    unsafe { libc::abort() }


please add a SAFETY comment here, on the write() and on the exit() too

sylvestre · 2026-10-18T08:05:12Z

-    if let Ok(val) = env::var("_STDBUF_O") {
-        set_buffer(unsafe { __stdbuf_get_stdout() }, &val);
-    }
+    set_buffer_from_env(c"_STDBUF_E", || __stdbuf_get_stderr());


the tests run in debug, so CI never runs the no_std version.
could you please add a CI step running test_stdbuf with --release?

could we build the no_std version in debug profile as well?

Added a CI job running test_stdbuf in release, with and without LTO.

Building the no_std version in debug too would mean forcing panic=abort for the nested libstdbuf build in stdbuf's build.rs.
Should I do it here or in a follow-up?

Since we define our own panic_handler, do we actually need to gate on panic = "abort"?

I'd also prefer to use no_std consistently across all build configurations rather than switching between std in debug and no_std in release. Keeping the configurations consistent would make debugging more predictable. Is there a reason we can't build with no_std in both?

The gate is still needed: even with our own panic_handler, a no_std cdylib fails to build with panic=unwind, and Cargo doesn't allow setting panic per package.
stdbuf's build.rs now forces panic=abort for the nested build, so the embedded library is no_std in every profile,debug included.
Only the external library built in an unwinding profile keeps std.

sylvestre · 2026-10-18T08:05:31Z

@Ecordonnier wdyt ? :)

Ecordonnier · 2026-10-18T15:07:24Z


-use core::ffi::{c_char, c_int};
+// Release builds (no debug assertions, panics abort, LTO) leave std out: the
+// library is preloaded into every process stdbuf runs, and embedded in stdbuf


"and embedded in stdbuf itself" is not true when building with feat_external_libstdbuf (which I recommend using for all cases except when installing from cargo), so I would remove this sentence.

Ecordonnier · 2026-10-18T15:39:49Z

+// library is preloaded into every process stdbuf runs, and embedded in stdbuf
+// itself. A no_std cdylib needs panics to abort and LTO on stable Rust, so
+// other builds keep std.
+#![cfg_attr(all(panic = "abort", not(debug_assertions)), no_std)]


I've checked locally and a build with CARGO_PROFILE_RELEASE_LTO=off fails to link with undefined reference to rust_eh_personality. Something like this could be added to avoid the issue:

#[cfg(all(panic = "abort", not(debug_assertions)))] #[unsafe(no_mangle)] extern "C" fn rust_eh_personality() {}

Ecordonnier · 2026-10-18T15:48:12Z

@Ecordonnier wdyt ? :)

This looks good. I would however change the PR title and description. Unless I am missing something, this is mostly about making binaries smaller. The few microseconds gained to load libstdbuf are nice to have, but aren't that relevant on any process doing serious work.

xtqqczze · 2026-10-18T16:08:33Z

Unless I am missing something, this is mostly about making binaries smaller.

The following result also look quite significant:

before after GNU 9.4

shell script running 100 commands 21.0 ms 17.3 ms (−18 %) 17.4 ms

@charly-mp That being said, could you compare against GNU 9.12 please.

Ecordonnier · 2026-10-18T17:42:15Z

Unless I am missing something, this is mostly about making binaries smaller.

The following result also look quite significant:

before after GNU 9.4

shell script running 100 commands 21.0 ms 17.3 ms (−18 %) 17.4 ms

@charly-mp That being said, could you compare against GNU 9.12 please.

Just keep in mind the use case for stdbuf is long running processes which are fully buffered / block buffered, so saving a few ms in the context of spawning 100 processes isn't very significant IMO.

On the other hand the significant size reduction is quite critical to drive adoption on small embedded targets.

charly-mp · 2026-10-18T18:12:56Z

Agreed, size is the main point, especially for small embedded targets: I retitled the PR and updated the description.
@xtqqczze the comparison with GNU 9.12 is in the table: the library is now on par with it, in size and load cost.

xtqqczze · 2026-10-18T18:30:56Z

+# Its std feature would pull std back into libstdbuf, which leaves it out.
+ctor = { version = "1.0.0", default-features = false, features = [
+  "proc_macro",
+] }


Suggested change

] }

ctor = { version = "1.0.0", default-features = false, features = ["proc_macro"] }

I don't think the comment is necessary, it seems more like a narration of the diff. You could explain in the PR description instead.

Removed the comment.
Kept the wrapped form: the one-line version fails the taplo check.

xtqqczze · 2026-10-18T18:37:32Z

+    println!("cargo:rustc-link-arg=-lc");
+
+    // On ARM, unwind tables, including those of the prebuilt core, refer to
+    // routines of libgcc_s, which std would link too.


cspell needs resolving

sylvestre · 2026-10-18T21:23:33Z

-    if let Ok(val) = env::var("_STDBUF_O") {
-        set_buffer(unsafe { __stdbuf_get_stdout() }, &val);
-    }
+    set_buffer_from_env(c"_STDBUF_E", || __stdbuf_get_stderr());


the getters aren't unsafe anymore, so could you pass __stdbuf_get_stderr directly instead of the closure?

codspeed · 2026-10-19T08:43:36Z

Merging this PR will degrade performance by 15.86%

⚠️

121 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

cksum_crc

cksum_default

cksum_raw_output

sort_long_line[10000]

du_wide_tree[(5000, 500)]

cksum_sysv

cksum_crc32b

timeout_enforced

timer_expiry_latency[0.02]

du_deep_tree[(100, 3)]

…and 111 more

ℹ️ 1 benchmark has no flame graph

The flamegraph could not be generated because too much time was spent in runtime internals, making the visualization not useful for performance analysis.

sort_dictionary_order[500000]

❌ 1 regressed benchmark
✅ 400 untouched benchmarks
⏩ 70 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`true_consecutive_calls`	294.7 ns	350.2 ns	-15.86%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing charly-mp:libstdbuf-no-std (1ebe472) with main (6f60c70)²}

70 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on main (48f2a33) during the generation of this report, so 6f60c70 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

oech3 · 2026-10-19T08:45:42Z

        fi
        echo "OK: ariadne is not in the dependency tree"

+  test_stdbuf_release:


Can we remove this since no_std is used for all profiles now?

You're right, I removed it.

xtqqczze · 2026-10-19T18:54:39Z

+))]
+core::arch::global_asm!(".hidden rust_eh_personality");
+
+/// Unbuffered writes to file descriptor 2, whatever buffering stderr was given.


rewrite the comment now that we use libc::STDERR_FILENO

xtqqczze · 2026-10-19T18:56:20Z

error on powerpc64le-unknown-linux-gnu:

error[E0658]: inline assembly is not stable yet on this architecture
  --> src/uu/stdbuf/src/libstdbuf/src/libstdbuf.rs:40:1
   |
40 | core::arch::global_asm!(".hidden rust_eh_personality");
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: see issue #93335 <https://github.com/rust-lang/rust/issues/93335> for more information

libstdbuf is preloaded into every process stdbuf runs, and into their children. Linking std made it much larger than needed, which also slows down the start of each of these processes. Build it as no_std, which brings its size and load cost close to GNU's. no_std applies when panics abort, which stdbuf's build script forces for the embedded copy: a no_std cdylib cannot unwind on stable Rust, so other builds keep std. A hidden stub personality routine lets it link without LTO. libstdbuf's build script links the C library, as std did, and libgcc_s for the unwind tables of ARM. ctor's std feature is off. The stream getters lose their unsafe marker: they have no safety precondition. CI runs test_stdbuf in release, with and without LTO.

charly-mp changed the title ~~stdbuf: speed up loading of libstdbuf~~ WIP stdbuf: speed up loading of libstdbuf Oct 17, 2026

charly-mp marked this pull request as draft October 17, 2026 19:56

charly-mp force-pushed the libstdbuf-no-std branch from 029cd83 to 4645661 Compare October 17, 2026 21:23

charly-mp changed the title ~~WIP stdbuf: speed up loading of libstdbuf~~ stdbuf: speed up loading of libstdbuf Oct 17, 2026

charly-mp marked this pull request as ready for review October 17, 2026 21:46

charly-mp force-pushed the libstdbuf-no-std branch from 4645661 to 4fdf20c Compare October 17, 2026 22:13

xtqqczze reviewed Oct 17, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from 4fdf20c to b10fa18 Compare October 17, 2026 22:17

xtqqczze reviewed Oct 17, 2026

View reviewed changes

sylvestre reviewed Oct 18, 2026

View reviewed changes

Ecordonnier reviewed Oct 18, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from b10fa18 to 9ed5681 Compare October 18, 2026 17:59

charly-mp changed the title ~~stdbuf: speed up loading of libstdbuf~~ stdbuf: shrink libstdbuf by building it without std Oct 18, 2026

xtqqczze reviewed Oct 18, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from 9ed5681 to af77eff Compare October 18, 2026 18:58

sylvestre reviewed Oct 18, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from af77eff to 68fbbe6 Compare October 18, 2026 22:32

oech3 reviewed Oct 19, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from 68fbbe6 to b384781 Compare October 19, 2026 18:48

xtqqczze reviewed Oct 19, 2026

View reviewed changes

charly-mp force-pushed the libstdbuf-no-std branch from b384781 to 1ebe472 Compare October 19, 2026 18:57

	let n = unsafe { libc::write(2, bytes.as_ptr().cast(), bytes.len()) };
	let n = unsafe { libc::write(libc::STDERR_FILENO, bytes.as_ptr().cast(), bytes.len()) };

	] }
	ctor = { version = "1.0.0", default-features = false, features = ["proc_macro"] }

Conversation

charly-mp commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

github-actions Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xtqqczze Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xtqqczze Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charly-mp Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sylvestre commented Oct 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ecordonnier commented Oct 18, 2026

Uh oh!

xtqqczze commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ecordonnier commented Oct 18, 2026

charly-mp commented Oct 17, 2026 •

edited

Loading

github-actions Bot commented Oct 17, 2026 •

edited

Loading

xtqqczze Oct 17, 2026 •

edited

Loading

xtqqczze Oct 17, 2026 •

edited

Loading

charly-mp Oct 18, 2026 •

edited

Loading

xtqqczze commented Oct 18, 2026 •

edited

Loading

codspeed Bot commented Oct 19, 2026 •

edited

Loading