Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/+dev-pure.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-local dev --pure` runs `module purge` and unsets `LD_LIBRARY_PATH`, `LD_PRELOAD` and `LD_AUDIT` before the stack loads, so modules and loader variables the shell already had do not reach the build environment.
12 changes: 8 additions & 4 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ eb-local [OPTION]... --explain <easyconfig> [eb options]
eb-local eb [--] [eb options]
eb-local --explain eb [--] [eb options]
eb-local [OPTION]... env [--strict] [--json]
eb-local [OPTION]... dev [--strict] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local [OPTION]... dev [--strict] [--pure] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
eb-local check [--recursive] [--quiet] [--json] FILE...
eb-local check (--module NAME/VERSION... | --dev | --prefix DIR) [--recursive] [--quiet] [--json]
Expand Down Expand Up @@ -60,14 +60,15 @@ directory you own.
eb-local env [--strict] [--json] lines that make your installed modules visible;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1)
eb-local dev [--strict] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local dev [--strict] [--pure] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
the same, plus MODULEs and a build environment,
so gcc, make, cmake and meson just work;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1);
--buildenv names the build environment to load;
--json prints the change as JSON, for env too;
--envrc writes that line to .envrc for direnv
--envrc writes that line to .envrc for direnv;
--pure purges loaded modules and LD_* first
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
lines that put RPATH wrappers for the compilers
on PATH first; dev uses them when no build
Expand Down Expand Up @@ -299,7 +300,7 @@ When stdout is a terminal, it also says on stderr that the lines change nothing

With =--json=, eb-local runs the lines in a bash child and prints how that shell's environment then differs from the caller's: ={"schema_version":1,"kind":"env","set":[{"name":...,"value":...}],"unset":[...]}=, both lists sorted by name. A shell that cannot evaluate POSIX lines applies it; in nushell:

#+begin_src nu
#+begin_src text
def --env --wrapped eb [...args] {
let change = (^eb-local ...$args --json | from json)
$change.set | reduce --fold {} {|var, acc| $acc | upsert $var.name $var.value } | load-env
Expand Down Expand Up @@ -331,6 +332,9 @@ eval "$(/path/to/eb-local dev MODULE...)"
#+end_src

The program is named by its own path, and =--config=, =--profile=, =--project= and =--buildenv= are repeated on the line. A block already in the file is replaced and every other line is kept; a file without one gets it appended. A begin line without its end line is refused. Run =direnv allow= once per change. direnv evaluates the file in bash and gives the calling shell the change, so the environment follows the directory in bash, zsh, fish, nushell and the other shells direnv has a hook for, and is taken back on leaving it. As with =--json=, only variables reach the calling shell.
| =--pure= | Before anything loads, run =module purge= and unset =LD_LIBRARY_PATH=, =LD_PRELOAD= and =LD_AUDIT=. |

Without =--pure=, the modules the shell already has stay loaded beside the ones =dev= names, and the loader variables stay as they are, with a warning for =LD_LIBRARY_PATH=. With it, the build environment is the stack, the named modules and the build environment only. Sticky modules survive =module purge=. A site that preloads a library for every process (XALT through =LD_PRELOAD=) loses that in this shell.

A =MODULE= starts with a letter or digit and holds letters, digits, =.=, =_=, =+=, =-= and single slashes, with none at the end.
With =LD_LIBRARY_PATH= set in the shell, =dev= warns on stderr.
Expand Down
14 changes: 10 additions & 4 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ eb-local [OPTION]... --explain <easyconfig> [eb options]
eb-local eb [--] [eb options]
eb-local --explain eb [--] [eb options]
eb-local [OPTION]... env [--strict] [--json]
eb-local [OPTION]... dev [--strict] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local [OPTION]... dev [--strict] [--pure] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
eb-local check [--recursive] [--quiet] [--json] FILE...
eb-local check (--module NAME/VERSION... | --dev | --prefix DIR) [--recursive] [--quiet] [--json]
Expand Down Expand Up @@ -62,14 +62,15 @@ directory you own.
eb-local env [--strict] [--json] lines that make your installed modules visible;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1)
eb-local dev [--strict] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
eb-local dev [--strict] [--pure] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
the same, plus MODULEs and a build environment,
so gcc, make, cmake and meson just work;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1);
--buildenv names the build environment to load;
--json prints the change as JSON, for env too;
--envrc writes that line to .envrc for direnv
--envrc writes that line to .envrc for direnv;
--pure purges loaded modules and LD_* first
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
lines that put RPATH wrappers for the compilers
on PATH first; dev uses them when no build
Expand Down Expand Up @@ -301,7 +302,7 @@ When stdout is a terminal, it also says on stderr that the lines change nothing

With `--json`, eb-local runs the lines in a bash child and prints how that shell's environment then differs from the caller's: `{"schema_version":1,"kind":"env","set":[{"name":...,"value":...}],"unset":[...]}`, both lists sorted by name. A shell that cannot evaluate POSIX lines applies it; in nushell:

```nu
```text
def --env --wrapped eb [...args] {
let change = (^eb-local ...$args --json | from json)
$change.set | reduce --fold {} {|var, acc| $acc | upsert $var.name $var.value } | load-env
Expand Down Expand Up @@ -334,6 +335,11 @@ eval "$(/path/to/eb-local dev MODULE...)"

The program is named by its own path, and `--config`, `--profile`, `--project` and `--buildenv` are repeated on the line. A block already in the file is replaced and every other line is kept; a file without one gets it appended. A begin line without its end line is refused. Run `direnv allow` once per change. direnv evaluates the file in bash and gives the calling shell the change, so the environment follows the directory in bash, zsh, fish, nushell and the other shells direnv has a hook for, and is taken back on leaving it. As with `--json`, only variables reach the calling shell.

| `--pure` | Before anything loads, run `module purge` and unset `LD_LIBRARY_PATH`, `LD_PRELOAD` and `LD_AUDIT`. |
|---|---|

Without `--pure`, the modules the shell already has stay loaded beside the ones `dev` names, and the loader variables stay as they are, with a warning for `LD_LIBRARY_PATH`. With it, the build environment is the stack, the named modules and the build environment only. Sticky modules survive `module purge`. A site that preloads a library for every process (XALT through `LD_PRELOAD`) loses that in this shell.

A `MODULE` starts with a letter or digit and holds letters, digits, `.`, `_`, `+`, `-` and single slashes, with none at the end.
With `LD_LIBRARY_PATH` set in the shell, `dev` warns on stderr.

Expand Down
28 changes: 22 additions & 6 deletions src/main.zig
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,15 @@ const usage_text =
\\ eb-local env [--strict] [--json] lines that make your installed modules visible;
\\ --strict exits when the EESSI init file is missing
\\ (also when EB_LOCAL_ENV_STRICT=1)
\\ eb-local dev [--strict] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
\\ eb-local dev [--strict] [--pure] [--json] [--buildenv MODULE] [--envrc] [MODULE...]
\\ the same, plus MODULEs and a build environment,
\\ so gcc, make, cmake and meson just work;
\\ --strict exits when the EESSI init file is missing
\\ (also when EB_LOCAL_ENV_STRICT=1);
\\ --buildenv names the build environment to load;
\\ --json prints the change as JSON, for env too;
\\ --envrc writes that line to .envrc for direnv
\\ --envrc writes that line to .envrc for direnv;
\\ --pure purges loaded modules and LD_* first
\\ eb-local rpath-wrappers [--for FAMILY | --unwrapped]
\\ lines that put RPATH wrappers for the compilers
\\ on PATH first; dev uses them when no build
Expand Down Expand Up @@ -290,7 +291,16 @@ const env_strict_prelude =
\\
;

fn environmentLines(c: Ctx, s: config.Settings, p: Place, strict: bool) []const u8 {
/// `dev --pure`: start from no loaded modules and no loader variables, so
/// what the shell had before does not reach the build environment. Sticky
/// modules stay, as `module purge` leaves them.
const pure_prelude =
\\module purge >/dev/null 2>&1 || true
\\unset LD_LIBRARY_PATH LD_PRELOAD LD_AUDIT
\\
;

fn environmentLines(c: Ctx, s: config.Settings, p: Place, strict: bool, pure: bool) []const u8 {
const a = c.a;
var out: std.ArrayList(u8) = .empty;
const add = struct {
Expand All @@ -300,6 +310,7 @@ fn environmentLines(c: Ctx, s: config.Settings, p: Place, strict: bool) []const
}.f;
if (strict) add(a, &out, env_strict_prelude);
add(a, &out, lmod_init);
if (pure) add(a, &out, pure_prelude);
add(a, &out, str(a, "module unload {s} >/dev/null 2>&1 || true\n", .{q(a, s.extend_module.value)}));
add(a, &out,
\\unset EESSI_CVMFS_INSTALL EESSI_SITE_INSTALL EESSI_USER_INSTALL EESSI_PROJECT_INSTALL
Expand Down Expand Up @@ -367,6 +378,7 @@ fn replay(a: Allocator, command: []const u8, s: config.Settings, g: Global, cano
if (s.project.value != null) parts.appendSlice(a, &.{ "--project", q(a, canonical_prefix) }) catch oom();
parts.append(a, command) catch oom();
if (std.mem.eql(u8, command, "dev")) {
if (g.pure) parts.append(a, "--pure") catch oom();
if (g.buildenv) |p| parts.appendSlice(a, &.{ "--buildenv", q(a, p) }) catch oom();
for (extra) |m| parts.append(a, q(a, m)) catch oom();
}
Expand Down Expand Up @@ -480,6 +492,8 @@ const Global = struct {
json_out: bool = false,
/// `dev --envrc`: write the dev line to `.envrc` here, for direnv.
envrc: bool = false,
/// `dev --pure`: purge the loaded modules and the loader variables first.
pure: bool = false,
};

fn nonEmpty(v: ?[]const u8) ?[]const u8 {
Expand Down Expand Up @@ -684,7 +698,7 @@ fn emitEnvChange(c: Ctx, kind: []const u8, lines: []const u8) u8 {

fn cmdEnv(c: Ctx, s: config.Settings, g: Global) u8 {
const p = place(c, s);
const lines = environmentLines(c, s, p, envStrict(c, g));
const lines = environmentLines(c, s, p, envStrict(c, g), false);
if (g.json_out) return emitEnvChange(c, "env", lines);
evalHint(replay(c.a, "env", s, g, p.prefix, &.{}));
cli.put(1, lines);
Expand Down Expand Up @@ -753,7 +767,7 @@ fn cmdDev(c: Ctx, s: config.Settings, g: Global, extra: []const []const u8) u8 {
if (g.envrc) return writeEnvrc(c, replay(a, "dev", s, g, installed.prefix, extra));
if (!g.json_out) evalHint(replay(a, "dev", s, g, installed.prefix, extra));
var out: std.ArrayList(u8) = .empty;
out.appendSlice(a, environmentLines(c, s, installed, envStrict(c, g))) catch oom();
out.appendSlice(a, environmentLines(c, s, installed, envStrict(c, g), g.pure)) catch oom();
// The modules the program uses first, the build environment last, so
// its wrappers come first on PATH. When the program links, they record
// a run path for each directory on LIBRARY_PATH, which the modules set
Expand All @@ -776,7 +790,7 @@ fn cmdDev(c: Ctx, s: config.Settings, g: Global, extra: []const []const u8) u8 {
out.appendSlice(a, str(a, "export EB_LOCAL_DEV_PREFIX={s}\"/$EESSI_VERSION/$EESSI_SOFTWARE_SUBDIR${{EESSI_ACCELERATOR_TARGET:+/$EESSI_ACCELERATOR_TARGET}}\"\n", .{q(a, dev_root)})) catch oom();
if (g.json_out) return emitEnvChange(c, "dev", out.items);
cli.put(1, out.items);
if (nonEmpty(c.env.get("LD_LIBRARY_PATH")) != null) {
if (!g.pure and nonEmpty(c.env.get("LD_LIBRARY_PATH")) != null) {
warn("LD_LIBRARY_PATH is set in this shell; programs built here do not need it,", .{});
cli.err(" and it can mix host libraries into EESSI's. Unset it unless you know you need it.\n", .{});
}
Expand Down Expand Up @@ -4869,6 +4883,8 @@ pub fn main(init: std.process.Init) u8 {
g.json_out = true;
} else if (std.mem.eql(u8, x, "--envrc")) {
g.envrc = true;
} else if (std.mem.eql(u8, x, "--pure")) {
g.pure = true;
} else if (std.mem.eql(u8, x, "--buildenv")) {
g.buildenv = value(args, &i, "--buildenv");
} else if (joined(x, "--buildenv")) |v| {
Expand Down
15 changes: 15 additions & 0 deletions tests/test_dev_cli_modes.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,21 @@ def test_the_envrc_block_is_valid_bash_that_runs_the_dev_lines(self):
capture_output=True, text=True,
)
self.assertTrue(run.stdout.startswith(str(self.prefix) + "/dev/"), run.stdout + run.stderr)
def test_pure_purges_modules_and_loader_variables_before_the_stack_loads(self):
plain = self.run_dev(["FFTW/3.3.10-GCC-15.2.0"], LD_LIBRARY_PATH="/host/lib")
pure = self.run_dev(["--pure", "FFTW/3.3.10-GCC-15.2.0"], LD_LIBRARY_PATH="/host/lib")
self.assertEqual(pure.returncode, 0, pure.stderr)
purge = "module purge >/dev/null 2>&1 || true\nunset LD_LIBRARY_PATH LD_PRELOAD LD_AUDIT\n"
self.assertNotIn("module purge", plain.stdout)
self.assertEqual(pure.stdout.count(purge), 1)
# the rest of the lines is what dev prints without it
self.assertEqual(pure.stdout.replace(purge, ""), plain.stdout)
# after a module command exists, before anything is loaded
self.assertLess(pure.stdout.index("type module"), pure.stdout.index(purge))
self.assertLess(pure.stdout.index(purge), pure.stdout.index("module load"))
# the variable is unset by the lines, so there is nothing to warn about
self.assertIn("LD_LIBRARY_PATH is set", plain.stderr)
self.assertNotIn("LD_LIBRARY_PATH is set", pure.stderr)

def test_names_the_host_ca_bundle_without_replacing_one_the_shell_set(self):
got = self.run_dev([])
Expand Down
Loading