Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/+env-json.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`eb-local env --json` and `eb-local dev --json` print the change their lines make to the environment (`set` and `unset`, as one line of JSON) in place of the lines, so a shell that cannot evaluate POSIX text, such as nushell, can apply it. The reference has a nushell wrapper.
27 changes: 22 additions & 5 deletions docs/orgmode/reference/cli.org
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ eb-local [OPTION]... <easyconfig> [eb options]
eb-local [OPTION]... --explain <easyconfig> [eb options]
eb-local eb [--] [eb options]
eb-local --explain eb [--] [eb options]
eb-local [OPTION]... env [--strict]
eb-local [OPTION]... dev [--strict] [--buildenv MODULE] [MODULE...]
eb-local [OPTION]... env [--strict] [--json]
eb-local [OPTION]... dev [--strict] [--json] [--buildenv MODULE] [MODULE...]
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
eb-local check [--recursive] [--quiet] [--json] FILE...
eb-local check (--module NAME/VERSION... | --dev | --prefix DIR) [--recursive] [--quiet] [--json]
Expand Down Expand Up @@ -57,15 +57,16 @@ directory you own.
tree that variable selects
eb-local --explain eb [--] [eb options]
the script eb runs, with its arguments; runs nothing
eb-local env [--strict] lines that make your installed modules visible;
eb-local env [--strict] [--json] lines that make your installed modules visible;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1)
eb-local dev [--strict] [--buildenv MODULE] [MODULE...]
eb-local dev [--strict] [--json] [--buildenv MODULE] [MODULE...]
the same, plus MODULEs and a build environment,
so gcc, make, cmake and meson just work;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1);
--buildenv names the build environment to load
--buildenv names the build environment to load;
--json prints the change as JSON, for env too
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
lines that put RPATH wrappers for the compilers
on PATH first; dev uses them when no build
Expand Down Expand Up @@ -293,6 +294,21 @@ When stdout is a terminal, it also says on stderr that the lines change nothing
| Option | Meaning |
|------------+----------------------------------------------------------------------------------------------------------------------|
| =--strict= | Check that the EESSI init file exists in the emitted lines and exit 2 if absent (also when =EB_LOCAL_ENV_STRICT=1=). |
| =--json= | Print, in place of the lines, the change they make to the environment, as one line of JSON. |

With =--json=, eb-local runs the lines in a bash child and prints how that shell's environment then differs from the caller's: ={"schema_version":1,"kind":"env","set":[{"name":...,"value":...}],"unset":[...]}=, both lists sorted by name. A shell that cannot evaluate POSIX lines applies it; in nushell:

#+begin_src text
def --env --wrapped eb [...args] {
let change = (^eb-local ...$args --json | from json)
$change.set | reduce --fold {} {|var, acc| $acc | upsert $var.name $var.value } | load-env
hide-env --ignore-errors ...$change.unset
}
eb env
eb dev FFTW/3.3.10-GCC-15.2.0
#+end_src

Only variables are in the change. Shell functions and aliases, such as Lmod's =module=, stay in the bash child, so a later =module load= in the other shell is not possible this way; name the modules on the =dev= line. What the lines print goes to stderr. If the lines fail in bash, nothing is printed and the status is 1.

** dev

Expand All @@ -301,6 +317,7 @@ Print the lines =env= prints, then the lines that load the modules =dev.modules=
| Option | Meaning |
|---------------------+--------------------------------------------------------------------------------------------|
| =--strict= | Check that the EESSI init file exists in the emitted lines and exit 2 if absent (also when =EB_LOCAL_ENV_STRICT=1=). |
| =--json= | Print the change the lines make to the environment as one line of JSON, as =env --json= does, with =kind= =dev=. |
| =--buildenv MODULE= | Load MODULE as the build environment whatever the compilers, as =dev.buildenv= does. |

A =MODULE= starts with a letter or digit and holds letters, digits, =.=, =_=, =+=, =-= and single slashes, with none at the end.
Expand Down
27 changes: 22 additions & 5 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ eb-local [OPTION]... <easyconfig> [eb options]
eb-local [OPTION]... --explain <easyconfig> [eb options]
eb-local eb [--] [eb options]
eb-local --explain eb [--] [eb options]
eb-local [OPTION]... env [--strict]
eb-local [OPTION]... dev [--strict] [--buildenv MODULE] [MODULE...]
eb-local [OPTION]... env [--strict] [--json]
eb-local [OPTION]... dev [--strict] [--json] [--buildenv MODULE] [MODULE...]
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
eb-local check [--recursive] [--quiet] [--json] FILE...
eb-local check (--module NAME/VERSION... | --dev | --prefix DIR) [--recursive] [--quiet] [--json]
Expand Down Expand Up @@ -59,15 +59,16 @@ directory you own.
tree that variable selects
eb-local --explain eb [--] [eb options]
the script eb runs, with its arguments; runs nothing
eb-local env [--strict] lines that make your installed modules visible;
eb-local env [--strict] [--json] lines that make your installed modules visible;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1)
eb-local dev [--strict] [--buildenv MODULE] [MODULE...]
eb-local dev [--strict] [--json] [--buildenv MODULE] [MODULE...]
the same, plus MODULEs and a build environment,
so gcc, make, cmake and meson just work;
--strict exits when the EESSI init file is missing
(also when EB_LOCAL_ENV_STRICT=1);
--buildenv names the build environment to load
--buildenv names the build environment to load;
--json prints the change as JSON, for env too
eb-local rpath-wrappers [--for FAMILY | --unwrapped]
lines that put RPATH wrappers for the compilers
on PATH first; dev uses them when no build
Expand Down Expand Up @@ -295,6 +296,21 @@ When stdout is a terminal, it also says on stderr that the lines change nothing
| Option | Meaning |
|---|---|
| `--strict` | Check that the EESSI init file exists in the emitted lines and exit 2 if absent (also when `EB_LOCAL_ENV_STRICT=1`). |
| `--json` | Print, in place of the lines, the change they make to the environment, as one line of JSON. |

With `--json`, eb-local runs the lines in a bash child and prints how that shell's environment then differs from the caller's: `{"schema_version":1,"kind":"env","set":[{"name":...,"value":...}],"unset":[...]}`, both lists sorted by name. A shell that cannot evaluate POSIX lines applies it; in nushell:

```text
def --env --wrapped eb [...args] {
let change = (^eb-local ...$args --json | from json)
$change.set | reduce --fold {} {|var, acc| $acc | upsert $var.name $var.value } | load-env
hide-env --ignore-errors ...$change.unset
}
eb env
eb dev FFTW/3.3.10-GCC-15.2.0
```

Only variables are in the change. Shell functions and aliases, such as Lmod's `module`, stay in the bash child, so a later `module load` in the other shell is not possible this way; name the modules on the `dev` line. What the lines print goes to stderr. If the lines fail in bash, nothing is printed and the status is 1.

### dev

Expand All @@ -303,6 +319,7 @@ Print the lines `env` prints, then the lines that load the modules `dev.modules`
| Option | Meaning |
|---|---|
| `--strict` | Check that the EESSI init file exists in the emitted lines and exit 2 if absent (also when `EB_LOCAL_ENV_STRICT=1`). |
| `--json` | Print the change the lines make to the environment as one line of JSON, as `env --json` does, with `kind` `dev`. |
| `--buildenv MODULE` | Load MODULE as the build environment whatever the compilers, as `dev.buildenv` does. |

A `MODULE` starts with a letter or digit and holds letters, digits, `.`, `_`, `+`, `-` and single slashes, with none at the end.
Expand Down
97 changes: 92 additions & 5 deletions src/main.zig
Original file line number Diff line number Diff line change
Expand Up @@ -62,15 +62,16 @@ const usage_text =
\\ tree that variable selects
\\ eb-local --explain eb [--] [eb options]
\\ the script eb runs, with its arguments; runs nothing
\\ eb-local env [--strict] lines that make your installed modules visible;
\\ eb-local env [--strict] [--json] lines that make your installed modules visible;
\\ --strict exits when the EESSI init file is missing
\\ (also when EB_LOCAL_ENV_STRICT=1)
\\ eb-local dev [--strict] [--buildenv MODULE] [MODULE...]
\\ eb-local dev [--strict] [--json] [--buildenv MODULE] [MODULE...]
\\ the same, plus MODULEs and a build environment,
\\ so gcc, make, cmake and meson just work;
\\ --strict exits when the EESSI init file is missing
\\ (also when EB_LOCAL_ENV_STRICT=1);
\\ --buildenv names the build environment to load
\\ --buildenv names the build environment to load;
\\ --json prints the change as JSON, for env too
\\ eb-local rpath-wrappers [--for FAMILY | --unwrapped]
\\ lines that put RPATH wrappers for the compilers
\\ on PATH first; dev uses them when no build
Expand Down Expand Up @@ -599,10 +600,91 @@ fn envStrict(c: Ctx, g: Global) bool {
return v.len > 0 and !std.mem.eql(u8, v, "0") and !std.mem.eql(u8, v, "false");
}

/// One variable of an `EnvChange`.
const EnvVar = struct {
name: []const u8,
value: []const u8,
};

/// `env --json` and `dev --json`: what the lines change in the environment of
/// the shell that evaluates them, for a shell that cannot evaluate them.
/// Shell functions and aliases (Lmod's `module`) are not variables and are
/// not in it.
const EnvChange = struct {
schema_version: u32 = jsonout.schema_version,
kind: []const u8,
/// Variables the lines add or change, by name.
set: []const EnvVar,
/// Variables the lines remove, by name.
unset: []const []const u8,
};

/// Variables bash keeps for itself; they say nothing about the lines.
const shell_own = [_][]const u8{ "_", "SHLVL", "PWD", "OLDPWD", "BASH_EXECUTION_STRING" };

fn shellOwn(name: []const u8) bool {
if (std.mem.startsWith(u8, name, "BASH_FUNC_")) return true;
for (shell_own) |own| if (std.mem.eql(u8, name, own)) return true;
return false;
}

fn envVarLess(_: void, x: EnvVar, y: EnvVar) bool {
return std.mem.lessThan(u8, x.name, y.name);
}

/// Run `lines` in a bash child and print, as one JSON document, how its
/// environment then differs from this process's. The child's own output goes
/// to stderr, so stdout carries the document alone.
fn emitEnvChange(c: Ctx, kind: []const u8, lines: []const u8) u8 {
const a = c.a;
const bash = (proc.which(a, "bash", c.env.get("PATH") orelse "/usr/bin:/bin") catch null) orelse "/bin/bash";
const script = str(a, "exec 3>&1 1>&2\n{s}\nenv -0 >&3\n", .{lines});
const r = proc.run(a, bash, &.{ bash, "--noprofile", "--norc", "-c", script }, c.envp, .{
.stdout_limit = 8 << 20,
.stderr_limit = 1 << 20,
.timeout_ms = 300_000,
}) catch |e| {
warn("could not run the {s} lines in bash: {s}", .{ kind, @errorName(e) });
return 1;
};
if (r.stderr.len > 0) cli.put(2, r.stderr);
if (r.term.code() != 0) {
warn("the {s} lines failed in bash (status {d}); no change is printed", .{ kind, r.term.code() });
return 1;
}
var set: std.ArrayList(EnvVar) = .empty;
var after: std.StringHashMapUnmanaged(void) = .empty;
var it = std.mem.splitScalar(u8, r.stdout, 0);
while (it.next()) |entry| {
const eq = std.mem.indexOfScalar(u8, entry, '=') orelse continue;
const name = entry[0..eq];
const value = entry[eq + 1 ..];
if (name.len == 0 or shellOwn(name)) continue;
after.put(a, name, {}) catch oom();
if (c.env.get(name)) |before| {
if (std.mem.eql(u8, before, value)) continue;
}
set.append(a, .{ .name = name, .value = value }) catch oom();
}
var unset: std.ArrayList([]const u8) = .empty;
var before_it = c.env.iterator();
while (before_it.next()) |kv| {
const name = kv.key_ptr.*;
if (shellOwn(name) or after.contains(name)) continue;
unset.append(a, name) catch oom();
}
std.mem.sort(EnvVar, set.items, {}, envVarLess);
std.mem.sort([]const u8, unset.items, {}, lessThanStr);
jsonout.emit(a, EnvChange{ .kind = kind, .set = set.items, .unset = unset.items });
return 0;
}

fn cmdEnv(c: Ctx, s: config.Settings, g: Global) u8 {
const p = place(c, s);
const lines = environmentLines(c, s, p, envStrict(c, g));
if (g.json_out) return emitEnvChange(c, "env", lines);
evalHint(replay(c.a, "env", s, g, p.prefix, &.{}));
cli.put(1, environmentLines(c, s, p, envStrict(c, g)));
cli.put(1, lines);
return 0;
}

Expand All @@ -620,7 +702,7 @@ fn cmdDev(c: Ctx, s: config.Settings, g: Global, extra: []const []const u8) u8 {
} else mods.append(a, m) catch oom();
};
const installed = place(c, s);
evalHint(replay(a, "dev", s, g, installed.prefix, extra));
if (!g.json_out) evalHint(replay(a, "dev", s, g, installed.prefix, extra));
var out: std.ArrayList(u8) = .empty;
out.appendSlice(a, environmentLines(c, s, installed, envStrict(c, g))) catch oom();
// The modules the program uses first, the build environment last, so
Expand All @@ -643,6 +725,7 @@ fn cmdDev(c: Ctx, s: config.Settings, g: Global, extra: []const []const u8) u8 {
// has picked them.
const dev_root = std.fs.path.join(a, &.{ installed.prefix, "dev" }) catch oom();
out.appendSlice(a, str(a, "export EB_LOCAL_DEV_PREFIX={s}\"/$EESSI_VERSION/$EESSI_SOFTWARE_SUBDIR${{EESSI_ACCELERATOR_TARGET:+/$EESSI_ACCELERATOR_TARGET}}\"\n", .{q(a, dev_root)})) catch oom();
if (g.json_out) return emitEnvChange(c, "dev", out.items);
cli.put(1, out.items);
if (nonEmpty(c.env.get("LD_LIBRARY_PATH")) != null) {
warn("LD_LIBRARY_PATH is set in this shell; programs built here do not need it,", .{});
Expand Down Expand Up @@ -4537,6 +4620,8 @@ pub fn main(init: std.process.Init) u8 {
.env => {
if (std.mem.eql(u8, x, "--strict")) {
g.env_strict = true;
} else if (std.mem.eql(u8, x, "--json")) {
g.json_out = true;
} else die("env takes only --project, --profile, --config and --strict; got '{s}'", .{x});
},
.profiles => {
Expand Down Expand Up @@ -4731,6 +4816,8 @@ pub fn main(init: std.process.Init) u8 {
.dev => {
if (std.mem.eql(u8, x, "--strict")) {
g.env_strict = true;
} else if (std.mem.eql(u8, x, "--json")) {
g.json_out = true;
} else if (std.mem.eql(u8, x, "--buildenv")) {
g.buildenv = value(args, &i, "--buildenv");
} else if (joined(x, "--buildenv")) |v| {
Expand Down
12 changes: 12 additions & 0 deletions tests/test_dev_cli_modes.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env python3
"""Black-box CLI behavior tests for `eb-local dev` module validation and warnings."""
import json
import os
from pathlib import Path
import subprocess
Expand Down Expand Up @@ -45,6 +46,17 @@ def test_invalid_module_names_exit_two(self):
self.assertEqual(got.stdout, "")
self.assertIn("is not a module name", got.stderr)

def test_json_is_the_change_the_dev_lines_make(self):
got = self.run_dev(["--json"])
self.assertEqual(got.returncode, 0, got.stderr)
self.assertEqual(got.stdout.count("\n"), 1, got.stdout)
change = json.loads(got.stdout)
self.assertEqual(change["kind"], "dev")
values = {var["name"]: var["value"] for var in change["set"]}
self.assertIn("EB_LOCAL_DEV_PREFIX", values)
self.assertTrue(values["EB_LOCAL_DEV_PREFIX"].startswith(str(self.prefix) + "/dev/"), values)
self.assertNotIn("module load", got.stdout)

def test_names_the_host_ca_bundle_without_replacing_one_the_shell_set(self):
got = self.run_dev([])
self.assertEqual(got.returncode, 0, got.stderr)
Expand Down
33 changes: 33 additions & 0 deletions tests/test_env_cli_modes.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env python3
"""Black-box CLI behavior tests for `eb-local env` and `--env`."""
import json
from pathlib import Path
import subprocess
import tempfile
Expand Down Expand Up @@ -43,6 +44,38 @@ def run_env(self, args, **env_overrides):
text=True,
)

def test_json_is_the_change_the_lines_make_to_the_environment(self):
got = self.run_env(
["env", "--json"],
EESSI_SITE_INSTALL="1",
EASYBUILD_PREFIX="/elsewhere",
KEPT="as it was",
)
self.assertEqual(got.returncode, 0, got.stderr)
self.assertEqual(got.stdout.count("\n"), 1, got.stdout)
change = json.loads(got.stdout)
self.assertEqual(change["schema_version"], 1)
self.assertEqual(change["kind"], "env")
names = [var["name"] for var in change["set"]]
self.assertEqual(names, sorted(names))
values = {var["name"]: var["value"] for var in change["set"]}
# what the lines export, and nothing the caller already had with that value
self.assertEqual(values["EESSI_USER_INSTALL"], str(self.prefix))
self.assertEqual(values["EESSI_CUSTOM_SITE_OPT"], "alpha")
self.assertNotIn("KEPT", values)
self.assertNotIn("HOME", values)
# what the lines unset
self.assertIn("EESSI_SITE_INSTALL", change["unset"])
self.assertIn("EASYBUILD_PREFIX", change["unset"])
self.assertNotIn("KEPT", change["unset"])
self.assertEqual(change["unset"], sorted(change["unset"]))
# bash's own bookkeeping and exported functions are not part of the change
for name in list(values) + change["unset"]:
self.assertNotIn(name, ("_", "SHLVL", "PWD", "OLDPWD"))
self.assertFalse(name.startswith("BASH_FUNC_"), name)
# the lines themselves are not printed
self.assertNotIn("module load", got.stdout)

def test_global_env_flag_matches_env_subcommand(self):
sub = self.run_env(["env"])
flag = self.run_env(["--env"])
Expand Down
Loading