Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ tests/tmachine/ # QEMU test machines, Ansible playbooks, and the Session
tests/runner/ # openshell-test-runner: runs a test family in a tmachine environment
```

## Performance benchmarks

Run `mise run perf:seccomp` for the native/filtered seccomp broker
microbenchmark.

## Rust Tests

Unit tests live inline with `#[cfg(test)] mod tests` blocks. Integration tests
Expand Down
51 changes: 51 additions & 0 deletions crates/openshell-sandbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,57 @@ the last workload reference to a listener removes it without waiting for registr
collection. Broker source descriptors always use close-on-exec independently of
the workload's requested descriptor flags.

A blocking loopback `connect` waits on a small pool of reusable worker threads
rather than a thread per call. A job never queues behind a busy worker: it goes
to a parked worker or starts a new one, and a pool with neither refuses with
`EAGAIN`, so a slow or full listener cannot delay another connect or the
dispatcher. Three bounds hold together. Each job carries an admission slot
shared with external opens (256 in all), which its worker releases only after
the pool records that worker parked or retired. Live workers are capped at that
same number under the pool's own lock. At most four workers stay parked, for up
to 30 seconds; a worker that finishes beyond that exits at once.

`setsockopt` calls for `TCP_NODELAY`, `SO_REUSEADDR`, and `SO_KEEPALIVE` run in
the kernel without a notification. The broker decides `setsockopt` from its two
scalar arguments and would continue these pairs. Every other pair, and any
level or option that does not fit a non-negative C `int`, still notifies and
keeps the broker's `EPERM` and `EINVAL` results. The active qualification
probe exercises both sides.

## Provider files

Provider files appear to the workload beneath `/run/openshell/providers`.
Every workload `open`, `openat`, and `openat2` notifies the broker, which
serves a provider path by injecting a read-only descriptor for a sealed
in-memory file and continues every other open.

Each published file is an immutable revision backed by one in-memory file. The
runtime creates that file when the revision is published, not when the
workload opens it: it writes the content, seals the file against writes,
growing, shrinking, and further sealing, and keeps only a read-only descriptor
for it. Each workload open reopens that descriptor through the runtime's own
`/proc/self/fd`, so every open gets its own read-only open file description
and file offset, positioned at the start.

An update prepares every new revision before it replaces the published set in
one step, so a failed update leaves the previous set in place. A file whose
bytes did not change keeps its revision. Opens after an update see the new
revisions, descriptors opened earlier keep the bytes they opened, and a
detached file is `ENOENT` for later opens.

All opens of one revision refer to one file, which the workload owns: it can
change that file's mode or reopen a descriptor for writing. Neither reaches
the content, which the seals protect, and the runtime restores the mode if it
denies an open. The runtime holds one descriptor per published file, at most
64.

The broker classifies each open from an exact copy of the first six bytes of
its path. Six bytes identify a `/proc/` path and the start of a provider path;
only those candidates have their whole path read and decided. An open whose
first six bytes cannot be read, or match neither, continues, and unrelated
`openat2` calls do not require a copy of their `open_how`. Every continued
open remains subject to the kernel and Landlock.

A boundary configured with `RuntimeInitiatedTlsTcp` binds no listener. It dials
only the driver-provided supervisor addresses, which must be concrete,
non-loopback, and share one port, and serves TLS and gRPC over that outbound
Expand Down
73 changes: 63 additions & 10 deletions crates/openshell-sandbox/src/bin/seccomp-perf.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,51 @@ use clap::builder::{PossibleValue, PossibleValuesParser};
#[cfg(target_os = "linux")]
use clap::{Parser, Subcommand, ValueEnum as _};
#[cfg(target_os = "linux")]
use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol};
use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol, WorkerRun};

#[cfg(target_os = "linux")]
#[derive(Debug, Parser)]
#[command(
about = "Measure native and seccomp-filtered socket performance",
long_about = "Measure native and seccomp-filtered TCP socket performance."
about = "Measure native and seccomp-filtered socket and file-open performance",
long_about = "Measure native and seccomp-filtered TCP socket and file-open performance.\n\n\
Layer `notify-only` and `--landlock` attribute the cost of a workload file open; \
see the perf module documentation."
)]
struct Cli {
/// Benchmark layer: native, filtered, or all.
#[arg(long, default_value = "all", value_parser = ["native", "filtered", "all"])]
/// Benchmark layer. `all` is native and filtered; `attribution` adds the
/// harness-only notify-only layer.
#[arg(
long,
default_value = "all",
value_parser = ["native", "filtered", "notify-only", "all", "attribution"]
)]
layer: String,
/// Implemented protocol to benchmark, or all.
/// Protocol to benchmark. `all` is the TCP pair; `file` is the file-open
/// attribution set; `mixed` runs file opens and TCP operations at the
/// same time, on `concurrency` threads each, and reports each class.
#[arg(long, default_value = "all", value_parser = protocol_value_parser())]
protocol: String,
/// Confine each worker with a Landlock domain before it measures.
#[arg(long)]
landlock: bool,
/// File the file protocols open.
#[arg(long, default_value = "/etc/passwd")]
file_path: std::path::PathBuf,
/// Publish one synthetic provider file and make the file protocols open
/// it, so the broker serves each open by injection. Requires
/// `--layer filtered`; no other layer has virtual provider files.
#[arg(long, conflicts_with = "file_path")]
provider_file: bool,
/// Size of the synthetic provider file.
#[arg(long, default_value_t = 4_096, requires = "provider_file")]
provider_file_bytes: usize,
/// Experiment: request the kernel's synchronous-wakeup hint on the
/// listener of each filtered layer. The production runtime never does.
#[arg(long)]
sync_wakeup: bool,
/// Experiment: threads answering notifications in the notify-only layer.
#[arg(long, default_value_t = 1)]
notify_receivers: usize,
#[arg(long, default_value_t = 10_000)]
iterations: u64,
#[arg(long, default_value_t = 1_000)]
Expand All @@ -44,7 +74,11 @@ struct Cli {

#[cfg(target_os = "linux")]
fn protocol_value_parser() -> PossibleValuesParser {
let mut values = vec![PossibleValue::new("all")];
let mut values = vec![
PossibleValue::new("all"),
PossibleValue::new("file"),
PossibleValue::new("mixed"),
];
values.extend(
Protocol::value_variants()
.iter()
Expand All @@ -70,6 +104,12 @@ enum Command {
concurrency: usize,
#[arg(long)]
payload_bytes: usize,
#[arg(long)]
landlock: bool,
#[arg(long, default_value = "/etc/passwd")]
file_path: std::path::PathBuf,
#[arg(long)]
provider_file_bytes: Option<usize>,
},
}

Expand All @@ -83,17 +123,25 @@ fn main() -> anyhow::Result<()> {
warmup,
concurrency,
payload_bytes,
landlock,
file_path,
provider_file_bytes,
}) = cli.command
{
let report = openshell_sandbox::perf::run_worker(
let reports = openshell_sandbox::perf::run_worker(&WorkerRun {
protocol,
target,
iterations,
warmup,
concurrency,
payload_bytes,
)?;
println!("{}", serde_json::to_string(&report)?);
landlock,
file_path,
provider_file_bytes,
})?;
for report in reports {
println!("{}", serde_json::to_string(&report)?);
}
return Ok(());
}

Expand All @@ -104,6 +152,11 @@ fn main() -> anyhow::Result<()> {
warmup: cli.warmup,
concurrency: cli.concurrency,
payload_bytes: cli.payload_bytes,
landlock: cli.landlock,
file_path: cli.file_path,
sync_wakeup: cli.sync_wakeup,
notify_receivers: cli.notify_receivers,
provider_file_bytes: cli.provider_file.then_some(cli.provider_file_bytes),
};
for report in openshell_sandbox::perf::run(options)? {
println!("{}", serde_json::to_string(&report)?);
Expand Down
135 changes: 135 additions & 0 deletions crates/openshell-sandbox/src/linux/proc_fd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,80 @@ pub fn installed_socket_inodes_excluding(
Ok(inodes)
}

/// Count this process's open descriptors, as one enumeration of
/// `/proc/self/fd` reports them.
///
/// An enumeration opens the directory it lists, so its result is the number
/// of descriptors open before the call plus one. Callers size their headroom
/// against that value. Linux 6.2 and later report the open-descriptor count as
/// the size of `/proc/self/fd`, which costs one `stat` instead of an open,
/// one directory read per batch of entries, and a close. Support is
/// established once by an active probe, never from the kernel version, and
/// the fast count adds one to stay equal to an enumeration. Any kernel,
/// profile, or procfs that does not pass the probe keeps enumerating.
pub fn open_descriptor_count() -> io::Result<usize> {
static STAT_COUNT_SUPPORTED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
let supported = *STAT_COUNT_SUPPORTED.get_or_init(|| {
stat_count_tracks_enumeration(
stat_descriptor_count,
|| enumerate_descriptor_count().ok(),
|| rustix::event::eventfd(0, rustix::event::EventfdFlags::CLOEXEC).ok(),
)
});
if supported && let Some(open) = stat_descriptor_count() {
return Ok(open.saturating_add(1));
}
enumerate_descriptor_count()
}

fn enumerate_descriptor_count() -> io::Result<usize> {
fs::read_dir("/proc/self/fd").map(Iterator::count)
}

/// Open-descriptor count from the directory size, or `None` where the kernel
/// does not report one (older kernels report a size of zero).
fn stat_descriptor_count() -> Option<usize> {
let size = fs::metadata("/proc/self/fd").ok()?.len();
usize::try_from(size).ok().filter(|count| *count > 0)
}

/// Decide whether `stat` reports a live descriptor count.
///
/// A sample qualifies only when two `stat` counts bracketing an enumeration
/// agree, the enumeration reports exactly one more (its own directory
/// descriptor), and holding one extra descriptor raises the `stat` count by
/// exactly one. Other threads open and close descriptors concurrently, so a
/// disturbed sample is retried a few times; persistent disagreement selects
/// enumeration.
fn stat_count_tracks_enumeration<Held>(
mut stat: impl FnMut() -> Option<usize>,
mut enumerate: impl FnMut() -> Option<usize>,
mut hold_descriptor: impl FnMut() -> Option<Held>,
) -> bool {
const ATTEMPTS: usize = 8;
for _ in 0..ATTEMPTS {
let Some(before) = stat() else {
return false;
};
let Some(enumerated) = enumerate() else {
return false;
};
let Some(held) = hold_descriptor() else {
return false;
};
let with_held = stat();
drop(held);
let after = stat();
if after == Some(before)
&& with_held == before.checked_add(1)
&& Some(enumerated) == before.checked_add(1)
{
return true;
}
}
false
}

/// Return the socket inode currently installed at `fd` in `tid`'s descriptor
/// table.
///
Expand Down Expand Up @@ -184,4 +258,65 @@ mod tests {
.contains(&inode)
);
}

#[test]
fn stat_count_is_used_only_when_it_tracks_descriptors() {
let hold = || Some(());
// A kernel that reports no size never qualifies.
assert!(!stat_count_tracks_enumeration(|| None, || Some(8), hold));
// A size that ignores a newly held descriptor is not a count.
assert!(!stat_count_tracks_enumeration(|| Some(7), || Some(8), hold));
// An enumeration that does not include its own descriptor disagrees.
let mut counts = [7, 8, 7].into_iter().cycle();
assert!(!stat_count_tracks_enumeration(
|| counts.next(),
|| Some(7),
hold
));
// A consistent sample qualifies: before, with one held, after.
let mut counts = [7, 8, 7].into_iter().cycle();
assert!(stat_count_tracks_enumeration(
|| counts.next(),
|| Some(8),
hold
));
// A sample disturbed by concurrent descriptor churn is retried.
let mut counts = [7, 9, 8, 8, 9, 8].into_iter();
assert!(stat_count_tracks_enumeration(
|| counts.next(),
|| Some(9),
hold
));
// Failing to enumerate or to hold a descriptor never qualifies.
assert!(!stat_count_tracks_enumeration(|| Some(7), || None, hold));
assert!(!stat_count_tracks_enumeration(
|| Some(7),
|| Some(8),
|| None::<()>
));
}

#[test]
fn descriptor_count_follows_open_descriptors() {
// Other tests open and close descriptors concurrently, so compare
// against a large deliberate change rather than an exact value.
const HELD: usize = 64;
let before = open_descriptor_count().unwrap();
assert!(before > 3, "count includes the standard descriptors");
let held = (0..HELD)
.map(|_| File::open("/dev/null").expect("open descriptor"))
.collect::<Vec<_>>();
let during = open_descriptor_count().unwrap();
drop(held);
let after = open_descriptor_count().unwrap();
assert!(during >= before + HELD / 2, "{before} -> {during}");
assert!(after + HELD / 2 <= during, "{during} -> {after}");
// The fast count and an enumeration agree to within concurrent churn.
let enumerated = enumerate_descriptor_count().unwrap();
let counted = open_descriptor_count().unwrap();
assert!(
enumerated.abs_diff(counted) < HELD / 2,
"{enumerated} vs {counted}"
);
}
}
Loading
Loading