Run student code in its own sandboxed process, apart from the teacher's (OSS-10)

.github/workflows/ci.yml

@@ -15,6 +15,8 @@ concurrency:
   cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
+  # Student code runs in the operating system's sandbox — Landlock and seccomp here, Seatbelt
+  # on macOS — so the suite, its sandbox probes included, runs on both.
   check:
     runs-on: ubuntu-latest
     steps:
@@ -29,13 +31,27 @@ jobs:
       - run: cargo clippy --workspace --all-targets -- -D warnings
       - run: cargo test --workspace
 
-  # Releases ship macOS and Windows binaries; build them on every change, not first at tag time.
+  test-macos:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      # The runner's own python3 may be older than the 3.10 the harness needs.
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - run: cargo test --workspace
+
+  # Releases ship Windows binaries too; build them on every change, not first at tag time.
   build:
     name: build (${{ matrix.os }})
     strategy:
       fail-fast: false
       matrix:
-        os: [macos-latest, windows-latest]
+        os: [windows-latest]
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v7

CLAUDE.md

@@ -82,8 +82,9 @@ revisions) → Display/DB/HTML/Canvas. `rescore` adds a revision from saved evid
 ```
 core/src/models/     Data models, TOML spec parsing, grading policies
 core/src/discovery   Student file discovery + archive extraction
-core/src/runner/     PythonExecutor (subprocess), orchestrator, sandbox (setrlimit),
-                parametrize, expander, oracle, linter
+core/src/runner/     PythonExecutor (teacher + student processes), harness/*.py,
+                orchestrator, sandbox (Seatbelt / Landlock + seccomp, setrlimit),
+                records (framed), parametrize, expander, oracle, linter
 core/src/checker/    Checker trait + Rhai + Python checkers
 core/src/record      Versioned evidence, score revisions, rescore reuse checks
 cli/src/db/          SQLite: students, sessions (= revisions), results, similarity
@@ -93,11 +94,11 @@ cli/src/tui/         ratatui terminal UI: students/sessions/similarity tabs
 
 ## Key Design Decisions
 
-**No pytest.** ScriptMark is its own test engine. Student code runs in subprocess via embedded Python helper script (`python.rs::HELPER_SCRIPT`). The helper: syntax-checks with py_compile, suppresses input()/print() during module load, calls target function, returns JSON on stdout.
+**No pytest.** ScriptMark is its own test engine. A unit (a case or a scenario) runs as two Python processes from embedded scripts (`runner/harness/`): the teacher side (`teacher.py`: teacher modules, function checkers, the plan, per-request deadlines, the records) and the student side (`student.py`: the student module only, one call per request, under its timer and import guard). They share `common.py` (framing, the record and transfer value encodings, `call()`). ScriptMark joins the two with a pair of pipes and reads length-prefixed JSON records from the teacher side; values cross by value (shared-class instances rebuilt from the receiver's copy, other objects as snapshots).
 
-**Sandboxed execution.** Student subprocess runs with: (1) `env_clear()` — no access to host env vars, (2) import allowlist — only safe stdlib + teacher-specified `allowed_imports`, (3) `setrlimit` — kernel-enforced CPU/file size/fd/process limits, (4) `tokio::join!` pipe reading + explicit `child.kill()` on timeout. Python interpreter resolved to absolute path at construction time so `env_clear()` doesn't change which interpreter runs.
+**Sandboxed execution.** Each side runs confined by default (`runner/sandbox.rs`): macOS Seatbelt via `sandbox-exec` with a generated profile, Linux Landlock + seccomp applied in `pre_exec`. The student side reads its interpreter and its own directory only; the teacher side also reads the bundle and a read-only source copy. Fail closed: no sandbox, no grading, unless `--no-sandbox` (recorded as `isolation: unisolated`). Both also get `env_clear()`, `setrlimit` (RLIMIT_AS only as a backstop), and an absolute interpreter path; the student side is watched against `memory_mb` beyond what it takes idle with every module the submission imports loaded (spec, case or scenario; undeclared, estimated by `runner/memory.rs` from `space` — O(n^2) by default — and the test data's size, 16 MiB–1 GiB; `harness/idle.py` measures the idle share once per set of imports, never running the submission). The import allowlist is a teaching rule, not a boundary. Sandbox tests need a real sandbox: run `cargo test` outside any other Seatbelt sandbox (Claude Code's included), and on Linux from a filesystem with stable inodes (not a Docker Desktop bind mount).
 
-**TOML test specs** have three sections: `[vars]` (constants injected as Python globals), `[[setup]]` (function calls storing results for `$ref`), `[[cases]]` (scored test cases). Cases support `expect`, `expect_error`, checkers (`check = "sorted"` / `{ rhai = "..." }` / `{ python = "verifiers/check.py" }`), and `[cases.parametrize]` with oracle. `[meta]` supports `allowed_imports` for teacher-controlled package access.
+**TOML test specs** have three sections: `[vars]` (constants injected as Python globals), `[[setup]]` (function calls storing results for `$ref`), `[[cases]]` (scored test cases). Cases support `expect`, `expect_error`, checkers (`check = "sorted"` / `{ rhai = "..." }` / `{ python = "verifiers/check.py" }` / `{ function = "..." }`, which may take the reserved `stdout` and `source`), and `[cases.parametrize]` with oracle. `[meta]` supports `allowed_imports` (a teaching rule) and `shared` (modules whose classes cross between the two sides by value). Prepare refuses a path into the bundle handed to student code: `data_files` stages it instead.
 
 **Grading** uses declared per-item points and aggregation in `assignment.toml`. The
 default grade is the raw score scaled to the configured total; curves are opt-in.
@@ -109,8 +110,10 @@ Zero and withheld grades stay distinct. See the notes' teacher contract for poli
 
 ## Critical Files
 
-- `src/core/src/runner/python.rs` — PythonExecutor and embedded harness. Starts the isolated Python subprocess and checks its output.
-- `src/core/src/runner/sandbox.rs` — SandboxConfig + setrlimit application. Platform-conditional: RLIMIT_AS skipped on macOS, RLIMIT_* constants differ between Linux (c_uint) and macOS (c_int).
+- `src/core/src/runner/python.rs` — PythonExecutor: stages a unit, starts and joins both sides, reads the records, watches memory, probes the interpreter's read roots.
+- `src/core/src/runner/harness/{common,teacher,student}.py` — the two sides of a unit and what they share; `idle.py` is the student that measures what a submission's imports take, and `imports.py` reads those imports in a process of its own.
+- `src/core/src/runner/sandbox.rs` — `Isolation`, `Confinement`, the Seatbelt profile and Landlock/seccomp rules, plus setrlimit. Platform-conditional: RLIMIT_AS skipped on macOS, RLIMIT_* constants differ between Linux (c_uint) and macOS (c_int).
+- `src/core/src/runner/records.rs` — framed record reader with plan-derived bounds.
 - `src/core/src/runner/orchestrator.rs` — Runs prepared bundles per student, tokio parallel.
 - `src/core/src/models/spec.rs` — TOML test specification types.
 - `src/core/src/discovery.rs` — File discovery and archive extraction with size/count limits.

Cargo.toml

@@ -32,6 +32,7 @@ clap = { version = "4", features = ["derive", "env"] }
 
 # Scripting
 rhai = "1"
+shlex = "2"
 
 # Display
 comfy-table = "7"
@@ -53,6 +54,8 @@ calamine = "0.36"
 
 # System
 libc = "0.2"
+landlock = "0.4"
+seccompiler = "0.5"
 
 # Random
 rand = "0.9"

README.md

@@ -17,10 +17,16 @@ notes/                Private project documentation submodule
 ```
 
 Run Cargo commands from the repository root. The CLI depends on `scriptmark-core`;
-the core has no CLI, network, database or terminal UI dependencies. Python 3 is still
-required to execute student submissions, teacher checkers and reference solutions.
+the core has no CLI, network, database or terminal UI dependencies. Python 3.10 or newer
+is still required to execute student submissions, teacher checkers and reference
+solutions.
 ScriptMark is distributed as a Rust CLI; it does not expose a Python package/API.
 
+Student code and teacher code run in separate processes, each in the operating system's
+sandbox: Seatbelt on macOS, Landlock and seccomp on Linux (5.13 or later, Landlock
+enabled). Grading refuses where no sandbox can be applied; `--no-sandbox` grades without
+one — on Windows, for now — and the results are marked unisolated.
+
 ## Documentation and checkout
 
 Public documentation belongs in `docs/`. Private project documentation lives in
@@ -387,13 +393,13 @@ See [the scoring contract](https://github.com/Acture/obsidian-vault/blob/project
 | `sorted` / `set_eq` / `contains` | Collection checks |
 | Rhai expression | `check = { rhai = "result != () && result > 0" }` |
 | Python script | `check = { python = "verifiers/check.py" }` |
-| Teacher function | `check = { function = "is_valid" }`, run on the live value |
+| Teacher function | `check = { function = "is_valid" }`, run on the returned value |
 | Output and files | `expected_stdout = "..."`, `expect_files = { "out.txt" = "..." }` |
 
 ## Features
 
 - **Custom test engine** -- subprocess execution, no pytest dependency
-- **Isolated units** -- a process and directory per case, env isolation, import allowlist, setrlimit, timeout with kill; contains accidents, not a security sandbox ([details](https://github.com/Acture/obsidian-vault/blob/project/scriptmark/docs/test-bundles.md#what-grading-does-not-defend-against))
+- **Sandboxed units** -- every case runs as a teacher process and a student process, each in the OS sandbox: the student's reads only its own directory, with no network, no new programs and no signals to others; the teacher's reads the bundle but not other students' work or your environment. Resource limits, a memory watch and per-call deadlines contain accidents; built for introductory courses, not against a student attacking the grader ([details](https://github.com/Acture/obsidian-vault/blob/project/scriptmark/docs/test-bundles.md#what-grading-does-not-defend-against))
 - **Parallel** -- tokio orchestrator, grades 80+ students in seconds
 - **Parametrize + oracle** -- random inputs with teacher reference implementations
 - **Per-item scoring** -- declared points and aggregation; zero and withheld kept apart in every export

src/cli/src/main.rs

@@ -20,6 +20,7 @@ use scriptmark_core::runner::generation::SeedSource;
 use scriptmark_core::runner::orchestrator::{self, RunOptions};
 use scriptmark_core::runner::prepare::prepare;
 use scriptmark_core::runner::python::PythonExecutor;
+use scriptmark_core::runner::sandbox::{self, Isolation};
 use scriptmark_core::spec_loader::load_specs_from_dir;
 
 #[derive(Parser)]
@@ -80,6 +81,27 @@ struct FrozenArgs {
 	fresh: bool,
 }
 
+#[derive(clap::Args)]
+struct IsolationArgs {
+	/// Run student code without the operating system's sandbox. Each unit still runs as a
+	/// teacher side and a student side under resource limits, but student code can then
+	/// read and write whatever your user can — other students' files, the test bundle, your
+	/// home directory — and reach the network. The grading record marks the results
+	/// unisolated. For platforms ScriptMark cannot sandbox yet.
+	#[arg(long)]
+	no_sandbox: bool,
+}
+
+impl IsolationArgs {
+	fn isolation(&self) -> Isolation {
+		if self.no_sandbox {
+			Isolation::Unisolated
+		} else {
+			Isolation::Sandboxed
+		}
+	}
+}
+
 #[derive(Parser)]
 struct GradeArgs {
 	/// Directories containing student submissions
@@ -133,6 +155,9 @@ struct GradeArgs {
 	#[command(flatten)]
 	frozen: FrozenArgs,
 
+	#[command(flatten)]
+	isolation: IsolationArgs,
+
 	/// Replace the grading record at --output even when it holds rescored revisions,
 	/// discarding them. Long form only: there is no -f.
 	#[arg(long)]
@@ -183,6 +208,9 @@ struct RunArgs {
 	#[command(flatten)]
 	frozen: FrozenArgs,
 
+	#[command(flatten)]
+	isolation: IsolationArgs,
+
 	/// Replace the grading record at --output even when it holds rescored revisions,
 	/// discarding them. Long form only: there is no -f.
 	#[arg(long)]
@@ -650,9 +678,17 @@ fn scored(view: &View, path: &Path) -> Result<u32> {
 
 /// How a summary names what it shows.
 fn shown(path: &Path, view: &View) -> String {
-	match view.revision {
+	let title = match view.revision {
 		Some(n) => format!("{} (revision {n} of {})", path.display(), view.of),
 		None => format!("{} (unscored)", path.display()),
+	};
+	if view.sandboxed() {
+		title
+	} else {
+		format!(
+			"{title} — UNISOLATED: {}",
+			record::isolation_label(view.isolation)
+		)
 	}
 }
 
@@ -670,14 +706,33 @@ async fn run_bundles(
 	output: &Path,
 	options: &FrozenArgs,
 ) -> Result<(Vec<StudentReport>, Frozen, String)> {
+	if run_options.isolation.is_sandboxed() {
+		// Fail closed, before anything runs: a unit whose sandbox cannot be applied is
+		// never graded.
+		sandbox::available().map_err(anyhow::Error::msg).context(
+			"refusing to grade: student code cannot be sandboxed here (pass --no-sandbox to grade without isolation; the results will say so)",
+		)?;
+	} else {
+		eprintln!(
+			"warning: --no-sandbox: student code runs without a sandbox and can reach everything your user can; the results are marked unisolated"
+		);
+	}
 	let generation = match &options.replay {
 		Some(path) => Generation::Replay(Frozen::load(path)?),
 		None => Generation::fresh(),
 	};
-	let executor = Arc::new(PythonExecutor::with_python_cmd(&run_options.python));
-	let bundles = prepare(specs, &generation, executor.clone(), timeout)
-		.await
-		.context("refusing to grade: the test bundle is not ready")?;
+	let executor = Arc::new(
+		PythonExecutor::with_python_cmd(&run_options.python).with_isolation(run_options.isolation),
+	);
+	// Preparing runs teacher code in units of its own process groups too: Ctrl-C ends them.
+	let bundles = tokio::select! {
+		prepared = prepare(specs, &generation, executor.clone(), timeout) => prepared
+			.context("refusing to grade: the test bundle is not ready")?,
+		_ = tokio::signal::ctrl_c() => {
+			scriptmark_core::runner::python::kill_all_units();
+			anyhow::bail!("interrupted while preparing: every running unit was stopped")
+		}
+	};
 	println!(
 		"Prepared {} test bundle(s): {} unit(s) per student",
 		bundles.len(),
@@ -877,6 +932,7 @@ async fn execute(
 	let spec_versions = record::spec_versions(&specs)?;
 	let versions = record::submission_versions(&input.students)?;
 	let matching = run_options.matching.clone();
+	let isolation = run_options.isolation;
 	let (mut reports, frozen, python) = run_bundles(
 		&input.students,
 		specs,
@@ -891,6 +947,7 @@ async fn execute(
 		spec_versions,
 		timeout,
 		python,
+		isolation,
 		&frozen,
 		(!frozen.is_empty()).then(|| frozen::beside(output)),
 	);
@@ -940,6 +997,7 @@ async fn cmd_grade(args: GradeArgs) -> Result<()> {
 			concurrency: args
 				.concurrency
 				.map(|n| usize::try_from(n).unwrap_or(usize::MAX)),
+			isolation: args.isolation.isolation(),
 		},
 		args.timeout,
 		&args.output,
@@ -1053,6 +1111,7 @@ async fn cmd_run(args: RunArgs) -> Result<()> {
 			concurrency: args
 				.concurrency
 				.map(|n| usize::try_from(n).unwrap_or(usize::MAX)),
+			isolation: args.isolation.isolation(),
 		},
 		args.timeout,
 		&args.output,
@@ -1412,6 +1471,13 @@ async fn cmd_grades_push(args: GradesPushArgs) -> Result<()> {
 		}
 	}
 	let view = record.view(args.revision)?;
+	if !view.sandboxed() {
+		eprintln!(
+			"warning: {} is {}: student code was not isolated when these grades were made",
+			args.results.display(),
+			record::isolation_label(view.isolation)
+		);
+	}
 	let revision = scored(&view, &args.results)?;
 
 	// Only graded students are pushed — a real 0 included, a withheld grade never. The
@@ -1527,6 +1593,7 @@ fn cmd_similarity(args: SimilarityArgs) -> Result<()> {
 
 fn cmd_report(args: ReportArgs) -> Result<()> {
 	let (_, view) = load_view(&args.results, args.revision.revision)?;
+	let (sandboxed, isolation) = (view.sandboxed(), view.isolation);
 	let reports = view.reports;
 
 	let similarity = if let Some(sim_dir) = &args.similarity_dir {
@@ -1553,7 +1620,16 @@ fn cmd_report(args: ReportArgs) -> Result<()> {
 		None
 	};
 
-	report::generate_html_report(&reports, similarity.as_deref(), &args.title, &args.output)?;
+	let title = if sandboxed {
+		args.title.clone()
+	} else {
+		format!(
+			"{} — UNISOLATED: {}",
+			args.title,
+			record::isolation_label(isolation)
+		)
+	};
+	report::generate_html_report(&reports, similarity.as_deref(), &title, &args.output)?;
 
 	println!("HTML report generated: {}", args.output.display());
 	Ok(())

src/cli/tests/record.rs

@@ -6,6 +6,7 @@ use std::path::Path;
 use std::process::{Command, Output};
 
 use scriptmark_core::record::Record;
+use scriptmark_core::runner::sandbox::Isolation;
 
 const SPEC: &str = r#"
 [meta]
@@ -877,3 +878,42 @@ fn a_canvas_record_rescores_from_its_bundle_and_refuses_another_attempt_or_excus
 	std::fs::write(&payload_path, payload.to_string()).unwrap();
 	refused(rescore(dir, &[]), "2024002 is excused now");
 }
+
+/// OSS-10: a record says whether student code ran in its sandbox, and a summary of one
+/// that did not says so.
+#[test]
+fn a_run_without_the_sandbox_is_recorded_as_unisolated() {
+	let dir = bench();
+	for (flags, isolated) in [(&[][..], true), (&["--no-sandbox"][..], false)] {
+		let mut args = vec![
+			"grade",
+			"submissions",
+			"-t",
+			"tests",
+			"-o",
+			"out/result.json",
+			"--force",
+		];
+		args.extend(flags);
+		let output = scriptmark(dir.path(), &args);
+		assert!(
+			output.status.success(),
+			"{}",
+			String::from_utf8_lossy(&output.stderr)
+		);
+		let text = std::fs::read_to_string(dir.path().join("out/result.json")).unwrap();
+		let record = Record::load(&dir.path().join("out/result.json")).unwrap();
+		let expected = if isolated {
+			Isolation::Sandboxed
+		} else {
+			Isolation::Unisolated
+		};
+		assert_eq!(record.evidence.bundle.isolation, Some(expected));
+		assert!(text.contains("\"isolation\""), "{text}");
+		let summary = scriptmark(dir.path(), &["summarize", "out/result.json"]);
+		assert_eq!(
+			String::from_utf8_lossy(&summary.stdout).contains("UNISOLATED"),
+			!isolated
+		);
+	}
+}

src/core/Cargo.toml

@@ -37,4 +37,9 @@ rars = "0.9.4"
 bzip2-rs = "0.1.2"
 lzma-rs = "0.3.0"
 tempfile = "3"
+shlex = { workspace = true }
 sha2 = "0.10"
+
+[target.'cfg(target_os = "linux")'.dependencies]
+landlock = { workspace = true }
+seccompiler = { workspace = true }