From 9d21df4930ef560aad6364c699e4138fa4341f36 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:22:34 +0900 Subject: [PATCH 01/17] test: probe execution-unit observation across controller loss --- ...2026-11-06-observable-unit-design-draft.md | 163 ++++++++ .agents/docs/README.md | 5 + .github/probes/process_unit.cpp | 361 ++++++++++++++++++ .github/workflows/process-unit-probe.yml | 46 +++ 4 files changed, 575 insertions(+) create mode 100644 .agents/docs/2026-11-06-observable-unit-design-draft.md create mode 100644 .agents/docs/README.md create mode 100644 .github/probes/process_unit.cpp create mode 100644 .github/workflows/process-unit-probe.yml diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md new file mode 100644 index 0000000..7583c54 --- /dev/null +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -0,0 +1,163 @@ +--- +subject: process-unit-lifetime +status: draft +--- + +# Compiler unit lifetime: API inventory and minimum recovery design + +This is a design review draft. It does not qualify the current candidate for +noncooperative supervisor loss. No incomplete output-only recovery patch has +been applied. + +## Evidence and ownership + +The pinned OS 0.2.3 uses OpenKal 0.17.1, Linux 0.18.3, macOS 0.15.1 and +Windows 0.13.3. Its process API separates an owned process unit from a bound +program lifetime. A process handle can end its unit while that handle is owned. +It exposes neither a persistent unit reference nor post-controller observation. + +The Linux backend binds the directly started program through PDEATHSIG. It does +not bind every descendant. A controlled two-level compiler in the same unit +remains alive and writes its object after only the supervisor is killed. +Separately, killing a supervisor and its compiler after BMI publication, then +killing the top-level owner, admits the next generation but leaves it waiting +for a nonexistent old status rather than recompiling the interface. + +Published releases were independently checked: OS 0.2.9, OpenKal 0.23.0, +Linux 0.24.2, macOS 0.21.3 and Windows 0.19.0. OS 0.2.9 still exposes the same +process/unit lifetime boundary. OpenKal 0.23 includes job_usage, introduced in +0.20, but it requires a held job handle. It does not provide a recoverable unit +reference. A Linux numeric process group and a Windows HANDLE are not portable, +persistent capabilities. Simply upgrading the existing dependency is insufficient. + +Core owns build transactions, permits and output recovery. OS/OpenKal own unit +creation, observation and lifetime semantics. Plugins own neither of these +kernel responsibilities. Core must not inspect native PIDs/PGIDs, recreate a +Windows HANDLE from its numeric value, or interpret leader death as unit death. + +## Minimum generic OS capability + +The needed capability is an observable execution unit with an opaque reference. +The names below describe requirements rather than an approved public spelling. + +1. A helper enters a private execution unit before starting any external + compiler. It receives a reference that another invocation can observe. +2. The reference can report `occupied`, positively proven `empty`, or `unknown`. + Unsupported queries, stale/unreadable data and permission failures are not + `empty`. Observation of an expired reference never grants termination + authority over a different unit that reuses a native identifier. +3. Live owners retain the existing unit termination capability. Recovered + references may be observation-only. A remaining orphan unit keeps its + logical permit reserved; a bounded refusal is safer than guessing it died. +4. Actual compiler processes inherit that explicitly created private unit. This + is a deliberate composition of a declared unit and its child, not an + unqualified switch to sharing Ninja's unit. +5. Unit observation counts remaining writers after the leader exits. It remains + usable after the controller dies. Closing an observer does not itself claim + that the unit has stopped. + +Creating the private unit inside the existing compiler helper avoids an unknown +spawn-identity window: its reference and output record are committed before the +helper may launch the real compiler. If that publication fails or the helper +ends first, no compiler is permitted to have started. Phase one already launches +its supervisor in a separate unit; ordinary compiler wrappers need the same +explicit bootstrap. The generic launcher needs a way to run a child in a +provided current unit rather than automatically create another unobserved unit. + +Candidate backend implementations require experiments before approval: + +- POSIX: use the unit's own observer contract. An occupancy check must prove + emptiness without sending a terminating signal or treating an absent leader + as an empty group. Identifier reuse and incomplete process enumeration must + yield conservative results. Native details remain in the backend. +- Windows: an observable named job is a candidate. Verify native lifetime of its + name and membership after the last controller handle closes, including nested + jobs and termination in progress. Never serialize a HANDLE for another + process to use. + +The existing job_usage operation can supply live-owner statistics. It cannot +on its own provide the missing persistent observation guarantee. + +## Core transaction and permit protocol + +The generation and actor barrier remain runtime state outside build identity. +Each attempt records absolute output identities before its first state change: +BMI, object, slot status/log/marker, dependency output and the verified previous +BMI backup. Resolving these paths occurs in the original execution context; +recovery does not resolve them against its own cwd or a newly changed target +symlink/home. + +Phase one publishes its prepared transaction before clearing an old status or +copying a backup. A backup becomes recoverable only after its copy and identity +are complete. The supervisor publishes its observable unit reference before any +external compiler starts. Stage records have separate writers, avoiding a +phase-one update overwriting the supervisor's unit reference. + +A compiler slot has both its kernel lock and a durable reservation pointing at +this attempt. A free kernel lock is insufficient when its previous reservation +still names a potentially occupied unit. A new holder observes that unit first; +it never overwrites an occupied or unknown reservation. This applies within one +running generation as well as between generations. It therefore preserves the +actual compiler count even when a supervisor is forcibly killed. + +Normal completion clears the reservation only after the external compiler unit +has no remaining writers. The helper's final metadata writes remain covered by +its actor lease. A cooperative cancellation records intent and ends the entire +private unit. If the helper dies before publishing status, its durable record +still protects the outputs and capacity. + +A phase-one/join waiter notices an abandoned controller through the attempt +protocol. It reports failure promptly; it does not silently wait two hours for +an rc that no remaining controller can publish. It does not delete an output +while the old unit is occupied or unknown. + +A new generation waits for both all old actor leases and all referenced compiler +units. Only then may it recover outputs. A committed successful status with +complete verified outputs is retained. An incomplete published BMI/object/status +is invalidated so Ninja reruns phase one, rather than replaying an orphan join. +The previous verified BMI remains a protected baseline that the next attempt can +restore on failure; it must not be accidentally deleted by the ordinary backup +initialization. Runtime transaction identifiers are not Ninja command hashes. + +Recovery is idempotent: interruption after each file operation can be retried +from the durable stage record. Unknown schema, corrupted records, record-write +failure and ambiguous unit state refuse admission and retain evidence. They do +not remove a live reservation or fall back to an unbounded compiler. + +## Required fault matrix before one coherent implementation push + +| Boundary | Fault | Required observation | +|---|---|---| +| Before first phase-one modification | kill phase one/owner | no unregistered state write; retry remains valid | +| After status reset or during backup copy | kill phase one/owner | incomplete backup is not certified; previous complete BMI preserved | +| Private unit created, reference not committed | kill helper | real compiler never started | +| Reference committed, compiler not started | kill helper | safe unit observation and idempotent empty-attempt recovery | +| Compiler running, BMI absent | kill supervisor alone | descendants counted; slot remains reserved; no premature output deletion | +| BMI published, compiler running | kill supervisor/owner | no orphan await; old writers quiet before invalidation/new compile | +| Compiler leader exits, descendant remains | normal/forced exit | unit remains occupied despite leader death | +| Before rc atomic rename | kill supervisor | missing rc recovered only from the unit/transaction proof | +| After rc atomic rename | kill supervisor/owner | committed successful outputs retained; failed outputs invalidated safely | +| Phase-one settle/restore after compiler exit | delay/kill writer | actor barrier covers the final write, not only compiler capacity | +| Recovery between any two file operations | kill new owner | next recovery is safe and idempotent | +| Record corruption/write refusal | inject failure | no compiler starts without its record; no guessed deletion | +| Cwd/home/symlink changes | restart | original absolute output identities recovered exactly | +| Normal completion and cached normal/quiet replay | no fault | unchanged objects/BMIs/graph; no runtime identity in cache keys | + +The matrix must include an actual driver with a descendant, not only a direct +sleeping child. It must exercise caps one and two, late failures and cancellation +without awaiting a new full toolchain build for every local checkpoint. + +## Dependency sequence + +First approve and test the generic OS/backend observation contract. Reuse the +published job-usage capability where sufficient, but add the missing persistent +reference semantics through OpenKal and the native backends rather than native +helpers in core. Upgrade the OS package to expose it, publish and index that +coherent dependency group, then pin core and run the full matrix and native CI. +Versions follow each repository's release rules; published tags stay unchanged. + +If a new process API changes the OpenKal source/ABI floor, the matching runtime +and platform rows used by core's openkal-target CI must be included in that +version group. A source-only host success does not qualify those image rows. +Plugin sidecar work remains prepared independently and receives the final public +core pin only after this release is actually available. diff --git a/.agents/docs/README.md b/.agents/docs/README.md new file mode 100644 index 0000000..1fca3a3 --- /dev/null +++ b/.agents/docs/README.md @@ -0,0 +1,5 @@ +# Active design records + +- [Observable process units: inventory and recovery draft](2026-11-06-observable-unit-design-draft.md) + records the observed lifetime boundary, the required OS responsibilities and + the fault matrix that precedes a compiler-recovery implementation. diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp new file mode 100644 index 0000000..0bc81a1 --- /dev/null +++ b/.github/probes/process_unit.cpp @@ -0,0 +1,361 @@ +// Native experiments for an observable execution unit. This is not a new +// OpenKal API or a capability claim; it records the kernel facts an API needs. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef _WIN32 +#define NOMINMAX +#include +#else +#include +#include +#include +#include +#ifdef __linux__ +#include +#else +#include +#include +#include +#endif +#endif + +namespace fs = std::filesystem; +using namespace std::chrono_literals; +void require(bool value, const char* message) { if (!value) throw std::runtime_error(message); } +void report(const char* name, bool value) { + std::cout << "{\"check\":\"" << name << "\",\"ok\":" << (value ? "true" : "false") << "}\n"; + std::cout.flush(); + require(value, name); +} +template bool until(Predicate predicate, std::chrono::milliseconds bound = 5000ms) { + const auto end = std::chrono::steady_clock::now() + bound; + do { if (predicate()) return true; std::this_thread::sleep_for(5ms); } + while (std::chrono::steady_clock::now() < end); + return false; +} +std::string text(const fs::path& path) { + std::ifstream input(path); return {std::istreambuf_iterator(input), {}}; +} +void put(const fs::path& path, std::string_view value) { std::ofstream out(path); out << value; out.flush(); require(bool(out), "write_control_record"); } +enum class observation { empty, occupied, unknown }; + +#ifdef _WIN32 +struct handle { + HANDLE value = nullptr; + handle() = default; + explicit handle(HANDLE h) : value(h) {} + void reset() { if (value && value != INVALID_HANDLE_VALUE) CloseHandle(value); value = nullptr; } + ~handle() { reset(); } + handle(const handle&) = delete; + handle& operator=(const handle&) = delete; +}; +std::string quote(const std::string& value) { return '"' + value + '"'; } +std::string executable() { + char path[32768]; const auto size = GetModuleFileNameA(nullptr, path, sizeof path); + require(size > 0 && size < sizeof path, "self_executable"); return {path, size}; +} +HANDLE start(const std::vector& args) { + std::string command = quote(executable()); + for (const auto& arg : args) command += " " + quote(arg); + STARTUPINFOA startup{}; startup.cb = sizeof startup; + PROCESS_INFORMATION child{}; + require(CreateProcessA(nullptr, command.data(), nullptr, nullptr, FALSE, 0, + nullptr, nullptr, &startup, &child), "start_native_process"); + CloseHandle(child.hThread); return child.hProcess; +} +FILETIME created(HANDLE process) { + FILETIME start{}, end{}, kernel{}, user{}; + require(GetProcessTimes(process, &start, &end, &kernel, &user), "read_worker_birth"); + return start; +} +bool same_birth(FILETIME left, FILETIME right) { + return left.dwLowDateTime == right.dwLowDateTime && left.dwHighDateTime == right.dwHighDateTime; +} +DWORD members(HANDLE job) { + JOBOBJECT_BASIC_ACCOUNTING_INFORMATION usage{}; + require(QueryInformationJobObject(job, JobObjectBasicAccountingInformation, + &usage, sizeof usage, nullptr), "query_job_members"); + return usage.ActiveProcesses; +} +observation observe(const std::string& name) { + handle job(OpenJobObjectA(JOB_OBJECT_QUERY, FALSE, name.c_str())); + if (!job.value) { + return GetLastError() == ERROR_FILE_NOT_FOUND ? observation::empty : observation::unknown; + } + return members(job.value) == 0 ? observation::empty : observation::occupied; +} +int worker(const fs::path& root) { + put(root / "worker.ready", std::to_string(GetCurrentProcessId())); + require(until([&] { return fs::exists(root / "worker.release"); }, 20000ms), "worker_release"); + put(root / "late.write", "writer finished"); + return 0; +} +int controller(const fs::path& root, const std::string& name, bool bound) { + handle job(CreateJobObjectA(nullptr, name.c_str())); + require(job.value && GetLastError() != ERROR_ALREADY_EXISTS, "new_named_job"); + if (bound) { + JOBOBJECT_EXTENDED_LIMIT_INFORMATION limit{}; + limit.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + require(SetInformationJobObject(job.value, JobObjectExtendedLimitInformation, + &limit, sizeof limit), "set_bound_job"); + } + require(AssignProcessToJobObject(job.value, GetCurrentProcess()), "enter_private_job"); + handle child(start({"worker", root.string()})); + require(until([&] { return fs::exists(root / "worker.ready"); }), "worker_ready"); + put(root / "controller.ready", std::to_string(members(job.value))); + if (until([&] { return fs::exists(root / "normal.release"); }, 20000ms)) { + require(WaitForSingleObject(child.value, 5000) == WAIT_OBJECT_0, "normal_worker_exit"); + put(root / "owner.only", std::to_string(members(job.value))); + require(until([&] { return fs::exists(root / "owner.release"); }, 20000ms), "normal_owner_release"); + } + return 0; +} +void windows_case(const fs::path& root, bool bound, bool normal) { + fs::create_directories(root); + const auto suffix = std::to_string(GetCurrentProcessId()) + "-" + std::to_string(GetTickCount64()) + + "-" + std::to_string(bound) + "-" + std::to_string(normal); + const auto name = "Global\\openkal-unit-probe-" + suffix; + handle owner(start({"controller", root.string(), name, bound ? "1" : "0"})); + handle child; + DWORD worker_id = 0; + FILETIME worker_birth{}; + try { + require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); + report("private_job_contains_helper_and_writer", text(root / "controller.ready") == "2"); + const auto pid = static_cast(std::stoul(text(root / "worker.ready"))); + worker_id = pid; + child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); + require(child.value != nullptr, "hold_worker_process_identity"); + worker_birth = created(child.value); + if (!normal && !bound) child.reset(); + report("live_unit_is_not_empty", observe(name) == observation::occupied); + if (normal) { + put(root / "worker.release", "go"); put(root / "normal.release", "go"); + require(until([&] { return fs::exists(root / "owner.only"); }), "owner_only_ready"); + report("normal_completion_counts_helper_separately", text(root / "owner.only") == "1"); + report("helper_only_is_occupied_not_empty", observe(name) == observation::occupied); + put(root / "owner.release", "go"); + require(WaitForSingleObject(owner.value, 5000) == WAIT_OBJECT_0, "normal_controller_exit"); + } else { + require(TerminateProcess(owner.value, 137), "terminate_controller"); + require(WaitForSingleObject(owner.value, 5000) == WAIT_OBJECT_0, "controller_terminated"); + owner.reset(); + if (!bound) { + // Query before opening even a process handle in this observer: + // no retained controller/worker handle may mask name lifetime. + report("named_job_survives_without_observer_process_handles", observe(name) == observation::occupied); + child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); + require(child.value != nullptr && same_birth(created(child.value), worker_birth), "hold_orphan_process_identity"); + report("writer_survives_unbound_controller", WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT); + report("named_job_remains_observable_without_controller_handle", observe(name) == observation::occupied); + put(root / "worker.release", "go"); + require(until([&] { return fs::exists(root / "late.write"); }), "late_writer_output"); + } + } + bool false_empty = false; + require(until([&] { + const auto state = observe(name); + const bool ended = WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; + if (state == observation::empty && !ended) false_empty = true; + return ended && state == observation::empty; + }), "whole_job_empty_after_writer_exit"); + report("missing_named_job_never_precedes_live_writer", !false_empty); + report("complete_job_is_empty", observe(name) == observation::empty); + } catch (...) { + if (owner.value) TerminateProcess(owner.value, 137); + if (!child.value && worker_id) + child.value = OpenProcess(PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, worker_id); + if (child.value && same_birth(created(child.value), worker_birth)) TerminateProcess(child.value, 137); + throw; + } +} +#else +std::uint64_t born(pid_t pid) { +#ifdef __linux__ + auto stat = text(fs::path("/proc") / std::to_string(pid) / "stat"); + const auto end = stat.rfind(')'); + if (end == std::string::npos) return 0; + std::istringstream fields(stat.substr(end + 2)); std::string field; + for (int i = 0; i <= 19; ++i) if (!(fields >> field)) return 0; + return std::stoull(field); +#else + proc_bsdinfo info{}; + if (proc_pidinfo(pid, PROC_PIDTBSDINFO, 0, &info, sizeof info) != sizeof info) return 0; + return std::uint64_t(info.pbi_start_tvsec) * 1000000ull + info.pbi_start_tvusec; +#endif +} +std::string instance() { +#ifdef __linux__ + char ns[256]; const auto size = readlink("/proc/self/ns/pid", ns, sizeof ns); + if (size <= 0) return {}; + return text("/proc/sys/kernel/random/boot_id") + std::string(ns, static_cast(size)); +#else + char value[128]; std::size_t size = sizeof value; + if (sysctlbyname("kern.bootsessionuuid", value, &size, nullptr, 0)) return {}; + return std::string(value, size > 0 && value[size - 1] == '\0' ? size - 1 : size); +#endif +} +struct reference { pid_t group; std::uint64_t owner_born; std::string origin; }; +observation observe(const reference& ref) { + if (ref.group <= 0 || ref.owner_born == 0 || ref.origin.empty() || ref.origin != instance()) + return observation::unknown; + if (kill(-ref.group, 0) == -1) return errno == ESRCH ? observation::empty : observation::unknown; + const auto current = born(ref.group); + if (current && current != ref.owner_born) return observation::unknown; + return observation::occupied; +} +void reap() { while (waitpid(-1, nullptr, WNOHANG) > 0) {} } +void wait_reap(pid_t pid) { + while (waitpid(pid, nullptr, 0) < 0 && errno == EINTR) {} +} +pid_t launch(const fs::path& root) { + const auto owner = fork(); require(owner >= 0, "fork_controller"); + if (owner != 0) return owner; + if (setpgid(0, 0)) _exit(90); + const auto driver = fork(); if (driver < 0) _exit(91); + if (driver == 0) { +#ifdef __linux__ + const auto parent = getppid(); + if (prctl(PR_SET_PDEATHSIG, SIGKILL) || getppid() != parent) _exit(92); +#endif + const auto writer = fork(); if (writer < 0) _exit(93); + if (writer == 0) { + put(root / "writer.ready", std::to_string(getpid())); + if (!until([&] { return fs::exists(root / "writer.release"); }, 20000ms)) _exit(94); + put(root / "late.write", "writer finished"); _exit(0); + } + put(root / "driver.ready", std::to_string(getpid())); + wait_reap(writer); _exit(0); + } + put(root / "controller.ready", std::to_string(getpid())); + bool reaped = false; + if (!until([&] { + if (!reaped && waitpid(driver, nullptr, WNOHANG) == driver) { + reaped = true; put(root / "owner.only", "ready"); + } + return fs::exists(root / "owner.release"); + }, 20000ms)) _exit(95); + if (!reaped) wait_reap(driver); + _exit(0); +} +std::size_t members(pid_t group) { +#ifdef __linux__ + std::size_t result = 0; + for (const auto& entry : fs::directory_iterator("/proc")) { + const auto name = entry.path().filename().string(); + if (name.empty() || name[0] < '1' || name[0] > '9') continue; + const auto row = text(entry.path() / "stat"); const auto end = row.rfind(')'); + if (end == std::string::npos) continue; + std::istringstream fields(row.substr(end + 2)); std::string state; long parent = 0, id = 0; + if (fields >> state >> parent >> id && id == group) ++result; + } + return result; +#else + pid_t ids[2048]{}; + const int bytes = proc_listpgrppids(static_cast(group), ids, sizeof ids); + require(bytes > 0, "list_private_group_members"); + return static_cast(bytes) / sizeof(pid_t); +#endif +} +void posix_case(const fs::path& root, bool normal) { + fs::create_directories(root); + const auto owner = launch(root); + try { + require(until([&] { return fs::exists(root / "writer.ready") && fs::exists(root / "driver.ready") + && fs::exists(root / "controller.ready"); }), "private_unit_ready"); + reference ref{owner, born(owner), instance()}; + const auto writer = static_cast(std::stoi(text(root / "writer.ready"))); + const auto driver = static_cast(std::stoi(text(root / "driver.ready"))); + report("all_writers_in_declared_private_unit", getpgid(owner) == owner && getpgid(driver) == owner && getpgid(writer) == owner); + report("live_unit_is_not_empty", observe(ref) == observation::occupied); + auto wrong = ref; wrong.origin += "different"; + report("foreign_instance_is_unknown", observe(wrong) == observation::unknown); + if (normal) { + put(root / "writer.release", "go"); + require(until([&] { return fs::exists(root / "late.write"); }), "normal_writer_finished"); + require(until([&] { return fs::exists(root / "owner.only"); }), "normal_owner_only_ready"); + report("normal_completion_counts_helper_separately", members(owner) == 1); + report("live_helper_does_not_mean_empty", observe(ref) == observation::occupied); + put(root / "owner.release", "go"); + } else { + require(kill(owner, SIGKILL) == 0, "kill_controller"); wait_reap(owner); + report("leader_death_is_not_unit_empty", observe(ref) == observation::occupied); + put(root / "writer.release", "go"); + require(until([&] { return fs::exists(root / "late.write"); }), "orphan_writer_finished"); + report("writer_can_finish_after_controller_death", true); + } + if (normal) wait_reap(owner); + require(until([&] { reap(); return observe(ref) == observation::empty; }), "unit_reaped_empty"); + report("whole_unit_empty_after_all_members_finish", true); + } catch (...) { + kill(-owner, SIGKILL); kill(owner, SIGKILL); wait_reap(owner); reap(); throw; + } +} +#ifdef __linux__ +void reuse_case() { + require(getpid() == 1, "private_pid_namespace_required"); + const char* outside = std::getenv("PROBE_OUTER_PID_NAMESPACE"); + require(outside && instance().find(outside) == std::string::npos, "outer_namespace_must_differ"); + auto make = [] { + const auto child = fork(); require(child >= 0, "fork_reuse_leader"); + if (child == 0) { if (setpgid(0, 0)) _exit(96); for (;;) pause(); } + require(until([&] { return getpgid(child) == child; }), "reused_group_ready"); + return child; + }; + const auto first = make(); reference old{first, born(first), instance()}; + require(kill(-first, SIGKILL) == 0, "end_old_group"); wait_reap(first); + report("old_group_proven_empty", observe(old) == observation::empty); + std::this_thread::sleep_for(30ms); + put("/proc/sys/kernel/ns_last_pid", std::to_string(first - 1)); + const auto next = make(); + try { + report("native_group_identifier_actually_reused", next == first); + report("new_owner_has_different_birth", born(next) != old.owner_born); + report("reused_reference_does_not_claim_quiet", observe(old) == observation::unknown); + report("observation_never_terminates_unrelated_unit", kill(next, 0) == 0); + } catch (...) { kill(-next, SIGKILL); wait_reap(next); throw; } + kill(-next, SIGKILL); wait_reap(next); + report("reference_can_be_empty_after_replacement_ends", observe(old) == observation::empty); +} +#endif +#endif + +int main(int argc, char** argv) { + try { +#ifdef _WIN32 + if (argc > 2 && std::string(argv[1]) == "worker") return worker(argv[2]); + if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], std::string(argv[4]) == "1"); +#else +#ifdef __linux__ + if (argc > 1 && std::string(argv[1]) == "--pid-reuse") { reuse_case(); return 0; } + require(prctl(PR_SET_CHILD_SUBREAPER, 1) == 0, "test_harness_reaps_orphans"); +#endif +#endif + const auto nonce = std::chrono::steady_clock::now().time_since_epoch().count(); + const auto root = fs::temp_directory_path() / ("openkal-unit-probe-" + std::to_string(nonce)); + fs::create_directories(root); + try { +#ifdef _WIN32 + windows_case(root / "normal", false, true); + windows_case(root / "owner-lost", false, false); + windows_case(root / "bound-owner-lost", true, false); +#else + posix_case(root / "normal", true); + posix_case(root / "owner-lost", false); +#endif + } catch (...) { fs::remove_all(root); throw; } + fs::remove_all(root); + return 0; + } catch (const std::exception& error) { + std::cerr << "unit probe failed: " << error.what() << '\n'; return 1; + } +} diff --git a/.github/workflows/process-unit-probe.yml b/.github/workflows/process-unit-probe.yml new file mode 100644 index 0000000..9de1adc --- /dev/null +++ b/.github/workflows/process-unit-probe.yml @@ -0,0 +1,46 @@ +name: process unit observer experiments +on: + pull_request: + paths: + - '.github/probes/process_unit.cpp' + - '.github/workflows/process-unit-probe.yml' + workflow_dispatch: +concurrency: + group: unit-probe-${{ github.ref }} + cancel-in-progress: true +jobs: + native: + name: native unit lifetime (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + os: [ubuntu-24.04, macos-15, windows-2025] + steps: + - uses: actions/checkout@v4 + - uses: ilammy/msvc-dev-cmd@v1 + if: runner.os == 'Windows' + - name: Compile native experiment (Unix) + if: runner.os != 'Windows' + run: c++ -std=c++17 -O2 -pthread .github/probes/process_unit.cpp -o unit-probe + - name: Compile native experiment (Windows) + if: runner.os == 'Windows' + shell: cmd + run: cl /nologo /std:c++17 /EHsc /O2 .github\probes\process_unit.cpp /Fe:unit-probe.exe + - name: Observe real unit lifetime + shell: bash + run: | + set -o pipefail + ./unit-probe | tee unit-probe.jsonl + - name: Reuse a native identifier inside an isolated PID namespace + if: runner.os == 'Linux' + run: | + set -o pipefail + export PROBE_OUTER_PID_NAMESPACE=$(readlink /proc/self/ns/pid) + sudo --preserve-env=PROBE_OUTER_PID_NAMESPACE unshare --pid --fork --mount-proc ./unit-probe --pid-reuse | tee unit-reuse.jsonl + - uses: actions/upload-artifact@v4 + if: always() + with: + name: process-unit-${{ matrix.os }} + path: unit-*.jsonl From b7256b7901b65a26e69188639323ab617ba89cc3 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:23:41 +0900 Subject: [PATCH 02/17] test: record native stderr producer intervals --- .github/probes/process_unit.cpp | 40 +++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 0bc81a1..dfc6de2 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -18,6 +18,7 @@ #include #include #include +#include #include #ifdef __linux__ #include @@ -300,6 +301,43 @@ void posix_case(const fs::path& root, bool normal) { kill(-owner, SIGKILL); kill(owner, SIGKILL); wait_reap(owner); reap(); throw; } } +void stderr_intervals(bool external_sleep) { + int channel[2]; require(pipe(channel) == 0, "producer_pipe"); + const auto child = fork(); require(child >= 0, "producer_fork"); + if (child == 0) { + setpgid(0, 0); close(channel[0]); dup2(channel[1], 2); close(channel[1]); close(1); + if (external_sleep) { + execl("/bin/sh", "sh", "-c", "for i in 1 2 3 4 5 6 7 8; do printf x >&2; sleep .03; done; exit 7", nullptr); + _exit(97); + } + for (int i = 0; i < 24; ++i) { + write(2, "x", 1); std::this_thread::sleep_for(30ms); + } + _exit(7); + } + close(channel[1]); + auto previous = std::chrono::steady_clock::now(); + std::chrono::nanoseconds longest{0}; std::size_t bytes = 0; + try { + for (;;) { + pollfd ready{channel[0], POLLIN, 0}; + const auto waited = poll(&ready, 1, 5000); + if (waited < 0 && errno == EINTR) continue; + require(waited > 0, "stderr_progress_observed"); + char buffer[128]; const auto n = read(channel[0], buffer, sizeof buffer); + require(n >= 0, "stderr_read"); + const auto now = std::chrono::steady_clock::now(); + longest = std::max(longest, std::chrono::duration_cast(now - previous)); + previous = now; + if (n == 0) break; + bytes += static_cast(n); + } + close(channel[0]); wait_reap(child); + require(bytes == (external_sleep ? 8u : 24u), "all_stderr_ticks_observed"); + std::cout << "{\"producer\":\"" << (external_sleep ? "shell_with_external_sleep" : "one_process_ticks") + << "\",\"max_observed_gap_ns\":" << longest.count() << ",\"bytes\":" << bytes << "}\n"; + } catch (...) { close(channel[0]); kill(-child, SIGKILL); wait_reap(child); throw; } +} #ifdef __linux__ void reuse_case() { require(getpid() == 1, "private_pid_namespace_required"); @@ -351,6 +389,8 @@ int main(int argc, char** argv) { #else posix_case(root / "normal", true); posix_case(root / "owner-lost", false); + for (int repeat = 0; repeat < 5; ++repeat) stderr_intervals(true); + stderr_intervals(false); #endif } catch (...) { fs::remove_all(root); throw; } fs::remove_all(root); From 75a0fb68d6d661f180783ccbfd72ae3fe8fd33c4 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:24:21 +0900 Subject: [PATCH 03/17] test: make native probe setup explicit --- .github/probes/process_unit.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index dfc6de2..d32ba82 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -1,6 +1,7 @@ // Native experiments for an observable execution unit. This is not a new // OpenKal API or a capability claim; it records the kernel facts an API needs. #include +#include #include #include #include @@ -101,6 +102,7 @@ int worker(const fs::path& root) { return 0; } int controller(const fs::path& root, const std::string& name, bool bound) { + SetLastError(ERROR_SUCCESS); handle job(CreateJobObjectA(nullptr, name.c_str())); require(job.value && GetLastError() != ERROR_ALREADY_EXISTS, "new_named_job"); if (bound) { @@ -311,7 +313,8 @@ void stderr_intervals(bool external_sleep) { _exit(97); } for (int i = 0; i < 24; ++i) { - write(2, "x", 1); std::this_thread::sleep_for(30ms); + if (write(2, "x", 1) != 1) _exit(98); + std::this_thread::sleep_for(30ms); } _exit(7); } From 8a3a749e5b97b07845fd0002ccb914d258045b05 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:25:13 +0900 Subject: [PATCH 04/17] test: publish probe control records atomically --- .github/probes/process_unit.cpp | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index d32ba82..d8e97bf 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -47,7 +47,11 @@ template bool until(Predicate predicate, std::chrono::milliseco std::string text(const fs::path& path) { std::ifstream input(path); return {std::istreambuf_iterator(input), {}}; } -void put(const fs::path& path, std::string_view value) { std::ofstream out(path); out << value; out.flush(); require(bool(out), "write_control_record"); } +void put(const fs::path& path, std::string_view value) { + const auto pending = fs::path(path.string() + ".pending"); + { std::ofstream out(pending); out << value; out.flush(); require(bool(out), "write_control_record"); } + fs::rename(pending, path); +} enum class observation { empty, occupied, unknown }; #ifdef _WIN32 @@ -356,7 +360,8 @@ void reuse_case() { require(kill(-first, SIGKILL) == 0, "end_old_group"); wait_reap(first); report("old_group_proven_empty", observe(old) == observation::empty); std::this_thread::sleep_for(30ms); - put("/proc/sys/kernel/ns_last_pid", std::to_string(first - 1)); + { std::ofstream allocator("/proc/sys/kernel/ns_last_pid"); + allocator << first - 1; allocator.flush(); require(bool(allocator), "private_pid_allocator_write"); } const auto next = make(); try { report("native_group_identifier_actually_reused", next == first); From 5c2760df7ff847a7282ea0e23b3acd5fc939224d Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:29:24 +0900 Subject: [PATCH 05/17] test: distinguish missing job names from stopped writers --- .github/probes/process_unit.cpp | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index d8e97bf..999882a 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -159,13 +159,23 @@ void windows_case(const fs::path& root, bool bound, bool normal) { if (!bound) { // Query before opening even a process handle in this observer: // no retained controller/worker handle may mask name lifetime. - report("named_job_survives_without_observer_process_handles", observe(name) == observation::occupied); + const auto first = observe(name); + const auto query_error = GetLastError(); child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); require(child.value != nullptr && same_birth(created(child.value), worker_birth), "hold_orphan_process_identity"); - report("writer_survives_unbound_controller", WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT); - report("named_job_remains_observable_without_controller_handle", observe(name) == observation::occupied); + const bool writer_alive = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; + std::cout << "{\"post_controller_state\":" << static_cast(first) + << ",\"query_error\":" << query_error + << ",\"writer_alive\":" << (writer_alive ? "true" : "false") << "}\n"; put(root / "worker.release", "go"); - require(until([&] { return fs::exists(root / "late.write"); }), "late_writer_output"); + require(until([&] { return fs::exists(root / "late.write") + || WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; }), "post_controller_writer_outcome"); + const bool wrote_late = fs::exists(root / "late.write"); + std::cout << "{\"writer_completed_after_controller_loss\":" << (wrote_late ? "true" : "false") << "}\n"; + report("missing_job_does_not_hide_later_output_write", first != observation::empty || !wrote_late); + report("named_job_survives_without_observer_process_handles", first == observation::occupied); + report("writer_survives_unbound_controller", writer_alive); + require(wrote_late, "late_writer_output"); } } bool false_empty = false; @@ -268,9 +278,10 @@ std::size_t members(pid_t group) { return result; #else pid_t ids[2048]{}; - const int bytes = proc_listpgrppids(static_cast(group), ids, sizeof ids); - require(bytes > 0, "list_private_group_members"); - return static_cast(bytes) / sizeof(pid_t); + // proc_listpgrppids already converts proc_listpids' byte count to a PID count. + const int count = proc_listpgrppids(group, ids, sizeof ids); + require(count > 0, "list_private_group_members"); + return static_cast(count); #endif } void posix_case(const fs::path& root, bool normal) { From 8eb7cf1c4f12043ffa3a0f8097cb7439c2d629f4 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:39:54 +0900 Subject: [PATCH 06/17] test: compare named units with bound lifetime enabled --- .github/probes/process_unit.cpp | 35 +++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 999882a..5f01b5a 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -33,10 +33,11 @@ namespace fs = std::filesystem; using namespace std::chrono_literals; void require(bool value, const char* message) { if (!value) throw std::runtime_error(message); } +int failed_checks = 0; void report(const char* name, bool value) { std::cout << "{\"check\":\"" << name << "\",\"ok\":" << (value ? "true" : "false") << "}\n"; std::cout.flush(); - require(value, name); + if (!value) ++failed_checks; } template bool until(Predicate predicate, std::chrono::milliseconds bound = 5000ms) { const auto end = std::chrono::steady_clock::now() + bound; @@ -138,6 +139,8 @@ void windows_case(const fs::path& root, bool bound, bool normal) { try { require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); report("private_job_contains_helper_and_writer", text(root / "controller.ready") == "2"); + std::cout << "{\"bound_lifetime\":" << (bound ? "true" : "false") + << ",\"normal_completion\":" << (normal ? "true" : "false") << "}\n"; const auto pid = static_cast(std::stoul(text(root / "worker.ready"))); worker_id = pid; child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); @@ -178,6 +181,17 @@ void windows_case(const fs::path& root, bool bound, bool normal) { require(wrote_late, "late_writer_output"); } } + if (bound && !normal) { + const auto state = observe(name); + const bool live = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; + put(root / "worker.release", "go"); + require(until([&] { return fs::exists(root / "late.write") + || WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; }), "bound_writer_outcome"); + std::cout << "{\"bound_post_controller_state\":" << static_cast(state) + << ",\"writer_was_alive\":" << (live ? "true" : "false") + << ",\"writer_completed_after_close\":" << (fs::exists(root / "late.write") ? "true" : "false") << "}\n"; + report("bound_close_prevents_later_write", !fs::exists(root / "late.write")); + } bool false_empty = false; require(until([&] { const auto state = observe(name); @@ -318,7 +332,7 @@ void posix_case(const fs::path& root, bool normal) { kill(-owner, SIGKILL); kill(owner, SIGKILL); wait_reap(owner); reap(); throw; } } -void stderr_intervals(bool external_sleep) { +void stderr_intervals(bool external_sleep, bool absolute = false) { int channel[2]; require(pipe(channel) == 0, "producer_pipe"); const auto child = fork(); require(child >= 0, "producer_fork"); if (child == 0) { @@ -327,14 +341,17 @@ void stderr_intervals(bool external_sleep) { execl("/bin/sh", "sh", "-c", "for i in 1 2 3 4 5 6 7 8; do printf x >&2; sleep .03; done; exit 7", nullptr); _exit(97); } + auto next = std::chrono::steady_clock::now(); for (int i = 0; i < 24; ++i) { if (write(2, "x", 1) != 1) _exit(98); - std::this_thread::sleep_for(30ms); + if (absolute) { next += 30ms; std::this_thread::sleep_until(next); } + else std::this_thread::sleep_for(30ms); } _exit(7); } close(channel[1]); - auto previous = std::chrono::steady_clock::now(); + const auto start = std::chrono::steady_clock::now(); + auto previous = start; std::chrono::nanoseconds longest{0}; std::size_t bytes = 0; try { for (;;) { @@ -352,8 +369,9 @@ void stderr_intervals(bool external_sleep) { } close(channel[0]); wait_reap(child); require(bytes == (external_sleep ? 8u : 24u), "all_stderr_ticks_observed"); - std::cout << "{\"producer\":\"" << (external_sleep ? "shell_with_external_sleep" : "one_process_ticks") - << "\",\"max_observed_gap_ns\":" << longest.count() << ",\"bytes\":" << bytes << "}\n"; + std::cout << "{\"producer\":\"" << (external_sleep ? "shell_with_external_sleep" : absolute ? "one_process_absolute_ticks" : "one_process_ticks") + << "\",\"max_observed_gap_ns\":" << longest.count() << ",\"bytes\":" << bytes + << ",\"elapsed_ns\":" << std::chrono::duration_cast(previous - start).count() << "}\n"; } catch (...) { close(channel[0]); kill(-child, SIGKILL); wait_reap(child); throw; } } #ifdef __linux__ @@ -393,7 +411,7 @@ int main(int argc, char** argv) { if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], std::string(argv[4]) == "1"); #else #ifdef __linux__ - if (argc > 1 && std::string(argv[1]) == "--pid-reuse") { reuse_case(); return 0; } + if (argc > 1 && std::string(argv[1]) == "--pid-reuse") { reuse_case(); return failed_checks ? 1 : 0; } require(prctl(PR_SET_CHILD_SUBREAPER, 1) == 0, "test_harness_reaps_orphans"); #endif #endif @@ -410,10 +428,11 @@ int main(int argc, char** argv) { posix_case(root / "owner-lost", false); for (int repeat = 0; repeat < 5; ++repeat) stderr_intervals(true); stderr_intervals(false); + stderr_intervals(false, true); #endif } catch (...) { fs::remove_all(root); throw; } fs::remove_all(root); - return 0; + return failed_checks ? 1 : 0; } catch (const std::exception& error) { std::cerr << "unit probe failed: " << error.what() << '\n'; return 1; } From 0b37074f20c0ec85f0a431cff2406d6f91b55a8b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:45:28 +0900 Subject: [PATCH 07/17] test: contrast nested bound jobs after controller loss --- ...2026-11-06-observable-unit-design-draft.md | 24 +++++--- .github/probes/process_unit.cpp | 60 ++++++++++++++----- 2 files changed, 61 insertions(+), 23 deletions(-) diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 7583c54..8e8b3e4 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -52,9 +52,11 @@ The names below describe requirements rather than an approved public spelling. 4. Actual compiler processes inherit that explicitly created private unit. This is a deliberate composition of a declared unit and its child, not an unqualified switch to sharing Ninja's unit. -5. Unit observation counts remaining writers after the leader exits. It remains - usable after the controller dies. Closing an observer does not itself claim - that the unit has stopped. +5. Unit observation counts remaining writers after the leader exits. A + controller's death must not manufacture an empty result. A surviving + observer may retain the actual kernel reference; if no observer can prove + that the original unit has ended, recovery remains `unknown`. Closing an + observer does not itself claim that the unit has stopped. Creating the private unit inside the existing compiler helper avoids an unknown spawn-identity window: its reference and output record are committed before the @@ -70,10 +72,18 @@ Candidate backend implementations require experiments before approval: emptiness without sending a terminating signal or treating an absent leader as an empty group. Identifier reuse and incomplete process enumeration must yield conservative results. Native details remain in the backend. -- Windows: an observable named job is a candidate. Verify native lifetime of its - name and membership after the last controller handle closes, including nested - jobs and termination in progress. Never serialize a HANDLE for another - process to use. +- Windows: a job name alone is insufficient. The native experiment observes + `ERROR_FILE_NOT_FOUND` after the controller closes its last handle while a + writer is still alive. Without a lifetime limit, that writer subsequently + commits output. With `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`, a live writer is + still observable after the name disappears, so asynchronous termination is + not an emptiness proof either. The pinned backend deliberately leaves this + flag unset for `own_unit`; `bound_lifetime` instead creates a separate nested + job with the flag. The experiment compares these arrangements with a compiler + driver and descendant writer. A retained native handle can query the original + job; a recovered name that cannot be opened remains `unknown`. Never + serialize a HANDLE for another process to use, and include loss of every + proposed handle holder in the fault matrix. The existing job_usage operation can supply live-owner statistics. It cannot on its own provide the missing persistent observation guarantee. diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 5f01b5a..b070af0 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -70,13 +70,18 @@ std::string executable() { char path[32768]; const auto size = GetModuleFileNameA(nullptr, path, sizeof path); require(size > 0 && size < sizeof path, "self_executable"); return {path, size}; } -HANDLE start(const std::vector& args) { +HANDLE start(const std::vector& args, HANDLE bind_job = nullptr) { std::string command = quote(executable()); for (const auto& arg : args) command += " " + quote(arg); STARTUPINFOA startup{}; startup.cb = sizeof startup; PROCESS_INFORMATION child{}; - require(CreateProcessA(nullptr, command.data(), nullptr, nullptr, FALSE, 0, + require(CreateProcessA(nullptr, command.data(), nullptr, nullptr, FALSE, + bind_job ? CREATE_SUSPENDED : 0, nullptr, nullptr, &startup, &child), "start_native_process"); + if (bind_job) { + require(AssignProcessToJobObject(bind_job, child.hProcess), "bind_writer_job_before_start"); + require(ResumeThread(child.hThread) != static_cast(-1), "start_bound_writer"); + } CloseHandle(child.hThread); return child.hProcess; } FILETIME created(HANDLE process) { @@ -106,19 +111,35 @@ int worker(const fs::path& root) { put(root / "late.write", "writer finished"); return 0; } -int controller(const fs::path& root, const std::string& name, bool bound) { +int driver(const fs::path& root) { + handle child(start({"worker", root.string()})); + require(until([&] { return fs::exists(root / "worker.ready"); }), "descendant_writer_ready"); + put(root / "driver.ready", std::to_string(GetCurrentProcessId())); + require(WaitForSingleObject(child.value, 25000) == WAIT_OBJECT_0, "descendant_writer_exit"); + return 0; +} +int controller(const fs::path& root, const std::string& name, unsigned policy) { SetLastError(ERROR_SUCCESS); handle job(CreateJobObjectA(nullptr, name.c_str())); require(job.value && GetLastError() != ERROR_ALREADY_EXISTS, "new_named_job"); - if (bound) { + if (policy == 1) { JOBOBJECT_EXTENDED_LIMIT_INFORMATION limit{}; limit.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; require(SetInformationJobObject(job.value, JobObjectExtendedLimitInformation, &limit, sizeof limit), "set_bound_job"); } require(AssignProcessToJobObject(job.value, GetCurrentProcess()), "enter_private_job"); - handle child(start({"worker", root.string()})); - require(until([&] { return fs::exists(root / "worker.ready"); }), "worker_ready"); + handle nested; + if (policy == 2) { + nested.value = CreateJobObjectA(nullptr, nullptr); + require(nested.value != nullptr, "create_nested_bound_job"); + JOBOBJECT_EXTENDED_LIMIT_INFORMATION limit{}; + limit.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + require(SetInformationJobObject(nested.value, JobObjectExtendedLimitInformation, + &limit, sizeof limit), "bind_nested_writer_lifetime"); + } + handle child(start({"driver", root.string()}, nested.value)); + require(until([&] { return fs::exists(root / "worker.ready") && fs::exists(root / "driver.ready"); }), "worker_ready"); put(root / "controller.ready", std::to_string(members(job.value))); if (until([&] { return fs::exists(root / "normal.release"); }, 20000ms)) { require(WaitForSingleObject(child.value, 5000) == WAIT_OBJECT_0, "normal_worker_exit"); @@ -127,19 +148,21 @@ int controller(const fs::path& root, const std::string& name, bool bound) { } return 0; } -void windows_case(const fs::path& root, bool bound, bool normal) { +void windows_case(const fs::path& root, unsigned policy, bool normal) { + const bool bound = policy != 0; + bool false_empty = false; fs::create_directories(root); const auto suffix = std::to_string(GetCurrentProcessId()) + "-" + std::to_string(GetTickCount64()) - + "-" + std::to_string(bound) + "-" + std::to_string(normal); + + "-" + std::to_string(policy) + "-" + std::to_string(normal); const auto name = "Global\\openkal-unit-probe-" + suffix; - handle owner(start({"controller", root.string(), name, bound ? "1" : "0"})); + handle owner(start({"controller", root.string(), name, std::to_string(policy)})); handle child; DWORD worker_id = 0; FILETIME worker_birth{}; try { require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); - report("private_job_contains_helper_and_writer", text(root / "controller.ready") == "2"); - std::cout << "{\"bound_lifetime\":" << (bound ? "true" : "false") + report("private_job_contains_helper_driver_and_writer", text(root / "controller.ready") == "3"); + std::cout << "{\"lifetime_policy\":" << policy << ",\"bound_lifetime\":" << (bound ? "true" : "false") << ",\"normal_completion\":" << (normal ? "true" : "false") << "}\n"; const auto pid = static_cast(std::stoul(text(root / "worker.ready"))); worker_id = pid; @@ -167,6 +190,7 @@ void windows_case(const fs::path& root, bool bound, bool normal) { child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); require(child.value != nullptr && same_birth(created(child.value), worker_birth), "hold_orphan_process_identity"); const bool writer_alive = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; + false_empty = first == observation::empty && writer_alive; std::cout << "{\"post_controller_state\":" << static_cast(first) << ",\"query_error\":" << query_error << ",\"writer_alive\":" << (writer_alive ? "true" : "false") << "}\n"; @@ -184,6 +208,7 @@ void windows_case(const fs::path& root, bool bound, bool normal) { if (bound && !normal) { const auto state = observe(name); const bool live = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; + false_empty = state == observation::empty && live; put(root / "worker.release", "go"); require(until([&] { return fs::exists(root / "late.write") || WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; }), "bound_writer_outcome"); @@ -192,7 +217,6 @@ void windows_case(const fs::path& root, bool bound, bool normal) { << ",\"writer_completed_after_close\":" << (fs::exists(root / "late.write") ? "true" : "false") << "}\n"; report("bound_close_prevents_later_write", !fs::exists(root / "late.write")); } - bool false_empty = false; require(until([&] { const auto state = observe(name); const bool ended = WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; @@ -408,7 +432,8 @@ int main(int argc, char** argv) { try { #ifdef _WIN32 if (argc > 2 && std::string(argv[1]) == "worker") return worker(argv[2]); - if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], std::string(argv[4]) == "1"); + if (argc > 2 && std::string(argv[1]) == "driver") return driver(argv[2]); + if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], static_cast(std::stoul(argv[4]))); #else #ifdef __linux__ if (argc > 1 && std::string(argv[1]) == "--pid-reuse") { reuse_case(); return failed_checks ? 1 : 0; } @@ -420,9 +445,12 @@ int main(int argc, char** argv) { fs::create_directories(root); try { #ifdef _WIN32 - windows_case(root / "normal", false, true); - windows_case(root / "owner-lost", false, false); - windows_case(root / "bound-owner-lost", true, false); + windows_case(root / "normal", 0, true); + windows_case(root / "owner-lost", 0, false); + for (int repeat = 0; repeat < 8; ++repeat) { + windows_case(root / ("bound-owner-lost-" + std::to_string(repeat)), 1, false); + windows_case(root / ("nested-bound-owner-lost-" + std::to_string(repeat)), 2, false); + } #else posix_case(root / "normal", true); posix_case(root / "owner-lost", false); From e0a42451ca645fa65d70841748b5f7fd87c592d0 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:48:08 +0900 Subject: [PATCH 08/17] test: retain the original unit before admitting writers --- .github/probes/process_unit.cpp | 40 ++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index b070af0..e3a3ea0 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -118,7 +118,7 @@ int driver(const fs::path& root) { require(WaitForSingleObject(child.value, 25000) == WAIT_OBJECT_0, "descendant_writer_exit"); return 0; } -int controller(const fs::path& root, const std::string& name, unsigned policy) { +int controller(const fs::path& root, const std::string& name, unsigned policy, bool retain) { SetLastError(ERROR_SUCCESS); handle job(CreateJobObjectA(nullptr, name.c_str())); require(job.value && GetLastError() != ERROR_ALREADY_EXISTS, "new_named_job"); @@ -138,6 +138,10 @@ int controller(const fs::path& root, const std::string& name, unsigned policy) { require(SetInformationJobObject(nested.value, JobObjectExtendedLimitInformation, &limit, sizeof limit), "bind_nested_writer_lifetime"); } + if (retain) { + put(root / "unit.ready", name); + require(until([&] { return fs::exists(root / "observer.ready"); }), "observer_retains_before_compiler_start"); + } handle child(start({"driver", root.string()}, nested.value)); require(until([&] { return fs::exists(root / "worker.ready") && fs::exists(root / "driver.ready"); }), "worker_ready"); put(root / "controller.ready", std::to_string(members(job.value))); @@ -148,14 +152,26 @@ int controller(const fs::path& root, const std::string& name, unsigned policy) { } return 0; } -void windows_case(const fs::path& root, unsigned policy, bool normal) { +void windows_case(const fs::path& root, unsigned policy, bool normal, bool retain = false) { const bool bound = policy != 0; bool false_empty = false; fs::create_directories(root); const auto suffix = std::to_string(GetCurrentProcessId()) + "-" + std::to_string(GetTickCount64()) + "-" + std::to_string(policy) + "-" + std::to_string(normal); const auto name = "Global\\openkal-unit-probe-" + suffix; - handle owner(start({"controller", root.string(), name, std::to_string(policy)})); + handle owner(start({"controller", root.string(), name, std::to_string(policy), retain ? "1" : "0"})); + handle retained; + if (retain) { + require(until([&] { return fs::exists(root / "unit.ready"); }), "unit_published_before_compiler"); + retained.value = OpenJobObjectA(JOB_OBJECT_QUERY | JOB_OBJECT_TERMINATE, FALSE, name.c_str()); + require(retained.value != nullptr, "retain_actual_unit_before_ack"); + report("compiler_waits_for_observer_ack", !fs::exists(root / "worker.ready")); + put(root / "observer.ready", "retained"); + } + const auto observe_unit = [&] { + return retained.value ? (members(retained.value) == 0 ? observation::empty : observation::occupied) + : observe(name); + }; handle child; DWORD worker_id = 0; FILETIME worker_birth{}; @@ -163,6 +179,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal) { require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); report("private_job_contains_helper_driver_and_writer", text(root / "controller.ready") == "3"); std::cout << "{\"lifetime_policy\":" << policy << ",\"bound_lifetime\":" << (bound ? "true" : "false") + << ",\"retained_observer\":" << (retain ? "true" : "false") << ",\"normal_completion\":" << (normal ? "true" : "false") << "}\n"; const auto pid = static_cast(std::stoul(text(root / "worker.ready"))); worker_id = pid; @@ -170,12 +187,12 @@ void windows_case(const fs::path& root, unsigned policy, bool normal) { require(child.value != nullptr, "hold_worker_process_identity"); worker_birth = created(child.value); if (!normal && !bound) child.reset(); - report("live_unit_is_not_empty", observe(name) == observation::occupied); + report("live_unit_is_not_empty", observe_unit() == observation::occupied); if (normal) { put(root / "worker.release", "go"); put(root / "normal.release", "go"); require(until([&] { return fs::exists(root / "owner.only"); }), "owner_only_ready"); report("normal_completion_counts_helper_separately", text(root / "owner.only") == "1"); - report("helper_only_is_occupied_not_empty", observe(name) == observation::occupied); + report("helper_only_is_occupied_not_empty", observe_unit() == observation::occupied); put(root / "owner.release", "go"); require(WaitForSingleObject(owner.value, 5000) == WAIT_OBJECT_0, "normal_controller_exit"); } else { @@ -185,7 +202,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal) { if (!bound) { // Query before opening even a process handle in this observer: // no retained controller/worker handle may mask name lifetime. - const auto first = observe(name); + const auto first = observe_unit(); const auto query_error = GetLastError(); child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); require(child.value != nullptr && same_birth(created(child.value), worker_birth), "hold_orphan_process_identity"); @@ -206,7 +223,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal) { } } if (bound && !normal) { - const auto state = observe(name); + const auto state = observe_unit(); const bool live = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; false_empty = state == observation::empty && live; put(root / "worker.release", "go"); @@ -218,13 +235,13 @@ void windows_case(const fs::path& root, unsigned policy, bool normal) { report("bound_close_prevents_later_write", !fs::exists(root / "late.write")); } require(until([&] { - const auto state = observe(name); + const auto state = observe_unit(); const bool ended = WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; if (state == observation::empty && !ended) false_empty = true; return ended && state == observation::empty; }), "whole_job_empty_after_writer_exit"); report("missing_named_job_never_precedes_live_writer", !false_empty); - report("complete_job_is_empty", observe(name) == observation::empty); + report("complete_job_is_empty", observe_unit() == observation::empty); } catch (...) { if (owner.value) TerminateProcess(owner.value, 137); if (!child.value && worker_id) @@ -433,7 +450,7 @@ int main(int argc, char** argv) { #ifdef _WIN32 if (argc > 2 && std::string(argv[1]) == "worker") return worker(argv[2]); if (argc > 2 && std::string(argv[1]) == "driver") return driver(argv[2]); - if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], static_cast(std::stoul(argv[4]))); + if (argc > 4 && std::string(argv[1]) == "controller") return controller(argv[2], argv[3], static_cast(std::stoul(argv[4])), argc > 5 && std::string(argv[5]) == "1"); #else #ifdef __linux__ if (argc > 1 && std::string(argv[1]) == "--pid-reuse") { reuse_case(); return failed_checks ? 1 : 0; } @@ -447,6 +464,9 @@ int main(int argc, char** argv) { #ifdef _WIN32 windows_case(root / "normal", 0, true); windows_case(root / "owner-lost", 0, false); + windows_case(root / "retained-normal", 0, true, true); + windows_case(root / "retained-owner-lost", 0, false, true); + windows_case(root / "retained-nested-bound-owner-lost", 2, false, true); for (int repeat = 0; repeat < 8; ++repeat) { windows_case(root / ("bound-owner-lost-" + std::to_string(repeat)), 1, false); windows_case(root / ("nested-bound-owner-lost-" + std::to_string(repeat)), 2, false); From c2c9e60c569ab8e6e9deda103a479130285d1259 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:51:20 +0900 Subject: [PATCH 09/17] test: measure unit retirement and observer handoff --- ...2026-11-06-observable-unit-design-draft.md | 23 ++++++++++++++++ .github/probes/process_unit.cpp | 27 ++++++++++++++++++- 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 8e8b3e4..028280a 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -88,6 +88,29 @@ Candidate backend implementations require experiments before approval: The existing job_usage operation can supply live-owner statistics. It cannot on its own provide the missing persistent observation guarantee. +A possible composition uses the existing top-level owner and compiler helper, +without another keeper. Before compiler start, the helper publishes its unit; +the top-level owner opens and retains that original unit and acknowledges the +publication. The helper may then start the compiler. Loss of either holder is +observable through the other. Loss of both holders is a distinct boundary: +Windows name absence remains `unknown`, and admission fails within its bound +while retaining the reservation. This case is not automatic recovery. + +The helper also needs a normal-completion barrier. On POSIX it may stop +admitting children, move into a known surviving caller unit and ask the kernel +whether its former group is empty. Its own still-live identity prevents that +former identifier from being reused during this check. On Windows it cannot +leave a job; the original retained job must establish that no external members +remain. Process accounting, a process handle becoming signaled and the absence +of further writes are different observations; the native experiment records +these separately before a production completion contract is selected. + +The prototype measures its startup acknowledgement and repeated original-job +queries. Its file acknowledgement uses a five-millisecond polling interval; +this is an experiment parameter, not an accepted production mechanism or cost. +A final design must account for both steady-state build overhead and tail +latency, including any added thread, polling and handshake. + ## Core transaction and permit protocol The generation and actor barrier remain runtime state outside build identity. diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index e3a3ea0..0c8e16a 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -156,6 +156,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai const bool bound = policy != 0; bool false_empty = false; fs::create_directories(root); + const auto attempt_start = std::chrono::steady_clock::now(); const auto suffix = std::to_string(GetCurrentProcessId()) + "-" + std::to_string(GetTickCount64()) + "-" + std::to_string(policy) + "-" + std::to_string(normal); const auto name = "Global\\openkal-unit-probe-" + suffix; @@ -166,6 +167,12 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai retained.value = OpenJobObjectA(JOB_OBJECT_QUERY | JOB_OBJECT_TERMINATE, FALSE, name.c_str()); require(retained.value != nullptr, "retain_actual_unit_before_ack"); report("compiler_waits_for_observer_ack", !fs::exists(root / "worker.ready")); + const auto query_start = std::chrono::steady_clock::now(); + for (int repeat = 0; repeat < 1000; ++repeat) require(members(retained.value) == 1, "prepared_unit_has_only_helper"); + std::cout << "{\"unit_query_samples\":1000,\"unit_query_elapsed_ns\":" + << std::chrono::duration_cast(std::chrono::steady_clock::now() - query_start).count() + << ",\"pre_spawn_ack_ns\":" + << std::chrono::duration_cast(query_start - attempt_start).count() << "}\n"; put(root / "observer.ready", "retained"); } const auto observe_unit = [&] { @@ -178,6 +185,8 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai try { require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); report("private_job_contains_helper_driver_and_writer", text(root / "controller.ready") == "3"); + if (retain) std::cout << "{\"startup_through_ack_ns\":" + << std::chrono::duration_cast(std::chrono::steady_clock::now() - attempt_start).count() << "}\n"; std::cout << "{\"lifetime_policy\":" << policy << ",\"bound_lifetime\":" << (bound ? "true" : "false") << ",\"retained_observer\":" << (retain ? "true" : "false") << ",\"normal_completion\":" << (normal ? "true" : "false") << "}\n"; @@ -226,10 +235,17 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai const auto state = observe_unit(); const bool live = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; false_empty = state == observation::empty && live; + BOOL associated = FALSE; DWORD exit_code = 0; + const bool membership_known = !retained.value || IsProcessInJob(child.value, retained.value, &associated); + require(GetExitCodeProcess(child.value, &exit_code), "writer_exit_code"); put(root / "worker.release", "go"); require(until([&] { return fs::exists(root / "late.write") || WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; }), "bound_writer_outcome"); std::cout << "{\"bound_post_controller_state\":" << static_cast(state) + << ",\"retained_job_members\":" << (retained.value ? members(retained.value) : 0) + << ",\"membership_query_known\":" << (membership_known ? "true" : "false") + << ",\"writer_still_associated\":" << (associated ? "true" : "false") + << ",\"writer_exit_code\":" << exit_code << ",\"writer_was_alive\":" << (live ? "true" : "false") << ",\"writer_completed_after_close\":" << (fs::exists(root / "late.write") ? "true" : "false") << "}\n"; report("bound_close_prevents_later_write", !fs::exists(root / "late.write")); @@ -290,6 +306,7 @@ void wait_reap(pid_t pid) { while (waitpid(pid, nullptr, 0) < 0 && errno == EINTR) {} } pid_t launch(const fs::path& root) { + const auto previous_group = getpgrp(); const auto owner = fork(); require(owner >= 0, "fork_controller"); if (owner != 0) return owner; if (setpgid(0, 0)) _exit(90); @@ -309,11 +326,15 @@ pid_t launch(const fs::path& root) { wait_reap(writer); _exit(0); } put(root / "controller.ready", std::to_string(getpid())); - bool reaped = false; + bool reaped = false, retired = false; if (!until([&] { if (!reaped && waitpid(driver, nullptr, WNOHANG) == driver) { reaped = true; put(root / "owner.only", "ready"); } + if (reaped && !retired && fs::exists(root / "retire.request")) { + if (setpgid(0, previous_group)) _exit(99); + retired = true; put(root / "owner.retired", "ready"); + } return fs::exists(root / "owner.release"); }, 20000ms)) _exit(95); if (!reaped) wait_reap(driver); @@ -358,6 +379,10 @@ void posix_case(const fs::path& root, bool normal) { require(until([&] { return fs::exists(root / "owner.only"); }), "normal_owner_only_ready"); report("normal_completion_counts_helper_separately", members(owner) == 1); report("live_helper_does_not_mean_empty", observe(ref) == observation::occupied); + put(root / "retire.request", "go"); + require(until([&] { return fs::exists(root / "owner.retired"); }), "helper_retired_from_original_unit"); + report("retired_helper_still_has_original_identity", born(owner) == ref.owner_born); + report("retired_helper_cannot_hide_original_unit_writer", observe(ref) == observation::empty); put(root / "owner.release", "go"); } else { require(kill(owner, SIGKILL) == 0, "kill_controller"); wait_reap(owner); From 6e1b50abfd5320998bf053a323b7e18830464a82 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:53:24 +0900 Subject: [PATCH 10/17] test: separate job accounting from process completion --- .../docs/2026-11-06-observable-unit-design-draft.md | 9 ++++++--- .github/probes/process_unit.cpp | 12 +++++++----- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 028280a..65a905d 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -75,9 +75,12 @@ Candidate backend implementations require experiments before approval: - Windows: a job name alone is insufficient. The native experiment observes `ERROR_FILE_NOT_FOUND` after the controller closes its last handle while a writer is still alive. Without a lifetime limit, that writer subsequently - commits output. With `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`, a live writer is - still observable after the name disappears, so asynchronous termination is - not an emptiness proof either. The pinned backend deliberately leaves this + commits output. With `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`, a retained writer + process handle can remain nonsignaled after the name disappears. These + samples do not establish that the terminating writer can commit more output; + a process's final signal and job association accounting are recorded as + separate observations. Name absence still cannot identify or query the + original job. The pinned backend deliberately leaves this flag unset for `own_unit`; `bound_lifetime` instead creates a separate nested job with the flag. The experiment compares these arrangements with a compiler driver and descendant writer. A retained native handle can query the original diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 0c8e16a..9ddc473 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -185,8 +185,10 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai try { require(until([&] { return fs::exists(root / "controller.ready"); }), "controller_ready"); report("private_job_contains_helper_driver_and_writer", text(root / "controller.ready") == "3"); - if (retain) std::cout << "{\"startup_through_ack_ns\":" - << std::chrono::duration_cast(std::chrono::steady_clock::now() - attempt_start).count() << "}\n"; + std::cout << "{\"startup_to_writer_ready_ns\":" + << std::chrono::duration_cast(std::chrono::steady_clock::now() - attempt_start).count() + << ",\"retained_observer\":" << (retain ? "true" : "false") + << ",\"lifetime_policy\":" << policy << "}\n"; std::cout << "{\"lifetime_policy\":" << policy << ",\"bound_lifetime\":" << (bound ? "true" : "false") << ",\"retained_observer\":" << (retain ? "true" : "false") << ",\"normal_completion\":" << (normal ? "true" : "false") << "}\n"; @@ -246,7 +248,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai << ",\"membership_query_known\":" << (membership_known ? "true" : "false") << ",\"writer_still_associated\":" << (associated ? "true" : "false") << ",\"writer_exit_code\":" << exit_code - << ",\"writer_was_alive\":" << (live ? "true" : "false") + << ",\"writer_handle_not_signaled\":" << (live ? "true" : "false") << ",\"writer_completed_after_close\":" << (fs::exists(root / "late.write") ? "true" : "false") << "}\n"; report("bound_close_prevents_later_write", !fs::exists(root / "late.write")); } @@ -256,7 +258,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai if (state == observation::empty && !ended) false_empty = true; return ended && state == observation::empty; }), "whole_job_empty_after_writer_exit"); - report("missing_named_job_never_precedes_live_writer", !false_empty); + report("unit_empty_does_not_precede_process_signal", !false_empty); report("complete_job_is_empty", observe_unit() == observation::empty); } catch (...) { if (owner.value) TerminateProcess(owner.value, 137); @@ -491,10 +493,10 @@ int main(int argc, char** argv) { windows_case(root / "owner-lost", 0, false); windows_case(root / "retained-normal", 0, true, true); windows_case(root / "retained-owner-lost", 0, false, true); - windows_case(root / "retained-nested-bound-owner-lost", 2, false, true); for (int repeat = 0; repeat < 8; ++repeat) { windows_case(root / ("bound-owner-lost-" + std::to_string(repeat)), 1, false); windows_case(root / ("nested-bound-owner-lost-" + std::to_string(repeat)), 2, false); + windows_case(root / ("retained-nested-bound-owner-lost-" + std::to_string(repeat)), 2, false, true); } #else posix_case(root / "normal", true); From d75860e456a586ff578850d622e48b01ccda7b14 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 04:55:52 +0900 Subject: [PATCH 11/17] docs: record unit lifetime boundaries and handoff cost --- ...2026-11-06-observable-unit-design-draft.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 65a905d..11971cd 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -109,10 +109,30 @@ of further writes are different observations; the native experiment records these separately before a production completion contract is selected. The prototype measures its startup acknowledgement and repeated original-job -queries. Its file acknowledgement uses a five-millisecond polling interval; -this is an experiment parameter, not an accepted production mechanism or cost. -A final design must account for both steady-state build overhead and tail -latency, including any added thread, polling and handshake. +queries. In eight Windows pairs using the same nested-job arrangement, median +startup through descendant readiness was 60.393 ms without acknowledgement and +76.437 ms with it. Original-job queries took a median 1.00 microsecond each, +measured in batches of 1000. These are small controlled samples rather than a +build benchmark. The file acknowledgement polls at five milliseconds and adds +approximately 16 ms to median startup in this experiment; it is not an accepted +production transport. The native prototype adds no background thread. A final +design must measure its actual notification mechanism, any added thread, normal +and cached builds, and tail latency before release. + +Linux and macOS controls establish that a retired helper retains its original +identity while the kernel reports its former group empty. Linux also exercises +actual group-identifier reuse in a separate PID namespace. Windows retains the +negative name-loss controls; the hypothesis that unit emptiness always follows +the final process-handle signal is disproved and is not used as a completion +contract. An original job's active membership, final process signaling and +subsequent output commits remain distinct observations. + +The relevant native contracts are [job accounting](https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-jobobject_basic_accounting_information), +[process termination](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess) +and [job termination](https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-terminatejobobject). +Termination requests do not themselves certify completion. A retained original +job and a closed admission boundary are required for a job-wide completion +query; a missing name cannot supply either. ## Core transaction and permit protocol From f4da5e8fe8be09251638f920fae4effec29418e6 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 05:19:29 +0900 Subject: [PATCH 12/17] feat: declare observable units without a process registry --- .github/checks/build_job_observer.py | 44 +++++++ .github/checks/job_observer.cpp | 175 +++++++++++++++++++++++++ .github/workflows/process-unit-api.yml | 36 +++++ SURFACE.txt | 4 + include/openkal/process.h | 49 +++++++ src/process.cppm | 10 ++ 6 files changed, 318 insertions(+) create mode 100644 .github/checks/build_job_observer.py create mode 100644 .github/checks/job_observer.cpp create mode 100644 .github/workflows/process-unit-api.yml diff --git a/.github/checks/build_job_observer.py b/.github/checks/build_job_observer.py new file mode 100644 index 0000000..40ac419 --- /dev/null +++ b/.github/checks/build_job_observer.py @@ -0,0 +1,44 @@ +"""Build the real backend and C ABI consumer with the runner's native compiler.""" +import concurrent.futures +import os +from pathlib import Path +import subprocess +import sys + +root = Path.cwd() +backend = root / 'backend' +include = root / 'include' +out = root / 'unit-api-build' +out.mkdir(exist_ok=True) +windows = sys.platform == 'win32' + +def compile_backend(source): + obj = out / (source.stem + ('.obj' if windows else '.o')) + if windows: + command = ['cl', '/nologo', '/std:c++20', '/c', '/O2', '/GR-', '/EHs-c-', '/GS-', + '/I' + str(include), '/Fo' + str(obj), str(source)] + else: + command = ['c++', '-std=c++23', '-O2', '-ffreestanding', '-fno-exceptions', '-fno-rtti', + '-fno-stack-protector', '-I' + str(include), '-c', str(source), '-o', str(obj)] + result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + (out / (source.stem + '.log')).write_text(result.stdout) + if result.returncode: + raise RuntimeError(str(source.name) + '\n' + result.stdout) + return str(obj) + +with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool: + objects = list(pool.map(compile_backend, sorted((backend / 'src').glob('*.cpp')))) +consumer = root / '.github/checks/job_observer.cpp' +binary = out / ('job-observer.exe' if windows else 'job-observer') +if windows: + command = ['cl', '/nologo', '/std:c++17', '/EHsc', '/O2', '/I' + str(include), str(consumer), + *objects, '/Fe:' + str(binary), '/link', 'ntdll.lib', 'synchronization.lib', + 'shell32.lib', 'kernel32.lib', 'bcrypt.lib'] +else: + command = ['c++', '-std=c++17', '-O2', '-pthread', '-I' + str(include), str(consumer), + *objects, '-o', str(binary)] +subprocess.run(command, check=True) +with (out / 'job-observer.jsonl').open('w') as log: + result = subprocess.run([str(binary)], stdout=log, stderr=subprocess.STDOUT, timeout=30) +print((out / 'job-observer.jsonl').read_text()) +raise SystemExit(result.returncode) diff --git a/.github/checks/job_observer.cpp b/.github/checks/job_observer.cpp new file mode 100644 index 0000000..3ca95d7 --- /dev/null +++ b/.github/checks/job_observer.cpp @@ -0,0 +1,175 @@ +// Exercises the implemented C ABI, including a driver with a descendant. +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef __linux__ +#include +#include +#endif +using namespace std::chrono_literals; +namespace fs = std::filesystem; +void require(bool value,const char* name) { + std::printf("{\"check\":\"%s\",\"ok\":%s}\n",name,value?"true":"false"); + std::fflush(stdout); if(!value) throw std::runtime_error(name); +} +std::string image, relative_image; +kal_spawn place() { + kal_spawn how{}; + for(kal_uintptr i=0;iimage.size())continue; + bool same=true; + for(kal_uintptr j=0;j='A'&&a<='Z')a+=32; if(b>='A'&&b<='Z')b+=32; +#endif + if(a!=b)same=false; + } + if(same && (!how.base.h || n==1 || image[n-1]=='/' || image[n-1]=='\\')) { + how.base=how.work=d; relative_image=image.substr(n); + while(!relative_image.empty()&&(relative_image.front()=='/'||relative_image.front()=='\\'))relative_image.erase(0,1); + return how; + } + } + throw std::runtime_error("program_preopen"); +} +kal_process start(std::vector arguments,kal_job* job=nullptr, + const kal_spawn_streams* streams=nullptr) { + arguments.insert(arguments.begin(),image); + std::vector argv;std::vector lengths; + for(auto& s:arguments){argv.push_back(s.data());lengths.push_back(s.size());} + auto how=place();how.job=job; + kal_process child{}; + // An absolute name is resolved by the backend against the directory held. + const int rc=kal_process_spawn(&how,relative_image.data(),relative_image.size(),argv.data(),lengths.data(),argv.size(), + nullptr,nullptr,0,streams,&child); + if(rc)throw std::runtime_error("actual_spawn_"+std::to_string(rc));return child; +} +std::string read_line(kal_stream stream) { + std::string out;char c=0; + while(out.size()<512) { + const auto n=kal_stream_read(stream,&c,1); + if(n!=1)throw std::runtime_error("control_stream_eof"); + if(c=='\n')return out;out+=c; + } + throw std::runtime_error("control_stream_bound"); +} +void send_line(kal_stream stream,const std::string& value) { + const auto bytes=value+"\n"; + if(kal_stream_write(stream,bytes.data(),bytes.size())!=static_cast(bytes.size())) + throw std::runtime_error("control_write"); +} +int wait(kal_process child,bool success=true) { + int status=0,environment=0; + if(kal_process_wait(child,&status,&environment))throw std::runtime_error("actual_wait"); + kal_process_close(child); + if(success && (status||environment))throw std::runtime_error("actual_child_exit_"+std::to_string(status)); + return environment ? 128+status : status; +} +kal_u32 state(kal_job job,const std::string& ref) { + kal_u32 value=KAL_JOB_UNKNOWN; + if(kal_process_job_observe(job,ref.data(),ref.size(),&value))return KAL_JOB_UNKNOWN; + return value; +} +void await_empty(kal_job job,const std::string& ref) { + const auto limit=std::chrono::steady_clock::now()+5s; + for(;;) { +#ifdef __linux__ + while(waitpid(-1,nullptr,WNOHANG)>0){} +#endif + if(state(job,ref)==KAL_JOB_EMPTY)return; + if(std::chrono::steady_clock::now()>=limit)throw std::runtime_error("whole_unit_not_quiet"); + std::this_thread::sleep_for(1ms); + } +} +int helper(const std::string& mode,const fs::path& output) { + kal_job job{};char bytes[512]{};kal_uintptr used=0; + if(kal_process_job_create(&job,bytes,sizeof bytes,&used))return 41; + const std::string ref(bytes,used); + send_line(kal_stdout(),ref); + if(read_line(kal_stdin())!="ACK")return 42; + kal_stream read{},write{}; + if(kal_process_channel(&read,&write))return 43; + kal_spawn_streams streams{read,{}, {}}; + auto driver=start({"driver",output.string()},&job,&streams); + kal_process_channel_close(read); + kal_u32 retired=KAL_JOB_UNKNOWN; + if(kal_process_job_retire(job,ref.data(),ref.size(),&retired)||retired!=KAL_JOB_OCCUPIED)return 44; + if(mode=="kill") { + send_line(kal_stdout(),"BUSY"); + // Only the parent owns the acknowledgement's write end. Hard death + // also closes this helper's writer gate, allowing the orphan to finish. + (void)read_line(kal_stdin());return 45; + } + if(kal_stream_write(write,"R",1)!=1)return 46; + kal_process_channel_close(write);wait(driver); + if(kal_process_job_retire(job,ref.data(),ref.size(),&retired)||retired!=KAL_JOB_EMPTY)return 47; + if(kal_process_job_retire(job,ref.data(),ref.size(),&retired)||retired!=KAL_JOB_EMPTY)return 48; + send_line(kal_stdout(),"DONE");kal_process_job_close(job);return 0; +} +void exercise(const fs::path& root,const std::string& mode) { + kal_stream input{},gate{},results{},output{}; + if(kal_process_channel(&input,&gate)||kal_process_channel(&results,&output))throw std::runtime_error("actual_channels"); + kal_spawn_streams streams{input,output,{}}; + auto child=start({"helper",mode,(root/(mode+".object")).string()},nullptr,&streams); + kal_process_channel_close(input);kal_process_channel_close(output); + const auto ref=read_line(results);kal_job observed{}; + require(kal_process_job_open(ref.data(),ref.size(),&observed)==kal_ok,"original_unit_retained_before_admission"); + require(state(observed,ref)==KAL_JOB_OCCUPIED,"unstarted_helper_is_occupied"); + auto bad=ref;bad[0]='?';require(state(observed,bad)!=KAL_JOB_EMPTY,"malformed_reference_never_grants_quiet"); + if(mode=="before") { + require(kal_process_terminate(child)==kal_ok,"controller_ended_before_ack"); + kal_process_channel_close(gate);(void)wait(child,false); + require(!fs::exists(root/(mode+".object")),"no_writer_before_ack"); + } else { + send_line(gate,"ACK"); + if(mode=="kill") { + require(read_line(results)=="BUSY","retirement_detects_external_driver"); + require(kal_process_terminate(child)==kal_ok,"controller_ended_with_descendant"); + kal_process_channel_close(gate);(void)wait(child,false); + } else { + require(read_line(results)=="DONE","external_members_quiet_before_completion"); + kal_process_channel_close(gate);wait(child); + } + } + await_empty(observed,ref); + require(mode=="before"||fs::exists(root/(mode+".object")),"all_admitted_writer_outputs_observed"); + require(state(observed,ref)==KAL_JOB_EMPTY,"held_original_unit_positively_empty"); + kal_process_job_close(observed);kal_process_channel_close(results); + kal_job recovered{};const int opened=kal_process_job_open(ref.data(),ref.size(),&recovered); +#ifdef _WIN32 + require(opened!=kal_ok,"lost_windows_native_reference_is_not_recreated"); +#else + require(opened==kal_ok&&state(recovered,ref)==KAL_JOB_EMPTY,"expired_native_group_is_positively_empty"); +#endif + if(opened==kal_ok)kal_process_job_close(recovered); +} +int main(int argc,char** argv) { + try { + image=fs::absolute(argv[0]).string(); + if(argc>2&&std::string(argv[1])=="writer") { + char gate=0;const auto n=kal_stream_read(kal_stdin(),&gate,1); + if(n<0)return 50; + std::ofstream out(argv[2]);out<<"actual declared unit descendant\n";return out?0:51; + } + if(argc>2&&std::string(argv[1])=="driver") {return wait(start({"writer",argv[2]}));} + if(argc>3&&std::string(argv[1])=="helper")return helper(argv[2],argv[3]); +#ifdef __linux__ + require(prctl(PR_SET_CHILD_SUBREAPER,1)==0,"native_harness_reaps_fault_orphans"); +#endif + const auto root=fs::temp_directory_path()/("job-api-"+std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())); + fs::create_directories(root); + try {exercise(root,"normal");exercise(root,"before");exercise(root,"kill");} + catch(...){fs::remove_all(root);throw;} + fs::remove_all(root);return 0; + }catch(const std::exception& e){std::fprintf(stderr,"job observer API: %s\n",e.what());return 1;} +} diff --git a/.github/workflows/process-unit-api.yml b/.github/workflows/process-unit-api.yml new file mode 100644 index 0000000..6e6e960 --- /dev/null +++ b/.github/workflows/process-unit-api.yml @@ -0,0 +1,36 @@ +name: declared process unit API +on: + pull_request: + workflow_dispatch: +concurrency: + group: unit-api-${{ github.ref }} + cancel-in-progress: true +jobs: + native: + name: implemented unit API (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + include: + - { os: ubuntu-24.04, implementation: openkal-linux } + - { os: macos-15, implementation: openkal-macos } + - { os: windows-2025, implementation: openkal-windows } + steps: + - uses: actions/checkout@v4 + - uses: actions/checkout@v4 + with: + repository: mcpplibs/${{ matrix.implementation }} + ref: feat/observable-process-units + path: backend + - uses: ilammy/msvc-dev-cmd@v1 + if: runner.os == 'Windows' + - name: Run the implemented ABI against its actual native backend + shell: bash + run: python .github/checks/build_job_observer.py + - uses: actions/upload-artifact@v4 + if: always() + with: + name: declared-unit-${{ matrix.os }} + path: unit-api-build/*.jsonl diff --git a/SURFACE.txt b/SURFACE.txt index 87f9cf7..ef7e607 100644 --- a/SURFACE.txt +++ b/SURFACE.txt @@ -94,8 +94,12 @@ kal_process_close kal_process_end_as_requested kal_process_info kal_process_job_close +kal_process_job_create kal_process_job_enter kal_process_job_limit +kal_process_job_observe +kal_process_job_open +kal_process_job_retire kal_process_job_terminate kal_process_job_usage kal_process_open_id diff --git a/include/openkal/process.h b/include/openkal/process.h index 2a5feec..cb9f683 100644 --- a/include/openkal/process.h +++ b/include/openkal/process.h @@ -100,6 +100,8 @@ struct kal_preopen { /* KAL_SPAWN_CONFINE is honoured: the environment holds a started program to * its grants. Version 0.23. */ #define KAL_PROCESS_PROP_CONFINE ((kal_uintptr)1u << 14) +/* Declared units, portable references and retirement. Version 0.24. */ +#define KAL_PROCESS_PROP_JOB_OBSERVE ((kal_uintptr)1u << 15) /* HOW A PROGRAM IS STARTED, AND IT IS ONE OPERATION BECAUSE 0.11 STOPPED * MAKING IT A FAMILY. @@ -612,6 +614,53 @@ int kal_process_job_terminate(struct kal_job); * Terminating is `kal_process_job_terminate' and nothing else is. */ void kal_process_job_close(struct kal_job); +/* AN OBSERVABLE UNIT (0.24). The unit is declared before any program is + * started in it. `create' puts the caller in a fresh private unit and writes + * an opaque reference into `reference'; `used' is its byte count. A buffer + * smaller than the implementation requires is kal_err_no_space and leaves + * the caller outside a new unit. No terminator is included in the count. + * + * The reference is a value, not a handle: it can be retained and carried to + * another invocation. Its bytes shall not be interpreted or manufactured by + * a caller. `open' obtains an observation reference to the original unit; + * closing it ends nothing. An environment that cannot retain a native unit + * identity after its last observer is lost shall not report name absence as + * emptiness. It reports UNKNOWN. Neither a dead leader nor a stale reference + * proves that the unit's other programs have ended. + * + * An observer does not convey a right to terminate a replacement unit that + * reused a native identifier. `observe' validates the supplied reference and + * reports exactly one of the three states. Unsupported observation, identity + * ambiguity, an inaccessible unit or an expired native reference is UNKNOWN, + * never EMPTY. A positively empty unit has no remaining active member. + * + * The job returned by `create' is the job supplied to the EXISTING spawn: + * one start, with the declared unit stated by its caller. Before `retire', + * the caller shall cease all admission into that unit, including its own + * concurrent spawns. `retire' establishes that no member except the caller + * remains; where membership can be left it may leave the unit first. The + * caller is still alive and may complete its own bookkeeping afterwards. + * OCCUPIED and UNKNOWN can be retried. Retirement is idempotent, and after + * asking to retire the caller shall not admit another program to the unit. + * A library above this ABI enforces that rule through its owned unit object. + * + * These operations neither install a monitor nor reconstruct a registry. + * The reference is sufficient to validate the environment's own native unit; + * a retained job holds whatever native reference the environment supplies. + * An implementation not claiming JOB_OBSERVE reports kal_err_not_supported. + */ +#define KAL_JOB_UNKNOWN 0u +#define KAL_JOB_OCCUPIED 1u +#define KAL_JOB_EMPTY 2u +int kal_process_job_create(struct kal_job* out, char* reference, + kal_uintptr capacity, kal_uintptr* used); +int kal_process_job_open(const char* reference, kal_uintptr length, + struct kal_job* out); +int kal_process_job_observe(struct kal_job, const char* reference, + kal_uintptr length, kal_u32* state); +int kal_process_job_retire(struct kal_job, const char* reference, + kal_uintptr length, kal_u32* state); + /* WHAT A UNIT USES. Version 0.20, clause 11 entry 43. * * The shape is `kal_program_info's: the caller sets `self_size', and the diff --git a/src/process.cppm b/src/process.cppm index 08ab829..fe8daaa 100644 --- a/src/process.cppm +++ b/src/process.cppm @@ -36,6 +36,10 @@ export using ::kal_process_job_enter; export using ::kal_process_job_terminate; export using ::kal_process_job_close; export using ::kal_process_job_usage; +export using ::kal_process_job_create; +export using ::kal_process_job_open; +export using ::kal_process_job_observe; +export using ::kal_process_job_retire; export using ::kal_process_job_limit; export using ::kal_job_usage; export using ::kal_process_stop_requested; @@ -86,6 +90,12 @@ inline constexpr props job_limit_hard{KAL_PROCESS_PROP_JOB_LIMIT_HARD}; inline constexpr props job_limit_supervised{KAL_PROCESS_PROP_JOB_LIMIT_SUPERVISED}; // 0.20 inline constexpr props null_streams {KAL_PROCESS_PROP_NULL_STREAMS}; // 0.22 inline constexpr props confine {KAL_PROCESS_PROP_CONFINE}; // 0.23 +inline constexpr props job_observe {KAL_PROCESS_PROP_JOB_OBSERVE}; // 0.24 +namespace job_state { +inline constexpr kal_u32 unknown = KAL_JOB_UNKNOWN; +inline constexpr kal_u32 occupied = KAL_JOB_OCCUPIED; +inline constexpr kal_u32 empty = KAL_JOB_EMPTY; +} // The fields of kal_program_info, for `wanted' and `present' (0.17). namespace info { From cdb3d6f79bb5c3322c1d82d24c4e6443ffb13046 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 06:11:08 +0900 Subject: [PATCH 13/17] fix: anchor unit retirement to a live program identity --- ...2026-11-06-observable-unit-design-draft.md | 31 +++++++++++ .github/checks/build_job_observer.py | 7 +++ .github/checks/job_observer.cpp | 48 +++++++++++++---- .github/checks/retire_boundaries.py | 52 +++++++++++++++++++ .github/checks/retire_boundary.cpp | 23 ++++++++ .github/workflows/process-unit-api.yml | 12 +++-- include/openkal/process.h | 32 +++++++++--- src/process.cppm | 3 ++ 8 files changed, 188 insertions(+), 20 deletions(-) create mode 100644 .github/checks/retire_boundaries.py create mode 100644 .github/checks/retire_boundary.cpp diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 11971cd..1200b08 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -217,3 +217,34 @@ and platform rows used by core's openkal-target CI must be included in that version group. A source-only host success does not qualify those image rows. Plugin sidecar work remains prepared independently and receives the final public core pin only after this release is actually available. + +## 2026-11-06 continuation and observer contract correction + +The continuation is an explicit program identity containing both the id and +birth value reported by the process API. A saved former process group is not +a sufficient continuation: the BMI publisher may exit before its compiler +helper retires. The helper instead uses the still-live build owner as its +anchor. Retirement validates that same identity before and after migration, +and that the destination remains within the native scope it can join. A dead, +reused or inaccessible anchor, or an anchor in another session, yields unknown. +There is no fallback to a previous numeric group. On Windows the original +anchor process handle is retained through the original-job membership query. + +Retirement seals admission into the declared unit. Joining the anchor's unit +conveys neither admission nor termination authority over that unit. A native +unit already led by the declaring helper may be adopted; observation includes +its existing members. The original unit alone remains the termination target. + +A C job returned by `job_open` has an observe/close precondition. POSIX numeric +groups do not become kernel-enforced read-only capabilities. The OS observer +type hides the native job and exposes no admission or termination operations. +An intentional negative compile consumer verifies these absent control paths. + +The actual Linux backend consumer covers normal retirement, a forged birth, an +already-ended anchor and another session. A separate syscall-boundary control +kills the anchor at `setpgid` entry and immediately after its successful exit; +both yield unknown. Normal retirement followed by termination of the original +unit preserves the live anchor. These Linux results do not qualify the native +macOS/Windows continuation path; the three source consumers run independently +in native CI. A dedicated control also reaps the ephemeral publisher before +the helper retires against its live owner. diff --git a/.github/checks/build_job_observer.py b/.github/checks/build_job_observer.py index 40ac419..435977a 100644 --- a/.github/checks/build_job_observer.py +++ b/.github/checks/build_job_observer.py @@ -41,4 +41,11 @@ def compile_backend(source): with (out / 'job-observer.jsonl').open('w') as log: result = subprocess.run([str(binary)], stdout=log, stderr=subprocess.STDOUT, timeout=30) print((out / 'job-observer.jsonl').read_text()) + +if result.returncode == 0 and sys.platform == 'linux': + boundary = out / 'retire-boundary' + subprocess.run(['c++', '-std=c++17', '-O2', '-pthread', '-I' + str(include), + str(root / '.github/checks/retire_boundary.cpp'), *objects, '-o', str(boundary)], check=True) + subprocess.run([sys.executable, str(root / '.github/checks/retire_boundaries.py'), str(boundary)], check=True, timeout=30) + raise SystemExit(result.returncode) diff --git a/.github/checks/job_observer.cpp b/.github/checks/job_observer.cpp index 3ca95d7..febc3b7 100644 --- a/.github/checks/job_observer.cpp +++ b/.github/checks/job_observer.cpp @@ -21,6 +21,7 @@ void require(bool value,const char* name) { std::fflush(stdout); if(!value) throw std::runtime_error(name); } std::string image, relative_image; +kal_program_identity root_identity{}; kal_spawn place() { kal_spawn how{}; for(kal_uintptr i=0;i2&&std::string(argv[1])=="driver") {return wait(start({"writer",argv[2]}));} - if(argc>3&&std::string(argv[1])=="helper")return helper(argv[2],argv[3]); + if(argc>5&&std::string(argv[1])=="helper") { + const kal_program_identity anchor{std::stoull(argv[4]),std::stoull(argv[5])}; + kal_program_identity expired{}; + if(argc>7)expired={std::stoull(argv[6]),std::stoull(argv[7])}; + return helper(argv[2],argv[3],anchor,argc>7?&expired:nullptr); + } + if(argc>5&&std::string(argv[1])=="launcher") { + kal_job caller{};if(kal_process_job_enter(&caller))return 52; + kal_stream report{},published{};if(kal_process_channel(&report,&published))return 53; + kal_spawn_streams child_streams{{},{},published}; + kal_process self{};kal_program_info own{};own.self_size=sizeof own; + if(kal_process_self(&self)||kal_process_info(self,KAL_PINFO_ID|KAL_PINFO_STARTED,&own))return 55; + (void)start({"helper","launcher",argv[3],argv[4],argv[5],std::to_string(own.id),std::to_string(own.started_ns)},nullptr,&child_streams); + kal_process_channel_close(published); + send_line(kal_stdout(),read_line(report)); + kal_process_channel_close(report);return 0; + } + kal_process self{};kal_program_info who{};who.self_size=sizeof who; + require(kal_process_self(&self)==kal_ok + &&kal_process_info(self,KAL_PINFO_ID|KAL_PINFO_STARTED,&who)==kal_ok + &&(who.present&(KAL_PINFO_ID|KAL_PINFO_STARTED))==(KAL_PINFO_ID|KAL_PINFO_STARTED),"full_owner_identity_observed"); + root_identity={who.id,who.started_ns};kal_process_close(self); #ifdef __linux__ require(prctl(PR_SET_CHILD_SUBREAPER,1)==0,"native_harness_reaps_fault_orphans"); #endif const auto root=fs::temp_directory_path()/("job-api-"+std::to_string(std::chrono::steady_clock::now().time_since_epoch().count())); fs::create_directories(root); - try {exercise(root,"normal");exercise(root,"before");exercise(root,"kill");} + try {exercise(root,"normal");exercise(root,"before");exercise(root,"kill");exercise(root,"retire-parent");} catch(...){fs::remove_all(root);throw;} fs::remove_all(root);return 0; }catch(const std::exception& e){std::fprintf(stderr,"job observer API: %s\n",e.what());return 1;} diff --git a/.github/checks/retire_boundaries.py b/.github/checks/retire_boundaries.py new file mode 100644 index 0000000..3f6d16c --- /dev/null +++ b/.github/checks/retire_boundaries.py @@ -0,0 +1,52 @@ +"""Trace Linux x86-64 setpgid entry/exit; no implementation fault hooks.""" +import os,signal,subprocess,ctypes,json,time,sys,platform +if sys.platform != "linux" or platform.machine() != "x86_64": + raise SystemExit("retirement syscall boundary control requires Linux x86-64") +from pathlib import Path +binary=str(Path(sys.argv[1]).resolve()) +libc=ctypes.CDLL(None,use_errno=True);libc.ptrace.restype=ctypes.c_long +fields='r15 r14 r13 r12 rbp rbx r11 r10 r9 r8 rax rcx rdx rsi rdi orig_rax rip cs eflags rsp ss fs_base gs_base ds es fs gs'.split() +class Registers(ctypes.Structure):_fields_=[(n,ctypes.c_ulonglong) for n in fields] +def ptrace(op,pid,address=0,data=0): + r=libc.ptrace(op,pid,ctypes.c_void_p(address),data if isinstance(data,ctypes.c_void_p) else ctypes.c_void_p(data)) + if r==-1:raise OSError(ctypes.get_errno(),str((op,pid))) + return r +rows=[] +for mode in ['live','wrong-birth','dead','cross-session','before-migration','after-migration']: + anchor=subprocess.Popen(['python3','-c','import time;time.sleep(20)'],preexec_fn=os.setsid if mode=='cross-session' else lambda:os.setpgid(0,0)) + helper=None + try: + who=subprocess.check_output([binary,'identity',str(anchor.pid)],text=True).split();assert int(who[0])==anchor.pid + if mode=='wrong-birth':who[1]=str(int(who[1])+1) + if mode=='dead':anchor.kill();anchor.wait() + helper=subprocess.Popen([binary,*who],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True) + assert helper.stdout.readline().strip()=='READY' + if mode.endswith('migration'): + ptrace(16,helper.pid);os.waitpid(helper.pid,0);ptrace(0x4200,helper.pid,0,1) + helper.stdin.write('A');helper.stdin.flush() + if mode.endswith('migration'): + at_exit=False;hit=False + for step in range(500): + ptrace(24,helper.pid);pid,status=os.waitpid(helper.pid,0) + if not os.WIFSTOPPED(status):raise RuntimeError(('unexpected_exit',status)) + regs=Registers();ptrace(12,helper.pid,0,ctypes.cast(ctypes.byref(regs),ctypes.c_void_p)) + if regs.orig_rax==109: + if not at_exit: + hit=True + if mode=='before-migration':anchor.kill();anchor.wait();ptrace(17,helper.pid);break + at_exit=True + else: + assert regs.rax==0,regs.rax + anchor.kill();anchor.wait();ptrace(17,helper.pid);break + else:raise RuntimeError('retirement syscall boundary not reached') + assert hit + helper.stdin.close();output=helper.stdout.read();error=helper.stderr.read();code=helper.wait(timeout=5) + expected=2 if mode=='live' else 0 + assert code==0 and 'RESULT 0 '+str(expected) in output,(mode,code,output,error) + if mode=='live':assert anchor.poll() is None and 'OLD_UNIT_END ' in output + rows.append({'case':mode,'state':'empty' if expected==2 else 'unknown','exit':code,'anchor_preserved':anchor.poll() is None if mode in ['live','wrong-birth','cross-session'] else None,'actual_native_boundary':mode if mode.endswith('migration') else None}) + finally: + if helper and helper.poll() is None:helper.kill();helper.wait() + if anchor.poll() is None:anchor.kill();anchor.wait() +(Path(binary).parent/'retire-boundaries.json').write_text(json.dumps(rows,indent=2)+'\n') +print(json.dumps(rows,indent=2)) diff --git a/.github/checks/retire_boundary.cpp b/.github/checks/retire_boundary.cpp new file mode 100644 index 0000000..09cfe98 --- /dev/null +++ b/.github/checks/retire_boundary.cpp @@ -0,0 +1,23 @@ +#include +#include +#include +#include +int main(int argc,char** argv) { + if(argc==3&&std::string(argv[1])=="identity") { + kal_process p{};kal_program_info i{};i.self_size=sizeof i; + if(kal_process_open_id(std::stoull(argv[2]),&p)||kal_process_info(p,KAL_PINFO_ID|KAL_PINFO_STARTED,&i)||!i.started_ns)return 2; + std::printf("%llu %llu\n",(unsigned long long)i.id,(unsigned long long)i.started_ns);kal_process_close(p);return 0; + } + if(argc!=3)return 3; + kal_program_identity anchor{std::stoull(argv[1]),std::stoull(argv[2])}; + kal_job job{};char ref[512];kal_uintptr used=0; + if(kal_process_job_create(&job,ref,sizeof ref,&used))return 4; + std::puts("READY");std::fflush(stdout);char ack=0;if(kal_stream_read(kal_stdin(),&ack,1)!=1)return 5; + kal_u32 state=99;const auto rc=kal_process_job_retire(job,ref,used,&anchor,&state); + std::printf("RESULT %d %u\n",rc,state);std::fflush(stdout); + if(!rc&&state==KAL_JOB_EMPTY) { + const auto ended=kal_process_job_terminate(job); + std::printf("OLD_UNIT_END %d\n",ended);std::fflush(stdout); + } + kal_process_job_close(job);return 0; +} diff --git a/.github/workflows/process-unit-api.yml b/.github/workflows/process-unit-api.yml index 6e6e960..d4699ff 100644 --- a/.github/workflows/process-unit-api.yml +++ b/.github/workflows/process-unit-api.yml @@ -14,15 +14,15 @@ jobs: fail-fast: false matrix: include: - - { os: ubuntu-24.04, implementation: openkal-linux } - - { os: macos-15, implementation: openkal-macos } - - { os: windows-2025, implementation: openkal-windows } + - { os: ubuntu-24.04, implementation: openkal-linux, revision: 5d2217424c25f072eb7bb8a7f6c2ea95b4bfa0df } + - { os: macos-15, implementation: openkal-macos, revision: 0ad90826b6097c806bc7db7b0c3579acd231b3b7 } + - { os: windows-2025, implementation: openkal-windows, revision: 664628d0a01a38d028f59fd8c660891efa218679 } steps: - uses: actions/checkout@v4 - uses: actions/checkout@v4 with: repository: mcpplibs/${{ matrix.implementation }} - ref: feat/observable-process-units + ref: ${{ matrix.revision }} path: backend - uses: ilammy/msvc-dev-cmd@v1 if: runner.os == 'Windows' @@ -33,4 +33,6 @@ jobs: if: always() with: name: declared-unit-${{ matrix.os }} - path: unit-api-build/*.jsonl + path: | + unit-api-build/*.jsonl + unit-api-build/retire-boundaries.json diff --git a/include/openkal/process.h b/include/openkal/process.h index cb9f683..ff190f4 100644 --- a/include/openkal/process.h +++ b/include/openkal/process.h @@ -14,6 +14,10 @@ struct kal_process { kal_uintptr h; }; +/* A program's full identity, copied from kal_process_info's id/started_ns. + * Neither field is a native process identifier a caller may interpret. */ +struct kal_program_identity { kal_u64 id; kal_u64 started_ns; }; + /* How a started program's standard streams are supplied. A stream handle of * zero denotes that the program inherits the corresponding stream of its * parent, which is what an environment without a general mechanism for @@ -615,11 +619,15 @@ int kal_process_job_terminate(struct kal_job); void kal_process_job_close(struct kal_job); /* AN OBSERVABLE UNIT (0.24). The unit is declared before any program is - * started in it. `create' puts the caller in a fresh private unit and writes + * started in it. `create' declares the caller's private unit and writes * an opaque reference into `reference'; `used' is its byte count. A buffer * smaller than the implementation requires is kal_err_no_space and leaves * the caller outside a new unit. No terminator is included in the count. * + * The caller declares the unit before starting its external members. A + * native unit already led by this caller may be adopted; existing members + * are not excluded from observation. + * * The reference is a value, not a handle: it can be retained and carried to * another invocation. Its bytes shall not be interpreted or manufactured by * a caller. `open' obtains an observation reference to the original unit; @@ -628,8 +636,13 @@ void kal_process_job_close(struct kal_job); * emptiness. It reports UNKNOWN. Neither a dead leader nor a stale reference * proves that the unit's other programs have ended. * - * An observer does not convey a right to terminate a replacement unit that - * reused a native identifier. `observe' validates the supplied reference and + * A job obtained by `open' shall be passed only to `observe' and `close', + * never to spawn, enter, terminate, limit or another control operation. This + * is a caller precondition: an environment whose job is a numeric group does + * not turn that number into a kernel-enforced read-only capability. The C++ + * library above this ABI can enforce the distinction through its observer type. + * Such an observer conveys no right to control a replacement unit that reused + * a native identifier. `observe' validates the supplied reference and * reports exactly one of the three states. Unsupported observation, identity * ambiguity, an inaccessible unit or an expired native reference is UNKNOWN, * never EMPTY. A positively empty unit has no remaining active member. @@ -638,8 +651,14 @@ void kal_process_job_close(struct kal_job); * one start, with the declared unit stated by its caller. Before `retire', * the caller shall cease all admission into that unit, including its own * concurrent spawns. `retire' establishes that no member except the caller - * remains; where membership can be left it may leave the unit first. The - * caller is still alive and may complete its own bookkeeping afterwards. + * remains. `continuation' names the same still-live program that supplied + * its id and birth through kal_process_info. Where membership can be left, + * the caller joins that program's unit before observing its former unit. A + * dead/reused continuation or one outside the native scope the caller can + * join yields UNKNOWN; no former numeric unit is guessed as a fallback. + * The continuation conveys no right to terminate or admit into its unit, + * and moving there does not reopen admission to the retired unit. The caller + * remains alive and may complete its own bookkeeping afterwards. * OCCUPIED and UNKNOWN can be retried. Retirement is idempotent, and after * asking to retire the caller shall not admit another program to the unit. * A library above this ABI enforces that rule through its owned unit object. @@ -659,7 +678,8 @@ int kal_process_job_open(const char* reference, kal_uintptr length, int kal_process_job_observe(struct kal_job, const char* reference, kal_uintptr length, kal_u32* state); int kal_process_job_retire(struct kal_job, const char* reference, - kal_uintptr length, kal_u32* state); + kal_uintptr length, const struct kal_program_identity* continuation, + kal_u32* state); /* WHAT A UNIT USES. Version 0.20, clause 11 entry 43. * diff --git a/src/process.cppm b/src/process.cppm index fe8daaa..dac4f5a 100644 --- a/src/process.cppm +++ b/src/process.cppm @@ -21,6 +21,7 @@ export import openkal.fs; export import openkal.stream; export using ::kal_process; +export using ::kal_program_identity; export using ::kal_spawn_streams; export using ::kal_spawn; export using ::kal_job; @@ -51,6 +52,8 @@ export using ::kal_process_open_id; export using ::kal_process_info; static_assert(sizeof(kal_process) == sizeof(kal_uintptr), "clause 7.2"); +static_assert(sizeof(kal_program_identity) == 16, "clause 5.3"); +static_assert(__builtin_offsetof(kal_program_identity, started_ns) == 8, "clause 5.3"); static_assert(sizeof(kal_spawn_streams) == 3 * sizeof(kal_uintptr), "clause 5.3"); static_assert(sizeof(kal_preopen) == 3 * sizeof(kal_uintptr), "clause 5.3"); // kal_program_info grows by appending, as kal_node_info does; the offsets of From 9dee6627f78e2e0c755fb59b72c4e589c715a3d7 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:41:31 +0900 Subject: [PATCH 14/17] feat: publish OpenKal 0.24 observable unit contracts --- ...2026-11-06-observable-unit-design-draft.md | 11 ++++++ .github/workflows/ci.yml | 4 +- .github/workflows/process-unit-api.yml | 6 +-- README.md | 18 +++++++-- SPEC.md | 37 ++++++++++++++++++- SURFACE.txt | 2 +- conformance/mcpp.toml | 2 +- conformance/src/declarations.c | 4 ++ examples/portable/mcpp.toml | 10 ++--- include/openkal/version.h | 2 +- kit/mcpp.toml | 6 +-- mcpp.toml | 2 +- 12 files changed, 82 insertions(+), 22 deletions(-) diff --git a/.agents/docs/2026-11-06-observable-unit-design-draft.md b/.agents/docs/2026-11-06-observable-unit-design-draft.md index 1200b08..88e7ce9 100644 --- a/.agents/docs/2026-11-06-observable-unit-design-draft.md +++ b/.agents/docs/2026-11-06-observable-unit-design-draft.md @@ -248,3 +248,14 @@ unit preserves the live anchor. These Linux results do not qualify the native macOS/Windows continuation path; the three source consumers run independently in native CI. A dedicated control also reaps the ephemeral publisher before the helper retires against its live owner. + +## 2026-11-06 publication-scope correction + +The earlier phrases "durable reservation" and "durable stage record" describe +records that remain visible to another invocation after a process ends. They +do not establish power-loss durability. Core's accepted recovery contract +covers process loss while the filesystem continues operating. Runtime records +use complete checked write/close followed by same-directory atomic rename; +publication and acknowledgement occur afterwards. The OS library's durable +write operation retains its existing contract. Native unit observation and +the all-reference-holders-lost unknown boundary are unchanged. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ac921c2..c66d2b6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,8 +33,8 @@ env: # A version verified to build these packages, not a measured minimum. The pin # exists for reproducibility rather than because an older mcpp is known to # fail. - MCPP_VERSION: 2026.8.27.1 - XLINGS_VERSION: v2026.8.17.2 + MCPP_VERSION: 2026.11.5.1 + XLINGS_VERSION: v2026.10.26.1 XLINGS_NON_INTERACTIVE: '1' jobs: diff --git a/.github/workflows/process-unit-api.yml b/.github/workflows/process-unit-api.yml index d4699ff..4678d7e 100644 --- a/.github/workflows/process-unit-api.yml +++ b/.github/workflows/process-unit-api.yml @@ -14,9 +14,9 @@ jobs: fail-fast: false matrix: include: - - { os: ubuntu-24.04, implementation: openkal-linux, revision: 5d2217424c25f072eb7bb8a7f6c2ea95b4bfa0df } - - { os: macos-15, implementation: openkal-macos, revision: 0ad90826b6097c806bc7db7b0c3579acd231b3b7 } - - { os: windows-2025, implementation: openkal-windows, revision: 664628d0a01a38d028f59fd8c660891efa218679 } + - { os: ubuntu-24.04, implementation: openkal-linux, revision: 0e9f3ccd7a1f473a2aeb399f727efec7196bae46 } + - { os: macos-15, implementation: openkal-macos, revision: e1534d674c980ed839903bf4be880efee1a55e3a } + - { os: windows-2025, implementation: openkal-windows, revision: d09f316a35814c0cbde3a2add793659ea439c6a0 } steps: - uses: actions/checkout@v4 - uses: actions/checkout@v4 diff --git a/README.md b/README.md index 0722095..6eaf852 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,16 @@ words become types that cannot be mixed. The header includes nothing. openkal must be usable on a freestanding target, and a consumer compiled with `-nostdinc` has nothing to include. +## Observable process units + +OpenKal 0.24 declares a private process unit before its external members start. +The existing spawn record admits members to it. An opaque observation reference +reports `occupied`, positively established `empty`, or `unknown`; a dead leader +or a missing native name does not establish emptiness. Retirement closes +admission and permits a helper to finish its bookkeeping after the unit's +external members have ended. The caller requirements are stated in +[SPEC clause 7.18](SPEC.md#718-observable-process-units) and `openkal/process.h`. + ## How a program uses openkal A program declares two dependencies. The first fixes the version of the contract @@ -77,16 +87,16 @@ conditional on the target. ```toml [dependencies] -openkal = "0.23.0" +openkal = "0.24.0" [target.'cfg(os = "linux")'.dependencies] -openkal-linux = "0.24.0" +openkal-linux = "0.25.0" [target.'cfg(os = "macos")'.dependencies] -openkal-macos = "0.21.0" +openkal-macos = "0.22.0" [target.'cfg(windows)'.dependencies] -openkal-windows = "0.19.0" +openkal-windows = "0.20.0" ``` The program imports the interface and names no implementation. diff --git a/SPEC.md b/SPEC.md index 3cdb885..4dfe153 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,4 +1,4 @@ -# openkal Specification, version 0.23 +# openkal Specification, version 0.24 ## 1. Scope @@ -1088,6 +1088,29 @@ environment applies its own policy for memory where it has one. A balanced program sizes what it keeps by what `kal_memory_info` reports available. Clause 11, entry 57. +### 7.18 Observable process units + +An implementation claiming `KAL_PROCESS_PROP_JOB_OBSERVE` shall provide a +caller-owned unit before its external members start, an opaque observation +reference, and the three states `unknown`, `occupied` and `empty`. +`kal_process_job_create` adopts or establishes the caller's private unit; +subsequent starts use the existing `kal_spawn.job` field. Observation shall +validate the original native identity. A dead leader, a missing name or the +loss of every native reference is not evidence that all members have ended. + +`kal_process_job_open` obtains observation and close authority only. This is a +caller precondition where the native handle is numeric; it does not convert a +reusable numeric identifier into a kernel-enforced read-only capability. + +Before `kal_process_job_retire`, the caller shall close admission, including +concurrent starts. Retirement proves that no external member remains and, +where necessary, leaves the original unit for the unit of the same live +program identified by `continuation`. Its identifier and birth are verified +before and after that transition. An inaccessible, dead, reused or unreachable +continuation yields `unknown`. Retirement conveys no admission or termination +authority over the continuation's unit and never reopens the original unit. +The caller may remain alive to finish its own bookkeeping. Clause 11, entry 58. + ## 8. Evolution Each interface is versioned independently. A revision may add declarations and @@ -2075,6 +2098,18 @@ The following are recorded so that they are not mistaken for oversights. available. It is a preference and not a limit, and no operation answers differently for it, as `kal_task_set_priority` changes no result. +58. **A declared unit whose external writers have ended.** **Specified in + 0.24**, as `kal_process_job_create`, `kal_process_job_open`, + `kal_process_job_observe` and `kal_process_job_retire` (clause 7.18). + Termination of one program does not establish termination of its + descendants. A retained native reference can establish the original + unit's state without a process registry; when the environment cannot + establish that identity, the answer is `unknown`. A helper is itself a + member, so successful retirement first excludes external members and + then leaves its original unit where the environment requires it. A live + continuation names the surviving program, not a remembered numeric group. + The existing spawn form and immutable structure layouts remain unchanged. + ## 12. Images A program written against this specification is built for an environment: its diff --git a/SURFACE.txt b/SURFACE.txt index ef7e607..2d0c488 100644 --- a/SURFACE.txt +++ b/SURFACE.txt @@ -1,4 +1,4 @@ -# The C surface of openkal 0.23.0, one name per line. +# The C surface of openkal 0.24.0, one name per line. # # This file is normative and is the single source consulted by clause 9. A # conforming implementation exports the names of the interfaces it provides diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index b838ffd..c1fd5d9 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -18,7 +18,7 @@ repo = "https://github.com/mcpplibs/openkal" # What it depends upon is therefore openkal and the language. The formatting in # okc.report is the price of that, and it is sixty lines. [dependencies] -openkal = "0.10.0" +openkal = "0.24.0" # The implementation under examination is not named here. # diff --git a/conformance/src/declarations.c b/conformance/src/declarations.c index fc35540..8cf95cf 100644 --- a/conformance/src/declarations.c +++ b/conformance/src/declarations.c @@ -114,8 +114,12 @@ void okc_declarations_c(void) (void)sizeof(&kal_process_end_as_requested); (void)sizeof(&kal_process_info); (void)sizeof(&kal_process_job_close); + (void)sizeof(&kal_process_job_create); (void)sizeof(&kal_process_job_enter); (void)sizeof(&kal_process_job_limit); + (void)sizeof(&kal_process_job_observe); + (void)sizeof(&kal_process_job_open); + (void)sizeof(&kal_process_job_retire); (void)sizeof(&kal_process_job_terminate); (void)sizeof(&kal_process_job_usage); (void)sizeof(&kal_process_open_id); diff --git a/examples/portable/mcpp.toml b/examples/portable/mcpp.toml index 9300648..e20c44c 100644 --- a/examples/portable/mcpp.toml +++ b/examples/portable/mcpp.toml @@ -9,7 +9,7 @@ name = "portable" version = "0.1.0" [dependencies] -openkal = "0.12.0" +openkal = "0.24.0" # AND THE VERSIONS HAVE TO FOLLOW THE SOURCE, which is the reason this example # is a criterion rather than an assumption. @@ -25,13 +25,13 @@ openkal = "0.12.0" # so no repository's continuous integration compiled it. The platform legs # below exist so that it cannot go stale silently again. [target.'cfg(os = "linux")'.dependencies] -openkal-linux = "0.12.0" +openkal-linux = "0.25.0" [target.'cfg(os = "macos")'.dependencies] -openkal-macos = "0.9.0" +openkal-macos = "0.22.0" [target.'cfg(windows)'.dependencies] -openkal-windows = "0.7.0" +openkal-windows = "0.20.0" # ANDROID IS NOT A LINE HERE, AND THAT IS THE POINT. # @@ -57,7 +57,7 @@ openkal-windows = "0.7.0" # All three iOS rows match: the device and both simulator arches carry # `os = "ios"`. [target.'cfg(os = "ios")'.dependencies] -openkal-macos = "0.9.0" +openkal-macos = "0.22.0" # THE WEB IS THE ONE PLATFORM THAT NEEDED NEW SOFTWARE. # diff --git a/include/openkal/version.h b/include/openkal/version.h index 54904bc..369f489 100644 --- a/include/openkal/version.h +++ b/include/openkal/version.h @@ -42,7 +42,7 @@ * for the reason tools/check-readme-versions.sh exists --- the one fact a * consumer actually asks for was checked by nobody. */ #define KAL_VERSION_MAJOR 0u -#define KAL_VERSION_MINOR 23u +#define KAL_VERSION_MINOR 24u #define KAL_VERSION_PATCH 0u #define KAL_VERSION_MAKE(major, minor, patch) \ diff --git a/kit/mcpp.toml b/kit/mcpp.toml index 24c7cf2..4ef64db 100644 --- a/kit/mcpp.toml +++ b/kit/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-kit" -version = "0.2.0" +version = "0.2.1" description = "Facilities composed from openkal's interfaces. Not part of the specification, and structurally incapable of being mistaken for it." license = "Apache-2.0" authors = ["mcpplibs"] @@ -67,7 +67,7 @@ kind = "lib" # published packages, which is the only place this could be observed: no # continuous integration in this ecosystem resolves a published package, # because they all substitute working trees by design. -openkal = "0.10.0" +openkal = "0.24.0" # NO `include_dirs` HERE EITHER. It named `../include`, which reaches out of # this package into whatever happens to sit above it. With the specification @@ -98,4 +98,4 @@ openkal = "0.10.0" # same reason: a change spanning the specification and an implementation must be # tested against both halves as written, not against whichever half is published. [target.'cfg(os = "linux")'.dev-dependencies] -openkal-linux = { version = "0.9.0", features = ["standalone"] } +openkal-linux = { version = "0.25.0", features = ["standalone"] } diff --git a/mcpp.toml b/mcpp.toml index 33a42d5..d80a62b 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal" -version = "0.23.0" +version = "0.24.0" description = "openkal: a portable kernel ABI specification. This package carries the normative declarations; implementations are separate packages." license = "Apache-2.0" authors = ["mcpplibs"] From 989838724e1d249829e823ad870b616c7f32c3b4 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:44:16 +0900 Subject: [PATCH 15/17] test: align generated declarations and native unit boundaries --- .github/probes/process_unit.cpp | 20 +++++++++++++------- conformance/src/macros_reach.cpp | 6 ++++++ conformance/src/sections/process.cpp | 2 +- src/macros.cppm | 4 ++++ 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 9ddc473..55911ef 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -101,7 +101,8 @@ DWORD members(HANDLE job) { observation observe(const std::string& name) { handle job(OpenJobObjectA(JOB_OBJECT_QUERY, FALSE, name.c_str())); if (!job.value) { - return GetLastError() == ERROR_FILE_NOT_FOUND ? observation::empty : observation::unknown; + // A missing name can coexist with a live writer after the last handle closes. + return observation::unknown; } return members(job.value) == 0 ? observation::empty : observation::occupied; } @@ -154,7 +155,7 @@ int controller(const fs::path& root, const std::string& name, unsigned policy, b } void windows_case(const fs::path& root, unsigned policy, bool normal, bool retain = false) { const bool bound = policy != 0; - bool false_empty = false; + bool empty_before_process_signal = false; fs::create_directories(root); const auto attempt_start = std::chrono::steady_clock::now(); const auto suffix = std::to_string(GetCurrentProcessId()) + "-" + std::to_string(GetTickCount64()) @@ -218,7 +219,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai child.value = OpenProcess(SYNCHRONIZE | PROCESS_TERMINATE | PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid); require(child.value != nullptr && same_birth(created(child.value), worker_birth), "hold_orphan_process_identity"); const bool writer_alive = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; - false_empty = first == observation::empty && writer_alive; + empty_before_process_signal = first == observation::empty && writer_alive; std::cout << "{\"post_controller_state\":" << static_cast(first) << ",\"query_error\":" << query_error << ",\"writer_alive\":" << (writer_alive ? "true" : "false") << "}\n"; @@ -228,7 +229,9 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai const bool wrote_late = fs::exists(root / "late.write"); std::cout << "{\"writer_completed_after_controller_loss\":" << (wrote_late ? "true" : "false") << "}\n"; report("missing_job_does_not_hide_later_output_write", first != observation::empty || !wrote_late); - report("named_job_survives_without_observer_process_handles", first == observation::occupied); + report("unretained_reference_does_not_claim_empty", first != observation::empty); + std::cout << "{\"legacy_missing_name_would_claim_empty\":" + << (query_error == ERROR_FILE_NOT_FOUND && wrote_late ? "true" : "false") << "}\n"; report("writer_survives_unbound_controller", writer_alive); require(wrote_late, "late_writer_output"); } @@ -236,7 +239,7 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai if (bound && !normal) { const auto state = observe_unit(); const bool live = WaitForSingleObject(child.value, 0) == WAIT_TIMEOUT; - false_empty = state == observation::empty && live; + empty_before_process_signal = state == observation::empty && live; BOOL associated = FALSE; DWORD exit_code = 0; const bool membership_known = !retained.value || IsProcessInJob(child.value, retained.value, &associated); require(GetExitCodeProcess(child.value, &exit_code), "writer_exit_code"); @@ -255,10 +258,13 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai require(until([&] { const auto state = observe_unit(); const bool ended = WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; - if (state == observation::empty && !ended) false_empty = true; + if (state == observation::empty && !ended) empty_before_process_signal = true; return ended && state == observation::empty; }), "whole_job_empty_after_writer_exit"); - report("unit_empty_does_not_precede_process_signal", !false_empty); + // The job counts active members. A terminated process handle can remain + // unsignaled briefly; that ordering is an observation, not a writer proof. + std::cout << "{\"unit_empty_preceded_process_signal\":" + << (empty_before_process_signal ? "true" : "false") << "}\n"; report("complete_job_is_empty", observe_unit() == observation::empty); } catch (...) { if (owner.value) TerminateProcess(owner.value, 137); diff --git a/conformance/src/macros_reach.cpp b/conformance/src/macros_reach.cpp index 67f320d..35d8ebd 100644 --- a/conformance/src/macros_reach.cpp +++ b/conformance/src/macros_reach.cpp @@ -78,3 +78,9 @@ static_assert(kal::macros::KAL_PROF_PROP_TOUCHES_M == 2u); static_assert(kal::macros::KAL_PROF_SAMPLE_DEPTH_M == 64u); } // namespace + +// OpenKal 0.24: observation is distinct from process/job control. +static_assert(kal::macros::KAL_PROCESS_PROP_JOB_OBSERVE_M == 32768u); +static_assert(kal::macros::KAL_JOB_UNKNOWN_M == 0u); +static_assert(kal::macros::KAL_JOB_OCCUPIED_M == 1u); +static_assert(kal::macros::KAL_JOB_EMPTY_M == 2u); diff --git a/conformance/src/sections/process.cpp b/conformance/src/sections/process.cpp index d5b84fe..83e321b 100644 --- a/conformance/src/sections/process.cpp +++ b/conformance/src/sections/process.cpp @@ -401,7 +401,7 @@ void run() { | kal::process::job_limit_hard | kal::process::job_limit_supervised | kal::process::null_streams - | kal::process::confine).bits; + | kal::process::confine | kal::process::job_observe).bits; observe(kind::abi, sizeof(kal_preopen) == 3 * sizeof(kal_uintptr), "a directory grant occupies three machine words"); observe(kind::abi, sizeof(kal_spawn) == 6 * sizeof(kal_uintptr), diff --git a/src/macros.cppm b/src/macros.cppm index 9b3a4c6..041e3ab 100644 --- a/src/macros.cppm +++ b/src/macros.cppm @@ -96,7 +96,10 @@ inline constexpr kal_uintptr KAL_INFO_KIND_M = (kal_uintptr)( inline constexpr kal_uintptr KAL_INFO_MODIFIED_M = (kal_uintptr)(KAL_INFO_MODIFIED); inline constexpr kal_uintptr KAL_INFO_SIZE_M = (kal_uintptr)(KAL_INFO_SIZE); inline constexpr kal_uintptr KAL_INFO_WRITABLE_M = (kal_uintptr)(KAL_INFO_WRITABLE); +inline constexpr kal_uintptr KAL_JOB_EMPTY_M = (kal_uintptr)(KAL_JOB_EMPTY); inline constexpr kal_uintptr KAL_JOB_LIMIT_MEMORY_M = (kal_uintptr)(KAL_JOB_LIMIT_MEMORY); +inline constexpr kal_uintptr KAL_JOB_OCCUPIED_M = (kal_uintptr)(KAL_JOB_OCCUPIED); +inline constexpr kal_uintptr KAL_JOB_UNKNOWN_M = (kal_uintptr)(KAL_JOB_UNKNOWN); inline constexpr kal_uintptr KAL_LOCK_EXCLUSIVE_M = (kal_uintptr)(KAL_LOCK_EXCLUSIVE); inline constexpr kal_uintptr KAL_LOCK_SHARED_M = (kal_uintptr)(KAL_LOCK_SHARED); inline constexpr kal_uintptr KAL_LOCK_WAIT_M = (kal_uintptr)(KAL_LOCK_WAIT); @@ -129,6 +132,7 @@ inline constexpr kal_uintptr KAL_PROCESS_PROP_GRANT_DIR_M = (kal_uintptr)( inline constexpr kal_uintptr KAL_PROCESS_PROP_JOB_M = (kal_uintptr)(KAL_PROCESS_PROP_JOB); inline constexpr kal_uintptr KAL_PROCESS_PROP_JOB_LIMIT_HARD_M = (kal_uintptr)(KAL_PROCESS_PROP_JOB_LIMIT_HARD); inline constexpr kal_uintptr KAL_PROCESS_PROP_JOB_LIMIT_SUPERVISED_M = (kal_uintptr)(KAL_PROCESS_PROP_JOB_LIMIT_SUPERVISED); +inline constexpr kal_uintptr KAL_PROCESS_PROP_JOB_OBSERVE_M = (kal_uintptr)(KAL_PROCESS_PROP_JOB_OBSERVE); inline constexpr kal_uintptr KAL_PROCESS_PROP_JOB_USAGE_M = (kal_uintptr)(KAL_PROCESS_PROP_JOB_USAGE); inline constexpr kal_uintptr KAL_PROCESS_PROP_NULL_STREAMS_M = (kal_uintptr)(KAL_PROCESS_PROP_NULL_STREAMS); inline constexpr kal_uintptr KAL_PROCESS_PROP_ONE_STRING_M = (kal_uintptr)(KAL_PROCESS_PROP_ONE_STRING); From 4cbc223ef5ae78687b0784cab9c4df346a375d90 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:45:04 +0900 Subject: [PATCH 16/17] test: preserve unknown references and declare layout probe includes --- .github/probes/process_unit.cpp | 7 ++++--- .github/workflows/ci.yml | 6 ++---- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/.github/probes/process_unit.cpp b/.github/probes/process_unit.cpp index 55911ef..9a532a1 100644 --- a/.github/probes/process_unit.cpp +++ b/.github/probes/process_unit.cpp @@ -259,13 +259,14 @@ void windows_case(const fs::path& root, unsigned policy, bool normal, bool retai const auto state = observe_unit(); const bool ended = WaitForSingleObject(child.value, 0) == WAIT_OBJECT_0; if (state == observation::empty && !ended) empty_before_process_signal = true; - return ended && state == observation::empty; - }), "whole_job_empty_after_writer_exit"); + return ended && (state == observation::empty || (!retained.value && state == observation::unknown)); + }), "whole_job_writer_exit_observed"); // The job counts active members. A terminated process handle can remain // unsignaled briefly; that ordering is an observation, not a writer proof. std::cout << "{\"unit_empty_preceded_process_signal\":" << (empty_before_process_signal ? "true" : "false") << "}\n"; - report("complete_job_is_empty", observe_unit() == observation::empty); + if(retained.value)report("retained_complete_job_is_empty", observe_unit() == observation::empty); + else report("unretained_completed_job_may_remain_unknown", observe_unit() != observation::occupied); } catch (...) { if (owner.value) TerminateProcess(owner.value, 137); if (!child.value && worker_id) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c66d2b6..8426e69 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -250,10 +250,8 @@ jobs: [package] name = "abi" version = "0.1.0" - EOF - cat > .abi-probe/build.mcpp <<'EOF' - import mcpp; - int main() { mcpp::include_dir("../include"); return 0; } + [build] + include_dirs = ["../include"] EOF cat > .abi-probe/src/main.cpp <<'EOF' #include From 1f0571064ccb537a0f80d72e4d6a75ee534ef5cb Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:51:05 +0900 Subject: [PATCH 17/17] ci: verify the final observable-unit backend source group --- .github/workflows/process-unit-api.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/process-unit-api.yml b/.github/workflows/process-unit-api.yml index 4678d7e..3521228 100644 --- a/.github/workflows/process-unit-api.yml +++ b/.github/workflows/process-unit-api.yml @@ -14,9 +14,9 @@ jobs: fail-fast: false matrix: include: - - { os: ubuntu-24.04, implementation: openkal-linux, revision: 0e9f3ccd7a1f473a2aeb399f727efec7196bae46 } - - { os: macos-15, implementation: openkal-macos, revision: e1534d674c980ed839903bf4be880efee1a55e3a } - - { os: windows-2025, implementation: openkal-windows, revision: d09f316a35814c0cbde3a2add793659ea439c6a0 } + - { os: ubuntu-24.04, implementation: openkal-linux, revision: b429bac4368d1a3960efd6d9dd93c02f229fbeda } + - { os: macos-15, implementation: openkal-macos, revision: 5cad63b22b61df2af38d632ff1590f8cbb527130 } + - { os: windows-2025, implementation: openkal-windows, revision: 07994ed5d8f7741474b42aa83bcda0256fc71bcc } steps: - uses: actions/checkout@v4 - uses: actions/checkout@v4