Repository navigation
Fix clock-dependent RAM accounting identities - #164
Merged
Merged
Conversation
tavateva
approved these changes
Oct 21, 2026
tavateva
left a comment
There was a problem hiding this comment.
APPROVE at d1f4cb8fee0e99175e074b86e507ade7a79d73f6.
This fixes #162 at the identity owner rather than patching each symptom:
known,excludedand the per-sweep maps now holdpsutil.Processobjects, so psutil's own equality and hash are the identity._identity,root_createdand the manifest'spid/create_timepair are gone; nocreate_time()caller remains underpython/ouroboros.- The two cross-process carriers (the root handshake and the manager manifest) go through a single adapter in
memory_ipc. It exports_identtagged with the psutil version and platform, and receivers only use it to match a liveProcess.
This is the choice my comment on #162 asked to have named: private value, exact pin and native tests. It is declared in the PR body, pyproject.toml and run-statistics.md. It also covers both macOS (unadjusted native start) and Windows (native creation value) without the PR calculating its own platform identity.
Failure envelope, checked against the code:
- A psutil other than 7.1.3 makes
export_process_identityraise duringMemoryCollector.start.RunStatistics.start_memoryturns that into an unavailable RAM metric, and the operation still runs. I confirmed this with psutil 7.2.2:RuntimeError: Memory identity requires psutil 7.1.3; got 7.2.2. - An old-format or unverifiable manifest owner raises
ValueErrorinstead of quietly counting as owning no segments, so that sweep is incomplete rather than under-counted. process_is_runningvalidates the fresh identity before comparing, so a(pid, None)wrapper can't be cached as "PID reused" and drop a reparented worker.
Verification at the head (Linux, psutil 7.1.3 overlay):
- The four memory suites, including the new
test_memory_identity.py, ran with-W error::DeprecationWarning: 105 passed. The full Python suite: 407 passed, 2 skipped, which matches the body. - #162's own reproduction, adapted to the new constructor: 50 sleeping children, boot time moved +0…+30 s and then back to −10 s. The sampler reported 51 / 51 / 51 / 51 processes, with 51 unique PIDs throughout. Before the fix the same script printed 51 / 251 / 1551 at
4502a3c. - Boot time moved between the two discoveries inside a sweep: 5 of 5 sweeps were complete, where before they would have failed with "Process tree changed".
- All CI checks are green, including all six native Linux/Windows/macOS collector jobs.
I did not run Docker/WSL2. That and the overhead check are correctly moved to #163.
Non-blocking notes
- Wrong-repo link in the docs.
run-statistics.mdL410 links "#121" toChengLabResearch/ouroboros/issues/121, which is a closed Electron dependency-bump PR. The intended target is den-sq/ouroboros_pub#121, as in the PR body. Fixing the URL is enough. - "Transient identity access failure remains recoverable" holds for sweeps, not the idle poll. Inside
sample(), aValueErrorfromprocess_is_runningbecomes one incomplete sample. Inmemory_observer.main's waiting and idle loops,process_is_running(sampler.root)is outside the per-sampletry, so the same error ends observation and the run reports unavailable. Exposure is close to nil, because the root is the backend itself and readable by the same user on every platform, and failing to "unavailable" is the conservative direction. Narrowing the body claim, or catchingValueErrorthere, is either sufficient.
9 tasks
9 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clock adjustments could make the RAM collector count the same live worker repeatedly or discard a valid manager manifest. This replaces wall-clock identity keys throughout the collector with psutil's stable
Processidentity, so adjustments between discoveries or during a sweep neither multiply contributions nor create false process churn.The root handshake and manager manifests now share one validated IPC adapter. Missing, malformed, unsupported or unreadable identities invalidate coverage instead of falling back to PID/public creation time or silently dropping a worker. A per-sweep invariant rejects multiple live identities for one PID. Psutil is pinned to 7.1.3 in package metadata, the lockfile and focused CI; the adapter also rejects unverified runtime versions.
Closes #162.
Shared-basis audit
Processowns process identity and equality/hash semantics. Previously_identity,root_created, managerpid/create_timefields, and manifest matching reconstructed a competing wall-clock identity. Those writers, carriers and comparisons are removed.known,excluded, and per-sweep owner maps now useProcessobjects. Root startup consumes its token once and retains only the validatedProcess.memory_ipcowns the boundary adapter and existing file retry policy. It exports psutil's private_identwith its verified version/platform only when publishing control/manager JSON; receipts validate againstProcessidentity. No kernel start counter or clock-derived alternative is calculated. Manifest bytes retained during a sweep are only allocation-change evidence, not identity keys. The private API is the issue's chosen tradeoff: exact dependency pin plus native identity/IPC/clock upgrade tests. Generated wheel metadata requirespsutil==7.1.3, covering production pip installs as well as Poetry.Processidentity is validated before comparing it with the retained object, using the same native identity query asis_running(). This avoids psutil treating(pid, None)as exit/PID reuse and silently losing a reparented worker. Genuine exit/reuse remains detectable; transient identity access failure remains recoverable.Validation
poetry check --lockpasses. Its existing metadata-deprecation warnings match the base commit. Package metadata confirms the exact psutil constraint;git diff --checkpasses.Limits and follow-ups
Discovery continues to use psutil's
children(): its own public-timestamp filtering can omit a previously unseen child if the clock moves backward inside one enumeration. Previously observed live workers remain independently checked. This does not claim an atomic or complete process census.Collector overhead and real-run coverage validation remain in #163; replacement RAM measurements/publication tables remain in https://github.com/den-sq/ouroboros_pub/issues/121. This repair does not validate archived peaks or certify a 1% overhead bound. Existing #139 native-macOS workflow and macOS-host Docker deferrals remain unchanged.