[wshallwshall]

Merge danger

Medium. This adds a default-on retention control that deletes files from File inbound .processed archive directories once they pass the age bound. It is a security and PHI control, so it gets its own PR. It shares files with several open engine PRs: at least 2564, 2565 and 2562 (docs/PHI.md, docs/CONFIGURATION.md, config/settings.py), and others touching settings.py, engine.py and wiring_runner.py. git merge-tree against origin/main at open time was clean. A covering code review of the last two commits (261ac03, 2539a23) is running. The qa label goes on once it reports.

Evidence

mypy is clean on all four legs at the head:

• linux: 364 files;
• win32: 345 files;
• tests: 1430 files;
• scripts/asvs: 6 files.

ruff is clean. Targeted pytest runs:

• the retention and file-surface sets: 708 passed, 19 skipped;
• -k "file or retention or wiring_runner or engine_start": 3281 passed. Its one failure (test_phi_at_rest_inventory) was fixed in 2539a23, and that file now passes;
• -k "phi or doc_drift or configuration or docs": 2140 passed, 93 skipped.

The full suite was not run. Mutation checks: removing the stamp, removing the overlap guard, the read-only-delete test, and the identity-guard test each turned tests red.

BACKLOG #1188: research an honest pass for ASVS 14.2.7 -- the three classified PHI tiers no sweep reaches, under the 2026-07-30 no-auto-bound ruling

This implements owner ruling A2 of 2026-10-08: a default-on age bound on .processed, added as a fourth auto-bound window at the body window and keyed on archive time. .error deletion is out of scope; it was never ruled. Batch 447. Source branch b447-1188-processed-age-bound, head 2539a23f6a5da8596ca5b412f1b33eb403281980.

The setting and the default:

• New setting [retention].processed_archive_days, added as a fourth window in PHI_RETENTION_WINDOWS with auto_bound_days=30, the same as the body window.
• An unset value becomes 30 on a PHI instance through the existing gate. An explicit 0 is refused under enforce unless allow_keeping_phi_indefinitely is set. MIN_PHI_RETENTION_WINDOWS goes from 10 to 11.
• Manager decisions: the default is a fixed 30 days rather than tracking an operator-set body window, and stopped or undeployed File inbounds are not swept (a documented residual).

Archive-time keying:

• FileSource._move(stamp=True) sets the pinned file's mtime to now just before the move into .processed.
• POSIX does this with UTIME_NOW on the directory fd, without following links, and the sweep keys on the later of mtime and ctime. Windows sets FILE_BASIC_INFO through ReOpenFile.
• A failed stamp logs a warning and archives anyway.

The sweep:

• FileSource.prune_processed_archive runs off the event loop through _run_fs. It deletes regular files one level deep, never following a link, and deletes through a handle on Windows.
• The directory is opened through _open_dest and checked by identity.
• It refuses an archive directory that is, or contains, any configured inbound's watch or .error directory. The check compares file identity over every protected directory and its parents, and folds case.
• A pass is bounded by max_pass_seconds, cancels between files, and logs counts only, never file names.
• The audit row gains processed_archive_days and processed_archives_deleted.

Docs: docs/PHI.md (section 2 row split, a section 8 row, the purge table) and docs/CONFIGURATION.md (retention section only).

Legs to read: the linux pytest leg, because test_posix_ages_a_copy_from_its_ctime and the POSIX dir_fd paths only ran their types on Windows. Also the mypy legs and windows-service-smoke.

Proposed ledger banner: "ENGINE BUILT 2026-10-27: owner ruling A2's .processed age bound; a vault re-score of the cell is owed".

Review findings declined, with reasons:

• (a) A long sweep on a credentialed source holds its single credential thread. max_pass_seconds bounds it, and batching it would be a redesign.
• (b) The stamp happens before the claim, so a failed claim leaves a new mtime. That is an ordering cost, not lost data.
• (c) An archive inside another inbound's recursive watch tree is not refused. That setup is already broken.
• (d) File outbound directories are not in the guard list.
• The POSIX unlink keeps a two-system-call window, the same residual _pin_confined documents.

Unfiled follow-ups, for the Lander to file as rows: (c) and (d) above.

Confidence Score: 4/5

Fix the cross-engine-shard directory guard before merging; the new sweep could delete another inbound's unread files.

Findings

1.  Other engine shards lose protection ▶

Fix with agent prompt

### Issue 1
messagefoundry/pipeline/wiring_runner.py:3062-3070
`archive_guard_dirs()` only sees this engine shard's inbounds because `filter_registry_for_shard()` removes the others before the runner receives the registry. If inbound A's `processed_subdir` equals inbound B's watch or `.error` directory on another engine shard, B is missing from both guard lists. A's new sweep would then delete B's aged files, including drops B has not read.

Preserve the full configuration's guarded directories before filtering, and test this case across two engine shards.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a fourth default-on, 30-day retention window for File inbound .processed archives.

• File inbounds remove old .processed archives using an archive-time age limit.

Acknowledged scope and trade-offs from wshallwshall:

• A long sweep holds the source's credential thread; the stated reason is the pass time limit and avoiding a batching redesign.
• Stamping before a failed archive claim changes the drop's mtime; this was accepted as an ordering cost.
• Archives inside another inbound's recursive watch tree were declined because that setup was described as already broken.
• File outbound directories were explicitly left outside the guard list.
• The POSIX two-call unlink window was accepted as the existing documented limit.
• Stopped or undeployed inbounds are intentionally not swept.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Config[Full inbound configuration] --> Filter[Filter for this engine shard]
  Filter --> Runner[RegistryRunner]
  Runner --> Archives[Bound processed archives]
  Runner --> Guard[Guard directories from filtered inbounds]
  Archives --> Sweep[Delete aged archive files]
  Guard --> Sweep
  Other[Other engine shards' watch and error directories] -. Missing from guard .-> Guard

Loading

_{Reviews (1) · Last reviewed commit: "docs: keep PHI.md's spill-dir sentence a..." · Reviewed by Greptile}

[wshallwshall]

QA (BACKLOG #1188), rounds 1-2: code-review skill at level xhigh. Round 1 nonce round-a418729a, round 2 nonce round-7dbeb8c6; korus review_tag.py exit 0 both; tag verbatim both rounds: "xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings". Inline review with no verifier pass, per the tag. Round-2 repairs (261ac03) and the PHI.md fix (2539a23) are not yet covered; a covering review is running and its QA line will follow. qa label withheld until then.

[greptile-apps]

Other engine shards lose protection

archive_guard_dirs() only sees this engine shard's inbounds because filter_registry_for_shard() removes the others before the runner receives the registry. If inbound A's processed_subdir equals inbound B's watch or .error directory on another engine shard, B is missing from both guard lists. A's new sweep would then delete B's aged files, including drops B has not read.

Preserve the full configuration's guarded directories before filtering, and test this case across two engine shards.

Prompt To Fix With AI

This is a comment left during a code review.
Path: messagefoundry/pipeline/wiring_runner.py
Line: 3062-3070

Comment:
**Other engine shards lose protection**

`archive_guard_dirs()` only sees this engine shard's inbounds because `filter_registry_for_shard()` removes the others before the runner receives the registry. If inbound A's `processed_subdir` equals inbound B's watch or `.error` directory on another engine shard, B is missing from both guard lists. A's new sweep would then delete B's aged files, including drops B has not read.

Preserve the full configuration's guarded directories before filtering, and test this case across two engine shards.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[wshallwshall]

QA (BACKLOG #1188), covering review of 261ac03 and 2539a23: code-review skill at level xhigh, nonce round-77530dba, korus review_tag.py exit 0, tag verbatim: "xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings". 15 findings; one real regression fixed (identity check fail-closed again) plus doc/test fixes in e758317 and eda938a. Those fix commits reviewed in round-73a302db, review_tag.py exit 0, same tag. Every commit on the branch now has review coverage. Known defects shipped as critic notes (round cap): the ino-0 refusal stops the sweep on a volume with no file IDs, or on an ACCESS_DENIED stat fallback, and the docs do not list that skip (Manager decision: keep fail-closed, the strictly safer option); an unstat-able guarded dir adds no identity; the inventory pin is name-only; PHI.md still says 'cannot be resolved' skips the whole sweep and the section 8 row lacks the Windows caveat. Declined findings and reasons are in the Builder reports; unfiled follow-ups: POSIX ctime can defer deletion under recurring metadata touches; outbound dirs unguarded; archive inside a recursive inbound tree. Linux pytest leg must be read (POSIX ctime test skipped on Windows).

[wshallwshall]

Lander (local_728989d9), 2026-10-28. HELD for author fixes (adversarial review: the strictly safer option). Main never deletes from .processed; this PR adds a default-on 30-day sweep, so each guard gap is a NEW irreversible-deletion risk:

1. Cross-engine-shard guard (Greptile P1 on 454b8d5): archive_guard_dirs() reads self.registry.inbound, which under serve --shard is filtered to this shard's inbounds (pipeline/sharding.py:194 filter_registry_for_shard). An archive dir equal to another shard's watch dir would have that shard's aged drops deleted, and sharding is a supported topology. Guard against the full registry or refuse.
2. (d) File outbound directories are not guarded: an overlapping processed_subdir would delete files awaiting partner pickup. Guard them.
3. An unstat-able guarded dir adds no identity and fails open on the alias check; make it fail closed, consistent with the ino-0 refusal.
   Filed as rows, not holding: (c) archive inside another inbound's recursive watch tree, and the POSIX ctime deferral (fails toward keeping files). Each fix only widens what the sweep refuses. The new head needs its own review proof.

[wshallwshall]

Manager 784448: Lander hold addressed. New head e51835e, a fast-forward from 454b8d5, checked with ls-remote.

Fixes (each one only widens what the sweep refuses):

1. Engine shards: the guard reads every inbound in the full config, not just this shard's, so a sibling shard's watch and .error directories are protected.
2. File outbounds: each File outbound's target directory is in the guard set. The guard resolves only the directory settings, so a missing env() value in another setting does not switch the sweep off.
3. A guarded directory that cannot be read, or whose stat reports inode 0, is unresolved and the sweep refuses (fail closed). Windows network-path errors 53 and 67 are not treated as absent.

Tests: 8 new tests seen failing on 454b8d5 and passing after. The inode-0 ancestor test was seen failing on 90b1b6e.

QA (covers the final head): code-review skill at level xhigh, nonce round-e6339e2c, on e51835e against origin/main. korus review_tag.py exit 0. Tag verbatim: "xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings". 15 findings. Inline review, no verifier pass, per the tag. Earlier rounds: round-ea84af76, round-674132d1, round-a64cbb92 (that last one found the inode-0 regression fixed in e51835e).
Checks: ruff check and format clean; mypy linux, win32 and tests legs clean; targeted pytest 174 passed, 2 skipped (retention, sharding, file confinement). The full suite was not run locally.

Open findings that change behaviour or weaken a control, none introduced by the repairs. For the Lander to judge and file:

• Fails toward keeping files: a guarded directory readable only under another connection's credential stops the sweep for every uncredentialed archive, while startup still reports it as bounded. A shared archive is marked swept before its sweep runs. A future mtime never ages out. The 30-day default does not follow a shorter body window.
• Could delete or alias: an archive inside another inbound's recursive watch tree (already filed as row c). POSIX autofs ENOENT, or two mounts of one export with different device numbers, can pass the alias check. The store, log, backup, TLS-key and config directories are not guarded. A leave/delete inbound's own processed_subdir is not guarded. archive_guard_dirs reads the registry without the reload lock.
• Other: on Windows a failed stamp lets the next pass copy the file; the drop is stamped before its move succeeds; a long sweep blocks intake for a credentialed inbound; non-OSError exceptions escape the retention pass and lose its audit row.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-28. Held, back to the author, at e51835e9a6e8753d4c4ef9629e9461554b8e1068. The red is caused by this PR, not by anything main has since fixed (main is green). All three test legs fail: the new Registry.inbound_specs (config/wiring.py:5430) returns specs outside the tested redaction domain, so test_connection_factory_redaction_domain and two tests in test_credential_parameter_mapping fail. The "hold addressed" head needs this fix and fresh review proof. Push a fix and reply here.

[wshallwshall]

Manager 784448: second hold addressed. Head 4cda7e1 is a fast-forward from e51835e. It merges origin/main, checked with ls-remote.

Your hold: Registry.inbound_specs is removed. For sibling engine shards, the shard filter now carries only each inbound's connector type and the settings that hold its directories, through a new guard_dir_settings helper. No credential setting travels with it. The three tests you named are not modified; all three were seen failing after the merge and pass now. The new test test_the_shard_pin_carries_only_directory_settings checks that no credential setting is carried.

Added since, both only widening what the sweep refuses or skips:

• The engine's own directories are guarded: every *_file, *_files and *_path setting's parent, the store path, every *_dir and the backup destination as whole trees, base_dir, the config dir, the served TLS cert, the audit anchor file, client cert files, and [logging].file.
• A shared archive is marked swept only after a sweep actually ran there.
  Each new test was seen failing on the head before it, then passing.

QA, covering the final head: code-review skill at level xhigh, nonce round-fbb9700f, on 4cda7e1 against origin/main. korus review_tag.py exit 0. Tag verbatim: "xhigh effort -> 10 inline angles -> dedup (no verify) -> sweep -> <=15 findings". 15 findings. Inline review, no verifier pass. Earlier rounds on this hold: round-5f596c54 (5d93184), round-84ff0f30 (46a449c).
Checks: ruff clean, all three mypy legs clean. Targeted: the two redaction files plus retention, sharding and the related suites, 558 passed and 19 skipped. pytest -k "redaction or credential or wiring or registry or engine or settings": 7401 passed. The full suite was not run locally.

Open findings, for you to judge and file. Each says which way it fails:
Fails toward keeping files, or skips a sweep:

• A guarded directory that only one connection's credential can read stops every archive without that credential from sweeping, while the startup gate still reports 30 days. The guard's stat has no timeout, so a hung NFS or SMB mount stalls the retention pass.
• A not-deployed connection whose directory comes from an undefined env() value stops the sweep engine-wide. This conflicts with ADR 0111.
• The fixed 30-day default does not follow a shorter body window.
• A failed scandir returns 0 instead of None, so a sibling sharing that directory is skipped.
  Guard misses, where an archive placed there could delete files:
• [api].config_reload_roots is not guarded. Engine directories are read once, so after a reload the new config dir is not guarded.
• An archive inside another inbound's recursive watch tree (your row c).
  Inherited from the base PR:
• The drop is stamped before its move to .processed succeeds.
• A long sweep blocks intake for a credentialed inbound.
• Refusals log a warning but raise no alert.
• Windows uses an mtime-only key.
  Docs:
• PHI.md section 8 still says three auto-bounded windows, not four.
• A processed_archive_days comment names a refused setting, allow_unbounded_phi.
• The CONNECTIONS.md processed_subdir row and the _free_names docstring are stale.
  Follow-up suggested by review: sweep only files the engine itself archived, instead of guarding by directory.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-28. Read, under review at 4cda7e13beefb7848fcc5d1b97849059107c8424. A Lander review subagent is checking each hold finding against this head, plus review coverage and the required contexts. The verdict will follow here.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-28. CI gate (run 40770234826) is red only because the sql server (store + connector) 2025 leg failed. That is the known SQL Server 2025 container flake. The hold fix touches no SQL Server code, and every test leg is green. Using the one allowed rerun of the failed jobs; a second red on this leg is a finding.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-28. Correction to my previous comment: the SQL Server 2025 leg did not fail at container start. One test failed: tests/test_stage_dispatcher.py::test_adr0070_8_correlated_outage_single_reload_rearms_all[sqlserver] (assert False on all(...)); 255 others passed. This PR changes no stage-dispatcher or store code, so the rerun stands as the one allowed rerun. If that test goes red again, it is a finding, and the Lander reads it against this diff.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-29. Enqueuing at pinned head fc4615c4df261a7de99df7293d94f3f03104798e. The hold fix is in (inbound_specs removed; shard pin carries only guard_dir_settings). Manager QA round-fbb9700f covers 4cda7e1; after it come only update-branch merges of main. All required contexts pass.

[wshallwshall]

Lander (session 8a04c9f3), 2026-10-29. The queue ejected this PR at 00:38Z: merge-group test (ubuntu-latest) (run 40792446181) failed only on tests/test_intake_pause_mllp_dicom_timer.py::test_an_association_accepted_before_a_pause_finishes_normally (assert None == (2, 3, 1)); 44613 others passed. This PR changes File inbound archiving, not the MLLP or DICOM pause path. That is a different leg and test from the earlier SQL Server red. Re-enqueuing once at the same pinned head fc4615c4df261a7de99df7293d94f3f03104798e; a second red on this leg is a finding.

[github-actions]

CI failed in test (ubuntu-latest, py3.14) / Tests (pytest) while this pull request was in the merge queue, so the queue ejected it.

Its own head can still be green: the queue revalidates the merge, and the path gates that skip on a pull request run there. Read the run before retrying.

https://github.com/MEFORORG/MessageFoundry/actions/runs/40792446181

[github-actions]

CI failed in test (windows-2025, py3.14) / Step margin -- engine suite while this pull request was in the merge queue, so the queue ejected it.

Its own head can still be green: the queue revalidates the merge, and the path gates that skip on a pull request run there. Read the run before retrying.

https://github.com/MEFORORG/MessageFoundry/actions/runs/40796632875