Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion architecture/hardening.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Field order is insignificant by design (one `write_str` per field under the enti

## Local filesystem containment

`validate_local_fetch_path()` (`src/core/local.rs`) rejects empty, absolute, traversal (`..`), hidden (unless `include_hidden`), skipped-directory (`target`, `node_modules`, `.git`), binary-extension, oversized, and non-file targets. `safe_open_relative()` (`src/meta/safe_open.rs`) then opens race-free: component-wise walking, NUL rejection, and on Linux descriptor-relative `openat2` with kernel-enforced `RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS` (plus `RESOLVE_NO_SYMLINKS` when `follow_symlinks = false`), falling back to `openat` with `O_NOFOLLOW` only where `openat2` is unavailable. Symlink escapes, intermediate symlinks, loops, root replacement, and concurrent modification during validation are all rejected or handled; reads after open enforce a hard byte cap. Never bypass these two functions with a direct filesystem read for operator-supplied paths.
`validate_local_fetch_path()` (`src/core/local.rs`) rejects empty, absolute, traversal (`..`), hidden (unless `include_hidden`), skipped-directory (`target`, `node_modules`, `.git`), binary-extension, oversized, and non-file targets. `safe_open_relative()` (`src/meta/safe_open.rs`) then walks component-wise with NUL rejection: on Linux descriptor-relative `openat2` with kernel-enforced `RESOLVE_BENEATH | RESOLVE_NO_MAGICLINKS` (plus `RESOLVE_NO_SYMLINKS` when `follow_symlinks = false`), falling back to `openat` with `O_NOFOLLOW` only where `openat2` is unavailable; on non-Unix every component is `symlink_metadata`-checked at walk time and links are rejected with `SymlinkDetected` when following is disabled. Symlink escapes, intermediate symlinks, loops, and root replacement are rejected fail-closed on every platform; the exact variant is platform-specific (Windows reports `SymlinkDetected` from the walk-time check, macOS reports `Io` where `O_NOFOLLOW` surfaces `ENOTDIR`, Linux rejects from the untouched `openat2`/`openat` path with a non-`SymlinkDetected` variant). The regression test pins fail-closed on unix and `SymlinkDetected` on Windows, never success. Reads after open enforce a hard byte cap. Only Linux (and the non-Linux Unix `O_NOFOLLOW` fallback) eliminates the validation-to-open race atomically: on Windows a check-to-open swap window remains, contained by the user-owned jail-root trust boundary and recorded in the owning closure record. Never bypass these two functions with a direct filesystem read for operator-supplied paths.

`SafeOpenError` taxonomy (`Empty`, `AbsolutePath`, `PathTraversal`, `NullByte`, `NotFound`, `SymlinkDetected`, `NotAFile`, `FileTooLarge`, `FileContentLimitExceeded`, `SafeSymlinkFollowingUnsupported`, `RootOpenFailed`, `Io`) keeps every rejection machine-readable: callers map variants to retrieval attempts and absence kinds instead of string-matching messages. On non-Linux Unix and non-Unix platforms, `follow_symlinks = true` returns `SafeSymlinkFollowingUnsupported` rather than attempting containment without a race-safe primitive.

Expand Down
21 changes: 16 additions & 5 deletions architecture/local-workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,15 +144,26 @@ directory file descriptor:
- Non-Linux with `follow_symlinks=true`: `SafeSymlinkFollowingUnsupported`
— there is no race-safe containment primitive, so the backend reports
instead of approximating.
- Windows (`follow_symlinks=false`): every component is
`symlink_metadata`-checked at walk time and links are rejected with
`SymlinkDetected`; the root itself is followed, matching the unix root
policy.

Pre-walk validation rejects empty, absolute, `..`-traversal, and
null-byte components plus hidden components unless `include_hidden` is set.
The final descriptor is `fstat`-checked for regular file type and size
limits, eliminating TOCTOU races between validation and open.
On Linux the final descriptor is `fstat`-checked for regular file type
and size limits, eliminating TOCTOU races between validation and open;
on Windows the walk-time link checks leave a documented check-to-open
swap window (no atomic no-follow open in std), contained by the
user-owned jail-root trust boundary. Intermediate links are rejected
fail-closed everywhere; Windows reports `SymlinkDetected`, macOS
reports `Io` (`ENOTDIR` from `O_NOFOLLOW`), and Linux rejects from the
untouched `openat2`/`openat` path with a platform-specific variant.
`SafeOpenError` covers `Empty`, `AbsolutePath`, `PathTraversal`,
`NullByte`, `NotFound`, `RootOpenFailed`, and
`SafeSymlinkFollowingUnsupported`; `safe_read_file()` layers bounded reads
on top.
`NullByte`, `NotFound`, `SymlinkDetected`, `NotAFile`, `FileTooLarge`,
`FileContentLimitExceeded`, `RootOpenFailed`,
`SafeSymlinkFollowingUnsupported`, and `Io`; `safe_read_file()` layers
bounded reads on top.

---

Expand Down
22 changes: 20 additions & 2 deletions architecture/python-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,15 @@ Current handoff: M001's shared Rust facade and ten-operation parity are
closed at `plans/closure/python-native-library/001-status.md`; M002's native
asyncio bridge is closed at `plans/closure/python-native-library/002-status.md`;
M003's Python API, types, examples, and semantic parity work is closed at
`plans/closure/python-native-library/003-status.md`; M004 distribution
qualification is active.
`plans/closure/python-native-library/003-status.md`; M005 in-flight
cancellation and default-config corrective is closed at
`plans/closure/python-native-library/005-status.md`; M004 distribution
qualification, M006 exact-SHA closure, and the M007 Windows safe-read
corrective are closed together at `plans/closure/python-native-library/004-status.md`
(candidate `faa1da2`, qualifying run `38617598636`, unpublished by design),
superseded as the qualification basis by M008 advisory remediation and
symlink containment at `plans/closure/python-native-library/008-status.md`
(candidate `067468a`, fully-green run `38641009246`).

## Rust operation facade

Expand Down Expand Up @@ -40,6 +47,17 @@ in-flight calls at 32, and bounds JSON conversion to 1 MiB, 64 levels, and
waits for their Rust futures to drop; Python task cancellation drops its
corresponding future. The runtime intentionally remains process-scoped.

`AsyncEggsearch()` preserves no-arg `AppConfig::default()` semantics.
`AsyncEggsearch.from_default_config()` and `default_config_path()` use
Rust-owned `eggsearch::core::config::default_config_path()` plus
`AppConfig::load`, preserving explicit-path behavior and CLI missing-file
fallback without reimplementing platform selection in Python. Cancellation
after a started barrier is proven by a `src/sdk.rs` counted blocking engine
plus a debug-only bridge gate (`_test_blocking_execute`, `_test_gate_state`,
`_test_gate_reset`, `_test_gate_release`, `_test_active_count`) that shares the
production `ActiveCall`, 32-permit, and close-token path; the gate is compiled
out with `debug_assertions` off and never ships in release wheels.

Milestone status and exact-candidate evidence live in
`plans/registry.md` and `plans/closure/python-native-library/`. A compiled
extension alone is not evidence of a supported Python consumer. Wheel support
Expand Down
2 changes: 1 addition & 1 deletion architecture/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Per-suite counts live in `docs/test-inventory.md`; suite placement rules live in
| `tests/mcp_http.rs` | Loopback Streamable HTTP lifecycle and transport hardening |
| `packaging/test-install.sh`, `packaging/test-install.ps1` | Installer behavior (not `tests/`) |
| `src/**/mod.rs` unit tests | Private-function coverage at the bottom of the source file |
| `bindings/python/tests/test_native.py` | PyO3 asyncio consumer behavior, typed method delegation, bounded input conversion, cancellation and close lifecycle |
| `bindings/python/tests/test_native.py` | PyO3 asyncio consumer behavior, typed method delegation, bounded input conversion, started-barrier cancellation with Rust drop/permit recovery, concurrent close and 32-call saturation recovery, conversion-failure leak checks, sequential loop recovery, and default-config selection |

---

Expand Down
4 changes: 2 additions & 2 deletions bindings/python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions bindings/python/PYTHON_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,12 @@ async def main() -> None:
asyncio.run(main())
```

`AsyncEggsearch.from_default_config()` loads the standard installed
`eggsearch/config.toml` via Rust-owned path resolution, matching CLI fallback
semantics. Bare `AsyncEggsearch()` keeps `AppConfig` defaults and never reads
that path implicitly. Cancelling a pending operation drops its Rust future and
releases the shared 32-call permit; `aclose()` drains active work and reports
`client_closed` for close-triggered calls.

See [`docs/python-library.md`](../../docs/python-library.md) for configuration,
types, trust handling, and the framework-neutral search/fetch/evidence example.
10 changes: 10 additions & 0 deletions bindings/python/python/eggsearch/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,16 @@ def __init__(self, config_path: str | Path | None = None) -> None:
None if config_path is None else str(config_path)
)

@classmethod
def from_default_config(cls) -> AsyncEggsearch:
instance = cls.__new__(cls)
instance._native = _NativeAsyncEggsearch.from_default_config()
return instance

@staticmethod
def default_config_path() -> str:
return str(_NativeAsyncEggsearch.default_config_path())

async def __aenter__(self) -> AsyncEggsearch:
return self

Expand Down
4 changes: 4 additions & 0 deletions bindings/python/python/eggsearch/_native.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ class EggsearchError(Exception):

class AsyncEggsearch:
def __init__(self, config_path: str | None = ...) -> None: ...
@staticmethod
def default_config_path() -> str: ...
@staticmethod
def from_default_config() -> AsyncEggsearch: ...
def execute(self, tool: str, arguments: dict[str, Any]) -> Any: ...
def aclose(self) -> Any: ...
def tool_contracts(self) -> Any: ...
Expand Down
179 changes: 167 additions & 12 deletions bindings/python/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
use std::collections::HashSet;
use std::path::PathBuf;
#[cfg(debug_assertions)]
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, LazyLock, Mutex, OnceLock};

use eggsearch::core::config::AppConfig;
Expand All @@ -23,6 +25,40 @@ create_exception!(_native, EggsearchError, pyo3::exceptions::PyException);
static RUNTIME_INIT: OnceLock<()> = OnceLock::new();
static IN_FLIGHT: LazyLock<Arc<Semaphore>> =
LazyLock::new(|| Arc::new(Semaphore::new(MAX_IN_FLIGHT)));
#[cfg(debug_assertions)]
static TEST_GATE: LazyLock<TestGate> = LazyLock::new(TestGate::new);
#[cfg(debug_assertions)]
struct TestGate {
entered: AtomicUsize,
dropped: AtomicUsize,
release_count: AtomicUsize,
release: Notify,
}
#[cfg(debug_assertions)]
impl TestGate {
fn new() -> Self {
Self {
entered: AtomicUsize::new(0),
dropped: AtomicUsize::new(0),
release_count: AtomicUsize::new(0),
release: Notify::new(),
}
}
fn reset(&self) {
self.entered.store(0, Ordering::SeqCst);
self.dropped.store(0, Ordering::SeqCst);
self.release_count.fetch_add(1, Ordering::SeqCst);
self.release.notify_waiters();
}
}
#[cfg(debug_assertions)]
struct GateDropGuard;
#[cfg(debug_assertions)]
impl Drop for GateDropGuard {
fn drop(&mut self) {
TEST_GATE.dropped.fetch_add(1, Ordering::SeqCst);
}
}

fn initialize_runtime() {
RUNTIME_INIT.get_or_init(|| {
Expand Down Expand Up @@ -121,27 +157,50 @@ struct PyAsyncEggsearch {
state: Arc<ClientState>,
}

fn build_state(config: AppConfig) -> PyResult<Arc<ClientState>> {
let client = EggsearchClient::new(config).map_err(|error| {
PyValueError::new_err(format!("invalid Eggsearch configuration: {error}"))
})?;
Ok(Arc::new(ClientState {
client,
lifecycle: Mutex::new(Lifecycle::default()),
cancelled: CancellationToken::new(),
drained: Notify::new(),
}))
}

fn load_config_from_path(path: &PathBuf) -> PyResult<AppConfig> {
AppConfig::load(path)
.map_err(|error| PyValueError::new_err(format!("invalid Eggsearch configuration: {error}")))
}

#[pymethods]
impl PyAsyncEggsearch {
#[new]
#[pyo3(signature = (config_path=None))]
fn new(config_path: Option<PathBuf>) -> PyResult<Self> {
let config = match config_path {
Some(path) => AppConfig::load(&path).map_err(|error| {
PyValueError::new_err(format!("invalid Eggsearch configuration: {error}"))
})?,
Some(path) => load_config_from_path(&path)?,
None => AppConfig::default(),
};
let client = EggsearchClient::new(config).map_err(|error| {
PyValueError::new_err(format!("invalid Eggsearch configuration: {error}"))
})?;
Ok(Self {
state: Arc::new(ClientState {
client,
lifecycle: Mutex::new(Lifecycle::default()),
cancelled: CancellationToken::new(),
drained: Notify::new(),
}),
state: build_state(config)?,
})
}

#[staticmethod]
fn default_config_path() -> String {
eggsearch::core::config::default_config_path()
.to_string_lossy()
.into_owned()
}

#[staticmethod]
fn from_default_config() -> PyResult<Self> {
let path = eggsearch::core::config::default_config_path();
let config = load_config_from_path(&path)?;
Ok(Self {
state: build_state(config)?,
})
}

Expand Down Expand Up @@ -214,6 +273,102 @@ impl PyAsyncEggsearch {
let contracts = self.state.client.tool_contracts();
json_to_python(py, &contracts).map(Bound::unbind)
}

#[cfg(debug_assertions)]
#[staticmethod]
fn _test_gate_reset() {
TEST_GATE.reset();
}

#[cfg(debug_assertions)]
#[staticmethod]
fn _test_gate_release() {
TEST_GATE.release_count.fetch_add(1, Ordering::SeqCst);
TEST_GATE.release.notify_waiters();
}

#[cfg(debug_assertions)]
#[staticmethod]
fn _test_gate_state(py: Python<'_>) -> PyResult<Py<PyAny>> {
let snapshot = serde_json::json!({
"entered": TEST_GATE.entered.load(Ordering::SeqCst),
"dropped": TEST_GATE.dropped.load(Ordering::SeqCst),
"release_count": TEST_GATE.release_count.load(Ordering::SeqCst),
"permits_available": IN_FLIGHT.available_permits(),
});
json_to_python(py, &snapshot).map(Bound::unbind)
}

#[cfg(debug_assertions)]
fn _test_active_count(&self) -> usize {
self.state
.lifecycle
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.active
}

#[cfg(debug_assertions)]
fn _test_blocking_execute<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
initialize_runtime();
let active_call = self
.state
.register()
.map_err(|error| sdk_error(py, error))?;
let permit = IN_FLIGHT.clone().try_acquire_owned().map_err(|_| {
sdk_error(
py,
SdkError {
code: "budget_invalid".into(),
message: "native execution concurrency limit reached".into(),
repair: None,
data: None,
},
)
})?;
let state = self.state.clone();
pyo3_async_runtimes::tokio::future_into_py(py, async move {
let _active_call = active_call;
let _permit: OwnedSemaphorePermit = permit;
let _gate_guard = GateDropGuard;
TEST_GATE.entered.fetch_add(1, Ordering::SeqCst);
let start = TEST_GATE.release_count.load(Ordering::SeqCst);
let result = std::panic::AssertUnwindSafe(async {
loop {
tokio::select! {
_ = state.cancelled.cancelled() => {
return Err(SdkError {
code: "client_closed".into(),
message: "Eggsearch client closed while the operation was running"
.into(),
repair: None,
data: None,
});
}
_ = TEST_GATE.release.notified() => {
if TEST_GATE.release_count.load(Ordering::SeqCst) != start {
return Ok(serde_json::json!({"gate": "released"}));
}
}
}
}
})
.catch_unwind()
.await
.unwrap_or_else(|_| {
Err(SdkError {
code: "internal".into(),
message: "internal native operation failure".into(),
repair: None,
data: None,
})
});
Python::attach(|py| match result {
Ok(value) => json_to_python(py, &value).map(Bound::unbind),
Err(error) => Err(sdk_error(py, error)),
})
})
}
}

#[pyfunction]
Expand Down
Loading
Loading