Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -1,12 +1,7 @@
# Formatting-only commits excluded from `git blame` (git config:
# blame.ignoreRevsFile = .git-blame-ignore-revs).
#
# PLACEHOLDER — the ruff-format-only commit has NOT landed yet. Once the
# black -> ruff format switch is committed (~63 files reformatted, no code
# changes), append its full 40-char SHA below with a one-line note, e.g.:
#
# # style: switch formatter from black to ruff format (formatting-only, no code changes)
# <full-40-char-commit-sha>
#
# Then enable it repo-wide:
# git config blame.ignoreRevsFile .git-blame-ignore-revs
# style: apply ruff format across src, tests, scripts (52 files) [E2]
# Reviewed against its parent: identical ASTs and ordered comment tokens.
9cb2642687b5565d752762eb8e6c921b6c21608e

# Enable locally: git config blame.ignoreRevsFile .git-blame-ignore-revs
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Pre-commit gates — mirror the `gate` job in .github/workflows/test.yml:
# ruff check, ruff format, mypy on src (the same arms `run.py gate` runs).
# Replaces black: ruff format is the repo formatter (see [tool.ruff] in
# pyproject.toml); the black config/dev-dep removal is tracked in TODO.md.
# Ruff format is the repo formatter (see [tool.ruff] in pyproject.toml).
# The black configuration and development dependency were removed in E2.

repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
Expand Down
7 changes: 4 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

JOURNAL_DIR ?= ../ActiveInferenceJournal
MANIFEST ?= data/output/channel_videos.json
RAG_BASE_PATH ?= .

# Load .env file if it exists
ifneq (,$(wildcard .env))
Expand Down Expand Up @@ -69,8 +70,8 @@ lint:
uv run python run.py gate lint format

format:
uv run ruff format src/ tests/
uv run ruff check --fix src/ tests/
uv run ruff format src/ tests/ scripts/
uv run ruff check --fix src/ tests/ scripts/

db-start:
surreal start --log trace --user root --pass root --bind 0.0.0.0:8080 rocksdb://./data/database
Expand Down Expand Up @@ -112,7 +113,7 @@ extract-entities:
echo "Error: COHERE_API_KEY not found in .env file"; \
exit 1; \
fi
uv run python -m journal_utilities.rag.main
uv run python -m journal_utilities.rag.main --pending --base-path "$(RAG_BASE_PATH)"

enumerate-channel:
@echo "Enumerating videos on the Active Inference channel..."
Expand Down
40 changes: 38 additions & 2 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,44 @@

**Status:** Maintained
**Owner:** Active Inference Institute / Journal-Utilities maintainers
**Last reviewed:** 2026-11-12 (verification pass — Publishing contract migration, gate
re-run on this checkout, pin and formatter state confirmed; see the pass-5 entry below)
**Last reviewed:** 2026-11-13 (source-qualified CLI/API, disposable-database and
browser acceptance; earlier dated passes below retain their historical scope).

---

## Pass 6 — pipeline and graph reliability (2026-11-13)

- [x] **J1/J3 — Serve ownership and corpus** — occupied ports are refused without
killing their owners. The configured corpus reaches the actual web app;
browser navigation displayed its title and transcript with external media blocked.
- [x] **J2 — Required exports** — empty outcomes and mixed-success failures return
nonzero; required failure stops `full` and default orchestration.
- [x] **J4/J6 — Source identity and replay** — trusted source/type/name identity,
record-ID endpoints and deterministic edges prevent cross-source merging.
Explicit legacy migration preserves unknown-source rows and refuses conflicting
claims; failed edges remain pending and retry preserves historical dates.
Real in-memory SurrealDB 4.0.1 acceptance used SDK 3.0.0, including injected
write failure and replay. No production database was accessed or migrated.
- [x] **J5/A1 — Extraction entry point** — explicit file or pending selection performs
work, reports totals and propagates failures. Empty queues do not construct
the provider. `make extract-entities` selects pending work under `RAG_BASE_PATH`.
- [x] **J7 — Request boundaries** — bounded body ingestion, typed chat fields,
pre-stream errors and bounded public pagination are covered by focused tests.
- [x] **O3/O4/O5/A2 — Maintenance** — reviewed format-only commit recorded for blame;
stale test-count prose and Black commentary removed; both Makefile formatter
operations now include `scripts/`.

Focused Python 3.12 checks, actual database acceptance and browser navigation
passed. This is not a full-suite, coverage, hosted-CI, live-Cohere or production
migration claim. Historical M10 metadata stamping and ME2 module-path repair
were partial steps, not prior proof of complete identity or extraction behavior.
The dated measurement files remain unchanged.

Still open: hosted billing, qualified exact-pin GNN admission (no automatic pin
bump), canonical Publishing producer adoption, admitted full-gate measurements,
GPU/live-provider/YouTube-write qualification, the sibling translation migration,
and the owner-scheduled description backlog. Existing dated entries retain the
details of those external and ownership constraints.

---

Expand Down
12 changes: 12 additions & 0 deletions docs/RUN_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,15 @@ it runs the live download tests.
2. Add the row to the arms table above and any rule it enforces to the doc it protects.
3. Run `uv run python run.py gate <arm>` and, if the timing page should change,
`uv run python run.py gate report`.

## Listener ownership and corpus selection

`serve` refuses an occupied port without discovering or signalling its owner.
Stop that owner explicitly or choose another configured port.
`general.data_dir` is passed to the web application: relative paths resolve
against the project root; absolute paths remain absolute.

`make extract-entities` selects pending database work and resolves relative
transcript paths under `RAG_BASE_PATH` (default `.`).
`make format` runs both Ruff operations over `src/ tests/ scripts/`; it changes
files and is not an acceptance-only command.
8 changes: 8 additions & 0 deletions docs/export.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,11 @@ html = true
# Output directory relative to project root
output_dir = data/export
```

## Required export completion

`run.py export` returns nonzero when no outcomes are produced or any requested
export fails, including mixed-success batches. Successfully written or
already-existing nonempty outputs remain available. `full` and the default
pipeline propagate required export failure; the default pipeline does not
continue to tests or serving after that failure, independently of `test_strict`.
45 changes: 37 additions & 8 deletions docs/rag.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,31 +35,33 @@ Defines the ontology for extraction.

Manages interaction with SurrealDB.

- **Nodes**: Stores entities as graph nodes (e.g., `person:karl_friston`).
- **Edges**: Stores relationships as graph edges (e.g., `authored` -> `concept:free_energy_principle`).
- **Idempotency**: Ensures entities are merged rather than duplicated.
- **Nodes**: Entity records are identified by trusted transcript source, entity type and name.
- **Edges**: Relationships reference the returned entity record IDs, not generic display names.
- **Idempotency**: Scoped records and deterministic relationship identities are reused on replay.
- **SDK**: `surrealdb>=3.0.0` (the async-client generation). `SurrealDBClient` opens the
connection with `AsyncSurreal`, types it as `AsyncSurrealConnection`, and awaits
`connect()` / `signin()` / `use()` / `query()` / `close()`. `query()` answers one entry
per statement, so single-statement queries are unwrapped to their rows
(`_first_statement`) before use; `create()` on a table answers with the created records.
Server-side, the SDK accepts SurrealDB 2.x and newer and refuses older servers.
(`_first_statement`) before use. The qualified 3.0.0 SDK returns a single record
mapping from `create()` and singular-endpoint `relate()`, not a list.
The source-identity migration was exercised against SurrealDB 4.0.1; this is
not a qualification of every server version accepted by the SDK.

## Pipeline (`main.py`)

The `JournalRAGPipeline` class orchestrates the process:

1. **`process_transcript(transcript)`**:
- Creates a `Transcript` record in DB.
-Calls `extractor.extract_core_entities()`.
- Calls `extractor.extract_core_entities()`.
- Converts results via `EntityAdapter`.
- Upserts entities and relationships to SurrealDB.

## Usage

```bash
# Run the extraction pipeline on all new transcripts
make extract-entities
# Select pending database transcripts; resolve their relative paths under the corpus
make extract-entities RAG_BASE_PATH=/path/to/corpus
```

## Configuration
Expand All @@ -69,3 +71,30 @@ make extract-entities
| `COHERE_API_KEY` | API Key for Cohere AI |
| `COHERE_MODEL` | Model ID (default `command-a-03-2025`, from `rag/settings.py`) |
| `DB_URL` | SurrealDB connection string |

## Explicit extraction and source identity

Select files with `python -m journal_utilities.rag.main --file transcript.txt [more.txt ...]`
or pending database work with `python -m journal_utilities.rag.main --pending --base-path /path/to/corpus`.
No selection is a usage error (exit 2). Requested failures return exit 1;
successful runs, including an explicitly selected empty pending queue, return 0
and print processed/failed/entity/relationship totals. Extraction needs Cohere
credentials and SurrealDB; an empty queue does not construct the provider client.

Relationship endpoints are checked against the trusted source-qualified entity
identity. Missing required endpoints or failed edge writes leave the transcript
pending. Retry and replay reuse persisted records rather than duplicating edges.
Generic names or metadata claims never establish provenance for unknown-source
legacy records. File and database loaders parse dates from the full filename,
including dotted dates without a separate extension; invalid or absent dates use
`datetime.min`. Titles still use the filename stem; retries preserve stored historical dates.

## Legacy schema migration

Take a reviewed backup and stop concurrent writers before explicitly running
`python -m journal_utilities.rag.main --migrate-source-identity`.
Ordinary connection refuses a legacy schema rather than silently migrating it.
The migration changes schema, preserves records and performs no Cohere request.
It refuses ambiguous identities or contradictory attributed endpoints; it does
not backfill provenance. Qualify it against a disposable database before any
production migration.
23 changes: 23 additions & 0 deletions docs/web_interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,26 @@ The frontend files are served directly from `src/journal_utilities/interface/sta
- `src/journal_utilities/interface/app.py`: Main application logic.
- `src/journal_utilities/interface/static/app.js`: Core frontend logic (routing, rendering).
- `src/journal_utilities/interface/static/index.html`: Base layout.

## Request admission limits

Chat, streaming chat and clear-session requests must be JSON objects with
string `session_id` (default `default`, at most 200 characters), string `message`
(default empty, at most 32000 characters), and boolean `use_rag` (default true).
Chat and streaming require a nonblank message. Clear validates provided fields
but does not require a message. Malformed JSON or an empty required message
returns 400; invalid shapes/types return 422; byte or field limits return 413
before a provider request or streaming response starts.

The application retains at most 387424 body bytes before JSON decoding and
rejects an over-limit chunk before appending it. Streamed bytes are checked even
with missing or understated Content-Length. This does not bound upstream
server/ASGI transport allocations.

Video pagination requires offset >= 0 and limit 1..200. Search requires limit
1..100; internal unlimited search remains available through `limit=None`.

`run.py serve` uses the configured corpus and refuses occupied ports without
terminating another process. Browser fixtures must use canonical 11-character
YouTube IDs; external thumbnails, fonts and embedded video still need network
access independently of the locally served transcript.
59 changes: 16 additions & 43 deletions run.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
import json
import logging
import os
import signal
import socket
import subprocess
import sys
Expand Down Expand Up @@ -227,12 +226,19 @@ def cmd_export(config: configparser.ConfigParser, _args: argparse.Namespace) ->
print("⚠ No transcripts found to export")
return 1

failures = 0
outcomes = 0
for fmt_name, fmt_results in results.items():
ok = sum(1 for r in fmt_results if r.status == "success")
skip = sum(1 for r in fmt_results if r.status == "skipped")
fail = sum(1 for r in fmt_results if r.status == "failed")
failures += fail
outcomes += len(fmt_results)
print(f" {fmt_name}: {ok} exported, {skip} skipped, {fail} failed")

if failures or not outcomes:
print("\n✗ Export incomplete")
return 1
print("\n✓ Export complete")
return 0

Expand Down Expand Up @@ -287,36 +293,6 @@ def _port_in_use(port: int) -> bool:
return True


def _free_port(port: int) -> bool:
"""Try to kill whatever is occupying *port*. Return True on success."""
try:
result = subprocess.run(
["lsof", "-ti", f":{port}"],
capture_output=True,
text=True,
timeout=5,
)
pids = result.stdout.strip().split()
if not pids:
return True

my_pid = str(os.getpid())
for pid in pids:
if pid == my_pid:
continue
logger.info("Killing stale process %s on port %d", pid, port)
os.kill(int(pid), signal.SIGTERM)

# Brief wait for cleanup
import time as _t

_t.sleep(0.5)
return not _port_in_use(port)
except Exception as exc:
logger.warning("Could not free port %d: %s", port, exc)
return False


def cmd_serve(config: configparser.ConfigParser, _args: argparse.Namespace) -> int:
"""Start the web interface."""
import uvicorn
Expand All @@ -327,23 +303,19 @@ def cmd_serve(config: configparser.ConfigParser, _args: argparse.Namespace) -> i
port_str = get_str(config, "interface", "port", "8000")
port = int(port_str)

# ── Auto-resolve port conflicts ────────────────────────────────
# An existing listener belongs to its owner; never kill or reclaim it.
if _port_in_use(port):
logger.warning("Port %d already in use — attempting to free it", port)
if _free_port(port):
logger.info("Port %d freed successfully", port)
else:
logger.error("Cannot free port %d — aborting serve", port)
print(f" ✗ Port {port} is occupied and could not be freed.")
print(f" Try: lsof -ti :{port} | xargs kill -9")
return 1
logger.error("Port %d is already occupied; listener left untouched", port)
print(f" ✗ Port {port} is occupied. Stop its owner explicitly or choose another port.")
return 1

print(f"Starting web interface at http://{host}:{port}")

# Configure uvicorn logging to match ours if needed,
# but uvicorn has its own config. We'll let it handle itself.
try:
app = create_app()
data_dir = PROJECT_ROOT / get_str(config, "general", "data_dir", "data/output")
app = create_app(data_dir=data_dir)
# log_config=None tells uvicorn to use the existing logging configuration
# which allows our root logger (with FileHandler) to capture the logs.
uvicorn.run(app, host=host, port=port, log_config=None)
Expand Down Expand Up @@ -535,8 +507,9 @@ def cmd_default(config: configparser.ConfigParser, args: argparse.Namespace) ->
print("━" * 62)
rc = cmd_export(config, args)
if rc != 0:
logger.warning("Export step returned non-zero: %d", rc)
print(f" ⚠ Export finished with warnings (exit code {rc})\n")
logger.error("Export step failed: %d", rc)
print(f" ✗ Export failed (exit code {rc}) — aborting pipeline\n")
return rc
else:
print()

Expand Down
Loading
Loading