Personal Memory Assistant (PMA)

Local-First Semantic Search & Intelligence Engine

Personal Memory Assistant (PMA) is a high-performance, local-first search and retrieval system. It indexes your documents and projects with near-zero latency, allowing you to perform semantic queries and gain structural insights without your data ever leaving your machine.

What changed in each release: PMA Obsidian/05_CHANGELOGS.md.

---

1. Project Structure

├── app/                # Python Backend (FastAPI, Extraction, RAG, OCR)
├── frontend/           # React 19 Frontend & Tauri Desktop Shell
├── prompts/            # AI System Templates (RAG Logic)
├── scripts/            # Automated Development & Build Tooling (Windows)

---

2. Technology Stack

• Frontend: React 19 (Vite), TypeScript, TailwindCSS v4
• App Shell: Tauri v2 (Rust-based native shell)
• Backend: FastAPI (Python 3.12), Pydantic v2
• Storage Layer:
  • Metadata: SQLite (FTS5 for keyword search)
  • Vector Store: LanceDB (High-performance O(1) semantic retrieval; separate tables for chunk embeddings and document-summary embeddings)
• Extraction: Rust-powered parallel file walker & stream extractor
• Models: ONNX Runtime and tokenizers for local embedding generation and cross-encoder reranking
• GPU embedding (optional, Windows only, off by default): from Settings, PMA installs a DirectML runtime (about 150 MB) and the half-precision embedding model (about 67 MB) into a separate environment. Switching it on or off takes a restart and Diagnostics → Rebuild embeddings.
• OCR: optional, off by default. Reads scanned PDFs in an isolated subprocess venv — see §7
• AI Providers: Gemini, OpenAI, Anthropic, Groq, OpenRouter, NVIDIA NIM, Ollama, LM Studio, and generic OpenAI-compatible endpoints
• Spatial Engine: WebGPU-powered GPGPU compute for real-time indexing & Volumetric Visualization

---

3. Privacy & Security

Local-first is enforced in the code, not just claimed in the pitch.

• Your files never leave your machine by default. No telemetry, no phone-home. At first run PMA downloads its embedding model from Hugging Face, pinned to a revision and checked against the SHA-256 digests in models.lock.json. The download is currently about 460 MB, because every ONNX build in that revision is fetched while only the 34 MB INT8 one is used. Set PMA_EMBEDDING_ALLOW_DOWNLOAD=false to forbid it (.env.example sets it to true).
• Cloud providers are opt-in, and consent is checked against the resolved destination rather than a provider's label — a provider registered as "local" but pointed at a remote host is not exempt. PMA states plainly that free-tier cloud dispatch may use your inputs for model training before it sends anything.
• API keys live in the OS keyring. A key set in .env overrides the keyring and is managed outside the UI.
• Every /api/ route requires a local access token, sent as the X-Local-Access-Token header only and compared in constant time. There is no ?token= query fallback, so tokens never reach access logs, browser history, or Referer.
• A Content-Security-Policy with a per-request script nonce is applied to browser-served pages, alongside X-Content-Type-Options, X-Frame-Options and Referrer-Policy.
• Model output is sanitised before it renders. PMA searches documents you did not necessarily write, so a maliciously crafted file must not be able to steer the model into emitting working HTML inside the app.
• Destructive and expensive endpoints are rate-limited — /index/clear, /index/cleanup and /index/folder/remove at 3/minute, /index/export at 10/minute. /index/status is deliberately left unlimited so the Library page can poll it.

---

4. Getting Started

4.1 Prerequisites

• Python 3.12+ (Managed via uv recommended)
• Node.js 20+
• Rust via rustup; the toolchain is pinned to 1.97.1 in rust-toolchain.toml

Hardware. PMA targets budget machines on purpose: a ~4 GB VRAM GPU (GTX 1650 / RX 580 class) and an 8 GB RAM laptop. It holds roughly 196-248 MB resident when idle with models loaded (Windows working set, PMA_LANCEDB_MODE=portable, measured 2026-08-20 and 2026-10-13; resident size on Linux reads higher). Bulk indexing is a bounded transient that peaked 535 MB above idle on a 403 MB document corpus under the same conditions (567-632 MB on a 479 MB corpus, 2026-10). It does not return all the way: 60 s after a large ingest the working set sits about 190 MB above idle, mostly allocator fragmentation and database page caches, all of which Windows can page out under memory pressure. Serving queries holds about 500 MB, with brief spikes near 1.4 GB.

4.2 Quick Start (Windows)

Copy the example configuration, then choose the development workflow you need:

1. Create local configuration:

   Copy-Item .env.example .env

2. Launch browser-mode development:

   ./scripts/StartPMA.bat       # Backend + Vite in Windows Terminal tabs; asks for admin (NTFS fast scan)

3. Launch the Tauri desktop app:

   cd frontend
   npm run tauri dev

4.3 Manual Installation (Cross-Platform)

1. Initialize Backend:

   uv sync --all-extras
   cd app/scanner/rust_core && uv run maturin develop --release   # re-run after every uv sync

2. Initialize Frontend:

   cd frontend && npm install && npm run build

3. Run (from the repository root; the backend serves the built UI at http://127.0.0.1:8000):

   uv run python .

4.4 Credentials

PMA reads configuration from .env and checks the operating system keyring for provider API keys. Enter API keys during onboarding or in Settings → Providers; PMA stores those keys in the OS keyring by default. A provider API key set in .env takes precedence and is managed outside the UI, so it cannot be changed or deleted from Settings.

---

5. Development & Automation Scripts

The scripts/ directory contains tools to automate the development lifecycle.

Script	Purpose
StartPMA.bat	Starts the FastAPI backend and Vite frontend for browser-mode development.
Run-Tests.bat	Executes the full test suite (Python, Rust, and Frontend).
run_ci_checks.bat	Runs the project's lint, type, test, and security checks.
Build-Exe.bat	Builds the desktop app: frontend, PyInstaller backend (dist\PMA-sidecar.zip) and Windows MSI (dist\*.msi). See §8.
pin_models.py	Fetches the pinned models (embedder, reranker, OCR high-accuracy weights) into the Hugging Face cache and rewrites models.lock.json from them. Large: it pulls every ONNX build in each pinned revision (about 770 MB for the embedder and reranker alone).
Reindex-Embeddings.bat	Resets and regenerates the LanceDB vector store.
dev.bat	Launches the Vite development server for the frontend.

Testing and CI

./scripts/Run-Tests.bat       # Full Python, Rust, frontend, and E2E test suite
./scripts/run_ci_checks.bat   # Lint, type, test, and security checks

Neither script can run while the development backend is up — the dependency sync cannot replace the compiled rust_core extension while the server holds it.

To run the frontend test suite directly:

cd frontend
npm run test

---

6. Configuration

PMA uses a .env file in the root directory for critical configuration. See .env.example for details.

# Example .env configuration
PMA_HOST=127.0.0.1
PMA_PORT=8000
PMA_LOG_LEVEL=INFO
PMA_LANCEDB_MODE=portable
PMA_DB_PATH=data/pma_metadata.db
PMA_LANCEDB_PERSIST_DIR=data/lancedb
PMA_GEMINI_MODEL=gemini-2.5-flash-lite

# Optional: an API key here overrides the OS-keyring value for this provider.
# PMA_GEMINI_API_KEY=your_gemini_api_key_here

See .env.example for all supported provider and indexing settings. Values in .env override the built-in defaults, so anything you leave commented out stays on the tuned default rather than falling back to something worse.

Host-header guard. The server answers only requests whose Host header is 127.0.0.1, localhost, [::1], tauri.localhost or the value of PMA_HOST; anything else gets a 400. PMA_ALLOWED_HOSTS (comma-separated) adds names. It does not change where the server listens (PMA_HOST does). The guard is what stops a DNS-rebinding web page from reaching your index through the browser, so add only names you control.

---

7. OCR for Scanned Documents (optional)

A scanned PDF carries no text layer, so ordinary extraction finds nothing in it. OCR fills that gap. It is off by default and installs into its own isolated virtual environment from Settings → OCR, so leaving it switched off costs nothing — no extra dependencies reach the main application.

Option	What it does	Download
Standard	Runs on the processor. Around one page per second.	~230 MB
High accuracy	Larger, more accurate models on your graphics card. Windows only.	~430 MB
Your own AI model	Sends each page to a vision model you already run in Ollama or LM Studio.	nothing to install

High accuracy needs roughly 6 GB of free graphics memory — measured at 5.2 GB in use — so a 4 GB card is not enough and will fall back to the processor, ending up slower than Standard. Pick Standard on a 4 GB card.

Your own AI model downloads nothing: you pull the vision model yourself. It handles messy handwriting and unusual layouts better than the other options, but takes minutes per page rather than seconds, so it suits a few important documents rather than a whole library.

---

8. Packaging

Tauri is the supported desktop distribution path. After the setup in §4.3, build the Windows MSI installer with:

./scripts/Build-Exe.bat

It builds the frontend, packs the Python backend with PyInstaller into dist\PMA-sidecar.zip, then runs npx tauri build --bundles msi --config src-tauri/tauri.bundle.conf.json, which puts that zip inside the installer. The MSI lands in dist\, with an unbundled dist\app\pma-desktop.exe beside it. A bare npm run tauri build skips the bundle config, so its MSI has no backend.

Unproven. No MSI built this way has yet been installed and run on a clean machine. The installer bundles no models: the embedder downloads on first run (see §3) and the reranker is absent.

---

9. Architecture

9.1 Ingestion Pipeline

The system reads files in streams to ensure scalability without exceeding memory limits. Each file is chunked for keyword/semantic indexing and independently distilled into a structural summary that gets its own embedding, giving retrieval a document-level signal alongside chunk-level ones. Pages with no text layer are deferred to a durable OCR queue and rejoin the pipeline once recognised.

graph LR
    subgraph "Scanning"
        A[File System] --> B[Rust File Scanner]
    end

    subgraph "Processing"
        B --> C[Text Extractor]
        C --> D[SQLite Database]
        C --> E[Chunker]
        C --> S[Deep Summarizer]
        C -->|no text layer| O[OCR Queue]
        O --> W[OCR Worker]
        W --> E
    end

    subgraph "Indexing"
        E --> F[AI Embedder]
        F --> G[LanceDB Chunk Vectors]
        E --> H[FTS5 Keyword Index]
        S --> G2[LanceDB Summary Vectors]
    end

Loading

9.2 Unified Search Flow

A query planner first classifies intent into one of four modes: whole-collection inventory and project lookups bypass retrieval entirely, relationship questions that name code (an identifier, module.attr, f()) traverse the knowledge graph, and everything else runs the full RAG pipeline. Per-folder counts and newest / largest-file questions take the RAG path with the file statistics and file list attached. Routing is checked against a frozen, labelled query set (tests/eval/routing_set.json, scripts/eval_routing.py). The context budget comes from the model's size class and is capped at the model's reported context window, so a small model is never handed more than it can hold. FULL_RAG fuses three signals — keyword matching (SQLite FTS5), chunk-level semantic search, and document-summary semantic search — via Reciprocal Rank Fusion (RRF), then applies a cross-encoder reranker for maximum precision. There is no score-margin bypass (that heuristic was removed as unsound): on FULL_RAG the reranker is skipped when fewer than two candidates remain or when its model is not installed, and a 5-second deadline falls back to RRF order and marks the answer degraded. The reranker model is not downloaded automatically; the Diagnostics page shows whether it is installed.

Known limitation: English-first retrieval. The keyword index is trigram FTS5, so query terms shorter than three characters cannot match; most Chinese and Japanese words are two characters, which leaves a CJK query with no keyword signal. The pinned embedder (bge-small-en-v1.5) is English-only, so the semantic legs do not cover CJK either. CJK documents are indexed but retrieve poorly; this is untested and unmeasured.

graph LR
    A[Query] --> P[Query Planner]

    P -->|FAST_METADATA| M[SQLite Stats]
    P -->|FAST_PROJECT| J[Project Metadata Lookup]
    P -->|GRAPH_SEARCH| K[Knowledge Graph Traversal]

    P -->|FULL_RAG| B[Hybrid Retrieval]
    B -->|Keywords| C[SQLite FTS5]
    B -->|Chunk Meaning| D[LanceDB Semantic]
    B -->|Doc Meaning| N[LanceDB Summaries]
    C --> E[RRF Ranker]
    D --> E
    N --> E
    E --> Q[Cross-Encoder Reranker]
    Q --> F[Context Builder]
    F --> L[LLM Answer]

Loading

---

10. License

Distributed under the MIT License.

---

P.S. The Nested Volumetric Crystal Graph serves as a high-performance 3D Treemap alternative, providing recursive spatial depth for project structure visualization without the occlusion and scaling limits of traditional 2D/3D tiling methods.

Binit Varghese
GitHub Profile
Project: Personal Memory Assistant