Fix stale chunks and re-embed cascades, split classes at methods, auto-sync before search

AGENTS.md

@@ -23,14 +23,14 @@ Copy the skill into the chosen directory, e.g. (global Factory Droid):
 
 ```bash
 mkdir -p ~/.factory/skills/enowx-rag
-cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
+cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
 ```
 
 Or project-local:
 
 ```bash
 mkdir -p .agents/skills/enowx-rag
-cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md .agents/skills/enowx-rag/skill.md
+cp skill/enowx-rag.md .agents/skills/enowx-rag/skill.md
 ```
 
 If the tool requires a reload, restart the session or run any skill-refresh command available. After installing, follow the setup wizard in `skill/enowx-rag.md` whenever the user asks for RAG setup.
@@ -40,7 +40,7 @@ If the tool requires a reload, restart the session or run any skill-refresh comm
 The MCP server is written in Go and lives in `mcp-server/`.
 
 ```bash
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cd mcp-server
 go build ./cmd/mcp-server
 ```
 
@@ -51,7 +51,7 @@ go build ./cmd/mcp-server
 **Option B — Self-hosted TEI:** Start Qdrant + TEI with Docker:
 
 ```bash
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cd mcp-server
 docker compose up -d qdrant tei-embedding
 ```
 

CHANGELOG.md

@@ -8,6 +8,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **Auto-sync before search**: a directory indexed once with
+  `rag_index_project` is recorded (`~/.enowx-rag/project_dirs.json`) and
+  re-synced incrementally before a search when its index is older than
+  `RAG_AUTO_SYNC_INTERVAL` or `auto_sync_interval` in config.yaml (default
+  `60s`, `off` disables). Agents no longer
+  need to call `rag_index_project` after every task; the agent templates say
+  so. A recorded directory that no longer exists is skipped, not synced, and
+  a project deleted elsewhere is not recreated. The registry file is shared
+  safely by several processes (re-read before every read and change).
+- **Big classes split at their methods**: a declaration too long for one
+  chunk is cut at its members (Python, Rust, Ruby, Java/C#, Kotlin, Swift,
+  PHP, JS/TS class methods; decorators and annotations kept with their
+  method, closures kept inside it), each chunk's symbol `Class.method`,
+  before falling back to blank lines.
+- `rag.PayloadUpdater` (implemented by the SQLite store): rewrites a point's
+  content and metadata, keeping its vector.
 - **Built-in store (`vector_store: sqlite`)**: one SQLite file (pure Go, no
   cgo; `sqlite_path`, default `~/.enowx-rag/rag.db`, or `RAG_SQLITE_PATH`),
   vectors searched by cosine, text in FTS5 for hybrid search (RRF), so
@@ -21,6 +37,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   indexes re-embed once).
 - `Indexer.Files` lets a program embedding enowx-rag pass the files to index.
 
+### Fixed
+- **Stale chunks after editing a file**: a sync deleted chunks only when
+  their file was gone, so code removed from a file that still exists stayed
+  in the index and was served as current. A sync now deletes every chunk of
+  the directory it did not produce again (also replacing chunks written by
+  older chunk layouts).
+- **Re-embedding everything below an edit**: chunk IDs were their position
+  in the file (`#chunkN`), so adding a function shifted the IDs of all the
+  chunks after it. IDs are now derived from the chunk's content
+  (`dir/file#<hash>`, `chunk_version` v4; existing indexes re-embed once);
+  only small declarations are packed together, so an edit changes only the
+  chunks around it; a chunk that only moved gets its line numbers refreshed
+  without re-embedding where the store supports it (SQLite), re-indexed
+  otherwise.
+- `rag_index_project` on a directory that does not exist is an error instead
+  of an empty scan that deleted everything indexed from it.
+- A sync only ever deletes chunks under its own directory's ID prefix, even
+  when the store returns other points from `ListPoints` (pgvector ignores the
+  `source_dir` filter, so syncing one directory deleted another's chunks).
+- Indexing a second directory with the same folder name as one already in
+  the project (`/x/app`, `/y/app`) is refused instead of silently replacing
+  its chunks.
+- A file that exists but can't be read keeps its chunks instead of losing
+  them to the stale cleanup.
+- **Chroma**: points are written with `/upsert` instead of `/add`, which
+  skipped IDs already stored, so edited chunks were never updated.
+
 ### Security
 - Secrets are never indexed: `.env*`, keys and certificates, `credentials`
   and `*secret*` files by name, and any chunk holding a private key block or a

CLAUDE.md

@@ -8,15 +8,15 @@ If the user asks to install the skill or MCP server, run:
 
 ```bash
 mkdir -p ~/.factory/skills/enowx-rag
-cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
+cd mcp-server
 go build ./cmd/mcp-server
 ```
 
 For local backend (TEI fallback):
 
 ```bash
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cd mcp-server
 docker compose up -d qdrant tei-embedding
 ```
 
@@ -28,7 +28,7 @@ docker compose up -d qdrant tei-embedding
 {
   "mcpServers": {
     "enowx-rag": {
-      "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server",
+      "command": "/path/to/enowx-rag/mcp-server/mcp-server",
       "env": {
         "RAG_VECTOR_STORE": "qdrant",
         "RAG_QDRANT_URL": "http://localhost:6333",
@@ -46,7 +46,7 @@ docker compose up -d qdrant tei-embedding
 {
   "mcpServers": {
     "enowx-rag": {
-      "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server",
+      "command": "/path/to/enowx-rag/mcp-server/mcp-server",
       "env": {
         "RAG_VECTOR_STORE": "qdrant",
         "RAG_EMBEDDER": "tei",

README.md

@@ -334,7 +334,7 @@ Copy the skill file into the Droid skill directory:
 
 ```bash
 mkdir -p ~/.factory/skills/enowx-rag
-cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
+cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md
 ```
 
 If the skill must be reloaded, restart the Droid session or refresh skills.
@@ -344,11 +344,11 @@ If the skill must be reloaded, restart the Droid session or refresh skills.
 ## 2. Build the MCP server
 
 ```bash
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cd mcp-server
 go build ./cmd/mcp-server
 ```
 
-This produces the binary `/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server`.
+This produces the binary `mcp-server/mcp-server` inside your clone; use its absolute path as the MCP `command` below.
 
 ---
 
@@ -357,7 +357,7 @@ This produces the binary `/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server
 If the user does not already have a Qdrant + TEI backend, start the local one:
 
 ```bash
-cd /Users/enowdev/Project/enowx-rag/mcp-server
+cd mcp-server
 docker compose up -d qdrant tei-embedding
 ```
 
@@ -682,7 +682,7 @@ This project uses the `enowx-rag` MCP server for per-project memory.
 
 1. Summarize what you changed.
 2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`.
-3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically.
+3. No need to re-index files after each change: once indexed with `rag_index_project`, the directory is re-synced automatically before searches (`RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync.
 
 Keep chunks concise (one idea per chunk). Use metadata tags like `type:architecture`, `type:decision`, `type:api`, `type:bugfix`, `type:howto`, or `type:snippet`.
 ```
@@ -868,14 +868,15 @@ All configuration is via environment variables (or config file at `~/.enowx-rag/
 | `RAG_VECTOR_DIM` | `1024` | Embedding vector dimension (matches voyage-4 default). Override only if using a different model with a different dimension |
 | `RAG_RERANKER_MODEL` | *(empty)* | Reranker model name (e.g., `rerank-2.5`). When set and `RAG_VOYAGE_API_KEY` is available, reranking is enabled for search |
 | `RAG_ADMIN_TOKEN` | *(empty)* | Optional admin token. When set, all `/api/*` endpoints require `Authorization: Bearer <token>` header. When unset, no auth is required |
+| `RAG_AUTO_SYNC_INTERVAL` | `60s` | Directories indexed with `rag_index_project` are re-synced (incrementally) before a search once their index is older than this. A duration (`90s`, `5m`), a number of seconds, or `off` to disable; a tiny value like `1s` syncs before practically every search. Also settable as `auto_sync_interval` in `~/.enowx-rag/config.yaml` (the env var wins). The directories are recorded in `~/.enowx-rag/project_dirs.json` |
 | `RAG_CORS_ORIGIN` | *(empty)* | Optional CORS origin for the SSE event stream (`/api/events`). When set (e.g. `*` or `https://app.example.com`), it becomes the `Access-Control-Allow-Origin` header. When unset, no CORS header is sent and the stream stays same-origin only |
 
 ## Tools
 
 - `rag_create_project` — create a project collection
 - `rag_delete_project` — delete a project collection
 - `rag_index` — index individual documents into a project
-- `rag_index_project` — scan a directory and auto-index all code/text files. Handles insertions and deletions (removed files are purged from RAG). Skips node_modules, .git, vendor, dist, build.
+- `rag_index_project` — scan a directory and sync all code/text files: new and edited code is indexed, code removed from a file and removed files are purged. Honors `.gitignore`; skips node_modules, .git, vendor, dist, build and secrets. Run it once per directory: the directory is remembered and re-synced automatically before searches (see `RAG_AUTO_SYNC_INTERVAL`).
 - `rag_semantic_search` — semantic search across project memory
 - `rag_retrieve_context` — compact context string for LLM consumption
 

mcp-server/cmd/mcp-server/main.go

@@ -67,6 +67,9 @@ type RuntimeConfig struct {
 	OpenAIDim     int
 	RerankerModel string
 	VectorDim     int
+	// AutoSyncInterval is the raw auto-sync setting (config file
+	// auto_sync_interval, or RAG_AUTO_SYNC_INTERVAL); see core.ParseAutoSyncInterval.
+	AutoSyncInterval string
 }
 
 // resolveConfig builds a RuntimeConfig from the three-tier priority:
@@ -95,6 +98,8 @@ func resolveConfig() (*RuntimeConfig, error) {
 		OpenAIDim:     cfg.OpenAI.Dim,
 		RerankerModel: cfg.RerankerModel,
 		VectorDim:     cfg.Voyage.Dim,
+
+		AutoSyncInterval: cfg.AutoSyncInterval,
 	}
 
 	// Fall back to tei if voyage key is missing and embedder wasn't set
@@ -254,6 +259,21 @@ func main() {
 	svc.SetEmbedModel(cfg.VoyageModel)
 	svc.SetBackend(cfg.VectorStore)
 
+	// Auto-sync: directories indexed with rag_index_project are re-synced
+	// (incrementally) before a search once their index is older than the
+	// auto-sync interval (config auto_sync_interval or RAG_AUTO_SYNC_INTERVAL;
+	// default 60s; "off" disables), so agents don't have to re-run
+	// rag_index_project after every change.
+	if every, err := core.ParseAutoSyncInterval(cfg.AutoSyncInterval); err != nil {
+		fmt.Fprintf(os.Stderr, "auto-sync: disabled: %v\n", err)
+	} else if every > 0 {
+		if reg, err := core.NewDirRegistry(config.ProjectDirsPath()); err != nil {
+			fmt.Fprintf(os.Stderr, "auto-sync: disabled: %v\n", err)
+		} else {
+			svc.EnableAutoSync(reg, every)
+		}
+	}
+
 	// Durable metrics: a local SQLite file (pure-Go, no cgo) that works with any
 	// vector-store backend. If it can't be opened, fall back to in-memory
 	// metrics rather than failing startup.
@@ -391,7 +411,7 @@ func registerMCPTools(server *mcp.Server, svc *core.Service) {
 
 	mcp.AddTool(server, &mcp.Tool{
 		Name:        "rag_index_project",
-		Description: "Scan a project directory and auto-index all code/text files into RAG. Handles insertions (new/changed files) and deletions (removed files). Skips node_modules, .git, vendor, dist, build. Run this when the codebase changes.",
+		Description: "Scan a project directory and index all code/text files into RAG, syncing it: new and changed code is indexed, code removed from files and removed files are deleted. Honors .gitignore; skips node_modules, .git, vendor, dist, build and secret files. Run it once per project directory: the directory is remembered and re-synced automatically before searches (RAG_AUTO_SYNC_INTERVAL, default 60s). Run it again to force an immediate sync.",
 	}, func(ctx context.Context, req *mcp.CallToolRequest, in ScanProjectInput) (*mcp.CallToolResult, any, error) {
 		// Use a long-lived context so large projects don't time out under the
 		// MCP framework's short request deadline. 30 minutes is enough for
@@ -406,6 +426,7 @@ func registerMCPTools(server *mcp.Server, svc *core.Service) {
 			"status":         "synced",
 			"project_id":     in.ProjectID,
 			"chunks_indexed": result.Indexed,
+			"chunks_updated": result.Updated,
 			"points_deleted": result.Deleted,
 			"files_scanned":  result.FilesScanned,
 			"stale_error":    result.StaleError,

mcp-server/pkg/config/config.go

@@ -49,6 +49,10 @@ type Config struct {
 	ChromaURL     string `yaml:"chroma_url"`
 	TEIURL        string `yaml:"tei_url"`
 	RerankerModel string `yaml:"reranker_model"`
+	// AutoSyncInterval is how stale an indexed directory may get before a
+	// search re-syncs it: a duration ("60s", "5m"), seconds ("30"), or "off".
+	// Empty means the built-in default (60s). Env: RAG_AUTO_SYNC_INTERVAL.
+	AutoSyncInterval string `yaml:"auto_sync_interval,omitempty"`
 	// AdminToken, when set, gates /api and /mcp with a bearer token. The env var
 	// RAG_ADMIN_TOKEN takes precedence over this file value (see EffectiveAdminToken).
 	AdminToken string `yaml:"admin_token,omitempty"`
@@ -112,6 +116,13 @@ func MetricsDBPath() string {
 	return filepath.Join(home, ".enowx-rag", "metrics.db")
 }
 
+// ProjectDirsPath is where the directories indexed into each project are
+// recorded for auto-sync (~/.enowx-rag/project_dirs.json), next to the
+// metrics database.
+func ProjectDirsPath() string {
+	return filepath.Join(filepath.Dir(MetricsDBPath()), "project_dirs.json")
+}
+
 // Load reads the config file from Path() and returns a populated *Config.
 // If the file does not exist, Load returns an error (this triggers the
 // onboarding wizard in HTTP mode). Env var overrides are applied on top of
@@ -232,6 +243,9 @@ func applyEnvOverrides(cfg *Config) {
 	if v := os.Getenv("RAG_RERANKER_MODEL"); v != "" {
 		cfg.RerankerModel = v
 	}
+	if v := os.Getenv("RAG_AUTO_SYNC_INTERVAL"); v != "" {
+		cfg.AutoSyncInterval = v
+	}
 	if v := os.Getenv("RAG_VECTOR_DIM"); v != "" {
 		if d, err := strconv.Atoi(v); err == nil && d > 0 {
 			cfg.Voyage.Dim = d

mcp-server/pkg/config/config_test.go

@@ -35,7 +35,7 @@ func helperClearEnv(t *testing.T) func() {
 		"RAG_PGVECTOR_DSN",
 		"RAG_TEI_URL",
 		"RAG_VOYAGE_API_KEY", "RAG_VOYAGE_MODEL", "RAG_VECTOR_DIM",
-		"RAG_RERANKER_MODEL",
+		"RAG_RERANKER_MODEL", "RAG_AUTO_SYNC_INTERVAL",
 	}
 	saved := make(map[string]string)
 	for _, k := range keys {
@@ -550,3 +550,34 @@ func TestSave_OverwritesExisting(t *testing.T) {
 		t.Errorf("Voyage.APIKey = %q, want %q", loaded.Voyage.APIKey, "second-key")
 	}
 }
+
+// auto_sync_interval comes from the file, the env var wins, and it
+// survives a Save/Load round trip (the dashboard rewrites the file).
+func TestAutoSyncInterval_FileEnvAndRoundTrip(t *testing.T) {
+	cleanup := helperClearEnv(t)
+	defer cleanup()
+	tmpDir := t.TempDir()
+	t.Setenv("HOME", tmpDir)
+	if err := os.MkdirAll(filepath.Join(tmpDir, ".enowx-rag"), 0o700); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(Path(), []byte("vector_store: sqlite\nauto_sync_interval: 5s\n"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+
+	cfg, err := Resolve()
+	if err != nil || cfg.AutoSyncInterval != "5s" {
+		t.Fatalf("from file: %q, %v", cfg.AutoSyncInterval, err)
+	}
+	if err := Save(cfg); err != nil {
+		t.Fatal(err)
+	}
+	if again, err := Load(); err != nil || again.AutoSyncInterval != "5s" {
+		t.Fatalf("after round trip: %+v, %v", again, err)
+	}
+
+	t.Setenv("RAG_AUTO_SYNC_INTERVAL", "off")
+	if cfg, err := Resolve(); err != nil || cfg.AutoSyncInterval != "off" {
+		t.Fatalf("env override: %q, %v", cfg.AutoSyncInterval, err)
+	}
+}