diff --git a/AGENTS.md b/AGENTS.md index f7cfa34..022288c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,14 +23,14 @@ Copy the skill into the chosen directory, e.g. (global Factory Droid): ```bash mkdir -p ~/.factory/skills/enowx-rag -cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md +cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md ``` Or project-local: ```bash mkdir -p .agents/skills/enowx-rag -cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md .agents/skills/enowx-rag/skill.md +cp skill/enowx-rag.md .agents/skills/enowx-rag/skill.md ``` If the tool requires a reload, restart the session or run any skill-refresh command available. After installing, follow the setup wizard in `skill/enowx-rag.md` whenever the user asks for RAG setup. @@ -40,7 +40,7 @@ If the tool requires a reload, restart the session or run any skill-refresh comm The MCP server is written in Go and lives in `mcp-server/`. ```bash -cd /Users/enowdev/Project/enowx-rag/mcp-server +cd mcp-server go build ./cmd/mcp-server ``` @@ -51,7 +51,7 @@ go build ./cmd/mcp-server **Option B — Self-hosted TEI:** Start Qdrant + TEI with Docker: ```bash -cd /Users/enowdev/Project/enowx-rag/mcp-server +cd mcp-server docker compose up -d qdrant tei-embedding ``` diff --git a/CHANGELOG.md b/CHANGELOG.md index ddba061..26c7347 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- **Auto-sync before search**: a directory indexed once with + `rag_index_project` is recorded (`~/.enowx-rag/project_dirs.json`) and + re-synced incrementally before a search when its index is older than + `RAG_AUTO_SYNC_INTERVAL` or `auto_sync_interval` in config.yaml (default + `60s`, `off` disables). Agents no longer + need to call `rag_index_project` after every task; the agent templates say + so. A recorded directory that no longer exists is skipped, not synced, and + a project deleted elsewhere is not recreated. The registry file is shared + safely by several processes (re-read before every read and change). +- **Big classes split at their methods**: a declaration too long for one + chunk is cut at its members (Python, Rust, Ruby, Java/C#, Kotlin, Swift, + PHP, JS/TS class methods; decorators and annotations kept with their + method, closures kept inside it), each chunk's symbol `Class.method`, + before falling back to blank lines. +- `rag.PayloadUpdater` (implemented by the SQLite store): rewrites a point's + content and metadata, keeping its vector. - **Built-in store (`vector_store: sqlite`)**: one SQLite file (pure Go, no cgo; `sqlite_path`, default `~/.enowx-rag/rag.db`, or `RAG_SQLITE_PATH`), vectors searched by cosine, text in FTS5 for hybrid search (RRF), so @@ -21,6 +37,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 indexes re-embed once). - `Indexer.Files` lets a program embedding enowx-rag pass the files to index. +### Fixed +- **Stale chunks after editing a file**: a sync deleted chunks only when + their file was gone, so code removed from a file that still exists stayed + in the index and was served as current. A sync now deletes every chunk of + the directory it did not produce again (also replacing chunks written by + older chunk layouts). +- **Re-embedding everything below an edit**: chunk IDs were their position + in the file (`#chunkN`), so adding a function shifted the IDs of all the + chunks after it. IDs are now derived from the chunk's content + (`dir/file#`, `chunk_version` v4; existing indexes re-embed once); + only small declarations are packed together, so an edit changes only the + chunks around it; a chunk that only moved gets its line numbers refreshed + without re-embedding where the store supports it (SQLite), re-indexed + otherwise. +- `rag_index_project` on a directory that does not exist is an error instead + of an empty scan that deleted everything indexed from it. +- A sync only ever deletes chunks under its own directory's ID prefix, even + when the store returns other points from `ListPoints` (pgvector ignores the + `source_dir` filter, so syncing one directory deleted another's chunks). +- Indexing a second directory with the same folder name as one already in + the project (`/x/app`, `/y/app`) is refused instead of silently replacing + its chunks. +- A file that exists but can't be read keeps its chunks instead of losing + them to the stale cleanup. +- **Chroma**: points are written with `/upsert` instead of `/add`, which + skipped IDs already stored, so edited chunks were never updated. + ### Security - Secrets are never indexed: `.env*`, keys and certificates, `credentials` and `*secret*` files by name, and any chunk holding a private key block or a diff --git a/CLAUDE.md b/CLAUDE.md index 3f949d1..e7709a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,15 +8,15 @@ If the user asks to install the skill or MCP server, run: ```bash mkdir -p ~/.factory/skills/enowx-rag -cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md -cd /Users/enowdev/Project/enowx-rag/mcp-server +cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md +cd mcp-server go build ./cmd/mcp-server ``` For local backend (TEI fallback): ```bash -cd /Users/enowdev/Project/enowx-rag/mcp-server +cd mcp-server docker compose up -d qdrant tei-embedding ``` @@ -28,7 +28,7 @@ docker compose up -d qdrant tei-embedding { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -46,7 +46,7 @@ docker compose up -d qdrant tei-embedding { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_EMBEDDER": "tei", diff --git a/README.md b/README.md index 2e26c29..bac1336 100644 --- a/README.md +++ b/README.md @@ -334,7 +334,7 @@ Copy the skill file into the Droid skill directory: ```bash mkdir -p ~/.factory/skills/enowx-rag -cp /Users/enowdev/Project/enowx-rag/skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md +cp skill/enowx-rag.md ~/.factory/skills/enowx-rag/skill.md ``` If the skill must be reloaded, restart the Droid session or refresh skills. @@ -344,11 +344,11 @@ If the skill must be reloaded, restart the Droid session or refresh skills. ## 2. Build the MCP server ```bash -cd /Users/enowdev/Project/enowx-rag/mcp-server +cd mcp-server go build ./cmd/mcp-server ``` -This produces the binary `/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server`. +This produces the binary `mcp-server/mcp-server` inside your clone; use its absolute path as the MCP `command` below. --- @@ -357,7 +357,7 @@ This produces the binary `/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server If the user does not already have a Qdrant + TEI backend, start the local one: ```bash -cd /Users/enowdev/Project/enowx-rag/mcp-server +cd mcp-server docker compose up -d qdrant tei-embedding ``` @@ -682,7 +682,7 @@ This project uses the `enowx-rag` MCP server for per-project memory. 1. Summarize what you changed. 2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`. -3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically. +3. No need to re-index files after each change: once indexed with `rag_index_project`, the directory is re-synced automatically before searches (`RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync. Keep chunks concise (one idea per chunk). Use metadata tags like `type:architecture`, `type:decision`, `type:api`, `type:bugfix`, `type:howto`, or `type:snippet`. ``` @@ -868,6 +868,7 @@ All configuration is via environment variables (or config file at `~/.enowx-rag/ | `RAG_VECTOR_DIM` | `1024` | Embedding vector dimension (matches voyage-4 default). Override only if using a different model with a different dimension | | `RAG_RERANKER_MODEL` | *(empty)* | Reranker model name (e.g., `rerank-2.5`). When set and `RAG_VOYAGE_API_KEY` is available, reranking is enabled for search | | `RAG_ADMIN_TOKEN` | *(empty)* | Optional admin token. When set, all `/api/*` endpoints require `Authorization: Bearer ` header. When unset, no auth is required | +| `RAG_AUTO_SYNC_INTERVAL` | `60s` | Directories indexed with `rag_index_project` are re-synced (incrementally) before a search once their index is older than this. A duration (`90s`, `5m`), a number of seconds, or `off` to disable; a tiny value like `1s` syncs before practically every search. Also settable as `auto_sync_interval` in `~/.enowx-rag/config.yaml` (the env var wins). The directories are recorded in `~/.enowx-rag/project_dirs.json` | | `RAG_CORS_ORIGIN` | *(empty)* | Optional CORS origin for the SSE event stream (`/api/events`). When set (e.g. `*` or `https://app.example.com`), it becomes the `Access-Control-Allow-Origin` header. When unset, no CORS header is sent and the stream stays same-origin only | ## Tools @@ -875,7 +876,7 @@ All configuration is via environment variables (or config file at `~/.enowx-rag/ - `rag_create_project` — create a project collection - `rag_delete_project` — delete a project collection - `rag_index` — index individual documents into a project -- `rag_index_project` — scan a directory and auto-index all code/text files. Handles insertions and deletions (removed files are purged from RAG). Skips node_modules, .git, vendor, dist, build. +- `rag_index_project` — scan a directory and sync all code/text files: new and edited code is indexed, code removed from a file and removed files are purged. Honors `.gitignore`; skips node_modules, .git, vendor, dist, build and secrets. Run it once per directory: the directory is remembered and re-synced automatically before searches (see `RAG_AUTO_SYNC_INTERVAL`). - `rag_semantic_search` — semantic search across project memory - `rag_retrieve_context` — compact context string for LLM consumption diff --git a/mcp-server/cmd/mcp-server/main.go b/mcp-server/cmd/mcp-server/main.go index baebfc0..5cd924f 100644 --- a/mcp-server/cmd/mcp-server/main.go +++ b/mcp-server/cmd/mcp-server/main.go @@ -67,6 +67,9 @@ type RuntimeConfig struct { OpenAIDim int RerankerModel string VectorDim int + // AutoSyncInterval is the raw auto-sync setting (config file + // auto_sync_interval, or RAG_AUTO_SYNC_INTERVAL); see core.ParseAutoSyncInterval. + AutoSyncInterval string } // resolveConfig builds a RuntimeConfig from the three-tier priority: @@ -95,6 +98,8 @@ func resolveConfig() (*RuntimeConfig, error) { OpenAIDim: cfg.OpenAI.Dim, RerankerModel: cfg.RerankerModel, VectorDim: cfg.Voyage.Dim, + + AutoSyncInterval: cfg.AutoSyncInterval, } // Fall back to tei if voyage key is missing and embedder wasn't set @@ -254,6 +259,21 @@ func main() { svc.SetEmbedModel(cfg.VoyageModel) svc.SetBackend(cfg.VectorStore) + // Auto-sync: directories indexed with rag_index_project are re-synced + // (incrementally) before a search once their index is older than the + // auto-sync interval (config auto_sync_interval or RAG_AUTO_SYNC_INTERVAL; + // default 60s; "off" disables), so agents don't have to re-run + // rag_index_project after every change. + if every, err := core.ParseAutoSyncInterval(cfg.AutoSyncInterval); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync: disabled: %v\n", err) + } else if every > 0 { + if reg, err := core.NewDirRegistry(config.ProjectDirsPath()); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync: disabled: %v\n", err) + } else { + svc.EnableAutoSync(reg, every) + } + } + // Durable metrics: a local SQLite file (pure-Go, no cgo) that works with any // vector-store backend. If it can't be opened, fall back to in-memory // metrics rather than failing startup. @@ -391,7 +411,7 @@ func registerMCPTools(server *mcp.Server, svc *core.Service) { mcp.AddTool(server, &mcp.Tool{ Name: "rag_index_project", - Description: "Scan a project directory and auto-index all code/text files into RAG. Handles insertions (new/changed files) and deletions (removed files). Skips node_modules, .git, vendor, dist, build. Run this when the codebase changes.", + Description: "Scan a project directory and index all code/text files into RAG, syncing it: new and changed code is indexed, code removed from files and removed files are deleted. Honors .gitignore; skips node_modules, .git, vendor, dist, build and secret files. Run it once per project directory: the directory is remembered and re-synced automatically before searches (RAG_AUTO_SYNC_INTERVAL, default 60s). Run it again to force an immediate sync.", }, func(ctx context.Context, req *mcp.CallToolRequest, in ScanProjectInput) (*mcp.CallToolResult, any, error) { // Use a long-lived context so large projects don't time out under the // MCP framework's short request deadline. 30 minutes is enough for @@ -406,6 +426,7 @@ func registerMCPTools(server *mcp.Server, svc *core.Service) { "status": "synced", "project_id": in.ProjectID, "chunks_indexed": result.Indexed, + "chunks_updated": result.Updated, "points_deleted": result.Deleted, "files_scanned": result.FilesScanned, "stale_error": result.StaleError, diff --git a/mcp-server/pkg/config/config.go b/mcp-server/pkg/config/config.go index 650b88b..032d18b 100644 --- a/mcp-server/pkg/config/config.go +++ b/mcp-server/pkg/config/config.go @@ -49,6 +49,10 @@ type Config struct { ChromaURL string `yaml:"chroma_url"` TEIURL string `yaml:"tei_url"` RerankerModel string `yaml:"reranker_model"` + // AutoSyncInterval is how stale an indexed directory may get before a + // search re-syncs it: a duration ("60s", "5m"), seconds ("30"), or "off". + // Empty means the built-in default (60s). Env: RAG_AUTO_SYNC_INTERVAL. + AutoSyncInterval string `yaml:"auto_sync_interval,omitempty"` // AdminToken, when set, gates /api and /mcp with a bearer token. The env var // RAG_ADMIN_TOKEN takes precedence over this file value (see EffectiveAdminToken). AdminToken string `yaml:"admin_token,omitempty"` @@ -112,6 +116,13 @@ func MetricsDBPath() string { return filepath.Join(home, ".enowx-rag", "metrics.db") } +// ProjectDirsPath is where the directories indexed into each project are +// recorded for auto-sync (~/.enowx-rag/project_dirs.json), next to the +// metrics database. +func ProjectDirsPath() string { + return filepath.Join(filepath.Dir(MetricsDBPath()), "project_dirs.json") +} + // Load reads the config file from Path() and returns a populated *Config. // If the file does not exist, Load returns an error (this triggers the // onboarding wizard in HTTP mode). Env var overrides are applied on top of @@ -232,6 +243,9 @@ func applyEnvOverrides(cfg *Config) { if v := os.Getenv("RAG_RERANKER_MODEL"); v != "" { cfg.RerankerModel = v } + if v := os.Getenv("RAG_AUTO_SYNC_INTERVAL"); v != "" { + cfg.AutoSyncInterval = v + } if v := os.Getenv("RAG_VECTOR_DIM"); v != "" { if d, err := strconv.Atoi(v); err == nil && d > 0 { cfg.Voyage.Dim = d diff --git a/mcp-server/pkg/config/config_test.go b/mcp-server/pkg/config/config_test.go index 5af3410..eb23425 100644 --- a/mcp-server/pkg/config/config_test.go +++ b/mcp-server/pkg/config/config_test.go @@ -35,7 +35,7 @@ func helperClearEnv(t *testing.T) func() { "RAG_PGVECTOR_DSN", "RAG_TEI_URL", "RAG_VOYAGE_API_KEY", "RAG_VOYAGE_MODEL", "RAG_VECTOR_DIM", - "RAG_RERANKER_MODEL", + "RAG_RERANKER_MODEL", "RAG_AUTO_SYNC_INTERVAL", } saved := make(map[string]string) for _, k := range keys { @@ -550,3 +550,34 @@ func TestSave_OverwritesExisting(t *testing.T) { t.Errorf("Voyage.APIKey = %q, want %q", loaded.Voyage.APIKey, "second-key") } } + +// auto_sync_interval comes from the file, the env var wins, and it +// survives a Save/Load round trip (the dashboard rewrites the file). +func TestAutoSyncInterval_FileEnvAndRoundTrip(t *testing.T) { + cleanup := helperClearEnv(t) + defer cleanup() + tmpDir := t.TempDir() + t.Setenv("HOME", tmpDir) + if err := os.MkdirAll(filepath.Join(tmpDir, ".enowx-rag"), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(Path(), []byte("vector_store: sqlite\nauto_sync_interval: 5s\n"), 0o600); err != nil { + t.Fatal(err) + } + + cfg, err := Resolve() + if err != nil || cfg.AutoSyncInterval != "5s" { + t.Fatalf("from file: %q, %v", cfg.AutoSyncInterval, err) + } + if err := Save(cfg); err != nil { + t.Fatal(err) + } + if again, err := Load(); err != nil || again.AutoSyncInterval != "5s" { + t.Fatalf("after round trip: %+v, %v", again, err) + } + + t.Setenv("RAG_AUTO_SYNC_INTERVAL", "off") + if cfg, err := Resolve(); err != nil || cfg.AutoSyncInterval != "off" { + t.Fatalf("env override: %q, %v", cfg.AutoSyncInterval, err) + } +} diff --git a/mcp-server/pkg/core/autosync.go b/mcp-server/pkg/core/autosync.go new file mode 100644 index 0000000..1fcdc8f --- /dev/null +++ b/mcp-server/pkg/core/autosync.go @@ -0,0 +1,288 @@ +package core + +import ( + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +// DefaultAutoSyncInterval is how stale a project's index may get before a +// search re-syncs it from disk. +const DefaultAutoSyncInterval = 60 * time.Second + +// autoSyncTimeout bounds how long one search waits for a re-sync. A sync +// cut short keeps what it indexed (unchanged chunks are skipped next time), +// so a big change is caught up over the next searches. +const autoSyncTimeout = 30 * time.Second + +// DirRegistry remembers which directories were indexed into which project, +// in a JSON file, so a later process (every stdio session is one) can keep +// those directories in sync without being told again. Several processes may +// share the file (agent sessions, the dashboard), so it is re-read before +// every read and every change, and written atomically. +type DirRegistry struct { + mu sync.Mutex + path string + dirs map[string][]string // project → absolute directories +} + +// NewDirRegistry opens the registry stored at path (made on first write). +func NewDirRegistry(path string) (*DirRegistry, error) { + r := &DirRegistry{path: path, dirs: map[string][]string{}} + if err := r.load(); err != nil { + return nil, err + } + return r, nil +} + +// load re-reads the file, so changes by other processes are seen. A missing +// file is an empty registry. Caller holds mu (or owns r). +func (r *DirRegistry) load() error { + raw, err := os.ReadFile(r.path) + if os.IsNotExist(err) { + r.dirs = map[string][]string{} + return nil + } + if err != nil { + return fmt.Errorf("project dirs: %w", err) + } + dirs := map[string][]string{} + if len(raw) > 0 { + if err := json.Unmarshal(raw, &dirs); err != nil { + return fmt.Errorf("project dirs %s: %w", r.path, err) + } + } + r.dirs = dirs + return nil +} + +// Add records dir (made absolute) as indexed into project. +func (r *DirRegistry) Add(project, dir string) error { + abs, err := filepath.Abs(dir) + if err != nil { + return err + } + r.mu.Lock() + defer r.mu.Unlock() + if err := r.load(); err != nil { + return err + } + for _, d := range r.dirs[project] { + if d == abs { + return nil + } + } + r.dirs[project] = append(r.dirs[project], abs) + sort.Strings(r.dirs[project]) + return r.save() +} + +// Remove forgets every directory of project. +func (r *DirRegistry) Remove(project string) error { + r.mu.Lock() + defer r.mu.Unlock() + if err := r.load(); err != nil { + return err + } + if _, ok := r.dirs[project]; !ok { + return nil + } + delete(r.dirs, project) + return r.save() +} + +// Dirs returns the directories indexed into project, as the file has them +// now (if it can't be read, as last read). +func (r *DirRegistry) Dirs(project string) []string { + r.mu.Lock() + defer r.mu.Unlock() + if err := r.load(); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync: %v\n", err) + } + return append([]string(nil), r.dirs[project]...) +} + +// save writes the registry atomically (temp file + rename). Caller holds mu. +func (r *DirRegistry) save() error { + if err := os.MkdirAll(filepath.Dir(r.path), 0o700); err != nil { + return err + } + raw, err := json.MarshalIndent(r.dirs, "", " ") + if err != nil { + return err + } + tmp, err := os.CreateTemp(filepath.Dir(r.path), ".project_dirs-*.json") + if err != nil { + return err + } + defer os.Remove(tmp.Name()) + if _, err := tmp.Write(raw); err != nil { + tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmp.Name(), r.path) +} + +// autoSync holds the Service's auto-sync state. +type autoSync struct { + registry *DirRegistry + every time.Duration + + mu sync.Mutex + last map[string]time.Time // project\x00dir → last sync + locks map[string]*sync.Mutex // project\x00dir → one sync at a time +} + +// EnableAutoSync makes the Service keep indexed directories in sync: every +// directory indexed with IndexProject is recorded in registry, and a search +// on a project first re-syncs its directories whose last sync is older than +// every (incremental: only changed chunks are embedded). every <= 0, or a nil +// registry, disables it. +func (s *Service) EnableAutoSync(registry *DirRegistry, every time.Duration) { + if registry == nil || every <= 0 { + s.syncer = nil + return + } + s.syncer = &autoSync{registry: registry, every: every, + last: map[string]time.Time{}, locks: map[string]*sync.Mutex{}} +} + +// lock returns the mutex serializing syncs of one project directory. +func (a *autoSync) lock(key string) *sync.Mutex { + a.mu.Lock() + defer a.mu.Unlock() + l, ok := a.locks[key] + if !ok { + l = &sync.Mutex{} + a.locks[key] = l + } + return l +} + +func (a *autoSync) lastSync(key string) time.Time { + a.mu.Lock() + defer a.mu.Unlock() + return a.last[key] +} + +func (a *autoSync) markSynced(key string) { + a.mu.Lock() + a.last[key] = time.Now() + a.mu.Unlock() +} + +func syncKey(project, dir string) string { return project + "\x00" + dir } + +// checkDirNameClash returns an error when project already has a recorded +// directory with dir's base name but another path (see IndexProject). +func (s *Service) checkDirNameClash(project, dir string) error { + if s.syncer == nil { + return nil + } + abs, err := filepath.Abs(dir) + if err != nil { + return nil + } + for _, d := range s.syncer.registry.Dirs(project) { + if d != abs && filepath.Base(d) == filepath.Base(abs) { + return fmt.Errorf("project %q already indexes %s, which has the same folder name as %s: "+ + "their chunks would replace each other; index it into a separate project", project, d, abs) + } + } + return nil +} + +// recordIndexed notes a successful IndexProject of dir into project. +func (s *Service) recordIndexed(project, dir string) { + if s.syncer == nil { + return + } + abs, err := filepath.Abs(dir) + if err != nil { + return + } + if err := s.syncer.registry.Add(project, abs); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync: could not record %s: %v\n", abs, err) + } + s.syncer.markSynced(syncKey(project, abs)) +} + +// autoSyncProject re-syncs project's recorded directories that are due, +// before a search reads the index. Failures never fail the search: they are +// logged, and the directory is tried again after the interval. Skipped, never +// synced: a directory that no longer exists (that would wipe its index) and a +// project that was deleted (that would bring it back). Directories of one +// project sharing a base name are refused by IndexProject itself. +func (s *Service) autoSyncProject(ctx context.Context, project string) { + a := s.syncer + if a == nil { + return + } + checkedExists := false + for _, dir := range a.registry.Dirs(project) { + key := syncKey(project, dir) + l := a.lock(key) + l.Lock() + if time.Since(a.lastSync(key)) < a.every { + l.Unlock() + continue + } + if info, err := os.Stat(dir); err != nil || !info.IsDir() { + a.markSynced(key) // don't stat it on every search + l.Unlock() + continue + } + if !checkedExists { + if !s.ProjectExists(ctx, project) { + l.Unlock() + return // deleted (maybe by another process): don't recreate it + } + checkedExists = true + } + sctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), autoSyncTimeout) + if _, err := s.IndexProject(sctx, project, dir); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync %s (%s): %v\n", project, dir, err) + } + cancel() + a.markSynced(key) + l.Unlock() + } +} + +// ParseAutoSyncInterval reads the auto-sync interval (config file +// auto_sync_interval, or env RAG_AUTO_SYNC_INTERVAL): a duration ("90s", +// "5m"), a whole number of seconds, or "0" / "off" / "false" to disable. +// Empty means DefaultAutoSyncInterval; anything else is an error. +func ParseAutoSyncInterval(v string) (time.Duration, error) { + v = strings.ToLower(strings.TrimSpace(v)) + switch v { + case "": + return DefaultAutoSyncInterval, nil + case "0", "off", "false", "no", "disabled": + return 0, nil + } + if n, err := strconv.Atoi(v); err == nil { + if n < 0 { + return 0, fmt.Errorf("auto-sync interval %q: negative", v) + } + return time.Duration(n) * time.Second, nil + } + d, err := time.ParseDuration(v) + if err != nil { + return 0, fmt.Errorf("auto-sync interval %q: want a duration like 60s, seconds, or off", v) + } + if d < 0 { + return 0, fmt.Errorf("auto-sync interval %q: negative", v) + } + return d, nil +} diff --git a/mcp-server/pkg/core/autosync_test.go b/mcp-server/pkg/core/autosync_test.go new file mode 100644 index 0000000..5dd00a3 --- /dev/null +++ b/mcp-server/pkg/core/autosync_test.go @@ -0,0 +1,294 @@ +package core + +import ( + "context" + "hash/fnv" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/enowdev/enowx-rag/mcp-server/pkg/indexer" + "github.com/enowdev/enowx-rag/mcp-server/pkg/rag" +) + +// wordEmbedder embeds texts as hashed word buckets: enough for a search on +// a distinctive word to find the chunk holding it. +type wordEmbedder struct{} + +func (wordEmbedder) Embed(_ context.Context, texts []string) ([][]float32, error) { + out := make([][]float32, len(texts)) + for i, t := range texts { + v := make([]float32, 128) + for _, w := range strings.FieldsFunc(strings.ToLower(t), func(r rune) bool { + return !(r >= 'a' && r <= 'z' || r >= '0' && r <= '9') + }) { + h := fnv.New32a() + h.Write([]byte(w)) + v[h.Sum32()%128]++ + } + out[i] = v + } + return out, nil +} + +func (wordEmbedder) VectorSize() int { return 128 } + +func newSyncService(t *testing.T, every time.Duration) (*Service, *DirRegistry, string) { + t.Helper() + p, err := rag.NewSQLiteProvider(filepath.Join(t.TempDir(), "rag.db"), wordEmbedder{}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { p.Close() }) + svc := NewService(p, nil, indexer.NewIndexer(p, 1000)) + regPath := filepath.Join(t.TempDir(), "project_dirs.json") + reg, err := NewDirRegistry(regPath) + if err != nil { + t.Fatal(err) + } + svc.EnableAutoSync(reg, every) + return svc, reg, regPath +} + +func writeFile(t *testing.T, path, body string) { + t.Helper() + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } +} + +func searchHas(t *testing.T, svc *Service, query, want string) bool { + t.Helper() + res, err := svc.Search(context.Background(), "p", query, SearchOpts{K: 3}) + if err != nil { + t.Fatal(err) + } + for _, r := range res { + if strings.Contains(r.Content, want) { + return true + } + } + return false +} + +// After a directory is indexed once, a search sees later edits without the +// agent calling rag_index_project again, including code that was removed. +func TestAutoSync_SearchSeesEditsWithoutReindex(t *testing.T) { + svc, reg, regPath := newSyncService(t, time.Nanosecond) // always due + dir := t.TempDir() + file := filepath.Join(dir, "billing.go") + writeFile(t, file, "package billing\n\nfunc ChargeCard() {\n\t// legacy stripe charge flow\n}\n") + if _, err := svc.IndexProject(context.Background(), "p", dir); err != nil { + t.Fatal(err) + } + if got := reg.Dirs("p"); len(got) != 1 { + t.Fatalf("directory not recorded: %v", got) + } + + writeFile(t, file, "package billing\n\nfunc RefundInvoice() {\n\t// zebra refund flow\n}\n") + + if !searchHas(t, svc, "zebra refund", "RefundInvoice") { + t.Error("search did not see the new function") + } + if searchHas(t, svc, "legacy stripe charge", "ChargeCard") { + t.Error("search still returns the removed function") + } + + // The registry survives a restart (a new stdio session). + reopened, err := NewDirRegistry(regPath) + if err != nil { + t.Fatal(err) + } + if got := reopened.Dirs("p"); len(got) != 1 || got[0] != dir { + t.Errorf("registry not persisted: %v", got) + } +} + +// Within the interval no re-sync happens; with auto-sync off, never. +func TestAutoSync_RespectsIntervalAndOff(t *testing.T) { + for _, tc := range []struct { + name string + every time.Duration + }{{"interval not elapsed", time.Hour}, {"disabled", 0}} { + t.Run(tc.name, func(t *testing.T) { + svc, _, _ := newSyncService(t, tc.every) + dir := t.TempDir() + file := filepath.Join(dir, "a.go") + writeFile(t, file, "package a\n\nfunc Alpha() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", dir); err != nil { + t.Fatal(err) + } + writeFile(t, file, "package a\n\nfunc Quokka() {}\n") + if searchHas(t, svc, "quokka", "Quokka") { + t.Error("re-synced although not due / disabled") + } + }) + } +} + +// A recorded directory that disappeared is skipped: the search still works +// and the index is not wiped. +func TestAutoSync_MissingDirKeepsIndex(t *testing.T) { + svc, _, _ := newSyncService(t, time.Nanosecond) + parent := t.TempDir() + dir := filepath.Join(parent, "repo") + if err := os.Mkdir(dir, 0o755); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(dir, "a.go"), "package a\n\nfunc Pelican() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", dir); err != nil { + t.Fatal(err) + } + if err := os.RemoveAll(dir); err != nil { + t.Fatal(err) + } + if !searchHas(t, svc, "pelican", "Pelican") { + t.Error("index lost after its directory disappeared") + } +} + +// Deleting a project forgets its directories. +func TestAutoSync_DeleteProjectForgetsDirs(t *testing.T) { + svc, reg, _ := newSyncService(t, time.Hour) + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "a.go"), "package a\n\nfunc A() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", dir); err != nil { + t.Fatal(err) + } + if err := svc.DeleteProject(context.Background(), "p"); err != nil { + t.Fatal(err) + } + if got := reg.Dirs("p"); len(got) != 0 { + t.Errorf("dirs kept after delete: %v", got) + } +} + +func TestParseAutoSyncInterval(t *testing.T) { + for in, want := range map[string]time.Duration{ + "": DefaultAutoSyncInterval, " ": DefaultAutoSyncInterval, + "0": 0, "off": 0, "OFF": 0, "false": 0, + "90": 90 * time.Second, "90s": 90 * time.Second, "5m": 5 * time.Minute, + } { + got, err := ParseAutoSyncInterval(in) + if err != nil || got != want { + t.Errorf("ParseAutoSyncInterval(%q) = %v, %v; want %v", in, got, err, want) + } + } + for _, bad := range []string{"soon", "-5", "-1m"} { + if _, err := ParseAutoSyncInterval(bad); err == nil { + t.Errorf("ParseAutoSyncInterval(%q): want an error", bad) + } + } +} + +// Two processes sharing the registry file see each other's changes. +func TestDirRegistry_SharedBetweenProcesses(t *testing.T) { + path := filepath.Join(t.TempDir(), "project_dirs.json") + r1, err := NewDirRegistry(path) + if err != nil { + t.Fatal(err) + } + r2, err := NewDirRegistry(path) + if err != nil { + t.Fatal(err) + } + if err := r1.Add("one", "/tmp/one"); err != nil { + t.Fatal(err) + } + if err := r2.Add("two", "/tmp/two"); err != nil { + t.Fatal(err) + } + if got := r1.Dirs("two"); len(got) != 1 { + t.Errorf("r1 does not see r2's project: %v", got) + } + if err := r2.Remove("one"); err != nil { + t.Fatal(err) + } + if got := r1.Dirs("one"); len(got) != 0 { + t.Errorf("r1 still sees a project r2 removed: %v", got) + } + reread, _ := NewDirRegistry(path) + if len(reread.Dirs("two")) != 1 || len(reread.Dirs("one")) != 0 { + t.Errorf("file lost a change: one=%v two=%v", reread.Dirs("one"), reread.Dirs("two")) + } +} + +// A project deleted elsewhere (another process, the dashboard) is not +// recreated by a search that still has its directory registered. +func TestAutoSync_DeletedProjectNotRecreated(t *testing.T) { + svc, _, _ := newSyncService(t, time.Nanosecond) + dir := t.TempDir() + writeFile(t, filepath.Join(dir, "a.go"), "package a\n\nfunc Heron() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", dir); err != nil { + t.Fatal(err) + } + // Deleted behind this Service's back: the registry still lists the dir. + if err := svc.Provider().DeleteCollection(context.Background(), "p"); err != nil { + t.Fatal(err) + } + if _, err := svc.Search(context.Background(), "p", "heron", SearchOpts{K: 3}); err != nil { + t.Fatal(err) + } + if svc.ProjectExists(context.Background(), "p") { + t.Error("auto-sync recreated a deleted project") + } +} + +// A second directory with the same base name as one already indexed into +// the project is refused: their chunks share an ID prefix. +func TestIndexProject_RefusesSecondDirWithSameName(t *testing.T) { + svc, _, _ := newSyncService(t, time.Hour) + a := filepath.Join(t.TempDir(), "app") + b := filepath.Join(t.TempDir(), "app") + for _, d := range []string{a, b} { + if err := os.Mkdir(d, 0o755); err != nil { + t.Fatal(err) + } + } + writeFile(t, filepath.Join(a, "a.go"), "package a\n\nfunc Ibis() {}\n") + writeFile(t, filepath.Join(b, "b.go"), "package b\n\nfunc Kiwi() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", a); err != nil { + t.Fatal(err) + } + if _, err := svc.IndexProject(context.Background(), "p", b); err == nil { + t.Error("want an error for a second directory named app") + } + if !searchHas(t, svc, "ibis", "Ibis") { + t.Error("first directory's chunks lost") + } + // Re-indexing the same directory is fine. + if _, err := svc.IndexProject(context.Background(), "p", a); err != nil { + t.Errorf("re-index of the same directory refused: %v", err) + } +} + +// If the registry already holds two same-named directories for a project +// (recorded by an older version), auto-sync skips them rather than letting +// one replace the other's chunks on every search. +func TestAutoSync_SkipsDirsSharingABaseName(t *testing.T) { + svc, reg, _ := newSyncService(t, time.Nanosecond) + a := filepath.Join(t.TempDir(), "app") + b := filepath.Join(t.TempDir(), "app") + for _, d := range []string{a, b} { + if err := os.Mkdir(d, 0o755); err != nil { + t.Fatal(err) + } + } + writeFile(t, filepath.Join(a, "a.go"), "package a\n\nfunc Ibis() {}\n") + writeFile(t, filepath.Join(b, "b.go"), "package b\n\nfunc Kiwi() {}\n") + if _, err := svc.IndexProject(context.Background(), "p", a); err != nil { + t.Fatal(err) + } + if err := reg.Add("p", b); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(a, "a.go"), "package a\n\nfunc Ibis() {}\n\nfunc Narwhal() {}\n") + if searchHas(t, svc, "narwhal", "Narwhal") { + t.Error("auto-synced a directory whose base name is shared") + } + if !searchHas(t, svc, "ibis", "Ibis") { + t.Error("a directory's chunks were replaced by the other one's") + } +} diff --git a/mcp-server/pkg/core/service.go b/mcp-server/pkg/core/service.go index 8b4e5e9..0d0164f 100644 --- a/mcp-server/pkg/core/service.go +++ b/mcp-server/pkg/core/service.go @@ -6,6 +6,7 @@ package core import ( "context" "fmt" + "os" "strings" "sync" "time" @@ -108,6 +109,7 @@ type Service struct { metrics *Metrics metricsStore MetricsStore // optional durable metrics; nil = in-memory only backend string // vector store name (e.g. "qdrant"), set by main.go + syncer *autoSync // optional: keeps indexed directories in sync (EnableAutoSync) } // MetricsStore is an optional durable sink for query metrics, injected into @@ -264,6 +266,10 @@ func (s *Service) retrieveCandidates(ctx context.Context, projectID, query strin // // Defaults: K=5, Recall=40. func (s *Service) Search(ctx context.Context, projectID, query string, opts SearchOpts) ([]rag.Result, error) { + // Bring the project's indexed directories up to date first, when enabled + // and due, so results reflect the files as they are now. + s.autoSyncProject(ctx, projectID) + k := opts.K if k <= 0 { k = DefaultK @@ -418,6 +424,18 @@ func (s *Service) IndexProject(ctx context.Context, projectID, dir string) (*ind Data: map[string]any{"project_id": projectID, "directory": dir}, }) + // Chunk IDs are prefixed with the directory's base name, so two + // directories of one project with the same name (/x/app, /y/app) would + // replace each other's chunks. Refuse the second one. + if err := s.checkDirNameClash(projectID, dir); err != nil { + s.events.Publish(Event{ + Type: "index_failed", + Timestamp: time.Now(), + Data: map[string]any{"project_id": projectID, "error": err.Error()}, + }) + return nil, err + } + // Ensure the collection exists before indexing. The MCP flow calls // CreateProject first, but the HTTP reindex endpoint indexes a project // directly, so a first-time index would otherwise fail with a missing @@ -446,12 +464,15 @@ func (s *Service) IndexProject(ctx context.Context, projectID, dir string) (*ind return nil, err } + s.recordIndexed(projectID, dir) + s.events.Publish(Event{ Type: "index_completed", Timestamp: time.Now(), Data: map[string]any{ "project_id": projectID, "indexed": result.Indexed, + "updated": result.Updated, "deleted": result.Deleted, "files_scanned": result.FilesScanned, "skipped": result.Skipped, @@ -539,6 +560,11 @@ func (s *Service) DeleteProject(ctx context.Context, projectID string) error { if err := s.provider.DeleteCollection(ctx, projectID); err != nil { return err } + if s.syncer != nil { + if err := s.syncer.registry.Remove(projectID); err != nil { + fmt.Fprintf(os.Stderr, "auto-sync: could not forget %s: %v\n", projectID, err) + } + } s.events.Publish(Event{ Type: "project_deleted", Timestamp: time.Now(), diff --git a/mcp-server/pkg/httpapi/docs.go b/mcp-server/pkg/httpapi/docs.go index ffa3db0..e9bdee0 100644 --- a/mcp-server/pkg/httpapi/docs.go +++ b/mcp-server/pkg/httpapi/docs.go @@ -144,6 +144,11 @@ directory, or from the API: POST %s/api/projects//reindex { "directory": "/abs/path" } +Once indexed, the directory is remembered and re-synced automatically before +searches whose index is older than ` + "`auto_sync_interval`" + ` in config.yaml or +` + "`RAG_AUTO_SYNC_INTERVAL`" + ` (default 60s, ` + "`off`" + ` to disable): new and edited +code is indexed, removed code is deleted. + ## 5. Retrieve Ask your agent to use ` + "`rag_retrieve_context`" + `, or try the **Playground** in @@ -158,9 +163,11 @@ func docMCPTools(_, _ string) string { "## `rag_create_project`\nCreate a project collection. Input: `project_id`.\n\n" + "## `rag_index`\nIndex documents you pass directly. Input: `project_id`, `documents` " + "(each `{id?, content, meta?}`). Use for saving facts/decisions.\n\n" + - "## `rag_index_project`\nScan a directory and index all code/text files (insertions, edits, " + - "deletions handled incrementally; skips node_modules/.git/vendor/dist/build). " + - "Input: `project_id`, `directory`. Run this whenever the codebase changes.\n\n" + + "## `rag_index_project`\nScan a directory and sync all code/text files (new and edited code " + + "indexed, removed code and files deleted, incrementally; honors .gitignore, skips " + + "node_modules/.git/vendor/dist/build and secrets). Input: `project_id`, `directory`. Run it " + + "once per directory: it is then re-synced automatically before searches " + + "(`RAG_AUTO_SYNC_INTERVAL`, default 60s). Run it again to force an immediate sync.\n\n" + "## Read / search\n" + "## `rag_semantic_search`\nSearch a project. Input: `project_id`, `query`, `limit`, and " + "optionally `recall`, `hybrid`, `rerank`, `compress` (hybrid/rerank default on). Returns " + @@ -177,8 +184,9 @@ func docMCPTools(_, _ string) string { "re-index). Input: `project_id`, `point_ids`.\n\n" + "## `rag_delete_project`\nDelete a project collection and all its data. Input: `project_id`.\n\n" + "## `rag_stats`\nAggregate stats: projects, total chunks, embed model, latency, tokens. No input.\n\n" + - "**Typical loop:** `rag_list_projects` to see what exists → `rag_retrieve_context` before " + - "coding → do the work → `rag_index` new facts → `rag_index_project` to sync file changes." + "**Typical loop:** `rag_index_project` once per project directory → `rag_retrieve_context` " + + "before coding (file changes are synced automatically) → do the work → `rag_index` new facts " + + "and decisions." } func docAPIReference(base, _ string) string { diff --git a/mcp-server/pkg/httpapi/setup_probe.go b/mcp-server/pkg/httpapi/setup_probe.go index a7b861d..d7c5d1e 100644 --- a/mcp-server/pkg/httpapi/setup_probe.go +++ b/mcp-server/pkg/httpapi/setup_probe.go @@ -90,8 +90,9 @@ func agentsBlock(projectID string) string { This project uses the enowx-rag MCP server for per-project RAG memory. +- **First time**: if the project was never indexed, call `+"`rag_index_project`"+` once with the project directory. After that the server re-syncs it automatically before searches (`+"`RAG_AUTO_SYNC_INTERVAL`"+`, default 60s); call it again only to force a sync. - **Before coding**: call `+"`rag_retrieve_context`"+` with project ID `+"`%s`"+` and the user's query; use any relevant context. -- **After coding**: call `+"`rag_index`"+` with new facts/decisions/gotchas, then `+"`rag_index_project`"+` with the project directory to sync file changes. Keep chunks focused; tag with `+"`type:architecture|decision|api|bugfix|howto|snippet`"+`. +- **After coding**: call `+"`rag_index`"+` with new facts/decisions/gotchas (code changes are synced automatically; the reasons behind them are not). Keep chunks focused; tag with `+"`type:architecture|decision|api|bugfix|howto|snippet`"+`. - Each project has its own collection; do not mix project memories. %s`, agentsMarkerStart, projectID, projectID, agentsMarkerEnd) } diff --git a/mcp-server/pkg/indexer/chunks.go b/mcp-server/pkg/indexer/chunks.go index 17bcb61..bfd142a 100644 --- a/mcp-server/pkg/indexer/chunks.go +++ b/mcp-server/pkg/indexer/chunks.go @@ -43,30 +43,83 @@ var ( jsDecl = regexp.MustCompile(`^(?:export\s+)?(?:default\s+)?(?:async\s+)?(?:function\*?|class|interface|type|enum|const|let)\s+([A-Za-z_$][\w$]*)`) classDecl = regexp.MustCompile(`^(?:(?:public|private|protected|internal|static|final|abstract|open|override|data|sealed|export)\s+)*(?:class|interface|enum|struct|record|fun|func|function|object|protocol|extension)\s+([A-Za-z_]\w*)`) comment = regexp.MustCompile(`^\s*(?://|#|/\*|\*|--|"""|///)`) + // attached are the lines that belong to the declaration below them: + // comments, and decorators / annotations (@Override, @property, ...). + attached = regexp.MustCompile(`^\s*(?://|#|/\*|\*|--|"""|///|@)`) ) +// members find, by language, the declarations nested in a type: methods, +// nested classes. They match indented lines only, so never a top-level +// declaration. A declaration too long for one chunk (a big class) is split at +// its members before falling back to blank lines. Languages not listed (Go: +// its methods are top-level already) fall back to blank lines. +var members = map[string]*regexp.Regexp{ + ".py": regexp.MustCompile(`^\s+(?:async\s+)?(?:def|class)\s+([A-Za-z_]\w*)`), + ".rs": regexp.MustCompile(`^\s+(?:pub(?:\([\w:]+\))?\s+)?(?:(?:async|const|unsafe|extern\s+"\w+")\s+)*fn\s+([A-Za-z_]\w*)`), + ".rb": regexp.MustCompile(`^\s+def\s+([\w.?!=]+)`), + ".java": javaMember, + ".cs": javaMember, + ".kt": regexp.MustCompile(`^\s+(?:(?:public|private|protected|internal|override|open|abstract|final|suspend|inline|operator|infix|tailrec)\s+)*fun\s+(?:<[^>]*>\s*)?(?:[\w.]+\.)?([A-Za-z_]\w*)`), + ".swift": regexp.MustCompile(`^\s+(?:(?:public|private|fileprivate|internal|open|static|class|override|final|mutating|nonmutating|@\w+)\s+)*func\s+([A-Za-z_]\w*)`), + ".php": regexp.MustCompile(`^\s+(?:(?:public|private|protected|static|final|abstract)\s+)*function\s+&?([A-Za-z_]\w*)`), + ".js": jsMember, + ".jsx": jsMember, + ".ts": jsMember, + ".tsx": jsMember, +} + +var ( + // javaMember: a method or constructor with at least one modifier, so a + // call inside a method body ("foo(x);", "return bar(y)") never matches. + javaMember = regexp.MustCompile(`^\s+(?:@\w+\s+)*(?:(?:public|private|protected|internal|static|final|abstract|override|virtual|async|synchronized|sealed|partial|extern|unsafe|default)\s+)+(?:<[^>]+>\s+)?(?:[\w<>\[\],.?]+\s+)?([A-Za-z_]\w*)\s*\(`) + // jsMember: a class method whose signature fits on its line and opens its + // body: " async load(id: string): Promise {". Control-flow keywords + // that look the same (" if (x) {") are filtered out by notMember. + jsMember = regexp.MustCompile(`^\s+(?:(?:public|private|protected|static|async|readonly|override|abstract|get|set|declare)\s+)*\*?\s*(#?[A-Za-z_$][\w$]*)\s*(?:<[^>]*>)?\s*\([^)]*\)\s*(?::\s*[^={;]+)?\{\s*$`) + notMember = map[string]bool{"if": true, "for": true, "while": true, "switch": true, "catch": true, + "function": true, "return": true, "with": true, "do": true, "else": true, "try": true} +) + +// span is a range of lines (0-based, end excluded) and the symbol it holds. +type span struct { + start, end int + symbol string +} + // chunkCode cuts file path's text into chunks of at most maxChars, at its -// top-level declarations (with the comments right above them), several -// small ones together; a declaration too long is cut at blank lines, then -// at lines. A language enowX knows no declarations of is cut at blank -// lines. +// top-level declarations (with the comments right above them). Small +// declarations are packed together with the one that follows; a declaration +// too long is split at its members (methods), then at blank lines, then at +// lines. A language enowX knows no declarations of is cut at blank lines. func chunkCode(text, path string, maxChars int) []Chunk { lines := strings.Split(strings.ReplaceAll(text, "\r\n", "\n"), "\n") - decl := declarations[strings.ToLower(filepath.Ext(path))] - // Segments: line spans starting at a declaration (or a blank line when - // the language has none), comments above it pulled in. - type seg struct { - start, end int // 0-based, end excluded - symbol string - } - var segs []seg - cur := seg{} + ext := strings.ToLower(filepath.Ext(path)) + member := members[ext] + long := func(sg span) []span { return splitLong(lines, sg, maxChars, member) } + + var out []Chunk + for _, sp := range pack(lines, segments(lines, declarations[ext]), maxChars, long) { + body := strings.Join(lines[sp.start:sp.end], "\n") + if strings.TrimSpace(body) == "" { + continue + } + out = append(out, Chunk{Text: body, Start: sp.start + 1, End: sp.end, Symbol: sp.symbol}) + } + return out +} + +// segments are the line spans starting at a top-level declaration (or at a +// blank line when the language has no declarations known), the comments +// above a declaration pulled in. +func segments(lines []string, decl *regexp.Regexp) []span { + var segs []span + cur := span{} begin := func(i int, symbol string) { if i > cur.start { cur.end = i segs = append(segs, cur) } - cur = seg{start: i, symbol: symbol} + cur = span{start: i, symbol: symbol} } for i, line := range lines { if decl != nil { @@ -82,54 +135,122 @@ func chunkCode(text, path string, maxChars int) []Chunk { } } cur.end = len(lines) - segs = append(segs, cur) + return append(segs, cur) +} - size := func(s, e int) int { - n := 0 - for _, l := range lines[s:e] { - n += len(l) + 1 - } - return n - } - var out []Chunk - emit := func(s, e int, symbol string) { - body := strings.Join(lines[s:e], "\n") - if strings.TrimSpace(body) == "" { - return - } - out = append(out, Chunk{Text: body, Start: s + 1, End: e, Symbol: symbol}) - } - // Pack small segments together; cut long ones. - packS, packE, packSym := -1, -1, "" +// pack groups contiguous segments into chunk spans of at most maxChars: a +// small segment (under a quarter of the size: an import block, a const, a +// one-line type) joins the small ones next to it and the segment that +// follows; a segment of fair size stands alone; one too long goes to long. +// Packing only small segments keeps boundaries local: adding a function +// changes the chunks around it, not every chunk after it, so the others keep +// their content (and their IDs: nothing to re-embed). +func pack(lines []string, segs []span, maxChars int, long func(span) []span) []span { + small := maxChars / 4 + var out []span + cur := span{start: -1} flush := func() { - if packS >= 0 { - emit(packS, packE, packSym) - packS, packE, packSym = -1, -1, "" + if cur.start >= 0 { + out = append(out, cur) + cur = span{start: -1} } } for _, sg := range segs { - n := size(sg.start, sg.end) - if n > maxChars { + n := size(lines[sg.start:sg.end]) + switch { + case n > maxChars: flush() - for _, c := range cutLong(lines, sg.start, sg.end, maxChars) { - emit(c[0], c[1], sg.symbol) + out = append(out, long(sg)...) + case n < small: + if cur.start >= 0 && size(lines[cur.start:sg.end]) > maxChars { + flush() + } + if cur.start < 0 { + cur = sg + } else { + cur.end = sg.end + if cur.symbol == "" { + cur.symbol = sg.symbol + } + } + default: + if cur.start >= 0 && size(lines[cur.start:sg.end]) <= maxChars { + sym := sg.symbol // the small lead-in joins this declaration + if sym == "" { + sym = cur.symbol + } + out = append(out, span{start: cur.start, end: sg.end, symbol: sym}) + cur = span{start: -1} + continue } - continue - } - if packS >= 0 && size(packS, sg.end) > maxChars { flush() + out = append(out, sg) } - if packS < 0 { - packS, packSym = sg.start, sg.symbol - } else if packSym == "" { - packSym = sg.symbol - } - packE = sg.end } flush() return out } +// splitLong splits a segment too long for one chunk at its members (the +// shallowest ones: a closure inside a method stays in the method), each with +// the comments and decorators above it, symbol "Outer.member"; pieces are +// packed like top-level segments, and a piece still too long is cut at blank +// lines, then at lines. +func splitLong(lines []string, sg span, maxChars int, member *regexp.Regexp) []span { + byBlank := func(p span) []span { + var out []span + for _, c := range cutLong(lines, p.start, p.end, maxChars) { + out = append(out, span{start: c[0], end: c[1], symbol: p.symbol}) + } + return out + } + if member == nil { + return byBlank(sg) + } + type hit struct { + line, indent int + name string + } + var hits []hit + shallowest := -1 + for i := sg.start + 1; i < sg.end; i++ { + m := member.FindStringSubmatch(lines[i]) + if m == nil || notMember[m[len(m)-1]] { + continue + } + ind := len(lines[i]) - len(strings.TrimLeft(lines[i], " \t")) + hits = append(hits, hit{line: i, indent: ind, name: m[len(m)-1]}) + if shallowest < 0 || ind < shallowest { + shallowest = ind + } + } + pieces := []span{{start: sg.start, symbol: sg.symbol}} + for _, h := range hits { + if h.indent != shallowest { + continue + } + last := &pieces[len(pieces)-1] + at := h.line + for at > last.start+1 && attached.MatchString(lines[at-1]) { + at-- + } + if at <= last.start { + continue + } + last.end = at + name := h.name + if sg.symbol != "" { + name = sg.symbol + "." + name + } + pieces = append(pieces, span{start: at, symbol: name}) + } + pieces[len(pieces)-1].end = sg.end + if len(pieces) == 1 { + return byBlank(sg) + } + return pack(lines, pieces, maxChars, byBlank) +} + // cutLong cuts lines[s:e] into spans of at most maxChars, at blank lines // when it can, else at a line; a single line longer is cut alone. func cutLong(lines []string, s, e, maxChars int) [][2]int { diff --git a/mcp-server/pkg/indexer/chunks_member_test.go b/mcp-server/pkg/indexer/chunks_member_test.go new file mode 100644 index 0000000..f2d8dac --- /dev/null +++ b/mcp-server/pkg/indexer/chunks_member_test.go @@ -0,0 +1,154 @@ +package indexer + +import ( + "fmt" + "strings" + "testing" +) + +// body returns n indented filler lines at the given indent. +func body(indent string, n int, tag string) string { + var b strings.Builder + for i := 0; i < n; i++ { + fmt.Fprintf(&b, "%sx_%d = compute(%q, %d)\n", indent, i, tag, i) + } + return b.String() +} + +func symbols(chunks []Chunk) []string { + var out []string + for _, c := range chunks { + out = append(out, c.Symbol) + } + return out +} + +func chunkWith(t *testing.T, chunks []Chunk, sym string) Chunk { + t.Helper() + for _, c := range chunks { + if c.Symbol == sym { + return c + } + } + t.Fatalf("no chunk with symbol %q in %q", sym, symbols(chunks)) + return Chunk{} +} + +// A Python class too long for one chunk is split at its methods: each chunk +// starts at a method (decorators kept with it), named Class.method, and a +// function nested in a method stays inside that method. +func TestChunkCode_PythonClassSplitAtMethods(t *testing.T) { + src := "class UserService:\n" + + " \"\"\"Users.\"\"\"\n\n" + + " def __init__(self, db):\n" + body(" ", 8, "init") + "\n" + + " @property\n" + + " def active(self):\n" + body(" ", 8, "active") + "\n" + + " async def save(self, user):\n" + + " def validate(u):\n" + body(" ", 3, "validate") + + body(" ", 6, "save") + chunks := chunkCode(src, "svc.py", 400) + + active := chunkWith(t, chunks, "UserService.active") + if !strings.HasPrefix(active.Text, " @property\n def active") { + t.Errorf("decorator not kept with its method:\n%s", active.Text) + } + save := chunkWith(t, chunks, "UserService.save") + if !strings.Contains(save.Text, "def validate") { + t.Errorf("nested function split out of its method:\n%s", save.Text) + } + for _, c := range chunks { + if c.Symbol == "UserService.validate" { + t.Errorf("nested function became its own chunk: %q", symbols(chunks)) + } + if len(c.Text) > 400 { + t.Errorf("chunk over the size (%d): %s", len(c.Text), c.Symbol) + } + } + // Line numbers stay those of the file. + lines := strings.Split(src, "\n") + if !strings.HasPrefix(lines[active.Start-1], " @property") { + t.Errorf("active chunk starts at line %d: %q", active.Start, lines[active.Start-1]) + } +} + +// Java: methods and constructors with modifiers are members; a call inside +// a method body is not; annotations stay with their method. +func TestChunkCode_JavaClassSplitAtMethods(t *testing.T) { + src := "public class OrderService {\n" + + " private final Repo repo;\n\n" + + " public OrderService(Repo repo) {\n" + body(" ", 6, "ctor") + " }\n\n" + + " @Override\n" + + " public List findAll() {\n" + body(" ", 6, "find") + + " helper(repo);\n" + + " }\n\n" + + " private static T pick(List xs) {\n" + body(" ", 6, "pick") + " }\n" + + "}\n" + chunks := chunkCode(src, "OrderService.java", 350) + chunkWith(t, chunks, "OrderService.OrderService") + find := chunkWith(t, chunks, "OrderService.findAll") + if !strings.HasPrefix(strings.TrimSpace(find.Text), "@Override") { + t.Errorf("annotation not kept with method:\n%s", find.Text) + } + chunkWith(t, chunks, "OrderService.pick") + for _, s := range symbols(chunks) { + if s == "OrderService.helper" { + t.Errorf("a call was taken for a method: %q", symbols(chunks)) + } + } +} + +// TypeScript: class methods are members; control flow ("if (x) {") is not. +func TestChunkCode_TypeScriptClassSplitAtMethods(t *testing.T) { + src := "export class Cart {\n" + + " private items: Item[] = [];\n\n" + + " async load(id: string): Promise {\n" + body(" ", 6, "load") + + " if (id) {\n" + body(" ", 2, "if") + " }\n" + + " }\n\n" + + " get total(): number {\n" + body(" ", 6, "total") + " }\n" + + "}\n" + chunks := chunkCode(src, "cart.ts", 300) + load := chunkWith(t, chunks, "Cart.load") + if !strings.Contains(load.Text, "if (id) {") { + t.Errorf("if block split out of its method:\n%s", load.Text) + } + chunkWith(t, chunks, "Cart.total") + for _, s := range symbols(chunks) { + if s == "Cart.if" { + t.Errorf("control flow taken for a method: %q", symbols(chunks)) + } + } +} + +// Rust: fns in an impl block are members. +func TestChunkCode_RustImplSplitAtFns(t *testing.T) { + src := "impl Server {\n" + + " pub fn new(addr: &str) -> Self {\n" + body(" ", 6, "new") + " }\n\n" + + " /// Starts it.\n" + + " pub async fn start(&self) {\n" + body(" ", 6, "start") + " }\n" + + "}\n" + chunks := chunkCode(src, "server.rs", 300) + chunkWith(t, chunks, "Server.new") + start := chunkWith(t, chunks, "Server.start") + if !strings.HasPrefix(start.Text, " /// Starts it.") { + t.Errorf("doc comment not kept with fn:\n%s", start.Text) + } +} + +// A class that fits in one chunk is not split; a long declaration in a +// language without members known is still cut at blank lines. +func TestChunkCode_MembersOnlyWhenTooLong(t *testing.T) { + src := "class Small:\n def a(self):\n return 1\n\n def b(self):\n return 2\n" + if got := chunkCode(src, "s.py", 1000); len(got) != 1 || got[0].Symbol != "Small" { + t.Fatalf("small class split: %+v", got) + } + goSrc := "func Big() {\n" + body("\t", 10, "a") + "\n" + body("\t", 10, "b") + "}\n" + got := chunkCode(goSrc, "big.go", 300) + if len(got) < 2 { + t.Fatalf("long Go func not cut: %+v", got) + } + for _, c := range got { + if c.Symbol != "Big" || len(c.Text) > 300 { + t.Fatalf("cut piece: %+v", c) + } + } +} diff --git a/mcp-server/pkg/indexer/indexer.go b/mcp-server/pkg/indexer/indexer.go index d6a5426..4314b0e 100644 --- a/mcp-server/pkg/indexer/indexer.go +++ b/mcp-server/pkg/indexer/indexer.go @@ -53,14 +53,40 @@ var defaultIgnores = map[string]bool{ ".pytest_cache": true, } -// IndexProject scans the directory and indexes all code/text files. -// It handles insertions (new/changed files), and deletions (files removed since last index). -// Chunks whose content_hash matches an existing point are skipped (no re-embedding). +// chunkVersion tags the chunk layout. v4: chunk IDs are derived from the +// chunk's content ("dir/file#") instead of its position in the file. +const chunkVersion = "v4" + +// indexBatch bounds how many documents go to the provider per call. The TEI +// embedding client sub-batches these further to respect its own limit. +const indexBatch = 100 + +// IndexProject scans the directory and syncs its code/text files to the +// project collection, so the stored chunks mirror the files as they are now: +// +// - A chunk's ID is derived from its text ("dir/file#"), so a +// chunk keeps its ID when code is added or removed around it. +// - An unchanged chunk at the same lines is skipped (no re-embedding). +// - An unchanged chunk whose line numbers moved gets its header and line +// metadata refreshed, without re-embedding when the provider implements +// rag.PayloadUpdater (re-indexed otherwise). +// - Every previously indexed chunk of this directory that the scan did not +// produce again is deleted: chunks of removed files, of code removed from +// a file that still exists, and chunks written by older chunk layouts. +// +// Points without a source_file (memory notes written with rag_index) are +// never touched. A root that does not exist is an error, so a typo or a moved +// repository cannot wipe the index. func (idx *Indexer) IndexProject(ctx context.Context, projectID, rootDir string) (*SyncResult, error) { rootDir, err := filepath.Abs(rootDir) if err != nil { return nil, err } + if info, err := os.Stat(rootDir); err != nil { + return nil, fmt.Errorf("project directory: %w", err) + } else if !info.IsDir() { + return nil, fmt.Errorf("project directory %s is not a directory", rootDir) + } // Use the directory base name as a source_dir tag so stale-point // reconciliation is scoped per-directory. Without this, indexing a second @@ -69,27 +95,31 @@ func (idx *Indexer) IndexProject(ctx context.Context, projectID, rootDir string) // directory's file list. sourceDir := filepath.Base(rootDir) - // Fetch existing points for this source_dir so we can compare content hashes - // and skip re-embedding unchanged chunks. - existingPoints, err := idx.provider.ListPoints(ctx, projectID, map[string]string{"source_dir": sourceDir}) - if err != nil { - // If we can't list existing points, proceed without skip optimization - existingPoints = nil - } - existingHashes := make(map[string]string) // docID -> content_hash - for _, pt := range existingPoints { - key := pt.DocID - if key == "" { - key = pt.ID - } - if pt.ContentHash != "" { - existingHashes[key] = pt.ContentHash + // Existing points of this source_dir, keyed by their document ID, to skip + // unchanged chunks and find the stale ones. If they can't be listed, every + // chunk is indexed and stale cleanup is skipped (reported in StaleError). + // Only IDs under "/" are this directory's: not every provider + // applies the source_dir filter (pgvector filters by source_file only), and + // another directory's chunks must never be taken for stale ones. + existing := make(map[string]rag.PointInfo) + existingPoints, listErr := idx.provider.ListPoints(ctx, projectID, map[string]string{"source_dir": sourceDir}) + if listErr == nil { + for _, pt := range existingPoints { + key := pt.DocID + if key == "" { + key = pt.ID + } + if strings.HasPrefix(key, sourceDir+"/") { + existing[key] = pt + } } } // The files: the caller's, else git's (its ignores honored), else a // walk; never one that holds secrets. - var docs []rag.Document + var fresh, moved []rag.Document + keep := make(map[string]bool) // doc IDs produced by this scan + unreadable := make(map[string]bool) // files that exist but couldn't be read: keep their chunks var currentFiles []string skipped := 0 files, err := idx.listFiles(rootDir) @@ -107,24 +137,31 @@ func (idx *Indexer) IndexProject(ctx context.Context, projectID, rootDir string) } currentFiles = append(currentFiles, relPath) content, err := os.ReadFile(path) - if err != nil || bytes.IndexByte(content, 0) >= 0 { - continue // unreadable, or binary + if err != nil { + unreadable[relPath] = true // maybe transient: don't drop what we have + continue + } + if bytes.IndexByte(content, 0) >= 0 { + continue // binary } + seen := make(map[string]int) // content hash -> occurrences in this file for i, ch := range chunkCode(string(content), relPath, idx.chunkSize) { if secretText.MatchString(ch.Text) { continue // a key in the code: never sent to an embedder } - docID := fmt.Sprintf("%s/%s#chunk%d", sourceDir, relPath, i) contentHash := computeContentHash(ch.Text) - if existingHash, ok := existingHashes[docID]; ok && existingHash == contentHash { - skipped++ - continue + seen[contentHash]++ + docID := fmt.Sprintf("%s/%s#%s", sourceDir, relPath, contentHash) + if n := seen[contentHash]; n > 1 { + docID = fmt.Sprintf("%s-%d", docID, n) // identical chunks in one file } + keep[docID] = true + head := fmt.Sprintf("File: %s:%d-%d", relPath, ch.Start, ch.End) if ch.Symbol != "" { head += " (" + ch.Symbol + ")" } - docs = append(docs, rag.Document{ + doc := rag.Document{ ID: docID, Content: head + "\n\n" + ch.Text, Meta: map[string]string{ @@ -132,47 +169,94 @@ func (idx *Indexer) IndexProject(ctx context.Context, projectID, rootDir string) "source_dir": sourceDir, "chunk_index": fmt.Sprintf("%d", i), "content_hash": contentHash, - "chunk_version": "v3", + "chunk_version": chunkVersion, "start_line": fmt.Sprintf("%d", ch.Start), "end_line": fmt.Sprintf("%d", ch.End), "symbol": ch.Symbol, }, - }) + } + if pt, ok := existing[docID]; ok && pt.ContentHash == contentHash { + if sameHeader(pt.Content, head) { + skipped++ + continue + } + moved = append(moved, doc) // same code, new line numbers + continue + } + fresh = append(fresh, doc) } } - // Index new/changed documents. - if len(docs) > 0 { - // Batch index 100 docs per provider call to bound request size. The TEI - // embedding client sub-batches these further to respect its own limit. - for i := 0; i < len(docs); i += 100 { - end := i + 100 - if end > len(docs) { - end = len(docs) - } - if err := idx.provider.Index(ctx, projectID, docs[i:end]); err != nil { - return nil, fmt.Errorf("index batch %d: %w", i/100, err) + // Refresh moved chunks without re-embedding when the provider can; any + // batch it can't update is re-indexed instead. + updated := 0 + if len(moved) > 0 { + if up, ok := idx.provider.(rag.PayloadUpdater); ok { + for start := 0; start < len(moved); start += indexBatch { + batch := moved[start:min(start+indexBatch, len(moved))] + if err := up.UpdatePayload(ctx, projectID, batch); err != nil { + fresh = append(fresh, batch...) + continue + } + updated += len(batch) } + } else { + fresh = append(fresh, moved...) } } - // Find and delete stale points (files that no longer exist in THIS directory). - staleIDs, err := idx.findStalePoints(ctx, projectID, sourceDir, currentFiles) - if err != nil { - return &SyncResult{Indexed: len(docs), Deleted: 0, Skipped: skipped, StaleError: err.Error()}, nil + // Index new/changed documents. + for start := 0; start < len(fresh); start += indexBatch { + batch := fresh[start:min(start+indexBatch, len(fresh))] + if err := idx.provider.Index(ctx, projectID, batch); err != nil { + return nil, fmt.Errorf("index batch %d: %w", start/indexBatch, err) + } + } + + result := &SyncResult{ + Indexed: len(fresh), + Updated: updated, + FilesScanned: len(currentFiles), + Skipped: skipped, + } + if listErr != nil { + result.StaleError = fmt.Sprintf("list existing points: %v", listErr) + return result, nil + } + + // Delete every chunk of this directory the scan did not produce again. + var staleIDs []string + for key, pt := range existing { + if pt.SourceFile == "" || unreadable[pt.SourceFile] { + continue // a memory note, or a file we couldn't read this time + } + if !keep[key] { + staleIDs = append(staleIDs, pt.ID) + } } if len(staleIDs) > 0 { if err := idx.provider.DeletePoints(ctx, projectID, staleIDs); err != nil { - return &SyncResult{Indexed: len(docs), Deleted: 0, Skipped: skipped, StaleError: err.Error()}, nil + result.StaleError = err.Error() + return result, nil } } + result.Deleted = len(staleIDs) + return result, nil +} - return &SyncResult{ - Indexed: len(docs), - Deleted: len(staleIDs), - FilesScanned: len(currentFiles), - Skipped: skipped, - }, nil +// sameHeader reports whether a stored chunk's content (possibly a truncated +// preview) starts with the given header line. Providers that return no +// content preview can't be checked; the chunk is then treated as unchanged, +// since its ID and content hash already match. +func sameHeader(stored, head string) bool { + if stored == "" { + return true + } + line := head + "\n" + if len(stored) < len(line) { + return strings.HasPrefix(line, stored) // preview cut inside the header + } + return strings.HasPrefix(stored, line) } // listFiles are the files of rootDir to look at, relative with slashes. @@ -204,38 +288,14 @@ func (idx *Indexer) listFiles(rootDir string) ([]string, error) { // SyncResult holds stats from a project sync. type SyncResult struct { - Indexed int `json:"indexed"` - Deleted int `json:"deleted"` + Indexed int `json:"indexed"` // chunks embedded and stored + Updated int `json:"updated,omitempty"` // moved chunks refreshed without re-embedding + Deleted int `json:"deleted"` // stale chunks removed FilesScanned int `json:"files_scanned"` - Skipped int `json:"skipped"` + Skipped int `json:"skipped"` // unchanged chunks StaleError string `json:"stale_error,omitempty"` } -func (idx *Indexer) findStalePoints(ctx context.Context, projectID, sourceDir string, currentFiles []string) ([]string, error) { - currentSet := make(map[string]bool, len(currentFiles)) - for _, f := range currentFiles { - currentSet[f] = true - } - - // List only points belonging to this source_dir so that indexing a - // different directory into the same project doesn't wipe the first. - points, err := idx.provider.ListPoints(ctx, projectID, map[string]string{"source_dir": sourceDir}) - if err != nil { - return nil, err - } - - var stale []string - for _, pt := range points { - if pt.SourceFile == "" { - continue - } - if !currentSet[pt.SourceFile] { - stale = append(stale, pt.ID) - } - } - return stale, nil -} - // chunkText splits text into chunks of approximately maxChars. // It tries to split on newline boundaries. func chunkText(text string, maxChars int) []string { diff --git a/mcp-server/pkg/indexer/indexer_test.go b/mcp-server/pkg/indexer/indexer_test.go index ddf3e72..7a4e4cd 100644 --- a/mcp-server/pkg/indexer/indexer_test.go +++ b/mcp-server/pkg/indexer/indexer_test.go @@ -150,7 +150,7 @@ func TestIndexerAddsContentHash(t *testing.T) { } } -// TestIndexerAddsChunkVersion verifies that IndexProject adds chunk_version="v2" +// TestIndexerAddsChunkVersion verifies that IndexProject adds chunk_version="v4" // to each document's metadata. func TestIndexerAddsChunkVersion(t *testing.T) { dir := t.TempDir() @@ -173,8 +173,8 @@ func TestIndexerAddsChunkVersion(t *testing.T) { t.Errorf("doc %d: chunk_version not in metadata", i) continue } - if version != "v3" { - t.Errorf("doc %d: chunk_version = %q, want %q", i, version, "v3") + if version != "v4" { + t.Errorf("doc %d: chunk_version = %q, want %q", i, version, "v4") } } } diff --git a/mcp-server/pkg/indexer/sync_test.go b/mcp-server/pkg/indexer/sync_test.go new file mode 100644 index 0000000..b633bd5 --- /dev/null +++ b/mcp-server/pkg/indexer/sync_test.go @@ -0,0 +1,431 @@ +package indexer + +import ( + "context" + "fmt" + "hash/fnv" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/enowdev/enowx-rag/mcp-server/pkg/rag" +) + +// countingEmbedder embeds texts as hashed word buckets and counts how many +// texts it was asked to embed, so tests can tell what was re-embedded. +type countingEmbedder struct { + mu sync.Mutex + texts []string +} + +func (e *countingEmbedder) Embed(_ context.Context, texts []string) ([][]float32, error) { + e.mu.Lock() + e.texts = append(e.texts, texts...) + e.mu.Unlock() + out := make([][]float32, len(texts)) + for i, t := range texts { + v := make([]float32, 64) + for _, w := range strings.Fields(strings.ToLower(t)) { + h := fnv.New32a() + h.Write([]byte(w)) + v[h.Sum32()%64]++ + } + out[i] = v + } + return out, nil +} + +func (e *countingEmbedder) VectorSize() int { return 64 } + +func (e *countingEmbedder) reset() []string { + e.mu.Lock() + defer e.mu.Unlock() + got := e.texts + e.texts = nil + return got +} + +// goFunc returns a Go function of roughly 450 characters: with a 1000-char +// chunk size it is above the "small" threshold, so it gets a chunk of its own. +func goFunc(name string) string { + var b strings.Builder + fmt.Fprintf(&b, "func %s() {\n", name) + for i := 0; i < 12; i++ { + fmt.Fprintf(&b, "\t_ = %q // %s body line %d\n", name, name, i) + } + b.WriteString("}\n\n") + return b.String() +} + +func writeGoFile(t *testing.T, path string, names ...string) { + t.Helper() + var b strings.Builder + b.WriteString("package auth\n\n") + for _, n := range names { + b.WriteString(goFunc(n)) + } + if err := os.WriteFile(path, []byte(b.String()), 0o644); err != nil { + t.Fatal(err) + } +} + +func newSQLiteIndexer(t *testing.T) (*Indexer, *rag.SQLiteProvider, *countingEmbedder) { + t.Helper() + emb := &countingEmbedder{} + p, err := rag.NewSQLiteProvider(filepath.Join(t.TempDir(), "rag.db"), emb) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { p.Close() }) + idx := NewIndexer(p, 1000) + // No git in the temp dir: list every file under it. + idx.Files = nil + return idx, p, emb +} + +func storedContents(t *testing.T, p *rag.SQLiteProvider, project string) []string { + t.Helper() + docs, err := p.ExportPoints(context.Background(), project) + if err != nil { + t.Fatal(err) + } + out := make([]string, len(docs)) + for i, d := range docs { + out[i] = d.Content + } + return out +} + +// A file that shrinks must not leave chunks of the code it lost behind: +// they would be served to agents as if the code still existed. +func TestIndexProject_ShrinkingFileDropsStaleChunks(t *testing.T) { + ctx := context.Background() + idx, p, _ := newSQLiteIndexer(t) + dir := t.TempDir() + file := filepath.Join(dir, "auth.go") + + writeGoFile(t, file, "Login", "Logout", "Refresh", "VerifyToken", "HashPassword") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + + writeGoFile(t, file, "Login", "Logout") + res, err := idx.IndexProject(ctx, "p", dir) + if err != nil { + t.Fatal(err) + } + + for _, c := range storedContents(t, p, "p") { + for _, gone := range []string{"Refresh", "VerifyToken", "HashPassword"} { + if strings.Contains(c, gone) { + t.Errorf("stale chunk for removed func %s still stored:\n%s", gone, c) + } + } + } + if res.Deleted == 0 { + t.Errorf("expected stale chunks to be reported deleted, got %+v", res) + } +} + +// Inserting a function at the top of a file embeds only the chunks around +// it (the new function, and the old first one that loses the file header); +// the chunks further down kept their text and only get new line numbers. +func TestIndexProject_InsertAtTopOnlyEmbedsNearbyChunks(t *testing.T) { + ctx := context.Background() + idx, p, emb := newSQLiteIndexer(t) + dir := t.TempDir() + file := filepath.Join(dir, "auth.go") + + writeGoFile(t, file, "Login", "Logout", "Refresh") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + emb.reset() + + writeGoFile(t, file, "Signup", "Login", "Logout", "Refresh") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + + embedded := emb.reset() + joined := strings.Join(embedded, "\n---\n") + if !strings.Contains(joined, "func Signup") || len(embedded) > 2 { + t.Fatalf("want the Signup chunk (and at most the old first chunk) embedded, got %d texts:\n%s", len(embedded), joined) + } + for _, untouched := range []string{"func Logout", "func Refresh"} { + if strings.Contains(joined, untouched) { + t.Errorf("%s did not change but was re-embedded", untouched) + } + } + + // The moved chunks must carry their new line numbers. + content, _ := os.ReadFile(file) + lines := strings.Split(string(content), "\n") + wantStart := 0 + for i, l := range lines { + if strings.HasPrefix(l, "func Refresh") { + wantStart = i + 1 + } + } + found := false + for _, c := range storedContents(t, p, "p") { + if strings.Contains(c, "func Refresh") { + found = true + head := strings.SplitN(c, "\n", 2)[0] + if !strings.HasPrefix(head, fmt.Sprintf("File: auth.go:%d-", wantStart)) { + t.Errorf("Refresh chunk header has stale lines: %q (want start %d)", head, wantStart) + } + } + } + if !found { + t.Fatal("Refresh chunk missing") + } + if n := len(storedContents(t, p, "p")); n != 4 { + t.Errorf("want 4 chunks stored, got %d", n) + } +} + +// Points written by an older version (IDs "dir/file#chunkN") are replaced, +// and memory notes indexed with rag_index are never touched by a sync. +func TestIndexProject_ReplacesLegacyPointsKeepsNotes(t *testing.T) { + ctx := context.Background() + idx, p, _ := newSQLiteIndexer(t) + dir := t.TempDir() + file := filepath.Join(dir, "auth.go") + writeGoFile(t, file, "Login") + base := filepath.Base(dir) + + legacy := rag.Document{ + ID: base + "/auth.go#chunk0", + Content: "File: auth.go\n\nfunc OldLogin() {}", + Meta: map[string]string{"source_file": "auth.go", "source_dir": base, + "chunk_index": "0", "content_hash": "0000000000000000", "chunk_version": "v2"}, + } + note := rag.Document{ID: "decision-1", Content: "We hash passwords with argon2id.", + Meta: map[string]string{"type": "decision"}} + if err := p.Index(ctx, "p", []rag.Document{legacy, note}); err != nil { + t.Fatal(err) + } + + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + contents := strings.Join(storedContents(t, p, "p"), "\n---\n") + if strings.Contains(contents, "OldLogin") { + t.Errorf("legacy chunk not replaced:\n%s", contents) + } + if !strings.Contains(contents, "argon2id") { + t.Errorf("memory note was deleted by the sync:\n%s", contents) + } + if !strings.Contains(contents, "func Login") { + t.Errorf("current code not indexed:\n%s", contents) + } +} + +// A directory that does not exist (a typo, a moved repo) must be an error, +// not an empty scan that deletes everything indexed from it. +func TestIndexProject_MissingDirIsErrorAndKeepsPoints(t *testing.T) { + ctx := context.Background() + idx, p, _ := newSQLiteIndexer(t) + parent := t.TempDir() + dir := filepath.Join(parent, "repo") + if err := os.Mkdir(dir, 0o755); err != nil { + t.Fatal(err) + } + writeGoFile(t, filepath.Join(dir, "auth.go"), "Login", "Logout") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + before := len(storedContents(t, p, "p")) + + if err := os.RemoveAll(dir); err != nil { + t.Fatal(err) + } + if _, err := idx.IndexProject(ctx, "p", dir); err == nil { + t.Error("want an error for a missing directory") + } + if after := len(storedContents(t, p, "p")); after != before { + t.Errorf("missing directory wiped the index: %d -> %d points", before, after) + } +} + +// Two identical chunks in one file are both kept (distinct IDs). +func TestIndexProject_DuplicateChunksInOneFile(t *testing.T) { + ctx := context.Background() + idx, p, _ := newSQLiteIndexer(t) + dir := t.TempDir() + body := "# Notes\n\nsame paragraph\n\n# Notes\n\nsame paragraph\n" + // Force one chunk per heading with a tiny chunk size. + idx.chunkSize = 40 + if err := os.WriteFile(filepath.Join(dir, "a.md"), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + if n := len(storedContents(t, p, "p")); n != 2 { + t.Errorf("want 2 chunks for two identical sections, got %d", n) + } +} + +// storeWithoutUpdater is a provider that keeps points in memory and does not +// implement rag.PayloadUpdater, like most remote stores. +type storeWithoutUpdater struct { + recording + pts map[string]rag.Document +} + +func (s *storeWithoutUpdater) Index(_ context.Context, _ string, docs []rag.Document) error { + s.recording.Index(context.Background(), "", docs) + for _, d := range docs { + s.pts[d.ID] = d + } + return nil +} + +func (s *storeWithoutUpdater) ListPoints(_ context.Context, _ string, f map[string]string) ([]rag.PointInfo, error) { + var out []rag.PointInfo + for id, d := range s.pts { + if d.Meta["source_dir"] != f["source_dir"] { + continue + } + preview := d.Content + if len(preview) > 200 { + preview = preview[:200] + } + out = append(out, rag.PointInfo{ID: id, DocID: id, SourceFile: d.Meta["source_file"], ContentHash: d.Meta["content_hash"], Content: preview}) + } + return out, nil +} + +func (s *storeWithoutUpdater) DeletePoints(_ context.Context, _ string, ids []string) error { + for _, id := range ids { + delete(s.pts, id) + } + return nil +} + +// Without a PayloadUpdater, a chunk whose lines moved is re-indexed so its +// stored header stays right. +func TestIndexProject_MovedChunksReindexedWithoutUpdater(t *testing.T) { + ctx := context.Background() + store := &storeWithoutUpdater{pts: map[string]rag.Document{}} + idx := NewIndexer(store, 1000) + dir := t.TempDir() + file := filepath.Join(dir, "auth.go") + writeGoFile(t, file, "Login", "Logout", "Refresh") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + store.docs = nil + writeGoFile(t, file, "Signup", "Login", "Logout", "Refresh") + res, err := idx.IndexProject(ctx, "p", dir) + if err != nil { + t.Fatal(err) + } + reindexed := false + for _, d := range store.docs { + if strings.Contains(d.Content, "func Refresh") { + reindexed = true + } + } + if !reindexed || res.Updated != 0 { + t.Fatalf("moved Refresh chunk not re-indexed: %+v", res) + } + if len(store.pts) != 4 { + t.Fatalf("want 4 points, got %d", len(store.pts)) + } +} + +// A second sync of unchanged files embeds nothing and deletes nothing. +func TestIndexProject_UnchangedSecondSyncIsNoop(t *testing.T) { + ctx := context.Background() + idx, _, emb := newSQLiteIndexer(t) + dir := t.TempDir() + writeGoFile(t, filepath.Join(dir, "auth.go"), "Login", "Logout", "Refresh") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + emb.reset() + res, err := idx.IndexProject(ctx, "p", dir) + if err != nil { + t.Fatal(err) + } + if n := len(emb.reset()); n != 0 || res.Indexed != 0 || res.Deleted != 0 || res.Updated != 0 || res.Skipped == 0 { + t.Fatalf("second sync not a no-op: embedded=%d result=%+v", n, res) + } +} + +// ignoresDirFilter mimics pgvector, whose ListPoints applies only a +// source_file filter: it returns the points of every directory. +type ignoresDirFilter struct{ storeWithoutUpdater } + +func (s *ignoresDirFilter) ListPoints(ctx context.Context, project string, _ map[string]string) ([]rag.PointInfo, error) { + var all []rag.PointInfo + for _, dir := range []string{"repo-a", "repo-b"} { + pts, _ := s.storeWithoutUpdater.ListPoints(ctx, project, map[string]string{"source_dir": dir}) + all = append(all, pts...) + } + return all, nil +} + +// Syncing one directory never deletes another directory's chunks, even +// when the provider returns them from ListPoints. +func TestIndexProject_ProviderIgnoringDirFilterKeepsOtherDirs(t *testing.T) { + ctx := context.Background() + store := &ignoresDirFilter{storeWithoutUpdater{pts: map[string]rag.Document{}}} + idx := NewIndexer(store, 1000) + parent := t.TempDir() + a, b := filepath.Join(parent, "repo-a"), filepath.Join(parent, "repo-b") + for _, d := range []string{a, b} { + if err := os.Mkdir(d, 0o755); err != nil { + t.Fatal(err) + } + } + writeGoFile(t, filepath.Join(a, "a.go"), "Alpha") + writeGoFile(t, filepath.Join(b, "b.go"), "Bravo") + for _, d := range []string{a, b, a} { + if _, err := idx.IndexProject(ctx, "p", d); err != nil { + t.Fatal(err) + } + } + var hasB bool + for _, d := range store.pts { + if strings.Contains(d.Content, "func Bravo") { + hasB = true + } + } + if !hasB { + t.Fatal("syncing repo-a deleted repo-b's chunks") + } +} + +// A file that exists but can't be read keeps its chunks. +func TestIndexProject_UnreadableFileKeepsChunks(t *testing.T) { + if os.Geteuid() == 0 { + t.Skip("root can read any file") + } + ctx := context.Background() + idx, p, _ := newSQLiteIndexer(t) + dir := t.TempDir() + file := filepath.Join(dir, "auth.go") + writeGoFile(t, file, "Login") + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + if err := os.Chmod(file, 0o000); err != nil { + t.Fatal(err) + } + defer os.Chmod(file, 0o644) + if _, err := idx.IndexProject(ctx, "p", dir); err != nil { + t.Fatal(err) + } + if n := len(storedContents(t, p, "p")); n == 0 { + t.Fatal("chunks of an unreadable file were deleted") + } +} diff --git a/mcp-server/pkg/rag/chroma.go b/mcp-server/pkg/rag/chroma.go index ca1f3d5..5af4552 100644 --- a/mcp-server/pkg/rag/chroma.go +++ b/mcp-server/pkg/rag/chroma.go @@ -101,7 +101,9 @@ func (p *ChromaProvider) Index(ctx context.Context, projectID string, docs []Doc "metadatas": metas, "documents": texts, } - return p.do(ctx, http.MethodPost, "/api/v1/collections/"+name+"/add", body, nil) + // upsert, not add: /add silently skips IDs that already exist, so an edited + // chunk (same ID, new content) would never be stored. + return p.do(ctx, http.MethodPost, "/api/v1/collections/"+name+"/upsert", body, nil) } func (p *ChromaProvider) SemanticSearch(ctx context.Context, projectID, query string, limit int) ([]Result, error) { diff --git a/mcp-server/pkg/rag/provider.go b/mcp-server/pkg/rag/provider.go index 882bab3..cda6c4f 100644 --- a/mcp-server/pkg/rag/provider.go +++ b/mcp-server/pkg/rag/provider.go @@ -100,6 +100,18 @@ type HybridSearcher interface { SemanticSearchHybrid(ctx context.Context, projectID, query string, limit int) ([]Result, error) } +// PayloadUpdater is an optional interface for providers that can rewrite the +// content and metadata of existing points while keeping their vectors. The +// indexer uses it when a chunk's code is unchanged but its location moved +// (e.g. a function was inserted above it, so its line numbers shifted): the +// stored header and line metadata are refreshed without paying to re-embed. +// Each Document's ID must name an existing point; Meta is merged over the +// stored metadata. Providers that do not implement it get those chunks +// re-indexed (re-embedded) instead, which is correct but costs embeddings. +type PayloadUpdater interface { + UpdatePayload(ctx context.Context, projectID string, docs []Document) error +} + // RerankHit is a single reranked result: the Index of the original document // in the input slice and its relevance Score from the reranker. type RerankHit struct { diff --git a/mcp-server/pkg/rag/sqlite.go b/mcp-server/pkg/rag/sqlite.go index a90089c..6b31ec1 100644 --- a/mcp-server/pkg/rag/sqlite.go +++ b/mcp-server/pkg/rag/sqlite.go @@ -140,6 +140,47 @@ ON CONFLICT(project, id) DO UPDATE SET content = excluded.content, meta = exclud return nil } +// UpdatePayload rewrites the content and metadata of existing points, keeping +// their vectors (see PayloadUpdater). The full-text index is refreshed too. +func (p *SQLiteProvider) UpdatePayload(ctx context.Context, projectID string, docs []Document) error { + if len(docs) == 0 { + return nil + } + tx, err := p.db.BeginTx(ctx, nil) + if err != nil { + return err + } + defer tx.Rollback() + for _, d := range docs { + var raw string + err := tx.QueryRowContext(ctx, `SELECT meta FROM points WHERE project = ? AND id = ?`, projectID, d.ID).Scan(&raw) + if err == sql.ErrNoRows { + return fmt.Errorf("update payload: point %q not found", d.ID) + } + if err != nil { + return err + } + meta := map[string]string{} + _ = json.Unmarshal([]byte(raw), &meta) + for k, v := range d.Meta { + meta[k] = v + } + merged, _ := json.Marshal(meta) + if _, err := tx.ExecContext(ctx, `UPDATE points SET content = ?, meta = ? WHERE project = ? AND id = ?`, + d.Content, string(merged), projectID, d.ID); err != nil { + return err + } + if _, err := tx.ExecContext(ctx, `DELETE FROM points_fts WHERE project = ? AND id = ?`, projectID, d.ID); err != nil { + return err + } + if _, err := tx.ExecContext(ctx, `INSERT INTO points_fts(content, project, id) VALUES (?, ?, ?)`, d.Content, projectID, d.ID); err != nil { + return err + } + } + // Vectors are untouched, so the cached vectors stay valid. + return tx.Commit() +} + // vectors are project's points' vectors, normalized, read once. func (p *SQLiteProvider) vectors(ctx context.Context, projectID string) ([]cached, error) { p.mu.Lock() diff --git a/mcp-server/pkg/rag/sqlite_test.go b/mcp-server/pkg/rag/sqlite_test.go index e215da4..323eff0 100644 --- a/mcp-server/pkg/rag/sqlite_test.go +++ b/mcp-server/pkg/rag/sqlite_test.go @@ -79,3 +79,43 @@ func TestSQLiteProvider(t *testing.T) { t.Fatalf("deleted still found: %+v", got) } } + +// UpdatePayload rewrites a point's content and metadata in place: its vector +// is kept (no embedding), the metadata merged, the full-text index refreshed. +func TestSQLiteUpdatePayload(t *testing.T) { + ctx := context.Background() + p, err := NewSQLiteProvider(filepath.Join(t.TempDir(), "rag.db"), wordsEmbedder{}) + if err != nil { + t.Fatal(err) + } + defer p.Close() + doc := Document{ID: "a", Content: "File: x.go:1-3\n\nfunc ParseConfig reads the yaml config file", + Meta: map[string]string{"source_file": "x.go", "start_line": "1"}} + if err := p.Index(ctx, "proj", []Document{doc}); err != nil { + t.Fatal(err) + } + before, _ := p.SemanticSearch(ctx, "proj", "read the config file", 1) + + moved := Document{ID: "a", Content: "File: x.go:11-13\n\nfunc ParseConfig reads the yaml config file", + Meta: map[string]string{"start_line": "11"}} + if err := p.UpdatePayload(ctx, "proj", []Document{moved}); err != nil { + t.Fatal(err) + } + got, err := p.SemanticSearch(ctx, "proj", "read the config file", 1) + if err != nil || len(got) != 1 { + t.Fatalf("search: %+v %v", got, err) + } + if got[0].Content != moved.Content || got[0].Meta["start_line"] != "11" || got[0].Meta["source_file"] != "x.go" || got[0].Meta["embed_model"] == "" { + t.Fatalf("payload not updated/merged: %+v", got[0]) + } + if got[0].Score != before[0].Score { + t.Fatalf("vector changed: score %v -> %v", before[0].Score, got[0].Score) + } + hy, err := p.SemanticSearchHybrid(ctx, "proj", "x.go:11-13", 1) + if err != nil || len(hy) == 0 || hy[0].ID != "a" { + t.Fatalf("fts not refreshed: %+v %v", hy, err) + } + if err := p.UpdatePayload(ctx, "proj", []Document{{ID: "missing", Content: "x"}}); err == nil { + t.Fatal("want an error for a missing point") + } +} diff --git a/skill/enowx-rag.md b/skill/enowx-rag.md index c839347..087dfa0 100644 --- a/skill/enowx-rag.md +++ b/skill/enowx-rag.md @@ -6,7 +6,7 @@ Use this skill when the user wants to set up, configure, or deploy the enowx-rag - User says "setup RAG memory", "install enowx-rag", "buat project RAG", "buat MCP RAG", "connect RAG", "rag per project", or similar. - User asks about configuring a Model Context Protocol (MCP) server for project memory. -- User wants to deploy the RAG backend locally or connect to an existing Coolify RAG stack. +- User wants to deploy the RAG backend locally, on a server, or connect to an existing Qdrant / TEI backend. ## Goal @@ -15,7 +15,7 @@ Walk the user through installing the `enowx-rag` MCP server and optionally deplo ## Constraints - Do not expose secrets or API keys in plain text in the final output; store them in `.env` files. -- If deploying to Coolify, prefer managed Docker services or the existing `robloxkit-rag` stack if healthy. +- If deploying to a hosting platform (e.g. Coolify), prefer its managed Docker services, and reuse a Qdrant / TEI service the user already runs if it is healthy. - Do not modify the user's existing code repositories unless explicitly asked. - Use the exact config format documented by each tool. Do not guess formats. @@ -40,15 +40,15 @@ Walk the user through installing the `enowx-rag` MCP server and optionally deplo - Create `.env` from the chosen configuration. - Provide the MCP client config snippet for all supported tools using the correct format per tool. -4. **Optional Coolify deployment** - - If the user wants to deploy to Coolify, use the `coolify` MCP tools to create/update the service/app. - - Use the existing `robloxkit-rag` service (Qdrant + TEI) if it is healthy and matches the requirements. +4. **Optional server deployment** + - If the user wants the backend on a server (e.g. Coolify), use that platform's tools if available (such as a `coolify` MCP server) to create/update the services. + - Reuse an existing Qdrant + TEI service the user already runs if it is healthy and matches the requirements. ## Required user info Ask these questions one at a time or in a compact batch: -1. `project_path` - Where should the `enowx-rag` repository be created/cloned? (default: `/Users/enowdev/Project/enowx-rag`) +1. `project_path` - Where should the `enowx-rag` repository be created/cloned? (suggest `~/enowx-rag`; skip if the user installed the prebuilt binary) 2. `embedder` - Which embedding option? - **`voyage`** (default, recommended): hosted API, free at voyageai.com. Ask for `voyage_api_key`. - **`tei`**: self-hosted. Ask for Qdrant URL + TEI URL, or offer Docker setup. @@ -140,6 +140,8 @@ volumes: Each tool has a different config format and file location. Use these exact formats. +In every block below, replace `/path/to/enowx-rag/mcp-server/mcp-server` with the absolute path of the binary you built. If the user installed the prebuilt binary (install script, Homebrew, npm or `go install`), use `enowx-rag` (on the `PATH`) as the command instead. + Each config block below shows the Voyage AI setup (recommended). Replace with `RAG_EMBEDDER=tei` + `RAG_TEI_URL` if using self-hosted TEI instead. ### Claude Code (Anthropic CLI) @@ -152,7 +154,7 @@ claude mcp add --transport stdio enowx-rag \ --env RAG_VECTOR_STORE=qdrant \ --env RAG_QDRANT_URL=http://localhost:6333 \ --env RAG_VOYAGE_API_KEY=your-voyage-api-key \ - -- /Users/enowdev/Project/enowx-rag/mcp-server/mcp-server + -- /path/to/enowx-rag/mcp-server/mcp-server ``` **JSON format (`.mcp.json` or `~/.claude.json`):** @@ -160,7 +162,7 @@ claude mcp add --transport stdio enowx-rag \ { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -182,7 +184,7 @@ Source: https://code.claude.com/docs/en/mcp { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -205,7 +207,7 @@ Source: https://code.claude.com/docs/en/mcp-quickstart { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -231,7 +233,7 @@ Source: https://docs.cline.bot/mcp/configuring-mcp-servers "mcpServers": { "enowx-rag": { "type": "stdio", - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -259,7 +261,7 @@ OpenCode uses a different schema: `mcp` key (not `mcpServers`), `command` as arr "mcp": { "enowx-rag": { "type": "local", - "command": ["/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server"], + "command": ["/path/to/enowx-rag/mcp-server/mcp-server"], "enabled": true, "environment": { "RAG_VECTOR_STORE": "qdrant", @@ -284,13 +286,13 @@ codex mcp add enowx-rag \ --env RAG_VECTOR_STORE=qdrant \ --env RAG_QDRANT_URL=http://localhost:6333 \ --env RAG_VOYAGE_API_KEY=your-voyage-api-key \ - -- /Users/enowdev/Project/enowx-rag/mcp-server/mcp-server + -- /path/to/enowx-rag/mcp-server/mcp-server ``` **TOML format (`config.toml`):** ```toml [mcp_servers.enowx-rag] -command = "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server" +command = "/path/to/enowx-rag/mcp-server/mcp-server" [mcp_servers.enowx-rag.env] RAG_VECTOR_STORE = "qdrant" @@ -305,7 +307,7 @@ Source: https://developers.openai.com/codex/mcp **CLI command:** ```bash -droid mcp add enowx-rag /Users/enowdev/Project/enowx-rag/mcp-server/mcp-server +droid mcp add enowx-rag /path/to/enowx-rag/mcp-server/mcp-server ``` Source: Factory Droid CLI documentation @@ -319,7 +321,7 @@ Source: Factory Droid CLI documentation { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -343,7 +345,7 @@ Source: https://docs.roocode.com/features/mcp/using-mcp-in-roo { "context_servers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "args": [], "env": { "RAG_VECTOR_STORE": "qdrant", @@ -368,7 +370,7 @@ Source: https://zed.dev/docs/ai/mcp { "mcpServers": { "enowx-rag": { - "command": "/Users/enowdev/Project/enowx-rag/mcp-server/mcp-server", + "command": "/path/to/enowx-rag/mcp-server/mcp-server", "env": { "RAG_VECTOR_STORE": "qdrant", "RAG_QDRANT_URL": "http://localhost:6333", @@ -393,7 +395,7 @@ Continue uses a `mcpServers` list (array of objects, not a map). Each entry has ```yaml mcpServers: - name: enowx-rag - command: /Users/enowdev/Project/enowx-rag/mcp-server/mcp-server + command: /path/to/enowx-rag/mcp-server/mcp-server env: RAG_VECTOR_STORE: qdrant RAG_QDRANT_URL: http://localhost:6333 @@ -459,15 +461,16 @@ This project uses `enowx-rag` MCP server for per-project memory (RAG). ## Before you start coding -1. Call `rag_retrieve_context` with the user's task/query and the project ID `PROJECT_ID`. -2. Read the returned context. If it is empty or irrelevant, continue as normal. -3. If the context changes how you would approach the task, explain the relevant insights briefly. +1. If the project has never been indexed (`rag_project_exists` returns false), call `rag_index_project` once with the project directory. +2. Call `rag_retrieve_context` with the user's task/query and the project ID `PROJECT_ID`. +3. Read the returned context. If it is empty or irrelevant, continue as normal. +4. If the context changes how you would approach the task, explain the relevant insights briefly. ## After you finish coding 1. Summarize what you changed and why. -2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`. -3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically. +2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID` — code is synced automatically, but the reasons behind it are only saved if you save them. +3. You don't need to re-index files after each change: once a directory has been indexed with `rag_index_project`, the server re-syncs it automatically before searches (when the index is older than `RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync, or when auto-sync is turned off. 4. Keep chunks focused and concise (one idea per chunk). Include metadata tags when helpful. ## Project ID @@ -509,15 +512,16 @@ You are working with a project that has an `enowx-rag` MCP server installed. ### Before making significant changes -1. Call `rag_retrieve_context` with the project ID `PROJECT_ID` and the user's query. -2. Read the returned context. If it is empty or irrelevant, continue as normal. -3. If the context changes how you would approach the task, explain the relevant insights briefly. +1. If the project has never been indexed (`rag_project_exists` returns false), call `rag_index_project` once with the project directory. +2. Call `rag_retrieve_context` with the project ID `PROJECT_ID` and the user's query. +3. Read the returned context. If it is empty or irrelevant, continue as normal. +4. If the context changes how you would approach the task, explain the relevant insights briefly. ### After completing work 1. Summarize what you changed and why. -2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`. -3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically. +2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID` — code is synced automatically, but the reasons behind it are only saved if you save them. +3. You don't need to re-index files after each change: once a directory has been indexed with `rag_index_project`, the server re-syncs it automatically before searches (when the index is older than `RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync, or when auto-sync is turned off. 4. Keep chunks focused and concise (one idea per chunk). Include metadata tags when helpful. ## Project ID @@ -600,7 +604,7 @@ Created: - `/path/to/project/CLAUDE.md` ### Next steps -1. Build MCP server: `cd /Users/enowdev/Project/enowx-rag/mcp-server && go build ./cmd/mcp-server` +1. Build MCP server: `cd /path/to/enowx-rag/mcp-server && go build ./cmd/mcp-server` 2. Restart your coding tool or reload MCP server list. 3. Test by asking the agent to retrieve project context. ``` @@ -617,7 +621,7 @@ curl -f http://localhost:6333/healthz curl -f http://localhost:8081/health # Build MCP server -cd /Users/enowdev/Project/enowx-rag/mcp-server && go build ./cmd/mcp-server +cd /path/to/enowx-rag/mcp-server && go build ./cmd/mcp-server ``` ## Notes diff --git a/skill/templates/AGENTS.md b/skill/templates/AGENTS.md index 3dc92de..30f80a9 100644 --- a/skill/templates/AGENTS.md +++ b/skill/templates/AGENTS.md @@ -4,15 +4,16 @@ This project uses `enowx-rag` MCP server for per-project memory (RAG). ## Before you start coding -1. Call `rag_retrieve_context` with the user's task/query and the project ID `PROJECT_ID`. -2. Read the returned context. If it is empty or irrelevant, continue as normal. -3. If the context changes how you would approach the task, explain the relevant insights briefly. +1. If the project has never been indexed (`rag_project_exists` returns false), call `rag_index_project` once with the project directory. +2. Call `rag_retrieve_context` with the user's task/query and the project ID `PROJECT_ID`. +3. Read the returned context. If it is empty or irrelevant, continue as normal. +4. If the context changes how you would approach the task, explain the relevant insights briefly. ## After you finish coding 1. Summarize what you changed and why. -2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`. -3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically. +2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID` — code is synced automatically, but the reasons behind it are only saved if you save them. +3. You don't need to re-index files after each change: once a directory has been indexed with `rag_index_project`, the server re-syncs it automatically before searches (when the index is older than `RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync, or when auto-sync is turned off. 4. Keep chunks focused and concise (one idea per chunk). Include metadata tags when helpful. ## Project ID diff --git a/skill/templates/CLAUDE.md b/skill/templates/CLAUDE.md index 7dc641b..280d4b3 100644 --- a/skill/templates/CLAUDE.md +++ b/skill/templates/CLAUDE.md @@ -6,15 +6,16 @@ You are working with a project that has an `enowx-rag` MCP server installed. ### Before making significant changes -1. Call `rag_retrieve_context` with the project ID `PROJECT_ID` and the user's query. -2. Read the returned context. If it is empty or irrelevant, continue as normal. -3. If the context changes how you would approach the task, explain the relevant insights briefly. +1. If the project has never been indexed (`rag_project_exists` returns false), call `rag_index_project` once with the project directory. +2. Call `rag_retrieve_context` with the project ID `PROJECT_ID` and the user's query. +3. Read the returned context. If it is empty or irrelevant, continue as normal. +4. If the context changes how you would approach the task, explain the relevant insights briefly. ### After completing work 1. Summarize what you changed and why. -2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID`. -3. Call `rag_index_project` with the project directory to sync all file changes into RAG. Always do this — it handles new files, edits, and deletions automatically. +2. Call `rag_index` with useful new facts, design decisions, gotchas, or patterns under project ID `PROJECT_ID` — code is synced automatically, but the reasons behind it are only saved if you save them. +3. You don't need to re-index files after each change: once a directory has been indexed with `rag_index_project`, the server re-syncs it automatically before searches (when the index is older than `RAG_AUTO_SYNC_INTERVAL`, default 60s). Call `rag_index_project` again only to force an immediate sync, or when auto-sync is turned off. 4. Keep chunks focused and concise (one idea per chunk). Include metadata tags when helpful. ## Project ID