Skip to content

Fix stale chunks and re-embed cascades, split classes at methods, auto-sync before search - #8

Open
SemutJahat wants to merge 2 commits into
enowdev:mainfrom
SemutJahat:fix/index-sync-chunking-autosync
Open

SemutJahat wants to merge 2 commits into
enowdev:mainfrom
SemutJahat:fix/index-sync-chunking-autosync

Conversation

@SemutJahat

Copy link
Copy Markdown

Summary

Fixes several problems in how rag_index_project keeps the index in sync with the code, and makes agents depend less on calling it after every task.

Bugs fixed

  • Stale chunks after editing a file. A sync only deleted chunks whose whole file was gone. When a file shrank (e.g. functions removed), its old chunks stayed and were served to agents as current code. A sync now deletes every chunk of the directory that the scan did not produce again.
  • Re-embedding everything below an edit. Chunk IDs were positional (#chunkN), so inserting a function shifted the ID of every chunk after it. IDs now derive from the chunk's content (dir/file#<hash>, chunk_version v4), and only small declarations are packed together, so an edit only touches nearby chunks. A chunk that only moved gets its line numbers refreshed without re-embedding on SQLite (new optional rag.PayloadUpdater), and is re-indexed on other stores.
  • pgvector: ListPoints ignores the source_dir filter, so a sync could delete another directory's chunks. The indexer now only deletes chunks under the directory's own ID prefix.
  • Chroma: Index used /add, which silently skips IDs that already exist, so edited chunks were never stored. It now uses /upsert.
  • Missing directory wiped the index. rag_index_project on a path that doesn't exist (a typo, a moved repo) did an empty scan and deleted everything. It now returns an error.
  • Unreadable file. A file that exists but can't be read keeps its chunks instead of losing them to the stale cleanup.
  • Same-named directories. A second directory with the same folder name in one project (/x/app, /y/app) is refused, since their chunks share an ID prefix and would replace each other.

Improvements

  • Big classes split at their methods. A declaration too long for one chunk is split at its members (Python, Rust, Ruby, Java/C#, Kotlin, Swift, PHP, JS/TS class methods), with decorators/annotations kept with each method and closures left inside it. Symbols look like Class.method. Before, these were cut at blank lines.
  • Auto-sync before search. A directory indexed once with rag_index_project is recorded in ~/.enowx-rag/project_dirs.json and re-synced incrementally before a search when its index is older than RAG_AUTO_SYNC_INTERVAL (default 60s, off disables). Missing directories and projects deleted elsewhere are skipped. The registry file is safe to share between processes. The agent templates (skill, skill/templates/*, write-agents-md, docs) no longer tell agents to run rag_index_project after every task.
  • Personal paths (/Users/...) and a personal deployment name removed from README, AGENTS.md, CLAUDE.md and the skill.

Behavior changes

  • One-time re-embed: chunk IDs changed, so the first sync after upgrading re-embeds each indexed project once. Old-format chunks are cleaned up automatically.
  • More, smaller chunks: functions of fair size now get a chunk each instead of being packed together.
  • SyncResult gains updated, and the rag_index_project tool returns chunks_updated.

Testing

  • 25 new tests: sync regressions (shrinking file, insert at top, legacy IDs and notes, missing directory, duplicate chunks, provider ignoring the dir filter, unreadable file, no-op second sync, fallback without PayloadUpdater), member splitting per language, SQLite UpdatePayload, and auto-sync (edits seen without re-index, interval/off, missing dir, deleted project not recreated, same-name refusal, shared registry, interval parsing).
  • The new regression tests fail when their fix is reverted.
  • pkg/rag, pkg/indexer and pkg/core pass go test -race and go vet locally on Go 1.24, using a test-only SQLite driver build because the Go module proxy wasn't reachable from the dev environment.
  • The changes in cmd/mcp-server, pkg/config and pkg/httpapi (wiring, one path helper, template text) could not be compiled locally and were reviewed by hand. CI on this PR is their first real build with Go 1.26.
  • Not run against live Qdrant, Chroma or pgvector servers.

SemutJahat and others added 2 commits November 3, 2026 11:52
…o-sync

Indexer sync
- Delete every chunk of the synced directory that the scan did not produce
  again, so code removed from a file that still exists leaves the index
  (before, chunks were only deleted when their whole file was gone).
- Chunk IDs derive from the chunk's content ("dir/file#<hash>",
  chunk_version v4) instead of its position, and only small declarations
  are packed together, so an edit re-embeds only the chunks around it.
- A chunk that only moved gets its header/line metadata refreshed without
  re-embedding via the new optional rag.PayloadUpdater (SQLite), and is
  re-indexed on other stores.
- Only delete chunks under the directory's own ID prefix (pgvector's
  ListPoints ignores the source_dir filter); keep chunks of files that
  exist but can't be read; a missing root directory is an error instead of
  an empty scan that wiped its index.

Chunker
- A declaration too long for one chunk is split at its members (methods of
  Python, Rust, Ruby, Java/C#, Kotlin, Swift, PHP, JS/TS classes), with
  decorators/annotations kept and closures left inside; symbol Class.method.

Auto-sync
- Directories indexed with rag_index_project are recorded in
  ~/.enowx-rag/project_dirs.json and re-synced incrementally before a
  search when older than RAG_AUTO_SYNC_INTERVAL (default 60s, "off"
  disables). Missing directories and projects deleted elsewhere are
  skipped; a second same-named directory in one project is refused.
- Agent templates no longer require rag_index_project after every task.

Also: Chroma writes with /upsert (/add skipped existing IDs, so edits were
never stored); personal paths removed from README, AGENTS.md, CLAUDE.md
and the skill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ws2zVFMvWyoj2kXBC6ecxt
The auto-sync interval can now be set once in ~/.enowx-rag/config.yaml
(auto_sync_interval: 1s, 5m, 30, off) instead of only through the
RAG_AUTO_SYNC_INTERVAL env var in every MCP client config. The env var
still wins. Survives Save/Load, so the dashboard keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ws2zVFMvWyoj2kXBC6ecxt
@SemutJahat
SemutJahat force-pushed the fix/index-sync-chunking-autosync branch from 09ba182 to 94fe95b Compare November 3, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant