Repository navigation
Fix stale chunks and re-embed cascades, split classes at methods, auto-sync before search - #8
Open
SemutJahat wants to merge 2 commits into
Open
SemutJahat wants to merge 2 commits into
SemutJahat wants to merge 2 commits into
Conversation
…o-sync
Indexer sync
- Delete every chunk of the synced directory that the scan did not produce
again, so code removed from a file that still exists leaves the index
(before, chunks were only deleted when their whole file was gone).
- Chunk IDs derive from the chunk's content ("dir/file#<hash>",
chunk_version v4) instead of its position, and only small declarations
are packed together, so an edit re-embeds only the chunks around it.
- A chunk that only moved gets its header/line metadata refreshed without
re-embedding via the new optional rag.PayloadUpdater (SQLite), and is
re-indexed on other stores.
- Only delete chunks under the directory's own ID prefix (pgvector's
ListPoints ignores the source_dir filter); keep chunks of files that
exist but can't be read; a missing root directory is an error instead of
an empty scan that wiped its index.
Chunker
- A declaration too long for one chunk is split at its members (methods of
Python, Rust, Ruby, Java/C#, Kotlin, Swift, PHP, JS/TS classes), with
decorators/annotations kept and closures left inside; symbol Class.method.
Auto-sync
- Directories indexed with rag_index_project are recorded in
~/.enowx-rag/project_dirs.json and re-synced incrementally before a
search when older than RAG_AUTO_SYNC_INTERVAL (default 60s, "off"
disables). Missing directories and projects deleted elsewhere are
skipped; a second same-named directory in one project is refused.
- Agent templates no longer require rag_index_project after every task.
Also: Chroma writes with /upsert (/add skipped existing IDs, so edits were
never stored); personal paths removed from README, AGENTS.md, CLAUDE.md
and the skill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ws2zVFMvWyoj2kXBC6ecxt
The auto-sync interval can now be set once in ~/.enowx-rag/config.yaml (auto_sync_interval: 1s, 5m, 30, off) instead of only through the RAG_AUTO_SYNC_INTERVAL env var in every MCP client config. The env var still wins. Survives Save/Load, so the dashboard keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ws2zVFMvWyoj2kXBC6ecxt
SemutJahat
force-pushed
the
fix/index-sync-chunking-autosync
branch
from
November 3, 2026 13:57
09ba182 to
94fe95b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes several problems in how
rag_index_projectkeeps the index in sync with the code, and makes agents depend less on calling it after every task.Bugs fixed
#chunkN), so inserting a function shifted the ID of every chunk after it. IDs now derive from the chunk's content (dir/file#<hash>,chunk_versionv4), and only small declarations are packed together, so an edit only touches nearby chunks. A chunk that only moved gets its line numbers refreshed without re-embedding on SQLite (new optionalrag.PayloadUpdater), and is re-indexed on other stores.ListPointsignores thesource_dirfilter, so a sync could delete another directory's chunks. The indexer now only deletes chunks under the directory's own ID prefix.Indexused/add, which silently skips IDs that already exist, so edited chunks were never stored. It now uses/upsert.rag_index_projecton a path that doesn't exist (a typo, a moved repo) did an empty scan and deleted everything. It now returns an error./x/app,/y/app) is refused, since their chunks share an ID prefix and would replace each other.Improvements
Class.method. Before, these were cut at blank lines.rag_index_projectis recorded in~/.enowx-rag/project_dirs.jsonand re-synced incrementally before a search when its index is older thanRAG_AUTO_SYNC_INTERVAL(default60s,offdisables). Missing directories and projects deleted elsewhere are skipped. The registry file is safe to share between processes. The agent templates (skill,skill/templates/*,write-agents-md, docs) no longer tell agents to runrag_index_projectafter every task./Users/...) and a personal deployment name removed from README, AGENTS.md, CLAUDE.md and the skill.Behavior changes
SyncResultgainsupdated, and therag_index_projecttool returnschunks_updated.Testing
PayloadUpdater), member splitting per language, SQLiteUpdatePayload, and auto-sync (edits seen without re-index, interval/off, missing dir, deleted project not recreated, same-name refusal, shared registry, interval parsing).pkg/rag,pkg/indexerandpkg/corepassgo test -raceandgo vetlocally on Go 1.24, using a test-only SQLite driver build because the Go module proxy wasn't reachable from the dev environment.cmd/mcp-server,pkg/configandpkg/httpapi(wiring, one path helper, template text) could not be compiled locally and were reviewed by hand. CI on this PR is their first real build with Go 1.26.