You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(SESF-67): configurable issue ID patterns and retag-issues - #135
issue_patterns.py decides which tokens count as issue IDs. The dashed preset (SESF-25, OPS-50) is always on. ~/.sessionflow/issue_ids.conf, or the path in SESSIONFLOW_ISSUE_ID_PATTERNS, adds lines:
preset:servicenow: INC|CHG|REQ|RITM|CTASK|PRB|SCTASK|TASK plus seven digits.
regex:<pattern>: any other ticket system.
deny:<prefix>: added to the built-in denylist.
A bad line logs a WARN with its line number and is skipped. The file reloads when its mtime changes.
rag_engine._extract_issue_ids (ingest and memory tags) and provider_adapters.is_valid_issue_token (the issue_id filter and get_issue_timeline) both read the active patterns.
A stored token holds only letters, digits and inner dashes. A user regex therefore cannot put a comma or a LIKE wildcard into issue_ids or into a filter.
get_stats and get_session_stats list the patterns in force, the config file and how many lines were skipped.
It re-extracts from every stored document. Passages take their parent turn's tags, as at ingest.
It also lists rows whose FTS value alone is stale.
It writes through update_row_metadata, so vectors are kept and nothing is re-embedded.
It is a dry run by default, and --apply refuses without --yes before any read.
issue_ids.conf.example, an Issue IDs section in the README, the env var row, and the operational note in CLAUDE.md and AGENTS.md.
Deviations from the issue text
The CLI is retag-issues [--apply --yes] rather than {scan|apply --yes}, to match retag-project, restamp-codex and dedupe-*.
The issue text has no way to turn the dashed preset off, so it stays on and preset:dashed is accepted as a no-op. A file with only preset:servicenow therefore keeps Plane, Linear and Jira tags.
Patterns match case-insensitively, including regex: lines, because tokens are stored and validated in upper case.
Memories are not re-tagged. They keep the tags they were written with, and new memories use the new patterns.
OPS-50 finding
On the live index, read-only:
15 turns and 17 passages contain OPS-50, and every one is tagged. A total of 47 rows carry the tag.
get_issue_timeline("OPS-50") returns all 15 turns.
The single-turn report does not reproduce. It predates the tag backfill.
Live dry runs (read-only)
Default config: 0 of 159,222 rows change, including 0 that are stale only in FTS. Extraction and the passage rule agree with ingest.
preset:servicenow: 400 rows would gain tags across 30 IDs. Nothing was applied.
Tests
Tests written first: tests/test_issue_patterns.py and tests/test_issue_retag.py. The retag tests use an autouse in-memory Milvus, so they never reach the live index.
The autouse _default_issue_patterns fixture keeps a developer's config file out of every test.
Locally, ruff check . is clean, and pytest -rs gives 1321 passed and 155 skipped. The skipped tests are the Postgres ones, because no local DSN was set; CI runs them.
Vale reports no alerts on the lines this PR adds.
Version: 0.21.0 (minor): adds the SESSIONFLOW_ISSUE_ID_PATTERNS env var, a config file and the retag-issues CLI command.
Summary by CodeRabbit
New Features
Added configurable issue ID recognition, with support for built-in and optional patterns, denied prefixes, and automatic reloading when configuration changes.
Issue IDs are used consistently for tagging and filtering, and active pattern details are included in session statistics.
Added a retagging command to preview or update issue IDs on existing records, including related passages. Updates preserve embeddings and require explicit confirmation.
Documentation
Added guidance and an example configuration for issue ID patterns and retagging.
Issue IDs come from issue_patterns.active(): the dashed preset stays on, and
~/.sessionflow/issue_ids.conf (or SESSIONFLOW_ISSUE_ID_PATTERNS) adds
preset:servicenow, regex: and deny: lines. Extraction, memory tags and every
issue_id check follow it; get_session_stats lists the patterns in force.
cleanup.py retag-issues re-extracts from stored documents, gives passages
their parent's tags, and rewrites changed rows through update_row_metadata
without re-embedding. Dry run by default; --apply refuses without --yes.
No actionable comments were generated in the recent review.
Walkthrough
Issue-ID extraction and validation now use configurable patterns. The change also adds a cleanup command to recalculate issue tags for stored rows and update corresponding FTS metadata.
Configurable patterns support the built-in dashed format, ServiceNow preset, custom regexes, and denied prefixes. Extraction and token validation use the active patterns. Statistics report the active configuration. Tests cover parsing, matching, reloads, and reporting.
Stored-row retag planning and updates issue_retag.py, tests/test_issue_retag.py
The planner derives issue IDs from stored documents, assigns a passage its parent’s computed tags when available, and identifies differences in Milvus and FTS. Applying a plan updates metadata while retaining vectors. Tests cover planning, writes, and failures.
The retag-issues command reports planned changes by default and requires --apply --yes to write updates. Documentation describes pattern configuration and retagging. The version changes to 0.21.0.
sequenceDiagram
actor Operator
participant CleanupCLI
participant IssueRetag
participant RAGEngine
participant IssuePatterns
participant Milvus
participant FTS
Operator->>CleanupCLI: Run retag-issues
CleanupCLI->>IssueRetag: Plan row changes
IssueRetag->>Milvus: Scan stored rows
IssueRetag->>RAGEngine: Extract IDs from documents
RAGEngine->>IssuePatterns: Match using active patterns
IssueRetag->>FTS: Read indexed issue IDs
IssueRetag-->>CleanupCLI: Return planned changes
CleanupCLI->>IssueRetag: Apply after --apply --yes
IssueRetag->>Milvus: Update row metadata
IssueRetag->>FTS: Update indexed issue IDs
Loading
Possibly related PRs
lbruton/SessionFlow#78: Its memory-tagging path calls the extractor that now uses configurable issue patterns.
lbruton/SessionFlow#106: It introduces passage rows and their parent relationship, which the retag planner uses when assigning passage tags.
lbruton/SessionFlow#91: It adds stored-row rewriting in cleanup.py, which is also used by the retagging change.
Merge Risk: Minimal · previous review 4e3f9
No merge-blocking issue was identified in the reviewed changes. The retagging workflow is ready for normal checks.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title names the two main changes: configurable issue ID patterns and the retag-issues command. Both appear across the changeset (issue_patterns.py, cleanup.py, issue_retag.py, docs). The title als…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the two main changes: configurable issue ID patterns and the retag-issues command. Both appear across the changeset (issue_patterns.py, cleanup.py, issue_retag.py, docs). The title also carries a SESF-67 ticket prefix and conventional-commit form, which is acceptable.
Finishing Touches
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Dashed IDs meet new patterns in the stream ServiceNow joins the matching team Stored tags are scanned and compared FTS and vectors are handled with care A confirmed command completes the scheme
Reviewing files that changed from the base of the PR and between 25150ae and 4e3f952.
Files selected for processing: 15
AGENTS.md
CHANGELOG.md
CLAUDE.md
README.md
cleanup.py
issue_ids.conf.example
issue_patterns.py
issue_retag.py
provider_adapters.py
rag_engine.py
tests/conftest.py
tests/test_issue_patterns.py
tests/test_issue_retag.py
tools.py
version.py
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer TIP This summary will be updated as you push new changes.
_rewrite_fts_row deletes before it inserts, so a failed insert leaves Milvus
right and FTS without the row. The planner skipped such rows, and a re-run
never restored them. When the sidecar reads, a missing row now counts as
stale.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Active moderate and critical findings remain, including an FTS read-error path that can report success after leaving stale rows unresolved and an unused import that will fail the Ruff gate.
This pull request adds configurable issue-ID recognition and a retroactive retagging workflow for existing Milvus and FTS records without re-embedding vectors.
Changes:
Adds built-in, ServiceNow, custom regex, and deny-prefix issue patterns with automatic configuration reloads.
Integrates active patterns into extraction, validation, filtering, timelines, statistics, and memory tagging.
Adds the dry-run and confirmed-apply cleanup.py retag-issues command.
Updates tests, documentation, operational guidance, changelog, and version metadata.
File
Description
version.py
Bumps the project version to 0.21.0.
tools.py
Displays active issue-pattern configuration in statistics output.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Active moderate issues remain: a failed FTS scan can allow a partial apply to report success, and invalid UTF-8 pattern configuration can escape loading instead of falling back safely.
open(..., encoding="utf-8") can raise UnicodeDecodeError, which is not an OSError; an invalid config therefore escapes load() and breaks any ingest, search, or stats call that refreshes the active patterns. Treat decode failures like other unreadable config files and fall back to the dashed preset.
- Extraction keeps only tokens a pattern matches on their own, the check
matches() uses, and lookaround lines are refused, so a stored token always
passes an issue_id filter.
- An unreadable configured file keeps its path and the error in describe().
- An orphan passage keeps its stored tags but is still compared with FTS.
- retag-issues exits 1 before any write when the FTS sidecar cannot be read.
- Drop the unused re import from provider_adapters.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A critical race condition can apply tags from stale scan data to a newer document, potentially creating inconsistent metadata and falsely reporting success.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Stale scan results can apply tags to updated documents
issue_retag.py:139
The plan computes tags from a scan snapshot, but apply passes those strings to update_row_metadata by doc_id without checking that the document is still the one scanned. If the watcher or backfill updates an open turn between the scan and apply, the metadata rewrite preserves the newer document while attaching tags computed from the old text, and the command can report success with inconsistent data. Revalidate and extract from the current row before writing, or add a document/version compare-and-swap and retry the row.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because a concurrent watcher or backfill can cause the run to overwrite newer document tags with metadata computed from stale data.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the review identified multiple hidden correctness and concurrency risks affecting numeric prefix handling, newline validation, and stale writes during document updates.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the change contains a critical configuration-error path that can remove non-dashed tags and a moderate documented prefix-matching defect.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the change is unusually complex and hidden findings include a deny-rule parsing edge case and a critical non-atomic migration write that could overwrite newer data.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The live Milvus/FTS rewrite requires operator-controlled validation for locking, partial-write recovery, and concurrent ingestion scenarios that dry-run and in-memory tests cannot cover.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because a critical concurrency risk could allow a stale document snapshot to overwrite newer data while the operation reports success.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Approval should wait because retag-issues performs a large live Milvus/FTS dual-store rewrite whose production throughput, schema behavior, and concurrent-writer consistency are not validated by fake-backed tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
SESF-67 (parent SESF-61, item 5).
What changes
issue_patterns.pydecides which tokens count as issue IDs. The dashed preset (SESF-25,OPS-50) is always on.~/.sessionflow/issue_ids.conf, or the path inSESSIONFLOW_ISSUE_ID_PATTERNS, adds lines:preset:servicenow:INC|CHG|REQ|RITM|CTASK|PRB|SCTASK|TASKplus seven digits.regex:<pattern>: any other ticket system.deny:<prefix>: added to the built-in denylist.rag_engine._extract_issue_ids(ingest and memory tags) andprovider_adapters.is_valid_issue_token(theissue_idfilter andget_issue_timeline) both read the active patterns.issue_idsor into a filter.get_statsandget_session_statslist the patterns in force, the config file and how many lines were skipped.cleanup.py retag-issues [--apply --yes](issue_retag.py):document. Passages take their parent turn's tags, as at ingest.update_row_metadata, so vectors are kept and nothing is re-embedded.--applyrefuses without--yesbefore any read.issue_ids.conf.example, an Issue IDs section in the README, the env var row, and the operational note in CLAUDE.md and AGENTS.md.Deviations from the issue text
retag-issues [--apply --yes]rather than{scan|apply --yes}, to matchretag-project,restamp-codexanddedupe-*.preset:dashedis accepted as a no-op. A file with onlypreset:servicenowtherefore keeps Plane, Linear and Jira tags.regex:lines, because tokens are stored and validated in upper case.OPS-50 finding
On the live index, read-only:
OPS-50, and every one is tagged. A total of 47 rows carry the tag.get_issue_timeline("OPS-50")returns all 15 turns.Live dry runs (read-only)
preset:servicenow: 400 rows would gain tags across 30 IDs. Nothing was applied.Tests
tests/test_issue_patterns.pyandtests/test_issue_retag.py. The retag tests use an autouse in-memory Milvus, so they never reach the live index._default_issue_patternsfixture keeps a developer's config file out of every test.ruff check .is clean, andpytest -rsgives 1321 passed and 155 skipped. The skipped tests are the Postgres ones, because no local DSN was set; CI runs them.Version: 0.21.0 (minor): adds the
SESSIONFLOW_ISSUE_ID_PATTERNSenv var, a config file and theretag-issuesCLI command.Summary by CodeRabbit