Skip to content

Commit 70e7f1d

Browse files
icecrasher321claude
andcommitted
feat(file): search workspace files by regular expression
Search read its query as literal text. It now reads it as a line-oriented regular expression by default, with a Match setting on the block to go back to verbatim text. The segment store and its `gin_trgm_ops` index already support this: pg_trgm extracts trigrams from a regex source too, so `~` / `~*` plan as a bitmap index scan exactly like `LIKE` / `ILIKE`. No migration, no new index. One compiled pattern owns every mode-specific decision — how PostgreSQL matches a segment, whether the segment must hold a whole line, and where the match sits inside it — so the repository builds one query shape and the preview renderer one preview shape. Compilation happens in the application use case, not the route adapter, so every surface gets the same semantics. The supported syntax is the intersection of PostgreSQL ARE and JavaScript RegExp, because the same source drives both the indexed predicate and the client-side match location a preview centres on. Anything the two engines read differently is rejected by name rather than silently reinterpreted, and `\b` is rewritten to `\y` on the way to PostgreSQL. Safety, in four independent layers: - A pattern must contain 3 consecutive literal characters every match will include. pg_trgm indexes nothing shorter, and an unextractable pattern plans as a sequential scan across every workspace's segments. - `new RegExp` proves it compiles in JavaScript. - PostgreSQL proves it compiles in ARE; 2201B becomes a 400, not a 500. - `statement_timeout` bounds the read. This one covers exact matching too, which has always been able to reach the same scan through a punctuation-only or non-ASCII query. `mode` is a builder setting, withheld from the model like `maxResults`. The model cannot see it and the two readings disagree on every metacharacter, so `toolEnrichment` replaces the declared syntax with the active mode's — a regex sent to a block set to exact matching would otherwise be searched for verbatim and silently find nothing. Verified against PostgreSQL 17: 14 behavioural checks end to end, a live search over 150,012 segments in 14ms, 6/6 representative patterns reaching the trigram index, and the guard cutting a 12s pattern at 10.08s into an actionable message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 23e1663 commit 70e7f1d

17 files changed

Lines changed: 1492 additions & 209 deletions

File tree

apps/sim/blocks/blocks/file.test.ts

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,15 +72,29 @@ describe('FileV5Block', () => {
7272
query: '',
7373
maxResults: '25',
7474
})
75-
).toEqual({ query: '', maxResults: 25 })
75+
).toEqual({ query: '', mode: 'regex', maxResults: 25 })
7676

7777
const query = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'query')
78+
const mode = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'mode')
7879
const maxResults = FileV5Block.subBlocks.find((subBlock) => subBlock.id === 'maxResults')
7980
expect(query?.paramVisibility).toBe('user-or-llm')
81+
expect(mode?.paramVisibility).toBe('user-only')
8082
expect(maxResults?.paramVisibility).toBe('user-only')
8183
expect(query?.canonicalParamId).toBeUndefined()
8284
expect(maxResults?.canonicalParamId).toBeUndefined()
8385
expect(maxResults?.value?.()).toBe('50')
86+
expect(mode?.value?.()).toBe('regex')
87+
})
88+
89+
it.each([
90+
[undefined, 'regex'],
91+
['exact', 'exact'],
92+
['regex', 'regex'],
93+
['glob', 'regex'],
94+
])('resolves the builder-configured match mode %s to %s', (mode, expected) => {
95+
expect(buildParams({ operation: 'file_search', query: 'needle', mode })).toMatchObject({
96+
mode: expected,
97+
})
8498
})
8599

86100
it('uses the default search cap when the builder field is cleared', () => {
@@ -90,7 +104,7 @@ describe('FileV5Block', () => {
90104
query: 'needle',
91105
maxResults: '',
92106
})
93-
).toEqual({ query: 'needle', maxResults: 50 })
107+
).toEqual({ query: 'needle', mode: 'regex', maxResults: 50 })
94108
})
95109

96110
it.each(['10.5', '10results', '0', '201'])(

apps/sim/blocks/blocks/file.ts

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -908,7 +908,9 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
908908
- Get Content is how you read file text. It accepts file objects or canonical file IDs and returns a "contents" array with one extracted text string per file (PDF, DOCX, CSV, etc. are parsed automatically).
909909
- To read the text of files produced by another block, chain into Get Content: set its file input to the upstream file output, e.g. <file.files>, <agent.files>, or <start.files>. Never assume Read (or any file-object output) already contains the text.
910910
- Get Content's "contents" can be large; it is persisted through the execution large-value system automatically, so prefer it over inlining file text any other way.
911-
- Search finds literal text across all active workspace files and returns structured results with fileId, lineNumber, and text. Lowercase queries are case-insensitive; adding any uppercase letter makes the search case-sensitive.
911+
- Search finds text across all active workspace files and returns structured results with fileId, lineNumber, and text. Lowercase queries are case-insensitive; adding any uppercase letter makes the search case-sensitive.
912+
- Search reads the query as a line-oriented regular expression: quantifiers, character classes, \\d \\w \\s, alternation, groups, "^" and "$" anchors, and \\b word boundaries. Lookaround, backreferences and patterns spanning a line break are not supported, and a pattern needs at least 3 consecutive literal characters that every match will contain. Set Match to "Exact match" to search for the query text verbatim instead.
913+
- Match is a builder setting, not an agent one: the agent writes the query, and Match decides how every query from that block is read.
912914
- Search is eventually consistent. Check "complete" and "indexStatus" when pending, failed, skipped, or partially indexed files matter to the task.
913915
- Use Fetch for external file URLs. Add headers for authenticated downloads, for example Slack private file URLs require an Authorization Bearer token.
914916
- Use Write to create a new workspace file and Append to add content to an existing one. Write adds a numeric suffix when the name is taken; turn on "Overwrite Existing File" to replace the contents of the file at that exact path (folder and name) instead — a same-named file in another folder is left alone.
@@ -1007,12 +1009,26 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
10071009
condition: { field: 'operation', value: 'file_get_content' },
10081010
required: { field: 'operation', value: 'file_get_content' },
10091011
},
1012+
{
1013+
id: 'mode',
1014+
title: 'Match',
1015+
type: 'dropdown' as SubBlockType,
1016+
options: [
1017+
{ label: 'Regular expression', id: 'regex' },
1018+
{ label: 'Exact match', id: 'exact' },
1019+
],
1020+
description:
1021+
'How the query is read. Regular expressions match one line at a time and need at least 3 consecutive literal characters.',
1022+
value: () => 'regex',
1023+
condition: { field: 'operation', value: 'file_search' },
1024+
paramVisibility: 'user-only',
1025+
},
10101026
{
10111027
id: 'query',
10121028
title: 'Query',
10131029
type: 'short-input' as SubBlockType,
1014-
placeholder: 'Text to find across workspace files',
1015-
description: 'Literal search text, 3-512 characters. Leave blank for the agent to supply.',
1030+
placeholder: 'Pattern to find across workspace files',
1031+
description: 'Search pattern, 3-512 characters. Leave blank for the agent to supply.',
10161032
condition: { field: 'operation', value: 'file_search' },
10171033
required: { field: 'operation', value: 'file_search' },
10181034
paramVisibility: 'user-or-llm',
@@ -1268,6 +1284,7 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
12681284
}
12691285
return {
12701286
query: params.query,
1287+
mode: params.mode === 'exact' ? 'exact' : 'regex',
12711288
maxResults,
12721289
}
12731290
}
@@ -1500,7 +1517,11 @@ export const FileV5Block: BlockConfig<FileParserV3Output> = {
15001517
type: 'string',
15011518
description: 'Operation to perform (read, search, get content, fetch, write, or append)',
15021519
},
1503-
query: { type: 'string', description: 'Literal workspace file search query' },
1520+
query: { type: 'string', description: 'Workspace file search query' },
1521+
mode: {
1522+
type: 'string',
1523+
description: 'How the search query is read: a regular expression (default) or an exact match',
1524+
},
15041525
maxResults: { type: 'number', description: 'Hard maximum search results (1-200)' },
15051526
readFileInput: {
15061527
type: 'json',

apps/sim/lib/internal/file/execute-tool.test.ts

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,13 @@ describe('executeFileTool', () => {
148148
})
149149
expect(mocks.searchContent).toHaveBeenCalledWith({
150150
principal: expect.objectContaining({ serviceId: 'executor' }),
151-
input: { workspaceId: 'workspace-1', query: 'needle', maxResults: 25, signal: undefined },
151+
input: {
152+
workspaceId: 'workspace-1',
153+
query: 'needle',
154+
mode: 'regex',
155+
maxResults: 25,
156+
signal: undefined,
157+
},
152158
})
153159
expect(mocks.executeManage).not.toHaveBeenCalled()
154160
})
@@ -168,7 +174,7 @@ describe('executeFileTool', () => {
168174

169175
expect(mocks.searchContent).toHaveBeenCalledWith(
170176
expect.objectContaining({
171-
input: { workspaceId: 'workspace-1', query: 'needle', maxResults: 50 },
177+
input: { workspaceId: 'workspace-1', query: 'needle', mode: 'regex', maxResults: 50 },
172178
})
173179
)
174180
expect(mocks.getProvenance).toHaveBeenCalledWith(
@@ -191,6 +197,7 @@ describe('executeFileTool', () => {
191197
[{ query: 'abc\0def', maxResults: 50 }, 400],
192198
[{ query: 'needle', maxResults: 201 }, 400],
193199
[{ query: 'needle', maxResults: 0 }, 400],
200+
[{ query: 'needle', mode: 'glob' }, 400],
194201
])('rejects invalid search input before authorization', async (input, status) => {
195202
const response = await executeFileTool(request('file_search', input))
196203

@@ -199,6 +206,16 @@ describe('executeFileTool', () => {
199206
expect(mocks.searchContent).not.toHaveBeenCalled()
200207
})
201208

209+
it('forwards an explicitly configured exact-match mode', async () => {
210+
await executeFileTool(request('file_search', { query: 'needle', mode: 'exact' }))
211+
212+
expect(mocks.searchContent).toHaveBeenCalledWith(
213+
expect.objectContaining({
214+
input: expect.objectContaining({ query: 'needle', mode: 'exact' }),
215+
})
216+
)
217+
})
218+
202219
it('does not expose unexpected search infrastructure errors', async () => {
203220
mocks.searchContent.mockRejectedValueOnce(new Error('database host and query details'))
204221

apps/sim/lib/internal/file/execute-tool.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ import {
3131
FILE_SEARCH_MAX_RESULTS,
3232
FILE_SEARCH_MIN_QUERY_LENGTH,
3333
} from '@/lib/workspace-files/search/constants'
34+
import { FILE_SEARCH_MODES } from '@/lib/workspace-files/search/pattern'
3435

3536
const logger = createLogger('FileToolExecution')
3637

@@ -57,6 +58,7 @@ const fileSearchInputSchema = z
5758
.min(FILE_SEARCH_MIN_QUERY_LENGTH)
5859
.max(FILE_SEARCH_MAX_QUERY_LENGTH)
5960
.refine((query) => !query.includes('\0'), 'Search query cannot contain NUL characters'),
61+
mode: z.enum(FILE_SEARCH_MODES).default('regex'),
6062
maxResults: z
6163
.number()
6264
.int()
@@ -110,6 +112,7 @@ export const executeFileTool: InternalToolOperationHandler = async (request) =>
110112
input: {
111113
workspaceId,
112114
query: searchInput.data.query,
115+
mode: searchInput.data.mode,
113116
maxResults: searchInput.data.maxResults,
114117
signal: request.signal,
115118
},

apps/sim/lib/workspace-files/application/search-workspace-file-content.ts

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,17 @@ import { OrchestrationError } from '@/lib/core/orchestration/types'
22
import { loadActiveWorkspaceContext } from '@/lib/uploads/contexts/workspace'
33
import { defineAuthorizedWorkspaceFileUseCase } from '@/lib/workspace-files/application/authorized-workspace-file-use-case'
44
import { fileOperations } from '@/lib/workspace-files/application/operations'
5+
import {
6+
compileFileSearchPattern,
7+
type FileSearchMode,
8+
FileSearchPatternError,
9+
} from '@/lib/workspace-files/search/pattern'
510
import { searchWorkspaceFileIndex } from '@/lib/workspace-files/search/repository'
6-
import { isFileSearchCaseSensitive } from '@/lib/workspace-files/search/text'
711

812
export interface SearchWorkspaceFileContentInput {
913
workspaceId: string
1014
query: string
15+
mode: FileSearchMode
1116
maxResults: number
1217
signal?: AbortSignal
1318
}
@@ -24,12 +29,24 @@ export const searchWorkspaceFileContent = defineAuthorizedWorkspaceFileUseCase({
2429
operation: fileOperations.searchContent,
2530
resolveContext: ({ input }: { input: SearchWorkspaceFileContentInput }) =>
2631
resolveSearchWorkspaceFileContext(input),
27-
execute: ({ input, context }) =>
28-
searchWorkspaceFileIndex({
29-
workspaceId: context.workspaceId,
30-
query: input.query,
31-
maxResults: input.maxResults,
32-
caseSensitive: isFileSearchCaseSensitive(input.query),
33-
signal: input.signal,
34-
}),
32+
execute: async ({ input, context }) => {
33+
try {
34+
return await searchWorkspaceFileIndex({
35+
workspaceId: context.workspaceId,
36+
pattern: compileFileSearchPattern(input.query, input.mode),
37+
maxResults: input.maxResults,
38+
signal: input.signal,
39+
})
40+
} catch (error) {
41+
/**
42+
* A rejected or too-expensive pattern is the caller's to fix, and the
43+
* message names the construct and the supported alternative — so it is
44+
* classified rather than left to become the surface's generic failure text.
45+
*/
46+
if (error instanceof FileSearchPatternError) {
47+
throw new OrchestrationError('validation', error.message)
48+
}
49+
throw error
50+
}
51+
},
3552
})

apps/sim/lib/workspace-files/search/constants.ts

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,29 @@
1+
/**
2+
* `pg_trgm` can only extract a trigram from three consecutive characters, so a
3+
* shorter query has nothing for the segment GIN index to probe and degrades to a
4+
* scan of every tenant's segments. It bounds the literal query length and, in
5+
* regex mode, the shortest literal run every match is guaranteed to contain.
6+
*/
17
export const FILE_SEARCH_MIN_QUERY_LENGTH = 3
28
export const FILE_SEARCH_MAX_QUERY_LENGTH = 512
9+
10+
/**
11+
* Caps the analyzer's bookkeeping strings so a bounded repeat cannot expand a
12+
* short pattern into a large intermediate. Only {@link FILE_SEARCH_MIN_QUERY_LENGTH}
13+
* characters are ever needed, so truncating past this loses no decision.
14+
*/
15+
export const FILE_SEARCH_PATTERN_LITERAL_CAP = 512
16+
export const FILE_SEARCH_PATTERN_MAX_REPEAT = 1000
17+
export const FILE_SEARCH_PATTERN_MAX_DEPTH = 20
18+
19+
/**
20+
* Backstop for a pattern whose trigrams the planner cannot use — a punctuation-only
21+
* or non-ASCII literal, or a regex whose guaranteed run yields no trigram. Those
22+
* plan as a sequential scan across every workspace's segments, so the search must
23+
* not be able to hold a pooled connection open indefinitely.
24+
*/
25+
export const FILE_SEARCH_STATEMENT_TIMEOUT_MS = 10 * 1000
26+
export const FILE_SEARCH_LOCK_TIMEOUT_MS = 5 * 1000
327
export const FILE_SEARCH_DEFAULT_MAX_RESULTS = 50
428
export const FILE_SEARCH_MAX_RESULTS = 200
529

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
import { describe, expect, it } from 'vitest'
2+
import {
3+
compileFileSearchPattern,
4+
escapeFileSearchLikePattern,
5+
FileSearchPatternError,
6+
isFileSearchCaseSensitive,
7+
} from '@/lib/workspace-files/search/pattern'
8+
9+
describe('compileFileSearchPattern', () => {
10+
describe('shared query rules', () => {
11+
it.each(['exact', 'regex'] as const)('bounds and screens the query in %s mode', (mode) => {
12+
expect(() => compileFileSearchPattern('ab', mode)).toThrow(/at least 3 characters/)
13+
expect(() => compileFileSearchPattern('a'.repeat(513), mode)).toThrow(/at most 512/)
14+
expect(() => compileFileSearchPattern('abc\0def', mode)).toThrow(/NUL/)
15+
})
16+
})
17+
18+
describe('exact mode', () => {
19+
it('implements Unicode smart-case and escapes LIKE metacharacters', () => {
20+
expect(isFileSearchCaseSensitive('résumé')).toBe(false)
21+
expect(isFileSearchCaseSensitive('Résumé')).toBe(true)
22+
expect(isFileSearchCaseSensitive('東京A')).toBe(true)
23+
expect(escapeFileSearchLikePattern('100%_done\\')).toBe('100\\%\\_done\\\\')
24+
})
25+
26+
it('wraps the escaped query for LIKE and keeps the raw text for ranking', () => {
27+
const pattern = compileFileSearchPattern('100%_done', 'exact')
28+
29+
expect(pattern).toMatchObject({
30+
mode: 'exact',
31+
caseSensitive: false,
32+
sqlPattern: '%100\\%\\_done%',
33+
literalText: '100%_done',
34+
wholeLineOnly: false,
35+
})
36+
})
37+
38+
it('reads regex metacharacters as text', () => {
39+
const pattern = compileFileSearchPattern('a.c', 'exact')
40+
41+
expect(pattern.sqlPattern).toBe('%a.c%')
42+
expect(pattern.findMatchRange('xxabcxx')).toEqual({ start: 0, end: 3 })
43+
})
44+
})
45+
46+
describe('regex mode', () => {
47+
it('matches through PostgreSQL spelling while ranking has no fixed literal', () => {
48+
const pattern = compileFileSearchPattern('\\bTODO\\b', 'regex')
49+
50+
expect(pattern).toMatchObject({
51+
mode: 'regex',
52+
sqlPattern: '\\yTODO\\y',
53+
literalText: null,
54+
caseSensitive: true,
55+
})
56+
})
57+
58+
it('derives smart case from literals, not from metacharacters', () => {
59+
expect(compileFileSearchPattern('error \\d+', 'regex').caseSensitive).toBe(false)
60+
expect(compileFileSearchPattern('error \\D+', 'regex').caseSensitive).toBe(false)
61+
expect(compileFileSearchPattern('[A-Z]+ error', 'regex').caseSensitive).toBe(false)
62+
expect(compileFileSearchPattern('Error \\d+', 'regex').caseSensitive).toBe(true)
63+
})
64+
65+
it('restricts an anchored pattern to segments that hold a whole line', () => {
66+
expect(compileFileSearchPattern('^import x', 'regex').wholeLineOnly).toBe(true)
67+
expect(compileFileSearchPattern('import x;$', 'regex').wholeLineOnly).toBe(true)
68+
expect(compileFileSearchPattern('import x', 'regex').wholeLineOnly).toBe(false)
69+
})
70+
71+
it('requires a literal run long enough for the trigram index to be used', () => {
72+
expect(() => compileFileSearchPattern('\\w+ \\d+', 'regex')).toThrow(FileSearchPatternError)
73+
expect(() => compileFileSearchPattern('\\w+ \\d+', 'regex')).toThrow(
74+
/at least 3 consecutive literal characters/
75+
)
76+
expect(() => compileFileSearchPattern('\\d{4}-\\d{2}-\\d{2}', 'regex')).toThrow(
77+
/at least 3 consecutive literal characters/
78+
)
79+
expect(() => compileFileSearchPattern('error \\d+', 'regex')).not.toThrow()
80+
})
81+
82+
it('locates the match so a long line previews around it', () => {
83+
const pattern = compileFileSearchPattern('needle\\d+', 'regex')
84+
85+
expect(pattern.findMatchRange('xxx needle42 yyy')).toEqual({ start: 4, end: 12 })
86+
expect(pattern.findMatchRange('no match here')).toBeNull()
87+
})
88+
89+
it('never returns a range that splits a surrogate pair', () => {
90+
const pattern = compileFileSearchPattern('.needle', 'regex')
91+
const line = `a🙂needle`
92+
const range = pattern.findMatchRange(line)
93+
94+
expect(range).not.toBeNull()
95+
expect([...line.slice(range?.start, range?.end)].join('')).not.toContain('�')
96+
expect(line.slice(range?.start, range?.end)).toBe('🙂needle')
97+
})
98+
99+
it('does not carry match state between calls', () => {
100+
const pattern = compileFileSearchPattern('needle', 'regex')
101+
102+
expect(pattern.findMatchRange('a needle')).toEqual({ start: 2, end: 8 })
103+
expect(pattern.findMatchRange('a needle')).toEqual({ start: 2, end: 8 })
104+
})
105+
})
106+
})

0 commit comments

Comments
 (0)