Tool performance improvements - #115
Open
aron-cf wants to merge 11 commits into
Open
Conversation
coalesceChanges resolved every touched inode with pathsOf, which walks vfs_dirents parent-by-parent issuing one statement per ancestor. That is O(N x depth) round-trips per push tick — ~74k for a 20k-node tree. Every one of those lookups already hit a covering index; the cost was the statement count, not the plan. Add pathsOfMany: one recursive CTE resolves an entire batch of inodes to all of their hardlink names, seeded from json_each over the inode list and walking child_inode -> parent_inode upward. Hardlinks that share a parent directory need the seed's (parent, name) pair as the grouping key, not the parent inode alone — keying on the parent collapsed /one.txt and /two.txt into /one.txt/two.txt. Covered by a regression test. Unreachable inodes produce no seed row and are absent from the result, matching pathsOf returning [] for them.
find and grep had no way to skip a subtree. grep's `include` glob was applied to entries the walk had already visited, so `grep --include='*.ts' /` still walked all of node_modules and read every file before discarding the results. Add `exclude`: whole-segment names that are neither yielded nor descended into. The check runs before the recursion, so an excluded directory's subtree costs zero statements rather than being walked and filtered. Matching is exact per segment — "node_modules" does not prune "node_modules_extra". Tests cover depth, multiple patterns, interaction with the pattern/include glob, and assert the walk does not issue statements for pruned subtrees. On an 18,963-node tree: find(/) 143.4 ms -> 0.5 ms (287x) grep(/) over the same tree 11412.3 ms -> 4.4 ms (2594x) with identical results for the non-excluded paths.
A recursive grep issued three statements per file: the recursive-CTE
path resolve inside readFile, the chunk range read, and the blob fetch.
The traversal had already read the inode and size from vfs_dirents, so
the resolve was pure duplicate work — and it was the most expensive of
the three.
Carry inode and size out of the find walk (internally only; find()'s
public {path, type} shape is unchanged) and add
readCommittedFileByInode, which pulls a committed file's bytes straight
from the chunk store. Open write buffers are still honoured so an
in-flight write is not missed; pending creates have no inode and keep
the resolve path, as does grepping a single file directly.
Line framing is shared with the streaming decoder's semantics: a
trailing fragment without a newline is still a line and an empty file
yields nothing. Verified byte-identical against the streaming path for
empty files, missing trailing newlines, CRLF, unicode, and matches
straddling a 512KiB chunk boundary, with and without context lines.
statements per file 3.0 -> 2.0 (the resolve CTE: 501 -> 1 for 500 files)
grep over 18,963 nodes 11412 ms -> 814 ms (14x)
The push tick reads tombstones with `WHERE rev > ? AND op = 'delete' GROUP BY path`. Neither existing index serves it: vfs_changes_by_rev(rev) can drive the range but leaves GROUP BY path to a sort, so the planner instead scanned vfs_changes_by_path in path order and never applied the rev predicate — reading the entire table to return the few rows in the watermark window. Add vfs_changes_by_op_rev(op, rev): equality column first, range column second, which is the shape SQLite can drive both halves from. Harmless while node_modules was ignored and the table stayed small. Now every reinstall appends thousands of tombstones and the table only grows, so the scan cost grew with history rather than with the window. Schema v5 -> v6. Index-only migration: no table rewrite, no row touched. Tests assert the index exists on a fresh install, that the migrator adds it to an existing v5 database without disturbing tombstones, and that the planner actually uses it. 100,000 rows, 5 in window: 29.64 ms -> 0.01 ms cost is now O(window) rather than O(table)
just-bash's find and grep are filesystem-agnostic: they walk through
IFileSystem, one readdirWithFileTypes per directory. Against the
workspace stub every one is an RPC to the DO, so a recursive command
over a tree containing node_modules costs hundreds to thousands of
round-trips before any matching work happens.
Give the adapter an opt-in prefetch scope. On the first listing inside
a scope it reads the whole subtree with one server-side find() and
answers later readdirWithFileTypes calls from that snapshot. The shell
entrypoint opens a scope only for commands that actually traverse
(find, grep -r), picked by prefetch-policy.
Correctness constraints, each covered by a test:
* the snapshot lives only for the command that opened it, so no
listing is ever reused across commands;
* every mutating adapter method drops it, so a walk that writes
re-reads instead of trusting the snapshot;
* paths outside the prefetched root fall through to a direct listing;
* a failed prefetch degrades to direct listings rather than failing
the command;
* dofs's find reports "symlink" at runtime even though the published
type is narrower, so links keep isSymbolicLink rather than being
mislabelled as files.
The policy declines mutating traversals (-delete, -exec) where a
snapshot taken before the walk could describe entries the command
removes, and declines non-recursive greps that would not benefit.
find / 412 -> 9 stub calls (45.8x)
grep -rl / 832 -> 429 (1.9x; the traversal is gone, per-file reads
remain)
Output is byte-identical with the scope on and off in every case.
Content prefetch was prototyped and rejected: file bodies can only be
reconstructed from grep matches lossily (a file with no trailing
newline comes back with one), and silently altering file content is
not worth the remaining round-trips.
diffWith computed the full worktree status matrix and then discarded
everything outside opts.paths with makePathFilter. isomorphic-git can
prune the walk itself, so the traversal was visiting — and stat'ing and
hashing — the entire tree to produce a result the caller had already
narrowed.
Pass the caller's paths through as filepaths. The library's matching
rule (exact path or directory prefix) is the same one makePathFilter
implements, so the filter stays the authority on what is emitted and
this is purely a traversal hint. Callers that name no paths keep
isomorphic-git's default of walking everything.
This only became expensive once node_modules was synced into the
worktree: a repo whose .gitignore does not cover it makes every scoped
diff pay for the vendor tree.
300-package vendor tree, :
unscoped 294 ms, 4,865 fs calls
scoped 14 ms, 64 fs calls (21x, 76x fewer calls)
Scoped rows verified identical to the full run's src subset.
Prevent an invalidated in-flight subtree load from publishing a stale directory snapshot after a concurrent mutation. A generation check now limits cache publication to the active prefetch scope.
Keep the path-resolution savings from inode-based grep reads without materializing whole files. Chunk bytes are loaded only as the line scanner pulls them, preserving bounded memory use for large files.
Document the exact path-segment exclusion behavior for find and grep, and add release metadata for the large-tree performance changes.
🦋 Changeset detectedLatest commit: 0f0f397 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Collaborator
Author
|
@agent-think can you fix the broken tests |
ReadableStream's default queue can pull a chunk as soon as the stream is created. Use a zero high water mark so blob reads begin only when a consumer requests data across both supported stream runtimes.
aron-cf
marked this pull request as ready for review
August 21, 2026 19:16
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #113
node_moduleshas now become a default part of workspace sync. Performance of routine file operations, grep/find/git etc have degraded. A sync pass looked up paths one at a time, recursive searches visited directories the caller did not care about, the Worker shell asked the durable object for each directory separately, and a focused Git diff still inspected the whole project. These costs add up quickly after a package install.This change makes those operations focus on the relevant part of the workspace. Sync resolves paths in groups and can find recent deletions without searching the full history. Recursive shell commands reuse one fresh directory listing for the command, while writes clear that listing so later reads stay current. Grep reuses information collected during its directory walk and still reads large files a piece at a time. A Git diff limited to a path now avoids walking unrelated directories.
The user-facing addition is an
excludeoption forfindandgrep. It matches a complete directory or file name, sonode_modulesis skipped whilenode_modules_extrais not.On a test workspace with 18,963 entries, excluding the package directory reduced
findfrom about 143 ms to under 1 ms and reducedgrepfrom about 11 seconds to about 4 ms. A scoped Git diff over a project with 300 packages fell from 294 ms to 14 ms. These numbers are workload-specific, but they show that the commands now avoid the unrelated work rather than making the same walk faster by a small amount.The tests cover grouped path lookup, files with more than one name, upgrades from existing workspaces, excluded directories, matches that cross file chunks, large-file reading, fresh directory listings after writes, and focused Git diffs. The filesystem documentation now describes the new option and its exact-name behavior, and the changeset records it for the next release.