Skip to content

feat(ai): steer the AI off shell edits and onto Edit/Write - #3124

Merged
abose merged 2 commits into
mainfrom
ai
Aug 22, 2026
Merged

feat(ai): steer the AI off shell edits and onto Edit/Write#3124
abose merged 2 commits into
mainfrom
ai

Conversation

@abose

@abose abose commented Aug 21, 2026

Copy link
Copy Markdown
Member

Two related fixes to how the in-app AI panel treats the user's files, both in src-node/claude-code-agent.js.

1. The AI stopped using the live preview

It only inspected the preview when explicitly asked. That is a regression, not model drift — two commits in late April 2026 removed both things that pointed it there:

  • ee1201f17 (let SDK run native Read/Edit/Write, preserve undo) dropped the per-edit hint that rode on the PreToolUse deny reason. isLivePreviewRelated survived the refactor but became UI-only — forwarded to the diff card and never shown to the model.
  • 21226fc5a deleted the last verification directive from appendSystemPrompt, leaving a capability catalogue with no instruction to ever use it. One line even pushed the other way: "These tools are for active iteration, not just final verification".

The only surviving nudge fired on plan approval, which is why the behaviour still showed up when you went through a plan.

Fix. A prompt line naming the preview as worth considering after editing livePreviewFile or a file it links to, plus a throttled runtime nudge reusing the isLivePreviewRelated flag the Edit/Write hooks already compute. It fires on the first unverified preview edit, then not until five more accumulate, resets whenever the model inspects the preview itself, and is capped at two per request.

Emitted from a new PostToolBatch hook. PostToolUse cannot own this state — per the SDK, it "may run concurrently for parallel tool calls", so a read-and-clear there would race. The PostToolUse catch-all keeps a fallback path for Claude CLI versions predating PostToolBatch, since Phoenix runs the user's global CLI.

2. Shell rewrites silently destroy Undo

Found while testing the above: the model reaches for sed -i to edit files. In Phoenix that is not equivalent to Edit/Write, which run through hooks that refresh the open buffer, paint the diff card backing the panel's Undo button, and carry the live preview signal. Measured on the same edit:

sed -i Edit
Diff card / Undo button 0 1
Buffer refresh skipped yes
Live preview signal never fires fires

So a shell edit lands on disk with no way for the user to undo it.

Fix. The prompt names the project root and scopes the rule to it — files under the root go through Edit/Write, scratch and temp files outside it are fair game. Framed as a default, not a ban: a mechanical change across many files is a fair reason to stay in the shell, but when the saving is marginal, Edit wins, since one shell call and one Edit call cost about the same.

Behind that, a PreToolUse speed bump: the first file-rewriting command in a request is denied with an explanation, and re-running it unchanged goes through.

Design notes

Three approaches were tried and measured, in this order:

  1. Hard deny — refused a sed -i that had been explicitly asked for by name. Rejected as user-hostile.
  2. Prompt only — measured, and it does not work. With the project-root rule and no enforcement the model still ran findcatsed and never touched Edit. (Notably the prompt was sufficient for the live preview change; overriding an ingrained tool habit is harder than suggesting an unused capability.)
  3. After-the-fact hint — fires once undo is already gone, so it cannot protect the edit that prompted it.

Hence deny-with-retry: the first edit is genuinely protected, and an intended command costs one extra round trip. One confirmation covers the rest of the request, because the model often has to fix its own command (BSD sed -i '' failing on GNU sed) and keying on the exact string charged a second bump for one logical operation.

Detection covers sed -i (GNU and BSD), perl -i, awk -i inplace, ed/ex, PowerShell Set-Content/Add-Content/Out-File, tee, and >/>> redirection. Redirection targets are found with a quote-aware scan so echo "a > b" and python -c "print(1 > 0)" are not misread. Device and scratch sinks are exempt on all three platforms: /dev/null, /var/folders and /private/tmp on macOS, NUL, $null, %TEMP%, AppData\Local\Temp and Git Bash's /c/temp on Windows.

Testing

A nine-prompt eval in one conversation against a scratch project — CSS, JS, Markdown, JSON, a 1602-line file, and a 12-file bulk change — with live preview active and style.css/script.js confirmed as related documents.

Prompt Bump Nudge Outcome
h1 colour in previewed CSS 1 → switched to Edit fired, ignored
tagline italic (follow-up) none attempted fired, ignored
button restyle + hover none attempted fired → took a screenshot
notes.md edit none none (correct)
one line in a 1602-line file none attempted none ✓ grep + targeted Read + Edit
12-file bulk replace 1 → retried none ✓ 12/12
node --version > /tmp/ver.txt none (exempt) none
create a new file none none ✓ Write

Two behaviours worth calling out. The bump teaches: after one denial the model used Edit directly for every later edit in the session, with no further attempts. The nudge discriminates: ignored for one-line tweaks, acted on for the substantive restyle — which is the intended judgement call, since it is advisory rather than mandatory.

Also verified by a 55-case corpus over the detection helpers (17 must-block, 38 must-allow), covering BSD and GNU sed, PowerShell, and Windows/macOS temp paths.

Limitations

  • Live-app testing was Linux only. macOS and Windows/Git Bash are covered by the corpus and by the CLI's own shell-resolution strings, but were never exercised in a running Phoenix.
  • The nudge depends on the preview having actually loaded the related resource. If the preview frame is not rendering, a linked stylesheet is not registered as related and the nudge correctly does not fire.
  • Which route the model takes at moderate scale (~12 files) is not deterministic — observed as both one sed and twelve Edits on the same prompt.
  • Detection is a heuristic, not a shell parser. False negatives are expected and acceptable; a false positive costs one round trip, not a refusal.
  • None of this applies to the panel's CLI mode (AIChatCLI.js), which spawns a bare claude with no MCP server and no appendSystemPrompt.

abose added 2 commits August 21, 2026 23:27
Two commits in April 2026 removed everything that pointed the model at
the live preview: ee1201f dropped the per-edit hint that rode on the
PreToolUse deny reason (isLivePreviewRelated survived, but only as a UI
flag for the diff card), and 21226fc deleted the last verification
line from appendSystemPrompt. Since then the AI inspected the preview
only when explicitly asked.

Restore the signal in two places:

- appendSystemPrompt now names the live preview as something worth
  considering after editing livePreviewFile or a file it links to,
  framed as the model's own judgement call rather than an obligation —
  aadad78 deliberately walked back the directive form, so this does
  not re-litigate it.

- A throttled runtime nudge reuses the isLivePreviewRelated flag the
  Edit/Write PostToolUse hooks already compute. It fires on the first
  unverified preview edit, then stays quiet until five more pile up,
  resets whenever the model inspects the preview itself, and is capped
  at two per request.

The nudge is emitted from a new PostToolBatch hook, which fires exactly
once after a batch resolves. PostToolUse cannot own this state: it may
run concurrently for parallel tool calls, so the read-and-clear would
race. The PostToolUse catch-all keeps a fallback path for Claude CLI
versions predating PostToolBatch, since we run the user's global CLI.
The model reaches for `sed -i` to change files, which in Phoenix is not
equivalent to Edit/Write. Edit and Write run through PostToolUse hooks
that refresh the open buffer, paint the diff card backing the panel's
Undo button, and carry the live preview signal. A shell rewrite skips
all three, so the change lands on disk with no way to undo it from the
panel — measured on a real run: the same edit produced a diff card via
Edit and none via sed.

Two parts, because neither alone was enough:

- appendSystemPrompt names the project root and scopes the rule to it.
  Files under the root go through Edit/Write; scratch and temp files
  outside it are fair game for the shell. Framed as a default rather
  than a ban: a mechanical change across many files, or one on a large
  file, is a fair reason to stay in the shell. When the saving would be
  marginal, Edit wins — one shell call and one Edit call cost about the
  same, so a handful of files is not a reason to give up undo.

- A PreToolUse speed bump. The first file-rewriting shell command in a
  request is denied with an explanation; re-running it unchanged goes
  through, and one confirmation covers the rest of that request. This
  is what actually protects the edit — the prompt alone was tried and
  measured first, and the model still ran find/cat/sed without ever
  touching Edit.

Not a hard block. An outright deny was tried and rejected: when the
user asks for a specific shell command by name, refusing it is worse
than the lost undo. Confirmation is by retry, so an intended command
costs one extra round trip and nothing else. Confirming once unlocks
the request because the model often has to fix its own command — BSD
`sed -i ''` failing on GNU sed — and keying on the exact string charged
a second bump for what is one operation.

Detection covers sed -i (GNU and BSD), perl -i, awk -i inplace, ed/ex,
PowerShell Set-Content/Add-Content/Out-File, tee, and `>`/`>>`
redirection. Targets are found with a quote-aware scan so `echo "a > b"`
and `python -c "print(1 > 0)"` are not misread, and device/scratch sinks
are exempt on all three platforms (/dev/null, /var/folders and
/private/tmp on macOS, NUL, $null, %TEMP%, AppData\Local\Temp and Git
Bash's /c/temp on Windows). Since a false positive costs one round trip
rather than a refusal, the heuristics stay simple.

Verified against a nine-prompt eval covering CSS, JS, Markdown, JSON, a
1602-line file and a 12-file bulk change: the bump fired once, the model
used Edit unprompted for every later edit in the session, `> /tmp/x` and
Write were untouched, and a demanded shell command went through on
retry.
@sonarqubecloud

Copy link
Copy Markdown

@abose
abose merged commit 7f188d1 into main Aug 22, 2026
10 of 21 checks passed
@abose
abose deleted the ai branch August 22, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant