Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ anchor: '@file'
created: 2026-10-20T17:21:33Z
norm: '1'
sig: 1574ff353918c008
body_hash: 334362b01e56eab9
raw_hash: 3dd4db75427ff129
body_hash: 6243d6c3b242a4db
raw_hash: 5bd16c78dfed21ca
vouched: 2026-10-20T17:37:48Z
lines: 1-182
---

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
id: 01M5D67S6R6K06W6RAC521M5X6
anchor: '@file'
created: 2026-10-20T17:37:38Z
norm: '1'
sig: 1574ff353918c008
body_hash: 6243d6c3b242a4db
raw_hash: 5bd16c78dfed21ca
lines: 1-191
---

Safari focuses its address field when it opens a page in the background, so Copy after a marker selection copied the URL; focus the web area before selecting, since focusing can move the selection
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ anchor: '@file'
created: 2026-10-20T16:41:34Z
norm: '1'
sig: 4949218de757741f
body_hash: f04bf517db564cdd
raw_hash: 4b9cffb06024b51f
vouched: 2026-10-20T17:21:46Z
body_hash: e345c7c433c48cd2
raw_hash: 3a89db569859919a
vouched: 2026-10-20T17:37:48Z
lines: 1-1211
---

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
id: 01M5D67S7ETJK13EGGXAS4PCRP
anchor: '@file'
created: 2026-10-20T17:37:38Z
norm: '1'
sig: 4949218de757741f
body_hash: e345c7c433c48cd2
raw_hash: 3a89db569859919a
lines: 1-1217
---

Safari disables Copy and has no in-window keyboard focus while inactive, so the #88 copy runs Edit > Copy after desktop_activate; background Command-C and Edit > Copy are refused
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
id: 01M5D682W73BEYBRAPTSK672XW
anchor: '@file'
created: 2026-10-20T17:37:48Z
norm: '1'
sig: 78473a106add9cfc
body_hash: 4ecc23ca1544c692
raw_hash: 4973bb5ae4651c19
lines: 1-971
supersedes: 01M5CT5AR9NBSN9BZBSR9MQKXN
---

wire 2 text is what native2 serves (scripts/contract.mjs runs dist/mcp.js --wire 2, so regenerate ContractText after editing it); wire 1 text is only for the Peekaboo stack a release can switch back to
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no

## [Unreleased]

### Fixed

- `desktop_select` selects text in a web page that isn't a field, such as a paragraph or a link, through WebKit's text markers. An image pasted into a web field is refused with `PASTE_UNVERIFIABLE_RECEIVER` before the focus check, since the page's handling can't be read back. ([#88](https://github.com/githubnext/desktop-tools/issues/88), [#99](https://github.com/githubnext/desktop-tools/issues/99))
- Text selected in a Safari page with `desktop_select` can be copied: selecting focuses the page, as a click would, so Edit > Copy in the activated Safari copies the selection instead of the page's address. Safari copies only while active, so `desktop_activate` it and choose Edit > Copy with `desktop_menu`; the tool description and docs said to press Command-C, which Safari ignores in the background. ([#188](https://github.com/githubnext/desktop-tools/issues/188))

## [0.17.0] - 2026-10-20

### Changed
Expand Down
2 changes: 1 addition & 1 deletion dist/tools.js
Original file line number Diff line number Diff line change
Expand Up @@ -775,7 +775,7 @@ export function desktopTools({ place, app, wire = 1 }) {
{
name: "desktop_select",
op: "select",
description: `Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy, press c with command through desktop_key; Preview copies only while active, so desktop_activate it and use desktop_menu Edit > Copy.`,
description: `Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy a field's selection, press c with command through desktop_key; Safari and Preview copy only while active, so desktop_activate the app and use desktop_menu Edit > Copy.`,
parameters: {
type: "object",
required: ["snapshot", "element", "text"],
Expand Down
2 changes: 1 addition & 1 deletion docs/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ The host replaces the field's contents with key events. The fill is completed wh

`desktop_insert` also pastes an image or files into a native text area, such as TextEdit's rich text, Mail or Notes, in place of text; see [the details](#details-by-tool) for how it confirms the paste and when it refuses.

`desktop_select` sets a field's `AXSelectedTextRange` and reads it back. Text in a web page has no such range, so for an element inside a web area, such as a paragraph, heading or link, the host uses WebKit's text markers instead. It finds the requested range from the element's own start, sets the web area's selected marker range, and reads it back: the result is `completed` only when the selection covers exactly the requested text, and `unknown` otherwise. A long element may need the host to walk the text one marker at a time; that walk stops after 2,048 steps or 300 ms, and the selection is refused before anything is selected, with a hint to select within a smaller element. Offsets are UTF-16 units within the element, as for fields. To copy the selection, press `c` with `command` through `desktop_key`, then `desktop_clipboard_read`. Text on a PDF page in Preview is selected the same way, through the page's text element, even though that element reports itself disabled and has no selection until one is set. Preview disables Copy while it is in the background, so `desktop_activate` it first, then choose Edit > Copy with `desktop_menu`. Not yet checked on a live system: other selectable text outside web pages that isn't a field, such as a read-only native text view, which works only if it exposes a settable `AXSelectedTextRange` and is refused otherwise.
`desktop_select` sets a field's `AXSelectedTextRange` and reads it back. Text in a web page has no such range, so for an element inside a web area, such as a paragraph, heading or link, the host uses WebKit's text markers instead. It focuses the web area, as a click in the page would, so that Copy acts on the page rather than on Safari's address field, finds the requested range from the element's own start, sets the web area's selected marker range, and reads it back: the result is `completed` only when the selection covers exactly the requested text, and `unknown` otherwise. A long element may need the host to walk the text one marker at a time; that walk stops after 2,048 steps or 300 ms, and the selection is refused before anything is selected, with a hint to select within a smaller element. Offsets are UTF-16 units within the element, as for fields. To copy a field's selection, press `c` with `command` through `desktop_key`, then `desktop_clipboard_read`. Safari disables Copy while it is in the background, and its keyboard focus isn't in the window then, so `desktop_activate` it first and choose Edit > Copy with `desktop_menu`. Text on a PDF page in Preview is selected the same way, through the page's text element, even though that element reports itself disabled and has no selection until one is set. Preview disables Copy while it is in the background, so `desktop_activate` it first, then choose Edit > Copy with `desktop_menu`. Not yet checked on a live system: other selectable text outside web pages that isn't a field, such as a read-only native text view, which works only if it exposes a settable `AXSelectedTextRange` and is refused otherwise.

`desktop_inspect` lists the actions each element offers in `actions`. A pop-up button has no settable value: click it, then pick the item from the next observation, which puts an open menu's items first. A `<select>` in a Chromium browser is the exception: its menu closes between tool calls, so picking an item from a later snapshot is refused, often with `SNAPSHOT_STALE`. Pick with one `desktop_key` call instead; see [the recipe](recipes.md#pick-from-a-native-select-in-a-browser).

Expand Down
2 changes: 2 additions & 0 deletions native2/Sources/DesktopMac/Text.swift
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,8 @@ extension Mac {
}
let selection = try resolveSelection(text, prefix: prefix, suffix: suffix, kind: kind, in: span.text)
let range = selection.selected
// Focus first, since focusing the page can move its selection.
_ = markers.focus()
if WebTextSelection.matches(range, in: span, markers: markers) {
return (Delivery(state: .confirmedNoChange, mechanism: nil, evidence: "verified_no_change", units: nil), selection)
}
Expand Down
9 changes: 9 additions & 0 deletions native2/Sources/DesktopMac/WebText.swift
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ protocol WebTextMarkerAccess {
func bounds(of range: CFTypeRef) -> (start: CFTypeRef, end: CFTypeRef)?
func selectedRange() -> CFTypeRef?
func select(_ range: CFTypeRef) -> Bool
/// Makes the web area its window's keyboard focus, as a click in the page does, so Copy and keys act on the page
/// instead of the address field Safari focuses when it opens a page in the background.
func focus() -> Bool
}

struct WebTextSpan {
Expand Down Expand Up @@ -173,6 +176,12 @@ struct AXWebTextMarkers: WebTextMarkerAccess {
AXUIElementSetAttributeValue(webArea, "AXSelectedTextMarkerRange" as CFString, range) == .success
}

func focus() -> Bool {
var value: CFTypeRef?
if AXUIElementCopyAttributeValue(webArea, kAXFocusedAttribute as CFString, &value) == .success, value as? Bool == true { return true }
return AXUIElementSetAttributeValue(webArea, kAXFocusedAttribute as CFString, kCFBooleanTrue) == .success
}

private func copy(_ attribute: String, _ parameter: CFTypeRef) -> CFTypeRef? {
var value: CFTypeRef?
guard AXUIElementCopyParameterizedAttributeValue(webArea, attribute as CFString, parameter, &value) == .success
Expand Down
2 changes: 1 addition & 1 deletion native2/Sources/DesktopProtocol/ContractText.swift
Original file line number Diff line number Diff line change
Expand Up @@ -2120,7 +2120,7 @@ Recipes for multi-step tasks, such as printing to PDF, installing from a DMG or
},
{
"name": "desktop_select",
"description": "Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy, press c with command through desktop_key; Preview copies only while active, so desktop_activate it and use desktop_menu Edit > Copy.",
"description": "Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy a field's selection, press c with command through desktop_key; Safari and Preview copy only while active, so desktop_activate the app and use desktop_menu Edit > Copy.",
"inputSchema": {
"type": "object",
"required": [
Expand Down
13 changes: 8 additions & 5 deletions scripts/live-check.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -851,22 +851,25 @@ check(92, "desktop_set_cell writes a focused web grid input and refuses an unfoc
);
});

check(88, "desktop_select selects a phrase in a Safari paragraph, and Command-C copies exactly it", async (s, m) => {
check(88, "desktop_select selects a phrase in a Safari paragraph, and Edit > Copy in the activated Safari copies exactly it", async (s, m) => {
await s.clipboard();
const win = await s.web();
const { snap, els } = await s.inspect(win);
const prose = els.find((e) => /quick brown fox/.test(`${label(e)} ${e.value ?? ""}`)) ?? skip("paragraph text not in the inspection");
const result = await m.call("desktop_select", { snapshot: snap, element: prose.id, text: "brown fox jumps" });
if (result.outcome !== "completed") return expect(false, evidence(result, "selection"));
const fresh = await s.inspect(win);
const copy = await m.call("desktop_key", { snapshot: fresh.snap, key: "c", modifiers: ["command"] });
// Safari disables Copy in the background; selecting focused the page, so Copy doesn't take the address field.
const safari = (await m.call("desktop_apps", { query: "Safari" })).data.apps?.find((a) => a.target?.pid === win.pid)
?? skip("Safari not in desktop_apps");
await m.call("desktop_activate", { target: safari.target });
const copy = await m.call("desktop_menu", { target: safari.target, path: ["Edit", "Copy"] });
await sleep(300);
const clip = await m.call("desktop_clipboard_read", { format: "text" });
return expect(
clip.data.text === "brown fox jumps",
`${evidence(result, "selection")}; Command-C ${evidence(copy)}; clipboard ${brief(clip.data.text, 60)}`,
`${evidence(result, "selection")}; Edit > Copy ${evidence(copy)}; clipboard ${brief(clip.data.text, 60)}`,
);
});
}, "Brings Safari to the front for the copy.");

check(88, "desktop_select selects a link's text", async (s, m) => {
const win = await s.web();
Expand Down
2 changes: 1 addition & 1 deletion src/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -838,7 +838,7 @@ export function desktopTools({ place, app, wire = 1 }: { place: string; app: str
name: "desktop_select",
op: "select",
description:
`Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy, press c with command through desktop_key; Preview copies only while active, so desktop_activate it and use desktop_menu Edit > Copy.`,
`Select literal text in one element from a fresh desktop_inspect snapshot, or place the caret before or after it (selection). The element is an editable field, web page text such as a paragraph or link, or the text of a PDF page in Preview. Background. Completed when the selection reads back as exactly that text. Missing or ambiguous text is refused; add prefix or suffix, the text immediately around it. To copy a field's selection, press c with command through desktop_key; Safari and Preview copy only while active, so desktop_activate the app and use desktop_menu Edit > Copy.`,
parameters: {
type: "object",
required: ["snapshot", "element", "text"],
Expand Down
Loading