diff --git a/.agents/skills/dg-create-engineering-ticket/SKILL.md b/.agents/skills/dg-create-engineering-ticket/SKILL.md index aeac85a71d..0e813dbaad 100644 --- a/.agents/skills/dg-create-engineering-ticket/SKILL.md +++ b/.agents/skills/dg-create-engineering-ticket/SKILL.md @@ -1,6 +1,6 @@ --- name: dg-create-engineering-ticket -description: Create concise Discourse Graphs Engineering tickets in Linear with the canonical template, correct readiness status, an active existing project, and existing labels. Use when the user asks to create or draft an Engineering ticket; do not use for Feedback tickets. +description: Create concise Discourse Graphs Engineering tickets in Linear with the canonical template, correct readiness status, an active existing project (none for unprojected Obsidian work), and existing labels (including `Obsidian plugin` for Obsidian work). Use when the user asks to create or draft an Engineering ticket; do not use for Feedback tickets. --- # DG Create Engineering Ticket @@ -57,21 +57,21 @@ Always use the Engineering team. This skill never creates or redirects to Feedba ## Project selection -Every ticket must use an existing project that is not completed, canceled, archived, or trashed. Resolve the project in this order: +Resolve the project in this order: 1. Use the project explicitly named by the user. 2. Use a clearly relevant active project from the current work context. -3. For Roam work, use `Roam Discourse Graph plugin assorted tasks`. -4. For Obsidian work, use `Obsidian Discourse Graphs plugin assorted tasks`. +3. For Obsidian work with no relevant project, leave the project empty and apply the existing `Obsidian plugin` label. Do not use `Obsidian Discourse Graphs plugin assorted tasks`; the Obsidian queue picks up unprojected issues by label (see DIS-617). +4. For Roam work, use `Roam Discourse Graph plugin assorted tasks`. 5. Otherwise, use `Engineering Ops assorted tasks`. -Confirm the selected project still exists, is active, and includes the Engineering team. Never create a project. If a user-specified project is missing, ambiguous, or inactive, ask the user for another project instead of silently substituting one. +Any project you select must exist, not be completed, canceled, archived, or trashed, and include the Engineering team. Never create a project. If a user-specified project is missing, ambiguous, or inactive, ask the user for another project instead of silently substituting one. Obsidian tickets that do belong to a real project still use that project, plus the `Obsidian plugin` label. ## Labels and optional fields -- Inspect existing issue labels and apply every clearly relevant label. +- Inspect existing issue labels and apply every clearly relevant label. Always apply `Obsidian plugin` to Obsidian work, since it is how unprojected Obsidian tickets reach the queue. - Never create a label or guess an uncertain label. -- Missing or uncertain optional labels do not block creation. If `Ticket Draft` or the required provenance label is missing or ambiguous, return the prepared draft and report the configuration gap instead of creating an unmarked ticket or substituting a status. +- Missing or uncertain optional labels do not block creation. If `Ticket Draft`, the required provenance label, or `Obsidian plugin` for Obsidian work is missing or ambiguous, return the prepared draft and report the configuration gap instead of creating an unmarked ticket or substituting a status. - Set priority only when the user or context makes it clear and useful. - Do not set a cycle or release. CI/CD handles releases. @@ -79,6 +79,6 @@ Confirm the selected project still exists, is active, and includes the Engineeri Draft without changing Linear when the user asks for a draft, review, or rewrite. Create the ticket without an additional confirmation when the user explicitly asks to create it. -Before creation, resolve the current Engineering statuses, selected project, and existing labels. Create the issue once with its title, completed description (including skill attribution), `Ticket Draft` status, project, provenance and relevant labels, and justified priority. If the result is ambiguous, search for the issue before retrying so a transient failure does not create a duplicate. +Before creation, resolve the current Engineering statuses, selected project, and existing labels. Create the issue once with its title, completed description (including skill attribution), `Ticket Draft` status, project (omitted for unprojected Obsidian work), provenance and relevant labels, and justified priority. If the result is ambiguous, search for the issue before retrying so a transient failure does not create a duplicate. -Read the created issue back and verify its status, labels, and attribution. Report any mismatch instead of claiming successful routing. Return the issue identifier, title, status, project, and link, and say that human review is pending. Include the generated branch name when Linear returns one. Do not restate the full description unless the user asks. +Read the created issue back and verify its status, labels, and attribution. For unprojected Obsidian work, confirm the project is empty and `Obsidian plugin` is applied. Report any mismatch instead of claiming successful routing. Return the issue identifier, title, status, project, and link, and say that human review is pending. Include the generated branch name when Linear returns one. Do not restate the full description unless the user asks. diff --git a/.github/workflows/obsidian-main.yaml b/.github/workflows/obsidian-main.yaml index 568d4b6009..2d56f6cf0f 100644 --- a/.github/workflows/obsidian-main.yaml +++ b/.github/workflows/obsidian-main.yaml @@ -57,14 +57,20 @@ jobs: # lookup needs an explicit --repo and a token that can read that repo. GH_TOKEN: ${{ env.OBSIDIAN_PLUGIN_REPO_TOKEN }} run: | - # manifest.json is the plugin's canonical version: it is what Obsidian - # and the community store read. package.json is private and its version - # is incidental. - STABLE_VERSION=$(node -p "require('./apps/obsidian/manifest.json').version") - LATEST_N=$(gh release list --repo "$PUBLISH_REPO" --limit 100 --json tagName --jq '.[].tagName' \ + # The publish repo's manifest.json is the live store version; CI can't push the + # bump to this repo's main, so the local manifest.json goes stale after a release. + STABLE_VERSION=$(gh api "repos/$PUBLISH_REPO/contents/manifest.json" --jq '.content' | base64 -d | jq -r '.version') + if ! [[ "$STABLE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "::error::Unexpected stable version in $PUBLISH_REPO manifest.json: '$STABLE_VERSION'" + exit 1 + fi + # Separate command so bash -e (no pipefail) fails here instead of restarting at beta.1. + TAGS=$(gh release list --repo "$PUBLISH_REPO" --limit 100 --json tagName --jq '.[].tagName') + LATEST_N=$(echo "$TAGS" \ | grep "^${STABLE_VERSION}-beta\." \ | sed "s/^${STABLE_VERSION}-beta\.//" \ | sort -n | tail -1) + # Deliberately below the stable version in semver, so automatic updates never pick a beta. NEXT_BETA="${STABLE_VERSION}-beta.$(( ${LATEST_N:-0} + 1 ))" echo "Stable version: $STABLE_VERSION, next beta: $NEXT_BETA" echo "version=$NEXT_BETA" >> "$GITHUB_OUTPUT" diff --git a/apps/obsidian/AGENTS.md b/apps/obsidian/AGENTS.md index 87514d5b30..33e1d32b49 100644 --- a/apps/obsidian/AGENTS.md +++ b/apps/obsidian/AGENTS.md @@ -91,6 +91,8 @@ for (let leaf of app.workspace.getActiveLeavesOfType(MY_VIEW_TYPE)) { - Node.js and Electron APIs (`fs`, `crypto`, `os`) are unavailable on mobile - If the plugin targets mobile, use web API equivalents: `SubtleCrypto` instead of `crypto`, `navigator.clipboard` for clipboard access - Regex lookbehind assertions are not supported on some mobile — avoid them if possible +- The plugin ships with `isDesktopOnly: false`, so it must load on mobile +- Node builtins and `electron` are deliberately not in the esbuild `external` array in `scripts/compile.ts`. Marking them external lets a dependency's `require("fs")` reach the bundle and throw on mobile; leaving them out fails the build instead. Replace the dependency rather than re-adding the external ### Code quality diff --git a/apps/obsidian/manifest.json b/apps/obsidian/manifest.json index 142ec82d9d..9d7e4c0672 100644 --- a/apps/obsidian/manifest.json +++ b/apps/obsidian/manifest.json @@ -6,5 +6,5 @@ "description": "Add semantic structure to your notes with the Discourse Graph protocol.", "author": "Discourse Graphs", "authorUrl": "https://discoursegraphs.com", - "isDesktopOnly": true + "isDesktopOnly": false } diff --git a/apps/obsidian/package.json b/apps/obsidian/package.json index 474f3dfa4d..47bc68a47b 100644 --- a/apps/obsidian/package.json +++ b/apps/obsidian/package.json @@ -22,15 +22,14 @@ "@octokit/core": "^6.1.2", "@repo/eslint-config": "workspace:*", "@repo/typescript-config": "workspace:*", - "@types/mime-types": "3.0.1", "@types/node": "catalog:", "@types/react": "catalog:obsidian", "@types/react-dom": "catalog:obsidian", "autoprefixer": "^10.4.21", - "builtin-modules": "3.3.0", "dotenv": "^16.4.5", "esbuild": "0.17.3", "eslint": "catalog:", + "gray-matter": "^4.0.3", "obsidian": "^1.7.2", "postcss": "^8.5.3", "tailwindcss": "^3.4.17", @@ -39,6 +38,7 @@ "typescript": "5.5.4", "uuidv7": "1.1.0", "vitest": "catalog:", + "yaml": "^2.8.1", "zod": "^3.24.1" }, "dependencies": { @@ -48,8 +48,6 @@ "@repo/utils": "workspace:*", "@supabase/supabase-js": "catalog:", "date-fns": "^4.1.0", - "gray-matter": "^4.0.3", - "mime-types": "^3.0.1", "nanoid": "^4.0.2", "react": "catalog:obsidian", "react-dom": "catalog:obsidian", diff --git a/apps/obsidian/scripts/compile.ts b/apps/obsidian/scripts/compile.ts index 84326b12d7..be07ac8a4b 100644 --- a/apps/obsidian/scripts/compile.ts +++ b/apps/obsidian/scripts/compile.ts @@ -2,7 +2,6 @@ import esbuild from "esbuild"; import fs from "fs"; import path from "path"; import { z } from "zod"; -import builtins from "builtin-modules"; import dotenv from "dotenv"; import postcss from "postcss"; import tailwindcss from "tailwindcss"; @@ -43,9 +42,11 @@ export const args = { format: "cjs", root: ".", mirror: process.env.OBSIDIAN_PLUGIN_PATH, + // Node builtins and "electron" are deliberately NOT external. Marking them external lets a + // dependency's `require("fs")` survive into the bundle, which throws at runtime on Obsidian + // mobile; leaving them out fails the build instead. Replace the dependency, do not re-add these. external: [ "obsidian", - "electron", "@codemirror/autocomplete", "@codemirror/collab", "@codemirror/commands", @@ -58,7 +59,6 @@ export const args = { "@lezer/highlight", "@lezer/lr", "tslib=window.TSLib", - ...builtins, ], } as CliOpts; @@ -115,6 +115,7 @@ export const compile = ({ outdir, bundle: true, format, + platform: "browser", sourcemap: isProd ? undefined : "inline", minify: isProd, entryNames: out, diff --git a/apps/obsidian/src/components/canvas/TextLinkDialog.tsx b/apps/obsidian/src/components/canvas/TextLinkDialog.tsx new file mode 100644 index 0000000000..de1f573e2a --- /dev/null +++ b/apps/obsidian/src/components/canvas/TextLinkDialog.tsx @@ -0,0 +1,228 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import { + TLBaseShape, + TLShape, + TldrawUiButton, + TldrawUiButtonLabel, + TldrawUiDialogBody, + TldrawUiDialogCloseButton, + TldrawUiDialogFooter, + TldrawUiDialogHeader, + TldrawUiDialogTitle, + TldrawUiInput, + track, + useEditor, +} from "tldraw"; +import { + getTextShapeLinkUrl, + isAllowedTextLinkUrl, + isObsidianUrl, +} from "~/components/canvas/utils/textShapeLink"; + +type ShapeWithUrl = TLBaseShape; + +// Text shapes keep their link in `meta` so the stock props schema is untouched; +// every other shape type uses tldraw's own `props.url`. +const isTextShape = (shape: TLShape): boolean => shape.type === "text"; + +const linkPatch = (shape: TLShape, url: string) => + isTextShape(shape) ? { meta: { url } } : { props: { url } }; + +const readLinkUrl = (shape: TLShape): string => + isTextShape(shape) + ? getTextShapeLinkUrl(shape) + : (((shape.props as { url?: unknown }).url as string | undefined) ?? ""); + +// Declared as a function property rather than reusing TLUiDialogProps, whose +// method-shorthand onClose trips @typescript-eslint/unbound-method. +type TextLinkDialogProps = { onClose: () => void }; + +type UrlValidity = { isValid: boolean; hasProtocol: boolean }; + +// Bookmarks link their title with a raw , not HyperlinkButton, so a +// page link there would reach the OS handler and skip the vault check. +const acceptsPageLinks = (shapeType: string): boolean => + shapeType !== "bookmark"; + +// Every shape type uses tldraw's linkUrl, patched to accept obsidian://open links. +const validateUrl = (url: string, shapeType: string): UrlValidity => { + // Kept away from the https:// fallback, which would accept `https://obsidian://…`. + if (isObsidianUrl(url)) { + const isValid = acceptsPageLinks(shapeType) && isAllowedTextLinkUrl(url); + return { isValid, hasProtocol: isValid }; + } + if (isAllowedTextLinkUrl(url)) return { isValid: true, hasProtocol: true }; + if (isAllowedTextLinkUrl(`https://${url}`)) + return { isValid: true, hasProtocol: false }; + return { isValid: false, hasProtocol: false }; +}; + +// A fork of tldraw's EditLinkDialog, which is not exported. Copy differs: the +// original's msg() keys are inlined, as this app has no i18n layer. +export const TextLinkDialog = track(({ onClose }: TextLinkDialogProps) => { + const editor = useEditor(); + const selectedShape = editor.getOnlySelectedShape(); + + const canEditLink = + selectedShape && + (isTextShape(selectedShape) || + ("url" in selectedShape.props && + typeof selectedShape.props.url === "string")); + + if (!canEditLink) return null; + + return ( + + ); +}); + +const TextLinkDialogInner = track( + ({ + onClose, + selectedShape, + }: TextLinkDialogProps & { selectedShape: ShapeWithUrl }) => { + const editor = useEditor(); + const rInput = useRef(null); + const shapeType = selectedShape.type; + + useEffect(() => { + editor.timers.requestAnimationFrame(() => rInput.current?.focus()); + }, [editor]); + + const rInitialValue = useRef(readLinkUrl(selectedShape)); + + const [urlInputState, setUrlInputState] = useState(() => { + const initialUrl = readLinkUrl(selectedShape); + const result = validateUrl(initialUrl, shapeType); + const initialValue = result.isValid + ? result.hasProtocol + ? initialUrl + : `https://${initialUrl}` + : "https://"; + return { actual: initialValue, safe: initialValue, valid: true }; + }); + + const handleChange = useCallback( + (rawValue: string) => { + // Drop the prefilled https:// when a full URL is pasted after it. + const fixedRawValue = rawValue.replace( + /https?:\/\/((?:https?|obsidian):\/\/)/i, + (_match, arg1: string) => arg1, + ); + const result = validateUrl(fixedRawValue, shapeType); + const safeValue = result.isValid + ? result.hasProtocol + ? fixedRawValue + : `https://${fixedRawValue}` + : "https://"; + + setUrlInputState({ + actual: fixedRawValue, + safe: safeValue, + valid: result.isValid, + }); + }, + [shapeType], + ); + + const handleClear = useCallback(() => { + const onlySelectedShape = editor.getOnlySelectedShape(); + if (!onlySelectedShape) return; + editor.updateShapes([ + { + id: onlySelectedShape.id, + type: onlySelectedShape.type, + ...linkPatch(onlySelectedShape, ""), + }, + ]); + onClose(); + }, [editor, onClose]); + + const handleComplete = useCallback(() => { + // Enter reaches this even while Save is disabled, and the fallback + // "https://" fails both validators. + if (!urlInputState.valid) return; + const onlySelectedShape = editor.getOnlySelectedShape(); + if (!onlySelectedShape) return; + if (readLinkUrl(onlySelectedShape) !== urlInputState.safe) { + editor.updateShapes([ + { + id: onlySelectedShape.id, + type: onlySelectedShape.type, + ...linkPatch(onlySelectedShape, urlInputState.safe), + }, + ]); + } + onClose(); + }, [editor, onClose, urlInputState]); + + const handleCancel = useCallback(() => { + onClose(); + }, [onClose]); + + const isRemoving = rInitialValue.current && !urlInputState.valid; + + return ( + <> + + Edit link + + + +
+ +
+ {!urlInputState.valid + ? "Invalid URL" + : acceptsPageLinks(shapeType) + ? "Enter a URL, or an obsidian:// link to a page." + : "Enter a URL."} +
+
+ + + + Cancel + + {isRemoving ? ( + + Clear + + ) : ( + + Save + + )} + + + ); + }, +); diff --git a/apps/obsidian/src/components/canvas/TldrawViewComponent.tsx b/apps/obsidian/src/components/canvas/TldrawViewComponent.tsx index 00cfea40b2..867849e108 100644 --- a/apps/obsidian/src/components/canvas/TldrawViewComponent.tsx +++ b/apps/obsidian/src/components/canvas/TldrawViewComponent.tsx @@ -51,6 +51,9 @@ import { DragHandleOverlay } from "./overlays/DragHandleOverlay"; import { NodeCardContextMenu } from "./NodeCardContextMenu"; import { WHITE_LOGO_SVG } from "~/icons"; import { CustomContextMenu } from "./CustomContextMenu"; +import { TextLinkDialog } from "~/components/canvas/TextLinkDialog"; +import { TextLinkOverlay } from "~/components/canvas/overlays/TextLinkOverlay"; +import { registerLinkNavigation } from "~/components/canvas/utils/linkNavigation"; import { openFileInSidebar, openFileInNewTab, @@ -129,6 +132,11 @@ export const TldrawPreviewComponent = ({ return () => clearTimeout(timer); }, []); + useEffect(() => { + if (!isEditorMounted || !editorRef.current) return; + return registerLinkNavigation(editorRef.current, plugin); + }, [isEditorMounted, plugin]); + // Add keyboard event listener for Meta+Alt+Enter when editor is mounted useEffect(() => { if (!isEditorMounted || !editorRef.current) return; @@ -463,6 +471,24 @@ export const TldrawPreviewComponent = ({ }, }} overrides={{ + // Swap only edit-link's dialog; the menu item, icon, placement + // and eligibility gate stay stock. + actions: (editor, actions, helpers) => { + const editLink = actions["edit-link"]; + if (editLink) { + actions["edit-link"] = { + ...editLink, + onSelect: () => { + // Same guard as stock: no-op outside the select tool. + if (!editor.isIn("select")) return; + if (!editor.getOnlySelectedShape()) return; + editor.markHistoryStoppingPoint("edit-link"); + helpers.addDialog({ component: TextLinkDialog }); + }, + }; + } + return actions; + }, tools: (editor, tools) => { tools["discourse-node"] = { id: "discourse-node", @@ -541,6 +567,7 @@ export const TldrawPreviewComponent = ({ InFrontOfTheCanvas: () => ( <> + ), }} diff --git a/apps/obsidian/src/components/canvas/overlays/TextLinkOverlay.tsx b/apps/obsidian/src/components/canvas/overlays/TextLinkOverlay.tsx new file mode 100644 index 0000000000..8982e1fcf1 --- /dev/null +++ b/apps/obsidian/src/components/canvas/overlays/TextLinkOverlay.tsx @@ -0,0 +1,59 @@ +import { HyperlinkButton, useEditor, useValue, TLShapeId } from "tldraw"; +import { getTextShapeLinkUrl } from "~/components/canvas/utils/textShapeLink"; + +// Matches tldraw's own HyperlinkButton, which hides itself when zoomed out. +const HIDE_BELOW_ZOOM = 0.32; + +type TextLink = { id: TLShapeId; url: string; left: number; top: number }; + +// Rendered here, not on a custom shape util, so the stock text shape stays +// untouched -- at the cost of rotation, frame clipping and stacking fidelity. +export const TextLinkOverlay = () => { + const editor = useEditor(); + + const links = useValue( + "textShapeLinks", + () => { + if (editor.getZoomLevel() < HIDE_BELOW_ZOOM) return []; + const viewport = editor.getViewportPageBounds(); + return editor.getCurrentPageShapes().flatMap((shape) => { + if (shape.type !== "text" || editor.isShapeHidden(shape)) return []; + const url = getTextShapeLinkUrl(shape); + if (!url) return []; + const bounds = editor.getShapePageBounds(shape.id); + if (!bounds || !viewport.includes(bounds)) return []; + const topRight = editor.pageToViewport({ + x: bounds.maxX, + y: bounds.minY, + }); + const bottomRight = editor.pageToViewport({ + x: bounds.maxX, + y: bounds.maxY, + }); + return [ + { + id: shape.id, + url, + left: topRight.x, + top: (topRight.y + bottomRight.y) / 2, + }, + ]; + }); + }, + [editor], + ); + + return ( +
+ {links.map(({ id, url, left, top }) => ( +
+ +
+ ))} +
+ ); +}; diff --git a/apps/obsidian/src/components/canvas/utils/externalContentHandlers.ts b/apps/obsidian/src/components/canvas/utils/externalContentHandlers.ts index 643fb2de4e..9db8c342fd 100644 --- a/apps/obsidian/src/components/canvas/utils/externalContentHandlers.ts +++ b/apps/obsidian/src/components/canvas/utils/externalContentHandlers.ts @@ -52,7 +52,7 @@ export const parseObsidianOpenUrl = (url: string): ParsedObsidianUrl | null => { * Resolve an obsidian URL to a TFile in the current vault. * Returns null if the URL points to another vault or the file is not found. */ -const resolveObsidianUrlToFile = ( +export const resolveObsidianUrlToFile = ( plugin: DiscourseGraphPlugin, parsed: ParsedObsidianUrl, ): TFile | null => { @@ -63,9 +63,7 @@ const resolveObsidianUrlToFile = ( let abstract = plugin.app.vault.getAbstractFileByPath(parsed.filePath); if (!(abstract instanceof TFile) && !parsed.filePath.endsWith(".md")) { - abstract = plugin.app.vault.getAbstractFileByPath( - `${parsed.filePath}.md`, - ); + abstract = plugin.app.vault.getAbstractFileByPath(`${parsed.filePath}.md`); } return abstract instanceof TFile ? abstract : null; }; @@ -80,7 +78,8 @@ const isDiscourseNodeFile = ( ): boolean => { if (!file.path.endsWith(".md")) return false; const frontmatter = getFrontmatterForFile(plugin.app, file); - const nodeTypeId = (frontmatter as { nodeTypeId?: string } | null)?.nodeTypeId; + const nodeTypeId = (frontmatter as { nodeTypeId?: string } | null) + ?.nodeTypeId; if (!nodeTypeId || typeof nodeTypeId !== "string") return false; return !!getNodeTypeById(plugin, nodeTypeId); }; @@ -116,7 +115,9 @@ export const handleExternalUrlContent = async ({ if (url.startsWith(OBSIDIAN_URL_PREFIX)) { const parsed = parseObsidianOpenUrl(url); if (!parsed) { - new Notice("Invalid Obsidian link. Only discourse nodes can be dropped on the canvas."); + new Notice( + "Invalid Obsidian link. Only discourse nodes can be dropped on the canvas.", + ); return; } @@ -182,7 +183,9 @@ const createDiscourseNodeShapeAtPoint = async ({ if (existing) { editor.setSelectedShapes([existing.id]); - editor.zoomToSelection({ animation: { duration: editor.options.animationMediumMs } }); + editor.zoomToSelection({ + animation: { duration: editor.options.animationMediumMs }, + }); return; } diff --git a/apps/obsidian/src/components/canvas/utils/linkNavigation.ts b/apps/obsidian/src/components/canvas/utils/linkNavigation.ts new file mode 100644 index 0000000000..3edb4eed23 --- /dev/null +++ b/apps/obsidian/src/components/canvas/utils/linkNavigation.ts @@ -0,0 +1,66 @@ +import type { Editor } from "tldraw"; +import DiscourseGraphPlugin from "~/index"; +import { + parseObsidianOpenUrl, + resolveObsidianUrlToFile, +} from "~/components/canvas/utils/externalContentHandlers"; +import { + openFileInNewLeaf, + openFileInNewTab, + openFileInSidebar, +} from "~/components/canvas/utils/openFileUtils"; +import { showToast } from "~/components/canvas/utils/toastUtils"; +import { isObsidianUrl } from "~/components/canvas/utils/textShapeLink"; + +type NavigateLinkDetail = { + url: string; + metaKey: boolean; + ctrlKey: boolean; + altKey: boolean; +}; + +// Patched tldraw dispatches this from every HyperlinkButton, so one listener +// covers text-shape links and geo/note/image links alike. +export const NAVIGATE_LINK_EVENT = "tldraw.navigate-link"; + +export const registerLinkNavigation = ( + editor: Editor, + plugin: DiscourseGraphPlugin, +): (() => void) => { + const container = editor.getContainer(); + + const onNavigate = (event: Event) => { + const { url, metaKey, ctrlKey, altKey } = ( + event as CustomEvent + ).detail; + if (!isObsidianUrl(url)) return; + // Cancelling tells tldraw to preventDefault, so the URI never reaches the + // OS handler and we keep the cross-vault check below. + event.preventDefault(); + + // The validator parses with `new URL`, so geo links may be stored as + // `Obsidian://…` or with leading spaces; the parser expects the normal form. + const parsed = parseObsidianOpenUrl(new URL(url).toString()); + const file = parsed ? resolveObsidianUrlToFile(plugin, parsed) : null; + if (!file) { + showToast({ + severity: "warning", + title: "Cannot open link", + description: "The linked file is not in this vault", + }); + return; + } + + // Mirrors the discourse-node gestures in TldrawViewComponent. + const open = altKey + ? openFileInNewLeaf + : metaKey || ctrlKey + ? openFileInNewTab + : openFileInSidebar; + void open(plugin.app, file); + editor.selectNone(); + }; + + container.addEventListener(NAVIGATE_LINK_EVENT, onNavigate); + return () => container.removeEventListener(NAVIGATE_LINK_EVENT, onNavigate); +}; diff --git a/apps/obsidian/src/components/canvas/utils/textShapeLink.ts b/apps/obsidian/src/components/canvas/utils/textShapeLink.ts new file mode 100644 index 0000000000..4c9159fdf0 --- /dev/null +++ b/apps/obsidian/src/components/canvas/utils/textShapeLink.ts @@ -0,0 +1,26 @@ +import { T, type TLShape } from "tldraw"; + +// The patched tldraw validator is the single definition of a valid link, so geo +// shapes and text shapes agree on what obsidian:// forms are accepted. +export const isAllowedTextLinkUrl = (value: string): boolean => + T.linkUrl.isValid(value); + +export const isObsidianUrl = (url: string): boolean => { + try { + return new URL(url).protocol.toLowerCase() === "obsidian:"; + } catch { + return false; + } +}; + +// Returns the parsed form: `new URL` strips leading control characters, so a +// raw value could pass the validator and fail a prefix test further on. +export const getTextShapeLinkUrl = (shape: TLShape): string => { + const raw = (shape.meta as { url?: unknown }).url; + if (typeof raw !== "string" || !isAllowedTextLinkUrl(raw)) return ""; + try { + return new URL(raw).toString(); + } catch { + return ""; + } +}; diff --git a/apps/obsidian/src/constants.ts b/apps/obsidian/src/constants.ts index 6b22873df8..8030e7dc44 100644 --- a/apps/obsidian/src/constants.ts +++ b/apps/obsidian/src/constants.ts @@ -34,7 +34,7 @@ export const DEFAULT_NODE_TYPES: Record = { source: { id: generateUid("node"), name: "Source", - format: "SRC - {content}", + format: "@{content}", color: "#9E9E9E", tag: "src-candidate", created: now, diff --git a/apps/obsidian/src/utils/__tests__/conceptConversion.test.ts b/apps/obsidian/src/utils/__tests__/conceptConversion.test.ts index 7456127269..db2505e7a7 100644 --- a/apps/obsidian/src/utils/__tests__/conceptConversion.test.ts +++ b/apps/obsidian/src/utils/__tests__/conceptConversion.test.ts @@ -14,9 +14,11 @@ const CONTEXT: SupabaseContext = { const conceptFor = ({ format, basename, + sourceDocument, }: { format: string; basename: string; + sourceDocument?: string; }): LocalConceptDataInput => { const file = new TFile(); file.basename = basename; @@ -31,6 +33,7 @@ const conceptFor = ({ created: "", last_modified: "", changeTypes: [], + sourceDocument, }, nodeTypesById: { "type-1": { @@ -62,3 +65,24 @@ describe("discourseNodeInstanceToLocalConcept core_title", () => { }); }); }); + +describe("discourseNodeInstanceToLocalConcept source", () => { + it("publishes the Source as local reference content", () => { + expect( + conceptFor({ + format: "EVD - {content}", + basename: "EVD - Evidence title", + sourceDocument: "source", + }), + ).toMatchObject({ local_reference_content: { sourceDocument: "source" } }); + }); + + it("omits reference content when the node has no Source", () => { + expect( + conceptFor({ + format: "EVD - {content}", + basename: "EVD - Evidence title", + }), + ).not.toHaveProperty("local_reference_content"); + }); +}); diff --git a/apps/obsidian/src/utils/__tests__/importNodesHarness.ts b/apps/obsidian/src/utils/__tests__/importNodesHarness.ts new file mode 100644 index 0000000000..b23731f625 --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/importNodesHarness.ts @@ -0,0 +1,321 @@ +import { vi } from "vitest"; +import { parseTestNote, serializeTestNote } from "../../../test/frontmatter"; +import { TFile } from "obsidian"; +import type DiscourseGraphPlugin from "~/index"; +import type { DGSupabaseClient } from "@repo/database/lib/client"; +import type { ImportableNode } from "~/types"; +import { spaceUriAndLocalIdToRid } from "@repo/database/lib/rid"; +import { + getLoggedInClient, + getSupabaseContext, + type SupabaseContext, +} from "~/utils/supabaseContext"; +import { importSelectedNodes } from "~/utils/importNodes"; + +export const LOCAL_URI = "obsidian:local-vault"; +export const REMOTE_URI = "https://roamresearch.com/#/app/research"; +export const SUPABASE_CONTEXT: SupabaseContext = { + spaceId: 1, + platform: "Obsidian", + userId: 1, + spacePassword: "test", +}; +export const SOURCE_UNAVAILABLE_NOTICE = + "Imported EVD - Evidence title, but its Source is unavailable. No source relation was created."; +export const evidenceRid = spaceUriAndLocalIdToRid( + REMOTE_URI, + "evidence", + "note", +); +export const sourceRid = spaceUriAndLocalIdToRid(REMOTE_URI, "source", "note"); +export const selectedNode: ImportableNode = { + nodeInstanceId: "evidence", + title: "Evidence title", + spaceId: 2, + spaceName: "Research", + groupId: "group", + selected: true, +}; + +type Row = Record; + +export const required = (value: T | undefined, label: string): T => { + if (value === undefined) throw new Error(`Missing ${label}`); + return value; +}; + +const setFilePath = (file: TFile, path: string): void => { + const name = path.slice(path.lastIndexOf("/") + 1); + const dot = name.lastIndexOf("."); + file.path = path; + file.name = name; + file.basename = dot > 0 ? name.slice(0, dot) : name; + file.extension = dot > 0 ? name.slice(dot + 1) : ""; +}; + +export const createHarness = () => { + const files = new Map(); + const contents = new Map(); + const readContent = (path: string): string => + required(contents.get(path), `file at ${path}`); + const schemas: Row[] = [ + { + id: 10, + space_id: 2, + source_local_id: "evidence-type", + name: "Evidence", + is_schema: true, + is_relation: false, + literal_content: {}, + }, + { + id: 11, + space_id: 2, + source_local_id: "source-type", + name: "Source", + is_schema: true, + is_relation: false, + literal_content: {}, + }, + ]; + const concepts: Row[] = [ + ...schemas, + { + id: 20, + space_id: 2, + source_local_id: "evidence", + schema_id: 10, + core_title: "Evidence title", + sourceDocument: 21, + is_schema: false, + is_relation: false, + }, + { + id: 21, + space_id: 2, + source_local_id: "source", + schema_id: 11, + core_title: "Source title", + sourceDocument: null, + is_schema: false, + is_relation: false, + }, + ]; + const contentRows: Row[] = ["evidence", "source"].flatMap((id) => [ + { + space_id: 2, + source_local_id: id, + variant: "direct", + text: `${id} incoming title`, + metadata: {}, + author_id: 1, + created: "2026-09-01T00:00:00", + last_modified: "2026-09-02T00:00:00", + }, + { + space_id: 2, + source_local_id: id, + variant: "full", + text: `${id} body`, + metadata: {}, + author_id: 1, + created: "2026-09-01T00:00:00", + last_modified: "2026-09-02T00:00:00", + }, + ]); + const spaces: Row[] = [ + { id: 1, url: LOCAL_URI, name: "Local" }, + { id: 2, url: REMOTE_URI, name: "Research" }, + ]; + const requests: { + table: string; + select: string; + filters: [string, unknown][]; + }[] = []; + let sourceQueryError = false; + const from = vi.fn((table: string) => { + const request = { table, select: "", filters: [] as [string, unknown][] }; + requests.push(request); + const rows = (): Row[] => { + const tableRows = + table === "my_concepts" + ? concepts + : table === "my_contents" || table === "Content" + ? contentRows + : table === "my_spaces" || table === "Space" + ? spaces + : []; + return tableRows.filter((row) => + request.filters.every(([key, value]) => + Array.isArray(value) ? value.includes(row[key]) : row[key] === value, + ), + ); + }; + const response = (single: boolean) => ({ + data: single ? (rows()[0] ?? null) : rows(), + error: + sourceQueryError && + table === "my_concepts" && + request.select === "id, source_local_id, space_id" + ? { message: "Source lookup failed" } + : null, + }); + const query = { + select: (columns: string) => { + request.select = columns; + return query; + }, + eq: (key: string, value: unknown) => { + request.filters.push([key, value]); + return query; + }, + in: (key: string, values: unknown[]) => { + request.filters.push([key, values]); + return query; + }, + maybeSingle: () => Promise.resolve(response(true)), + then: (resolve: (result: ReturnType) => unknown) => + Promise.resolve(response(false)).then(resolve), + }; + return query; + }); + const create = vi.fn((path: string, content: string) => { + if (files.has(path)) + return Promise.reject(new Error(`File already exists: ${path}`)); + const file = new TFile(); + setFilePath(file, path); + files.set(path, file); + contents.set(path, content); + return Promise.resolve(file); + }); + const renameFile = vi.fn((file: TFile, newPath: string) => { + const content = readContent(file.path); + files.delete(file.path); + contents.delete(file.path); + setFilePath(file, newPath); + files.set(newPath, file); + contents.set(newPath, content); + return Promise.resolve(); + }); + const getFileCache = vi.fn((file: TFile) => ({ + frontmatter: parseTestNote(contents.get(file.path) ?? "").frontmatter, + })); + const saveSettings = vi.fn(); + const plugin = { + app: { + plugins: { plugins: {} }, + vault: { + create, + getFileByPath: (path: string) => files.get(path) ?? null, + getAbstractFileByPath: (path: string) => files.get(path) ?? null, + getMarkdownFiles: () => + [...files.values()].filter((file) => file.extension === "md"), + read: (file: TFile) => Promise.resolve(readContent(file.path)), + modify: (file: TFile, content: string) => { + contents.set(file.path, content); + return Promise.resolve(); + }, + process: (file: TFile, callback: (content: string) => string) => { + contents.set(file.path, callback(readContent(file.path))); + return Promise.resolve(); + }, + createFolder: vi.fn(), + adapter: { exists: (path: string) => Promise.resolve(files.has(path)) }, + }, + metadataCache: { getFileCache, getFirstLinkpathDest: () => null }, + fileManager: { + renameFile, + processFrontMatter: ( + file: TFile, + callback: (frontmatter: Row) => void, + ) => { + const parsed = parseTestNote(readContent(file.path)); + callback(parsed.frontmatter); + contents.set( + file.path, + serializeTestNote(parsed.body, parsed.frontmatter), + ); + return Promise.resolve(); + }, + }, + }, + settings: { + nodeTypes: [ + { + id: "evidence-type", + name: "Evidence", + format: "EVD - {content}", + created: 0, + modified: 0, + }, + { + id: "source-type", + name: "Source", + format: "SRC - {content}", + created: 0, + modified: 0, + }, + ], + relationTypes: [ + { + id: "based-on", + label: "Based on", + complement: "Source of", + color: "black", + created: 0, + modified: 0, + }, + ], + discourseRelations: [ + { + id: "evidence-source", + sourceId: "evidence-type", + destinationId: "source-type", + relationshipTypeId: "based-on", + created: 0, + modified: 0, + }, + ], + }, + saveSettings, + } as unknown as DiscourseGraphPlugin; + vi.mocked(getLoggedInClient).mockResolvedValue({ + from, + } as unknown as DGSupabaseClient); + vi.mocked(getSupabaseContext).mockResolvedValue(SUPABASE_CONTEXT); + const pull = () => + importSelectedNodes({ plugin, selectedNodes: [selectedNode] }); + const seedSource = async (local = false): Promise => + create( + "My existing source.md", + serializeTestNote("Original source body", { + nodeInstanceId: "source", + nodeTypeId: "source-type", + ...(local ? {} : { importedFromRid: sourceRid }), + }), + ); + return { + plugin, + saveSettings, + concepts, + concept: (id: number): Row => + required( + concepts.find((row) => row.id === id), + `concept ${id}`, + ), + contentRows, + requests, + files, + file: (path: string): TFile => required(files.get(path), `file at ${path}`), + contents, + readContent, + create, + renameFile, + getFileCache, + pull, + seedSource, + failSourceQuery: () => { + sourceQueryError = true; + }, + }; +}; diff --git a/apps/obsidian/src/utils/__tests__/importNodesSource.test.ts b/apps/obsidian/src/utils/__tests__/importNodesSource.test.ts new file mode 100644 index 0000000000..e78da2b8d8 --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/importNodesSource.test.ts @@ -0,0 +1,420 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { parseTestNote, serializeTestNote } from "../../../test/frontmatter"; +import { Notice } from "obsidian"; +import type { RelationInstance } from "~/types"; +import { spaceUriAndLocalIdToRid } from "@repo/database/lib/rid"; +import { importSelectedNodes, refreshImportedFile } from "~/utils/importNodes"; +import { loadRelations, saveRelations } from "~/utils/relationsStore"; +import { + createHarness, + evidenceRid, + required, + sourceRid, + selectedNode, + SOURCE_UNAVAILABLE_NOTICE, +} from "./importNodesHarness"; + +vi.mock("obsidian", async (importOriginal) => ({ + ...(await importOriginal()), + Notice: vi.fn(), +})); +vi.mock("~/utils/supabaseContext", () => ({ + getLoggedInClient: vi.fn(), + getSupabaseContext: vi.fn(), + getVaultId: () => "local-vault", + getLocalSpaceUri: () => "obsidian:local-vault", +})); +vi.mock("~/utils/publishNode", () => ({ + publishNewRelation: vi.fn().mockResolvedValue(false), +})); +vi.mock("~/utils/templates", () => ({ createTemplateFile: vi.fn() })); +vi.mock("~/utils/importFolderMetadata", () => ({ + resolveFolderForSpaceUri: vi.fn().mockResolvedValue("import/Research"), +})); +vi.mock("~/utils/importRelations", () => ({ + fetchRelationInstancesForImport: vi.fn().mockResolvedValue([]), + importRelationsForImportedNodes: vi.fn().mockResolvedValue({ imported: 0 }), +})); + +beforeEach(() => { + vi.clearAllMocks(); + vi.spyOn(console, "warn").mockImplementation(() => undefined); +}); + +describe("source document import", () => { + it("imports an available Source and creates one local relation", async () => { + const h = createHarness(); + expect(await h.pull()).toEqual({ success: 1, failed: 0 }); + expect([...h.files.keys()]).toContain( + "import/Research/SRC - Source title.md", + ); + expect(Object.values((await loadRelations(h.plugin)).relations)).toEqual([ + expect.objectContaining({ + type: "based-on", + source: evidenceRid, + destination: sourceRid, + }), + ]); + expect(Notice).not.toHaveBeenCalled(); + expect( + h.requests.find( + (request) => request.select === "id, source_local_id, space_id", + )?.filters, + ).toEqual([ + ["is_schema", false], + ["is_relation", false], + ["id", [21]], + ]); + }); + + it.each([false, true])( + "reuses an existing Source (local=%s)", + async (local) => { + const h = createHarness(); + if (local) h.concept(21).space_id = 1; + const sourceFile = await h.seedSource(local); + const original = h.contents.get(sourceFile.path); + await h.pull(); + expect(h.contents.get(sourceFile.path)).toBe(original); + expect( + [...h.files.values()].filter((file) => file.extension === "md"), + ).toHaveLength(2); + expect( + Object.values((await loadRelations(h.plugin)).relations)[0] + ?.destination, + ).toBe(local ? "source" : sourceRid); + expect( + h.requests.filter((request) => request.table === "my_contents"), + ).toHaveLength(1); + }, + ); + + it.each(["triple", "type"])( + "imports the Source without creating a missing relation %s", + async (missing) => { + const h = createHarness(); + if (missing === "triple") h.plugin.settings.discourseRelations = []; + else h.plugin.settings.relationTypes = []; + const settings = structuredClone(h.plugin.settings); + expect(await h.pull()).toEqual({ success: 1, failed: 0 }); + expect( + [...h.files.values()].filter((file) => file.extension === "md"), + ).toHaveLength(2); + expect((await loadRelations(h.plugin)).relations).toEqual({}); + expect(h.plugin.settings).toEqual(settings); + expect(h.saveSettings).not.toHaveBeenCalled(); + }, + ); + + it.each(["triple", "type"])( + "does not materialize a relation through a provisional %s", + async (provisional) => { + const h = createHarness(); + const schema = + provisional === "triple" + ? required(h.plugin.settings.discourseRelations[0], "relation triple") + : required(h.plugin.settings.relationTypes[0], "relation type"); + schema.importedFromRid = "orn:obsidian.schema:remote/relation-type"; + schema.status = "provisional"; + expect(await h.pull()).toEqual({ success: 1, failed: 0 }); + expect( + [...h.files.values()].filter((file) => file.extension === "md"), + ).toHaveLength(2); + expect((await loadRelations(h.plugin)).relations).toEqual({}); + schema.status = "accepted"; + await h.pull(); + expect( + Object.values((await loadRelations(h.plugin)).relations), + ).toHaveLength(1); + }, + ); + + it("does nothing when the current node has no source value", async () => { + const h = createHarness(); + h.concept(20).sourceDocument = null; + expect(await h.pull()).toEqual({ success: 1, failed: 0 }); + expect(h.files.size).toBe(1); + expect( + h.requests.some( + (request) => request.select === "id, source_local_id, space_id", + ), + ).toBe(false); + expect(Notice).not.toHaveBeenCalled(); + }); + + it.each<[string, string, unknown[]]>([ + ["not-shared", SOURCE_UNAVAILABLE_NOTICE, [SOURCE_UNAVAILABLE_NOTICE]], + ["no-content", SOURCE_UNAVAILABLE_NOTICE, [SOURCE_UNAVAILABLE_NOTICE]], + [ + "query-error", + "Nodes imported, but their source relations could not be imported.", + [ + "Could not import source documents:", + expect.objectContaining({ message: "Source lookup failed" }), + ], + ], + ])( + "keeps the current node when the Source is unavailable: %s", + async (reason, notice, warning) => { + const h = createHarness(); + if (reason === "not-shared") + h.concepts.splice( + h.concepts.findIndex((row) => row.id === 21), + 1, + ); + if (reason === "no-content") h.contentRows.splice(2); + if (reason === "query-error") h.failSourceQuery(); + expect(await h.pull()).toEqual({ success: 1, failed: 0 }); + expect([...h.files.keys()]).toEqual([ + "import/Research/EVD - Evidence title.md", + ]); + expect((await loadRelations(h.plugin)).relations).toEqual({}); + expect(Notice).toHaveBeenCalledWith(notice); + expect(console.warn).toHaveBeenCalledWith(...warning); + }, + ); + + it("repeated pull and refresh reuse both nodes and the relation without renaming", async () => { + const h = createHarness(); + await h.pull(); + const firstRelations = await loadRelations(h.plugin); + const paths = [...h.files.keys()]; + await h.pull(); + const file = h.file("import/Research/EVD - Evidence title.md"); + expect(await refreshImportedFile({ plugin: h.plugin, file })).toEqual({ + success: true, + error: undefined, + }); + expect([...h.files.keys()]).toEqual(paths); + expect(await loadRelations(h.plugin)).toEqual(firstRelations); + expect(h.renameFile).not.toHaveBeenCalled(); + }); + + it.each([false, true])( + "reuses a manually created relation (local RIDs=%s)", + async (localRids) => { + const h = createHarness(); + await h.seedSource(); + const relation: RelationInstance = { + id: "manual", + type: "based-on", + source: localRids + ? spaceUriAndLocalIdToRid("obsidian:local-vault", "evidence", "note") + : "evidence", + destination: localRids + ? spaceUriAndLocalIdToRid("obsidian:local-vault", "source", "note") + : "source", + created: 1, + }; + await saveRelations(h.plugin, { + version: 1, + lastModified: 1, + relations: { manual: relation }, + }); + await h.pull(); + expect(Object.values((await loadRelations(h.plugin)).relations)).toEqual([ + relation, + ]); + }, + ); + + it("shares one Source across a batch even while the metadata cache is empty", async () => { + const h = createHarness(); + h.getFileCache.mockReturnValue({ frontmatter: {} }); + const evidence = h.concept(20); + h.concepts.push({ + ...evidence, + id: 22, + source_local_id: "second", + core_title: "Second evidence", + }); + h.contentRows.push( + ...h.contentRows + .slice(0, 2) + .map((row) => ({ ...row, source_local_id: "second" })), + ); + expect( + await importSelectedNodes({ + plugin: h.plugin, + selectedNodes: [ + selectedNode, + { ...selectedNode, nodeInstanceId: "second" }, + ], + }), + ).toEqual({ success: 2, failed: 0 }); + expect( + [...h.files.values()].filter((file) => file.extension === "md"), + ).toHaveLength(3); + expect( + Object.values((await loadRelations(h.plugin)).relations), + ).toHaveLength(2); + expect( + h.create.mock.calls.filter(([path]) => path.includes("SRC -")), + ).toHaveLength(1); + }); + + it("keeps distinct same-titled Sources and their relations through repeated pulls", async () => { + const h = createHarness(); + h.concepts.push( + { + ...h.concept(20), + id: 22, + source_local_id: "second-evidence", + core_title: "Second evidence", + sourceDocument: 23, + }, + { + ...h.concept(21), + id: 23, + source_local_id: "second-source", + }, + ); + h.contentRows.push( + ...h.contentRows.map((row) => ({ + ...row, + source_local_id: `second-${String(row.source_local_id)}`, + text: `Second ${String(row.text)}`, + })), + ); + const pull = () => + importSelectedNodes({ + plugin: h.plugin, + selectedNodes: [ + selectedNode, + { ...selectedNode, nodeInstanceId: "second-evidence" }, + ], + }); + expect(await pull()).toEqual({ success: 2, failed: 0 }); + const sources = [...h.files.values()].filter( + (file) => + parseTestNote(h.readContent(file.path)).frontmatter.nodeTypeId === + "source-type", + ); + expect(sources).toHaveLength(2); + const sourceContents = sources.map((file) => h.readContent(file.path)); + expect( + sourceContents.some( + (content) => parseTestNote(content).body.trim() === "source body", + ), + ).toBe(true); + expect( + sourceContents.some( + (content) => + parseTestNote(content).body.trim() === "Second source body", + ), + ).toBe(true); + const relations = Object.values((await loadRelations(h.plugin)).relations); + expect(relations).toHaveLength(2); + expect(new Set(relations.map((relation) => relation.destination))).toEqual( + new Set( + sourceContents.map((content) => + String(parseTestNote(content).frontmatter.importedFromRid), + ), + ), + ); + const paths = [...h.files.keys()]; + await pull(); + await refreshImportedFile({ + plugin: h.plugin, + file: required(sources[1], "second Source"), + }); + await pull(); + expect([...h.files.keys()]).toEqual(paths); + expect(Object.values((await loadRelations(h.plugin)).relations)).toEqual( + relations, + ); + expect(h.renameFile).not.toHaveBeenCalled(); + }); + + it.each([false, true])( + "resolves a local Source despite a same-ID import (Datacore=%s)", + async (datacore) => { + const h = createHarness(); + h.concept(21).space_id = 1; + await h.seedSource(); + const localSource = await h.create( + "Local source.md", + serializeTestNote("Local source", { + nodeInstanceId: "source", + nodeTypeId: "source-type", + }), + ); + if (datacore) + Object.assign(h.plugin.app, { + plugins: { + plugins: { + datacore: { + api: { + query: (query: string) => + [...h.files.values()] + .filter((file) => { + const frontmatter = parseTestNote( + h.readContent(file.path), + ).frontmatter; + return ( + file.extension === "md" && + [ + ...query.matchAll( + /(nodeInstanceId|importedFromRid) = "([^"]+)"/g, + ), + ].every( + ([, key, value]) => + key !== undefined && frontmatter[key] === value, + ) + ); + }) + .map((file) => ({ $path: file.path })), + }, + }, + }, + }, + }); + await h.pull(); + expect(Object.values((await loadRelations(h.plugin)).relations)).toEqual([ + expect.objectContaining({ source: evidenceRid, destination: "source" }), + ]); + expect(h.contents.get(localSource.path)).toContain("Local source"); + }, + ); + + it.each([false, true])( + "does not confuse same-ID local and imported relations (local RIDs=%s)", + async (localRids) => { + const h = createHarness(); + for (const id of ["evidence", "source"]) { + await h.create( + `Local ${id}.md`, + serializeTestNote(`Local ${id} body`, { + nodeInstanceId: id, + nodeTypeId: `${id}-type`, + }), + ); + } + const manual: RelationInstance = { + id: "local-relation", + type: "based-on", + source: localRids + ? spaceUriAndLocalIdToRid("obsidian:local-vault", "evidence", "note") + : "evidence", + destination: localRids + ? spaceUriAndLocalIdToRid("obsidian:local-vault", "source", "note") + : "source", + created: 1, + }; + await saveRelations(h.plugin, { + version: 1, + lastModified: 1, + relations: { [manual.id]: manual }, + }); + await h.pull(); + await h.pull(); + expect(Object.values((await loadRelations(h.plugin)).relations)).toEqual([ + manual, + expect.objectContaining({ + source: evidenceRid, + destination: sourceRid, + }), + ]); + }, + ); +}); diff --git a/apps/obsidian/src/utils/__tests__/mimeType.test.ts b/apps/obsidian/src/utils/__tests__/mimeType.test.ts new file mode 100644 index 0000000000..8e9022fc4d --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/mimeType.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from "vitest"; +import { DEFAULT_MIME_TYPE, getMimeTypeForPath } from "~/utils/mimeType"; + +describe("getMimeTypeForPath", () => { + it("types the attachment kinds Obsidian accepts", () => { + expect(getMimeTypeForPath("a/b.png")).toBe("image/png"); + expect(getMimeTypeForPath("a/b.pdf")).toBe("application/pdf"); + expect(getMimeTypeForPath("a/b.mp4")).toBe("video/mp4"); + expect(getMimeTypeForPath("a/b.mp3")).toBe("audio/mpeg"); + }); + + it("ignores extension case", () => { + expect(getMimeTypeForPath("A.PNG")).toBe("image/png"); + expect(getMimeTypeForPath("A.JpEg")).toBe("image/jpeg"); + }); + + // The caller skips every `text/` attachment, so an extension missing from the + // table would start being uploaded. Transcluded notes are the common case. + it("keeps transcluded text files under a text/ type", () => { + for (const path of ["note.md", "note.markdown", "a.mdx", "a.txt", "a.csv"]) + expect(getMimeTypeForPath(path).startsWith("text/")).toBe(true); + }); + + it("does not classify xml as text, which would stop it being uploaded", () => { + expect(getMimeTypeForPath("a.xml")).toBe("application/xml"); + }); + + it("uses the last extension of a multi-dot name", () => { + expect(getMimeTypeForPath("archive.tar.png")).toBe("image/png"); + }); + + it("ignores dots in parent directories", () => { + expect(getMimeTypeForPath("folder.v2/asset.png")).toBe("image/png"); + }); + + // A plain object literal would return an inherited member here, and the + // caller's `.startsWith` on it aborts the node's whole asset sync. + it("falls back for an extension that names an Object prototype member", () => { + for (const path of ["a.constructor", "a.__proto__", "a.CONSTRUCTOR"]) + expect(getMimeTypeForPath(path)).toBe(DEFAULT_MIME_TYPE); + }); + + it("falls back for unknown, extensionless and dotfile paths", () => { + expect(getMimeTypeForPath("a.unknownext")).toBe(DEFAULT_MIME_TYPE); + expect(getMimeTypeForPath("noextension")).toBe(DEFAULT_MIME_TYPE); + expect(getMimeTypeForPath("dir/.config")).toBe(DEFAULT_MIME_TYPE); + expect(getMimeTypeForPath("trailing.")).toBe(DEFAULT_MIME_TYPE); + }); +}); diff --git a/apps/obsidian/src/utils/__tests__/parseFrontmatter.test.ts b/apps/obsidian/src/utils/__tests__/parseFrontmatter.test.ts new file mode 100644 index 0000000000..321ff8ae9c --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/parseFrontmatter.test.ts @@ -0,0 +1,42 @@ +import { describe, expect, it } from "vitest"; +import { parseFrontmatter } from "~/utils/importNodes"; + +describe("parseFrontmatter", () => { + it("reads the fields callers depend on", () => { + const { frontmatter } = parseFrontmatter( + "---\nnodeTypeId: 7f3a9c12-4b8e\nimportedFromRid: rid://abc-123\n---\nbody", + ); + expect(frontmatter.nodeTypeId).toBe("7f3a9c12-4b8e"); + expect(frontmatter.importedFromRid).toBe("rid://abc-123"); + }); + + it("returns empty frontmatter when the file has no block", () => { + expect(parseFrontmatter("just body")).toEqual({ + frontmatter: {}, + body: "just body", + }); + }); + + // `gray-matter` threw here. Obsidian tolerates bad YAML, so the import keeps + // going and the node type falls back to the one the import supplies. + it("treats unparseable yaml as absent instead of throwing", () => { + expect(() => + parseFrontmatter("---\na: [unclosed\n---\nbody"), + ).not.toThrow(); + expect( + parseFrontmatter("---\na: [unclosed\n---\nbody").frontmatter, + ).toEqual({}); + }); + + it("strips the block from the body it returns, even for bad yaml", () => { + expect(parseFrontmatter("---\na: [unclosed\n---\nbody").body).toBe("body"); + }); + + // Obsidian's parseYaml is the `yaml` package, so an unquoted date stays a + // string. No caller reads a date field; this pins the engine's behaviour. + it("leaves an unquoted date as a string", () => { + expect( + parseFrontmatter("---\ncreated: 2026-08-07\n---\n").frontmatter, + ).toEqual({ created: "2026-08-07" }); + }); +}); diff --git a/apps/obsidian/src/utils/__tests__/publishNode.test.ts b/apps/obsidian/src/utils/__tests__/publishNode.test.ts index 40b27e6d36..90a09b25e4 100644 --- a/apps/obsidian/src/utils/__tests__/publishNode.test.ts +++ b/apps/obsidian/src/utils/__tests__/publishNode.test.ts @@ -221,6 +221,13 @@ const node = ( file: { basename: nodeInstanceId }, }) as unknown as DiscourseNodeInVault; +const TRIPLE = { + id: "claim-supports-claim", + relationshipTypeId: "supports", + sourceId: "claim", + destinationId: "claim", +}; + const relation = ( id: string, source: string, @@ -257,7 +264,9 @@ const runAccuracyCheck = async ({ await ensurePublishedRelationsAccuracy({ client: fake.client, context: { spaceId: SPACE_ID } as SupabaseContext, - plugin: {} as DiscourseGraphPlugin, + plugin: { + settings: { discourseRelations: [TRIPLE] }, + } as unknown as DiscourseGraphPlugin, allNodesById, relationInstancesData: { version: 1, @@ -273,6 +282,13 @@ const upsertedRows = (calls: Call[]): unknown => .filter((c) => c.table === "ResourceAccess" && c.operation === "upsert") .flatMap((c) => c.args[0] as unknown[]); +const accuracyRows = (group: string, relationIds: string[]): unknown[] => + [...relationIds, "supports", TRIPLE.id].map((source_local_id) => ({ + source_local_id, + space_id: SPACE_ID, + account_uid: group, + })); + describe("ensurePublishedRelationsAccuracy", () => { beforeEach(() => { vi.mocked(getPublishedGroupIdsByRid).mockResolvedValue({ @@ -286,9 +302,16 @@ describe("ensurePublishedRelationsAccuracy", () => { relations: [relation("rel", "localG1", "imported")], }); - expect(upsertedRows(calls)).toEqual([ - { source_local_id: "rel", space_id: SPACE_ID, account_uid: "g1" }, - ]); + expect(upsertedRows(calls)).toEqual(accuracyRows("g1", ["rel"])); + }); + + it("publishes the type and triple of an already published relation", async () => { + const calls = await runAccuracyCheck({ + relations: [relation("rel", "localG1", "imported")], + grantedLocalIdsByGroup: { g1: ["rel"] }, + }); + + expect(upsertedRows(calls)).toEqual(accuracyRows("g1", [])); }); it("publishes a relation between two imported nodes to their shared group", async () => { @@ -296,9 +319,7 @@ describe("ensurePublishedRelationsAccuracy", () => { relations: [relation("rel", "imported", "otherImported")], }); - expect(upsertedRows(calls)).toEqual([ - { source_local_id: "rel", space_id: SPACE_ID, account_uid: "g1" }, - ]); + expect(upsertedRows(calls)).toEqual(accuracyRows("g1", ["rel"])); expect(ensurePartialSpaceAccess).toHaveBeenCalledWith( expect.objectContaining({ groupIds: ["g1"], spaceId: SPACE_ID }), ); @@ -362,13 +383,6 @@ describe("ensurePublishedRelationsAccuracy", () => { }); }); -const TRIPLE = { - id: "claim-supports-claim", - relationshipTypeId: "supports", - sourceId: "claim", - destinationId: "claim", -}; - const pluginWithNodes = ( nodes: Record, ): DiscourseGraphPlugin => { diff --git a/apps/obsidian/src/utils/__tests__/sourceSlot.test.ts b/apps/obsidian/src/utils/__tests__/sourceSlot.test.ts new file mode 100644 index 0000000000..0b0b18ee35 --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/sourceSlot.test.ts @@ -0,0 +1,260 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { TFile } from "obsidian"; +import type { DGSupabaseClient } from "@repo/database/lib/client"; +import { spaceUriAndLocalIdToRid } from "@repo/database/lib/rid"; +import type { DiscourseNode, RelationInstance } from "~/types"; +import type { DiscourseNodeInVault } from "~/utils/getDiscourseNodes"; +import { + filterAvailableSourceSlotValues, + indexSourceSlotValues, +} from "~/utils/sourceSlot"; + +type LookupResult = { data: number | null; error: Error | null }; + +const SOURCE_RID = spaceUriAndLocalIdToRid( + "https://roamresearch.com/#/app/research", + "source", + "note", +); + +const LOCAL_URI = "obsidian:local-vault"; + +const NODE_TYPES_BY_ID: Record = { + "evidence-type": { + id: "evidence-type", + name: "Evidence", + format: "EVD - {content}", + created: 0, + modified: 0, + }, + "source-type": { + id: "source-type", + name: "Source", + format: "SRC - {content}", + created: 0, + modified: 0, + }, +}; + +const vaultNode = ({ + nodeInstanceId, + nodeTypeId, + importedFromRid, +}: { + nodeInstanceId: string; + nodeTypeId: string; + importedFromRid?: string; +}): DiscourseNodeInVault => ({ + file: {} as TFile, + frontmatter: importedFromRid ? { importedFromRid } : {}, + nodeTypeId, + nodeInstanceId, +}); + +const EVIDENCE = vaultNode({ + nodeInstanceId: "evidence", + nodeTypeId: "evidence-type", +}); + +const sourceRelation = ({ + id, + destination, + created, +}: { + id: string; + destination: string; + created: number; +}): RelationInstance => ({ + id, + type: "based-on", + source: "evidence", + destination, + created, +}); + +const clientWith = ( + rpc: (name: string, args: { rid: string }) => Promise, +): DGSupabaseClient => ({ rpc }) as unknown as DGSupabaseClient; + +beforeEach(() => { + vi.clearAllMocks(); + vi.spyOn(console, "warn").mockImplementation(() => undefined); +}); + +describe("indexSourceSlotValues", () => { + it("selects the earliest Source relation by created date, regardless of array order", () => { + expect( + indexSourceSlotValues({ + relations: [ + sourceRelation({ id: "later", destination: "newer", created: 20 }), + sourceRelation({ id: "earliest", destination: "source", created: 1 }), + ], + nodes: [ + EVIDENCE, + vaultNode({ nodeInstanceId: "source", nodeTypeId: "source-type" }), + vaultNode({ nodeInstanceId: "newer", nodeTypeId: "source-type" }), + ], + localSpaceUri: LOCAL_URI, + nodeTypesById: NODE_TYPES_BY_ID, + }), + ).toEqual({ evidence: "source" }); + }); + + it.each([ + ["its id", "imported-source"], + [ + "this vault's RID", + spaceUriAndLocalIdToRid(LOCAL_URI, "imported-source", "note"), + ], + ["its origin RID", SOURCE_RID], + ])( + "uses the origin RID of an imported Source stored by %s", + (_endpoint, destination) => { + expect( + indexSourceSlotValues({ + relations: [ + sourceRelation({ + id: "imported", + destination, + created: 1, + }), + ], + nodes: [ + EVIDENCE, + vaultNode({ + nodeInstanceId: "imported-source", + nodeTypeId: "source-type", + importedFromRid: SOURCE_RID, + }), + ], + localSpaceUri: LOCAL_URI, + nodeTypesById: NODE_TYPES_BY_ID, + }), + ).toEqual({ evidence: SOURCE_RID }); + }, + ); +}); + +describe("filterAvailableSourceSlotValues", () => { + it("keeps results attached to their Sources when parallel lookups finish out of order", async () => { + const responses = new Map void>(); + const rpc = vi.fn( + (_name: string, { rid }: { rid: string }) => + new Promise((resolve) => responses.set(rid, resolve)), + ); + const result = filterAvailableSourceSlotValues({ + sourceSlotByNodeId: { + first: "available-source", + second: "missing-source", + }, + client: clientWith(rpc), + spaceId: 42, + pendingNodeIds: new Set(), + }); + const respond = (rid: string, lookup: LookupResult): void => { + const resolve = responses.get(rid); + if (!resolve) throw new Error(`No lookup started for ${rid}`); + resolve(lookup); + }; + expect(rpc).toHaveBeenCalledTimes(2); + respond("missing-source", { data: null, error: null }); + respond("available-source", { data: 21, error: null }); + expect(await result).toEqual({ first: "available-source" }); + expect(console.warn).toHaveBeenCalledTimes(1); + expect(console.warn).toHaveBeenCalledWith( + "Source missing-source is not in the database yet; pushing node second without sourceDocument", + ); + }); + + it("resolves each distinct source once and preserves explicitly selected local sources", async () => { + const rpc = vi.fn((_name: string, { rid }: { rid: string }) => + Promise.resolve({ data: rid === SOURCE_RID ? 21 : null, error: null }), + ); + const values = await filterAvailableSourceSlotValues({ + sourceSlotByNodeId: { + first: SOURCE_RID, + second: SOURCE_RID, + third: "new-source", + fourth: "missing", + }, + client: clientWith(rpc), + spaceId: 42, + pendingNodeIds: new Set(["new-source"]), + }); + expect(values).toEqual({ + first: SOURCE_RID, + second: SOURCE_RID, + third: "new-source", + }); + expect(rpc.mock.calls).toEqual([ + [ + "rid_or_local_id_to_concept_db_id", + { rid: SOURCE_RID, default_space_id: 42 }, + ], + [ + "rid_or_local_id_to_concept_db_id", + { rid: "missing", default_space_id: 42 }, + ], + ]); + }); + + it("drops an unavailable earliest Source instead of falling back to a later available one", async () => { + const sourceSlotByNodeId = indexSourceSlotValues({ + relations: [ + sourceRelation({ id: "earliest", destination: SOURCE_RID, created: 1 }), + sourceRelation({ id: "later", destination: "later", created: 20 }), + ], + nodes: [ + EVIDENCE, + vaultNode({ + nodeInstanceId: "imported-source", + nodeTypeId: "source-type", + importedFromRid: SOURCE_RID, + }), + vaultNode({ nodeInstanceId: "later", nodeTypeId: "source-type" }), + ], + localSpaceUri: LOCAL_URI, + nodeTypesById: NODE_TYPES_BY_ID, + }); + const rpc = vi.fn((_name: string, { rid }: { rid: string }) => + Promise.resolve({ data: rid === "later" ? 21 : null, error: null }), + ); + expect( + await filterAvailableSourceSlotValues({ + sourceSlotByNodeId, + client: clientWith(rpc), + spaceId: 42, + pendingNodeIds: new Set(), + }), + ).toEqual({}); + expect(rpc).toHaveBeenCalledTimes(1); + expect(console.warn).toHaveBeenCalledWith( + `Source ${SOURCE_RID} is not in the database yet; pushing node evidence without sourceDocument`, + ); + }); + + it("does not mistake a lookup failure for an absent Source", async () => { + const error = new Error("lookup failed"); + await expect( + filterAvailableSourceSlotValues({ + sourceSlotByNodeId: { evidence: "source" }, + client: clientWith(() => Promise.resolve({ data: null, error })), + spaceId: 1, + pendingNodeIds: new Set(), + }), + ).rejects.toBe(error); + }); + + it("does not query when no node has a source", async () => { + const rpc = vi.fn(); + expect( + await filterAvailableSourceSlotValues({ + sourceSlotByNodeId: {}, + client: clientWith(rpc), + spaceId: 1, + pendingNodeIds: new Set(), + }), + ).toEqual({}); + expect(rpc).not.toHaveBeenCalled(); + }); +}); diff --git a/apps/obsidian/src/utils/__tests__/splitFrontmatter.test.ts b/apps/obsidian/src/utils/__tests__/splitFrontmatter.test.ts new file mode 100644 index 0000000000..02158f4892 --- /dev/null +++ b/apps/obsidian/src/utils/__tests__/splitFrontmatter.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it } from "vitest"; +import { splitFrontmatter } from "~/utils/splitFrontmatter"; + +describe("splitFrontmatter", () => { + it("splits a standard block", () => { + expect(splitFrontmatter("---\na: 1\n---\nbody\n")).toEqual({ + yaml: "a: 1", + body: "body\n", + }); + }); + + it("handles CRLF line endings", () => { + expect(splitFrontmatter("---\r\na: 1\r\n---\r\nbody")).toEqual({ + yaml: "a: 1", + body: "body", + }); + }); + + it("reports an empty block as empty yaml, not as absent", () => { + expect(splitFrontmatter("---\n---\nbody")).toEqual({ + yaml: "", + body: "body", + }); + }); + + it("returns null yaml when the file has no block", () => { + expect(splitFrontmatter("just body\n")).toEqual({ + yaml: null, + body: "just body\n", + }); + }); + + it("leaves a horizontal rule in the body alone", () => { + const content = "intro\n\n---\n\nafter the rule\n"; + expect(splitFrontmatter(content)).toEqual({ yaml: null, body: content }); + }); + + it("treats a file that is only frontmatter as having an empty body", () => { + expect(splitFrontmatter("---\na: 1\n---\n")).toEqual({ + yaml: "a: 1", + body: "", + }); + }); + + it("keeps multi-line yaml values intact", () => { + const { yaml } = splitFrontmatter( + "---\ntags:\n - one\n - two\n---\nbody", + ); + expect(yaml).toBe("tags:\n - one\n - two"); + }); + + it("tolerates trailing spaces on the fences", () => { + expect(splitFrontmatter("--- \na: 1\n--- \nbody")).toEqual({ + yaml: "a: 1", + body: "body", + }); + }); + + it("ignores a block that does not start at the first character", () => { + const content = "\n---\na: 1\n---\nbody"; + expect(splitFrontmatter(content)).toEqual({ yaml: null, body: content }); + }); +}); diff --git a/apps/obsidian/src/utils/__tests__/validateNodeType.test.ts b/apps/obsidian/src/utils/__tests__/validateNodeType.test.ts index e85e90334c..5c1e9f7145 100644 --- a/apps/obsidian/src/utils/__tests__/validateNodeType.test.ts +++ b/apps/obsidian/src/utils/__tests__/validateNodeType.test.ts @@ -1,8 +1,10 @@ import { describe, expect, it } from "vitest"; import { decorateTitle } from "@repo/database/lib/decorateTitle"; +import type { DiscourseNode } from "~/types"; import { checkInvalidChars, normalizeImportedNodeFormat, + validateNodeFormat, } from "~/utils/validateNodeType"; describe("normalizeImportedNodeFormat", () => { @@ -55,3 +57,49 @@ describe("normalizeImportedNodeFormat", () => { expect(normalizeImportedNodeFormat(format)).toBe(format); }); }); + +describe("validateNodeFormat", () => { + const currentNode: DiscourseNode = { + id: "node-1", + name: "Evidence", + format: "", + created: 0, + modified: 0, + }; + const validate = (format: string): ReturnType => + validateNodeFormat({ format, currentNode, allNodes: [currentNode] }); + + it.each([ + [ + "EVD - {content} - {Source}", + "Format contains unsupported placeholder: {Source}. Only {content} is supported.", + ], + [ + "{Author}: {content}", + "Format contains unsupported placeholder: {Author}. Only {content} is supported.", + ], + [ + "{Source} - {content} - {Author} - {Source}", + "Format contains unsupported placeholders: {Source}, {Author}. Only {content} is supported.", + ], + ])("rejects the unsupported placeholders in %s", (format, error) => { + expect(validate(format)).toEqual({ isValid: false, error }); + }); + + it.each(["CLM - {content}", "{content}", "@{content}"])( + "accepts the content-only format %s", + (format) => { + expect(validate(format)).toEqual({ isValid: true }); + }, + ); + + it.each(["CLM - title", "EVD - {Source}"])( + "reports the missing {content} for %s", + (format) => { + expect(validate(format)).toEqual({ + isValid: false, + error: 'Format must include the placeholder "{content}"', + }); + }, + ); +}); diff --git a/apps/obsidian/src/utils/importNodes.ts b/apps/obsidian/src/utils/importNodes.ts index dbc9429108..f236eb1fee 100644 --- a/apps/obsidian/src/utils/importNodes.ts +++ b/apps/obsidian/src/utils/importNodes.ts @@ -1,5 +1,4 @@ -import matter from "gray-matter"; -import { App, Notice, TFile } from "obsidian"; +import { App, Notice, TFile, parseYaml } from "obsidian"; import type { DGSupabaseClient } from "@repo/database/lib/client"; import { listGroupSharedNodes } from "@repo/database/lib/sharedNodes"; import type DiscourseGraphPlugin from "~/index"; @@ -39,6 +38,7 @@ import { } from "./importedNodeContent"; import { decorateTitle } from "@repo/database/lib/decorateTitle"; import { buildSchemaRid, findLocalNodeTypeMatch } from "./schemaMatching"; +import { splitFrontmatter } from "~/utils/splitFrontmatter"; type PublishedNode = { source_local_id: string; @@ -697,12 +697,18 @@ const updateMarkdownAssetLinks = ({ }, ); - // Match markdown links (non-image): [text](path) — internal paths resolved like wikilinks, href kept URL-encoded - const markdownLinkRegex = /(? { - if (!linkPath) return match; + // A rest parameter keeps this within the arrow-function parameter limit. + (match, ...groups: string[]) => { + const [imagePrefix, linkText, rawLinkPath] = groups; + // An `!` prefix makes this an image embed, handled by the image pass below. + if (imagePrefix) return match; + if (!rawLinkPath) return match; + let linkPath = rawLinkPath; // Resolve by row before looking at the shape of the link: a Roam-origin asset is // referenced by its storage URL. Matched on the raw link, because the decoding // below would mangle the percent-escaping a storage URL carries. @@ -1176,14 +1182,19 @@ type ParsedFrontmatter = { [key: string]: unknown; }; +// Unparseable frontmatter is treated as absent rather than thrown, matching how +// Obsidian itself tolerates bad YAML. `gray-matter` threw here instead. export const parseFrontmatter = ( content: string, ): { frontmatter: ParsedFrontmatter; body: string } => { - const { data, content: body } = matter(content); - return { - frontmatter: (data ?? {}) as ParsedFrontmatter, - body: body ?? "", - }; + const { yaml, body } = splitFrontmatter(content); + if (yaml === null) return { frontmatter: {}, body }; + try { + const data = parseYaml(yaml) as ParsedFrontmatter | null; + return { frontmatter: data ?? {}, body }; + } catch { + return { frontmatter: {}, body }; + } }; /** diff --git a/apps/obsidian/src/utils/mimeType.ts b/apps/obsidian/src/utils/mimeType.ts new file mode 100644 index 0000000000..2069a1f499 --- /dev/null +++ b/apps/obsidian/src/utils/mimeType.ts @@ -0,0 +1,151 @@ +// Replaces the `mime-types` package, which requires Node's `path` module and so +// cannot load on Obsidian mobile. Values are copied from mime-db so the type +// stored against an uploaded asset does not change. +// Null-prototyped so an attachment named `x.constructor` or `x.__proto__` misses +// rather than returning an inherited member, which is not a string. +const MIME_TYPES_BY_EXTENSION: Record = Object.assign( + Object.create(null) as Record, + { + // Text. Embeds include transcluded notes, and the caller skips every `text/` + // attachment, so an extension missing here would start being uploaded. + "3dml": "text/vnd.in3d.3dml", + appcache: "text/cache-manifest", + asm: "text/x-asm", + c: "text/x-c", + cc: "text/x-c", + coffee: "text/coffeescript", + conf: "text/plain", + cpp: "text/x-c", + css: "text/css", + csv: "text/csv", + curl: "text/vnd.curl", + cxx: "text/x-c", + dcurl: "text/vnd.curl.dcurl", + def: "text/plain", + dic: "text/x-c", + dsc: "text/prs.lines.tag", + etx: "text/x-setext", + f: "text/x-fortran", + f77: "text/x-fortran", + f90: "text/x-fortran", + flx: "text/vnd.fmi.flexstor", + fly: "text/vnd.fly", + for: "text/x-fortran", + ged: "text/vnd.familysearch.gedcom", + gv: "text/vnd.graphviz", + h: "text/x-c", + hbs: "text/x-handlebars-template", + hh: "text/x-c", + htc: "text/x-component", + htm: "text/html", + html: "text/html", + ics: "text/calendar", + ifb: "text/calendar", + in: "text/plain", + ini: "text/plain", + jad: "text/vnd.sun.j2me.app-descriptor", + jade: "text/jade", + java: "text/x-java-source", + js: "text/javascript", + jsx: "text/jsx", + less: "text/less", + list: "text/plain", + litcoffee: "text/coffeescript", + log: "text/plain", + lua: "text/x-lua", + man: "text/troff", + manifest: "text/cache-manifest", + markdown: "text/markdown", + mcurl: "text/vnd.curl.mcurl", + md: "text/markdown", + mdx: "text/mdx", + me: "text/troff", + mjs: "text/javascript", + mkd: "text/x-markdown", + mml: "text/mathml", + ms: "text/troff", + n3: "text/n3", + nfo: "text/x-nfo", + opml: "text/x-opml", + p: "text/x-pascal", + pas: "text/x-pascal", + pde: "text/x-processing", + roff: "text/troff", + rtx: "text/richtext", + s: "text/x-asm", + sass: "text/x-sass", + scss: "text/x-scss", + scurl: "text/vnd.curl.scurl", + sfv: "text/x-sfv", + sgm: "text/sgml", + sgml: "text/sgml", + shex: "text/shex", + shtml: "text/html", + slim: "text/slim", + slm: "text/slim", + spdx: "text/spdx", + spot: "text/vnd.in3d.spot", + styl: "text/stylus", + stylus: "text/stylus", + sub: "text/vnd.dvb.subtitle", + t: "text/troff", + text: "text/plain", + tr: "text/troff", + tsv: "text/tab-separated-values", + ttl: "text/turtle", + txt: "text/plain", + uri: "text/uri-list", + uris: "text/uri-list", + urls: "text/uri-list", + uu: "text/x-uuencode", + vcard: "text/vcard", + vcf: "text/x-vcard", + vcs: "text/x-vcalendar", + vtt: "text/vtt", + wgsl: "text/wgsl", + wml: "text/vnd.wap.wml", + wmls: "text/vnd.wap.wmlscript", + yaml: "text/yaml", + yml: "text/yaml", + ymp: "text/x-suse-ymp", + // Attachment types Obsidian accepts. + png: "image/png", + webp: "image/webp", + jpg: "image/jpeg", + jpeg: "image/jpeg", + gif: "image/gif", + bmp: "image/bmp", + svg: "image/svg+xml", + avif: "image/avif", + ico: "image/vnd.microsoft.icon", + tif: "image/tiff", + tiff: "image/tiff", + mp3: "audio/mpeg", + webm: "video/webm", + wav: "audio/wav", + m4a: "audio/mp4", + ogg: "audio/ogg", + oga: "audio/ogg", + "3gp": "video/3gpp", + flac: "audio/x-flac", + aac: "audio/aac", + mp4: "video/mp4", + ogv: "video/ogg", + mov: "video/quicktime", + mkv: "video/x-matroska", + pdf: "application/pdf", + json: "application/json", + xml: "application/xml", + }, +); + +export const DEFAULT_MIME_TYPE = "application/octet-stream"; + +export const getMimeTypeForPath = (filePath: string): string => { + const fileName = filePath.split("/").pop() ?? ""; + const lastDotIndex = fileName.lastIndexOf("."); + // A leading dot is a dotfile, not an extension. + if (lastDotIndex <= 0) return DEFAULT_MIME_TYPE; + const extension = fileName.slice(lastDotIndex + 1).toLowerCase(); + return MIME_TYPES_BY_EXTENSION[extension] ?? DEFAULT_MIME_TYPE; +}; diff --git a/apps/obsidian/src/utils/publishNode.ts b/apps/obsidian/src/utils/publishNode.ts index 6a527ddfb7..59afc4c1cb 100644 --- a/apps/obsidian/src/utils/publishNode.ts +++ b/apps/obsidian/src/utils/publishNode.ts @@ -21,7 +21,7 @@ import { syncAllNodesAndRelations, syncPublishedNodeAssets, } from "./syncDgNodesToSupabase"; -import { isProvisionalSchema } from "./typeUtils"; +import { isAcceptedSchema, isProvisionalSchema } from "./typeUtils"; import { intersection, difference } from "@repo/utils/setOperations"; import type { DiscourseNodeInVault } from "./getDiscourseNodes"; @@ -406,6 +406,19 @@ export const ensurePublishedRelationsAccuracy = async ({ console.error("Could not get the groups of imported nodes", error); return; } + const relationTriples = (plugin.settings.discourseRelations ?? []).filter( + isAcceptedSchema, + ); + // A group needs the relation's type and triple to read the relation. + const schemaIdsOf = (relation: RelationInstance): string[] => { + const triple = relationTriples.find( + (triple) => + triple.relationshipTypeId === relation.type && + triple.sourceId === allNodesById[relation.source]?.nodeTypeId && + triple.destinationId === allNodesById[relation.destination]?.nodeTypeId, + ); + return triple ? [relation.type, triple.id] : [relation.type]; + }; let changed = false; const missingPublishRecords: TablesInsert<"ResourceAccess">[] = []; for (const group of myGroups) { @@ -426,13 +439,19 @@ export const ensurePublishedRelationsAccuracy = async ({ console.error("Could not get synced relation ids", publishedIds.error); continue; } + const publishedLocalIds = new Set( + (publishedIds.data || []).map((x) => x.source_local_id), + ); const publishedRelationIds = intersection( syncedRelationIds, - new Set((publishedIds.data || []).map((x) => x.source_local_id)), + publishedLocalIds, ); const missingPublishableIds = difference( - publishableRelationIds, - publishedRelationIds, + new Set([ + ...publishableRelationIds, + ...publishableRelations.flatMap(schemaIdsOf), + ]), + publishedLocalIds, ); if (missingPublishableIds.size > 0) { missingPublishRecords.push( diff --git a/apps/obsidian/src/utils/splitFrontmatter.ts b/apps/obsidian/src/utils/splitFrontmatter.ts new file mode 100644 index 0000000000..f474870791 --- /dev/null +++ b/apps/obsidian/src/utils/splitFrontmatter.ts @@ -0,0 +1,20 @@ +// Replaces the frontmatter splitting `gray-matter` did. That package requires +// Node's `fs` at module load, so it cannot run on Obsidian mobile. Kept free of +// any `obsidian` import so it stays a pure, directly testable function; callers +// pass the returned YAML to Obsidian's own `parseYaml`. +const FRONTMATTER_BLOCK = + /^---[ \t]*\r?\n([\s\S]*?)\r?\n?^---[ \t]*(?:\r?\n|$)/m; + +export type SplitFrontmatter = { + /** Raw YAML source, or null when the content has no frontmatter block. */ + yaml: string | null; + body: string; +}; + +export const splitFrontmatter = (content: string): SplitFrontmatter => { + const match = FRONTMATTER_BLOCK.exec(content); + // The block only counts as frontmatter at the very start of the file; + // a `---` further down is a horizontal rule. + if (!match || match.index !== 0) return { yaml: null, body: content }; + return { yaml: match[1] ?? "", body: content.slice(match[0].length) }; +}; diff --git a/apps/obsidian/src/utils/syncDgNodesToSupabase.ts b/apps/obsidian/src/utils/syncDgNodesToSupabase.ts index 40880fc696..4098f9f0b3 100644 --- a/apps/obsidian/src/utils/syncDgNodesToSupabase.ts +++ b/apps/obsidian/src/utils/syncDgNodesToSupabase.ts @@ -1,7 +1,7 @@ import { Notice, TFile } from "obsidian"; import { addFile } from "@repo/database/lib/files"; import { isAssetTooLarge } from "@repo/database/lib/assetLimits"; -import mime from "mime-types"; +import { getMimeTypeForPath } from "~/utils/mimeType"; import { ensureNodeInstanceId } from "~/utils/nodeInstanceId"; import type { DGSupabaseClient } from "@repo/database/lib/client"; import type { Json } from "@repo/database/dbTypes"; @@ -939,7 +939,7 @@ export const syncPublishedNodeAssets = async ({ for (const { link, file: attachment } of attachments) { // The extension comes from the resolved file: a link may be written without one. - const mimetype = mime.lookup(attachment.path) || "application/octet-stream"; + const mimetype = getMimeTypeForPath(attachment.path); if (mimetype.startsWith("text/")) continue; // Do not use standard upload for large files if (isAssetTooLarge(attachment.stat.size)) { diff --git a/apps/obsidian/src/utils/validateNodeFormat.ts b/apps/obsidian/src/utils/validateNodeFormat.ts deleted file mode 100644 index 5e03281b89..0000000000 --- a/apps/obsidian/src/utils/validateNodeFormat.ts +++ /dev/null @@ -1,30 +0,0 @@ -export const validateNodeFormat = ( - format: string, -): { - isValid: boolean; - error?: string; -} => { - if (!format) { - return { - isValid: false, - error: "Format cannot be empty", - }; - } - - if (format.includes("[[") || format.includes("]]")) { - return { - isValid: false, - error: "Format should not contain double brackets [[ or ]]", - }; - } - - const hasVariable = /{[a-zA-Z]+}/.test(format); - if (!hasVariable) { - return { - isValid: false, - error: "Format must contain at least one variable in {varName} format", - }; - } - - return { isValid: true }; -}; diff --git a/apps/obsidian/src/utils/validateNodeType.ts b/apps/obsidian/src/utils/validateNodeType.ts index f577e7125a..da98c8a32a 100644 --- a/apps/obsidian/src/utils/validateNodeType.ts +++ b/apps/obsidian/src/utils/validateNodeType.ts @@ -1,3 +1,7 @@ +import { + CONTENT_PLACEHOLDER, + FORMAT_PLACEHOLDER, +} from "@repo/database/lib/decorateTitle"; import { DiscourseNode } from "~/types"; type ValidationResult = { @@ -35,6 +39,21 @@ export const validateNodeFormat = ({ }; } + // Same placeholder rule as decorateTitle, which can't build titles from other placeholders. + const unsupportedPlaceholders = [ + ...new Set( + (format.match(FORMAT_PLACEHOLDER) ?? []).filter( + (placeholder) => placeholder.toLowerCase() !== CONTENT_PLACEHOLDER, + ), + ), + ]; + if (unsupportedPlaceholders.length > 0) { + return { + isValid: false, + error: `Format contains unsupported placeholder${unsupportedPlaceholders.length > 1 ? "s" : ""}: ${unsupportedPlaceholders.join(", ")}. Only ${CONTENT_PLACEHOLDER} is supported.`, + }; + } + const invalidCharsResult = checkInvalidChars(format); if (!invalidCharsResult.isValid) { return invalidCharsResult; diff --git a/apps/obsidian/styles.css b/apps/obsidian/styles.css index 7f2fed184b..b3c625e247 100644 --- a/apps/obsidian/styles.css +++ b/apps/obsidian/styles.css @@ -136,3 +136,19 @@ body.dg-hide-frontmatter-ids .metadata-property[data-property-key^="rel_" i] { border-radius: 0; color: var(--interactive-accent); } + +/* The overlay positions a wrapper; tldraw's own button fills it. */ +.dg-text-link-overlay { + position: absolute; + inset: 0; + pointer-events: none; +} +.dg-text-link-anchor { + position: absolute; + /* Clears tldraw's edge resize handle, which reaches 6px past the shape. */ + margin-left: 8px; + width: 44px; + height: 44px; + transform: translateY(-50%); + pointer-events: all; +} diff --git a/apps/obsidian/test/frontmatter.test.ts b/apps/obsidian/test/frontmatter.test.ts new file mode 100644 index 0000000000..ac27803663 --- /dev/null +++ b/apps/obsidian/test/frontmatter.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from "vitest"; +import { parseTestNote, serializeTestNote } from "./frontmatter"; + +describe("test note frontmatter", () => { + it("keeps an unquoted date as a string", () => { + expect(parseTestNote("---\ncreated: 2026-08-07\n---\nbody")).toEqual({ + frontmatter: { created: "2026-08-07" }, + body: "body", + }); + }); + + it("treats invalid YAML as absent while preserving the body", () => { + expect(parseTestNote("---\na: [unclosed\n---\nbody")).toEqual({ + frontmatter: {}, + body: "body", + }); + }); + + it.each(["body", "---\nnodeTypeId: source-type\n---\nbody"])( + "does not retain mutated metadata when reading the same text again: %s", + (content) => { + parseTestNote(content).frontmatter.nodeInstanceId = "ghost-id"; + expect(parseTestNote(content).frontmatter).not.toHaveProperty( + "nodeInstanceId", + ); + }, + ); + + it("serializes fixture metadata without changing the body", () => { + const note = serializeTestNote("body\n", { + nodeInstanceId: "source", + created: "2026-08-07", + publishedToGroups: ["group"], + }); + expect(parseTestNote(note)).toEqual({ + frontmatter: { + nodeInstanceId: "source", + created: "2026-08-07", + publishedToGroups: ["group"], + }, + body: "body\n", + }); + }); +}); diff --git a/apps/obsidian/test/frontmatter.ts b/apps/obsidian/test/frontmatter.ts new file mode 100644 index 0000000000..b3c4b0c0f2 --- /dev/null +++ b/apps/obsidian/test/frontmatter.ts @@ -0,0 +1,23 @@ +import { parse, stringify } from "yaml"; +import { splitFrontmatter } from "~/utils/splitFrontmatter"; + +type TestNote = { + frontmatter: Record; + body: string; +}; + +export const parseTestNote = (content: string): TestNote => { + const { yaml, body } = splitFrontmatter(content); + if (yaml === null) return { frontmatter: {}, body }; + try { + const frontmatter = parse(yaml) as Record | null; + return { frontmatter: frontmatter ?? {}, body }; + } catch { + return { frontmatter: {}, body }; + } +}; + +export const serializeTestNote = ( + body: string, + frontmatter: Record, +): string => `---\n${stringify(frontmatter)}---\n${body}`; diff --git a/apps/obsidian/test/obsidianStub.conformance.ts b/apps/obsidian/test/obsidianStub.conformance.ts index a6d1b6d79b..b148b78cee 100644 --- a/apps/obsidian/test/obsidianStub.conformance.ts +++ b/apps/obsidian/test/obsidianStub.conformance.ts @@ -20,6 +20,7 @@ const conformant = { Platform: stub.Platform satisfies typeof ObsidianApi.Platform, normalizePath: stub.normalizePath satisfies typeof ObsidianApi.normalizePath, parseLinktext: stub.parseLinktext satisfies typeof ObsidianApi.parseLinktext, + parseYaml: stub.parseYaml satisfies typeof ObsidianApi.parseYaml, setIcon: stub.setIcon satisfies typeof ObsidianApi.setIcon, setTooltip: stub.setTooltip satisfies typeof ObsidianApi.setTooltip, debounce: stub.debounce satisfies typeof ObsidianApi.debounce, diff --git a/apps/obsidian/test/obsidianStub.ts b/apps/obsidian/test/obsidianStub.ts index 2a22a0ea77..88c997de7d 100644 --- a/apps/obsidian/test/obsidianStub.ts +++ b/apps/obsidian/test/obsidianStub.ts @@ -7,6 +7,7 @@ * "Keeping the stub honest" in AGENTS.md before adding one. */ +import { parse as parseYamlUpstream } from "yaml"; import type { Debouncer, FileStats, @@ -95,6 +96,12 @@ export const normalizePath = (path: string): string => { .normalize("NFC"); }; +/** + * Obsidian's `parseYaml` is `YAML.parse` from the `yaml` package, not js-yaml, + * so the stub delegates to that package rather than reimplementing it. + */ +export const parseYaml = (yaml: string): unknown => parseYamlUpstream(yaml); + export const parseLinktext = ( linktext: string, ): { path: string; subpath: string } => { diff --git a/apps/roam/scripts/__tests__/loadExtension.test.ts b/apps/roam/scripts/__tests__/loadExtension.test.ts index 8236ba6e64..6b4452465b 100644 --- a/apps/roam/scripts/__tests__/loadExtension.test.ts +++ b/apps/roam/scripts/__tests__/loadExtension.test.ts @@ -2,10 +2,10 @@ import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import type { BrowserContext, Locator, Page } from "playwright"; -import { describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { activateDeveloperExtension, - classifyBrowserMessage, + installDirectoryPickerShim, loadTestDefinition, waitForTestReady, } from "../playwright/loadExtension"; @@ -15,14 +15,89 @@ const createInvisibleLocator = (): Locator => first: vi.fn(() => ({ isVisible: vi.fn().mockResolvedValue(false) })), }) as unknown as Locator; +afterEach(() => vi.unstubAllGlobals()); + describe("loadExtension helpers", () => { - it("classifies only the fake directory-handle clone warning as expected", () => { - expect( - classifyBrowserMessage( - "Failed to execute 'put' on 'IDBObjectStore': object could not be cloned", + it("returns the native handle after staging files and removing stale optional files", async () => { + const write = vi.fn().mockResolvedValue(undefined); + const close = vi.fn().mockResolvedValue(undefined); + const directory = { + getFileHandle: vi.fn().mockResolvedValue({ + createWritable: vi.fn().mockResolvedValue({ write, close }), + }), + removeEntry: vi.fn().mockResolvedValue(undefined), + }; + const previews = { + getDirectoryHandle: vi.fn().mockResolvedValue(directory), + }; + const root = { getDirectoryHandle: vi.fn().mockResolvedValue(previews) }; + vi.stubGlobal("navigator", { + storage: { getDirectory: vi.fn().mockResolvedValue(root) }, + }); + const browserWindow = { showDirectoryPicker: vi.fn() }; + vi.stubGlobal("window", browserWindow); + const page = { + evaluate: vi.fn( + (callback: (args: unknown) => Promise, args: unknown) => + callback(args), ), - ).toBe("expected-directory-handle-warning"); - expect(classifyBrowserMessage("extension crashed")).toBe("error"); + } as unknown as Page; + await installDirectoryPickerShim({ + page, + dirName: "dist", + files: [ + { + name: "extension.js", + content: "bundle", + type: "text/javascript", + lastModified: 0, + }, + { + name: "README.md", + content: "readme", + type: "text/markdown", + lastModified: 0, + }, + ], + }); + expect(root.getDirectoryHandle).toHaveBeenCalledWith( + "dg-playwright-extensions", + { create: true }, + ); + expect(previews.getDirectoryHandle).toHaveBeenCalledWith("dist", { + create: true, + }); + expect(write.mock.calls).toEqual([["bundle"], ["readme"]]); + expect(close).toHaveBeenCalledTimes(2); + expect(directory.removeEntry.mock.calls).toEqual([ + ["extension.css"], + ["CHANGELOG.md"], + ["package.json"], + ]); + expect(await browserWindow.showDirectoryPicker()).toBe(directory); + }); + + it("fails without installing a fake handle when persistent storage is unavailable", async () => { + vi.stubGlobal("navigator", { + storage: { + getDirectory: vi + .fn() + .mockRejectedValue(new Error("storage unavailable")), + }, + }); + const picker = vi.fn(); + const browserWindow = { showDirectoryPicker: picker }; + vi.stubGlobal("window", browserWindow); + const page = { + evaluate: vi.fn( + (callback: (args: unknown) => Promise, args: unknown) => + callback(args), + ), + } as unknown as Page; + await expect( + installDirectoryPickerShim({ page, dirName: "dist", files: [] }), + ).rejects.toThrow("storage unavailable"); + expect(browserWindow.showDirectoryPicker).toBe(picker); }); it("loads caller modules through file URLs when paths contain spaces", async () => { diff --git a/apps/roam/scripts/__tests__/roamSession.test.ts b/apps/roam/scripts/__tests__/roamSession.test.ts index a9a93ef471..dad1cee41f 100644 --- a/apps/roam/scripts/__tests__/roamSession.test.ts +++ b/apps/roam/scripts/__tests__/roamSession.test.ts @@ -73,7 +73,7 @@ describe("openRoamSession", () => { expect(launchPersistentContext).toHaveBeenCalledWith( expect.any(String), - expect.objectContaining({ headless: false }), + expect.objectContaining({ headless: false, viewport: null }), ); expect(session.headless).toBe(false); await session.close(); @@ -83,6 +83,47 @@ describe("openRoamSession", () => { } }); + it.each([ + { + headless: true, + viewport: undefined, + expected: { width: 1440, height: 1000 }, + }, + { + headless: false, + viewport: { width: 900, height: 600 }, + expected: { width: 900, height: 600 }, + }, + { headless: true, viewport: null, expected: null }, + ])( + "uses the requested viewport: %j", + async ({ headless, viewport, expected }) => { + const profileRoot = await fs.mkdtemp( + path.join(os.tmpdir(), "dg-roam-viewport-"), + ); + const close = vi.fn().mockResolvedValue(undefined); + const launchPersistentContext = vi.fn().mockResolvedValue({ + close, + pages: () => [createPage()], + }); + try { + const session = await openRoamSession({ + headless, + viewport, + chromium: { launchPersistentContext } as unknown as BrowserType, + profileDir: path.join(profileRoot, "profile"), + }); + expect(launchPersistentContext).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ headless, viewport: expected }), + ); + await session.close(); + } finally { + await fs.rm(profileRoot, { recursive: true, force: true }); + } + }, + ); + it("closes the context when navigation fails", async () => { const profileRoot = await fs.mkdtemp( path.join(os.tmpdir(), "dg-roam-navigation-"), diff --git a/apps/roam/scripts/playwright/loadExtension.ts b/apps/roam/scripts/playwright/loadExtension.ts index 32fdbf6fa7..0ddc477b99 100644 --- a/apps/roam/scripts/playwright/loadExtension.ts +++ b/apps/roam/scripts/playwright/loadExtension.ts @@ -184,14 +184,6 @@ const clickNative = async ( await target.click({ timeout }); }; -export const classifyBrowserMessage = ( - message: string, -): "expected-directory-handle-warning" | "error" => - message.includes("Failed to execute 'put' on 'IDBObjectStore'") && - message.includes("could not be cloned") - ? "expected-directory-handle-warning" - : "error"; - export const loadTestDefinition = async ( testModulePath?: string, ): Promise<(ExtensionTestDefinition & { modulePath: string }) | null> => { @@ -280,117 +272,45 @@ const readFolderFiles = async (repoDir: string): Promise => { return files; }; -const installDirectoryPickerShim = async ({ +export const installDirectoryPickerShim = async ({ page, dirName, files, }: InstallDirectoryPickerShimOptions): Promise => { await page.evaluate( - ({ dirName: pageDirName, files: pageFiles }) => { - type BrowserExtensionFile = { - name: string; - content: string; - type: string; - lastModified: number; - }; - - type BrowserFileHandle = { - kind: "file"; - name: string; - getFile: () => Promise; - isSameEntry: (other: unknown) => Promise; - queryPermission: () => Promise; - requestPermission: () => Promise; - }; - - type BrowserDirectoryHandle = { - kind: "directory"; - name: string; - getFileHandle: (name: string) => Promise; - getDirectoryHandle: (name: string) => Promise; - resolve: ( - possibleDescendant: BrowserFileHandle, - ) => Promise; - isSameEntry: (other: unknown) => Promise; - queryPermission: () => Promise; - requestPermission: () => Promise; - entries: () => AsyncGenerator<[string, BrowserFileHandle]>; - keys: () => AsyncGenerator; - values: () => AsyncGenerator; - [Symbol.asyncIterator]: () => AsyncGenerator< - [string, BrowserFileHandle] - >; - }; - - const makeNotFound = (name: string): DOMException => - new DOMException( - `A requested file or directory could not be found: ${name}`, - "NotFoundError", - ); - - const makeFileHandle = ( - entry: BrowserExtensionFile, - ): BrowserFileHandle => { - const fileHandle: BrowserFileHandle = { - kind: "file", - name: entry.name, - getFile: () => - Promise.resolve( - new File([entry.content], entry.name, { - type: entry.type, - lastModified: entry.lastModified, - }), - ), - isSameEntry: (other: unknown) => - Promise.resolve(other === fileHandle), - queryPermission: () => Promise.resolve("granted"), - requestPermission: () => Promise.resolve("granted"), - }; - return fileHandle; - }; - - const handles = new Map( - pageFiles.map((entry) => [entry.name, makeFileHandle(entry)]), - ); - - const directoryHandle: BrowserDirectoryHandle = { - kind: "directory", - name: pageDirName, - getFileHandle: (name: string) => { - const handle = handles.get(name); - return handle - ? Promise.resolve(handle) - : Promise.reject(makeNotFound(name)); + async ({ dirName: pageDirName, files: pageFiles, rootFileNames }) => { + // Native origin-private handles can be saved by Roam and reopened after reload. + // Plain-object handles cannot be cloned into Roam's IndexedDB store. + const root = await navigator.storage.getDirectory(); + const previews = await root.getDirectoryHandle( + "dg-playwright-extensions", + { + create: true, }, - getDirectoryHandle: (name: string) => - Promise.reject(makeNotFound(name)), - resolve: (possibleDescendant: BrowserFileHandle) => { - for (const [name, handle] of handles) { - if (handle === possibleDescendant) return Promise.resolve([name]); + ); + const directoryHandle = await previews.getDirectoryHandle(pageDirName, { + create: true, + }); + for (const entry of pageFiles) { + const handle = await directoryHandle.getFileHandle(entry.name, { + create: true, + }); + const writable = await handle.createWritable(); + await writable.write(entry.content); + await writable.close(); + } + for (const name of rootFileNames) { + if (pageFiles.some((file) => file.name === name)) continue; + try { + await directoryHandle.removeEntry(name); + } catch (error) { + if ( + !(error instanceof DOMException && error.name === "NotFoundError") + ) { + throw error; } - return Promise.resolve(null); - }, - isSameEntry: (other: unknown) => - Promise.resolve(other === directoryHandle), - queryPermission: () => Promise.resolve("granted"), - requestPermission: () => Promise.resolve("granted"), - async *entries() { - await Promise.resolve(); - for (const entry of handles.entries()) yield entry; - }, - async *keys() { - await Promise.resolve(); - for (const key of handles.keys()) yield key; - }, - async *values() { - await Promise.resolve(); - for (const handle of handles.values()) yield handle; - }, - [Symbol.asyncIterator]() { - return this.entries(); - }, - }; - + } + } Object.defineProperty( window as unknown as DgPlaywrightWindow, "showDirectoryPicker", @@ -400,7 +320,7 @@ const installDirectoryPickerShim = async ({ }, ); }, - { dirName, files }, + { dirName, files, rootFileNames: ROOT_FILES.map(({ name }) => name) }, ); }; @@ -922,37 +842,8 @@ export const main = async (): Promise => { if (messages.length > 50) messages.shift(); }; - const warningKeys = new Set(); - const recordBrowserMessage = ({ - message, - source, - }: { - message: string; - source: "console" | "pageerror"; - }): void => { - if ( - classifyBrowserMessage(message) === "expected-directory-handle-warning" - ) { - if (!warningKeys.has(message)) result.knownWarnings.push(message); - warningKeys.add(message); - return; - } - if (source === "pageerror") result.pageErrors.push(message); - }; - page.on("console", (message) => { - const text = `${message.type()}: ${message.text()}`; - if (message.type() === "error") { - if ( - classifyBrowserMessage(message.text()) === - "expected-directory-handle-warning" - ) { - recordBrowserMessage({ message: message.text(), source: "console" }); - return; - } - recordBrowserMessage({ message: message.text(), source: "console" }); - } - pushBounded(result.consoleMessages, text); + pushBounded(result.consoleMessages, `${message.type()}: ${message.text()}`); }); page.on("requestfailed", (request) => { @@ -963,7 +854,7 @@ export const main = async (): Promise => { }); page.on("pageerror", (error) => { - recordBrowserMessage({ message: error.message, source: "pageerror" }); + result.pageErrors.push(error.message); }); const persistResult = async (): Promise => { diff --git a/apps/roam/scripts/playwright/roamSession.ts b/apps/roam/scripts/playwright/roamSession.ts index 97d4b9646c..266b4f9641 100644 --- a/apps/roam/scripts/playwright/roamSession.ts +++ b/apps/roam/scripts/playwright/roamSession.ts @@ -57,7 +57,7 @@ export type OpenRoamSessionOptions = { graphUrl?: string; profileDir?: string; headless?: boolean; - viewport?: ViewportSize; + viewport?: ViewportSize | null; timeout?: number; allowInteractiveLogin?: boolean; allowCredentialLogin?: boolean; @@ -342,7 +342,7 @@ export const openRoamSession = async ({ graphUrl, profileDir, headless, - viewport = { width: 1440, height: 1000 }, + viewport, timeout = 30_000, allowInteractiveLogin = false, allowCredentialLogin, @@ -366,7 +366,13 @@ export const openRoamSession = async ({ resolvedConfig.profileDir, { headless: resolvedHeadless, - viewport, + // Manual sessions follow the window; headless screenshots stay repeatable. + viewport: + viewport === undefined + ? resolvedHeadless + ? { width: 1440, height: 1000 } + : null + : viewport, }, ); } catch (error) { diff --git a/apps/roam/src/components/AdvancedNodeSearchDialog/AdvancedSearchDialog.tsx b/apps/roam/src/components/AdvancedNodeSearchDialog/AdvancedSearchDialog.tsx index 35dbf6a155..a6543eb886 100644 --- a/apps/roam/src/components/AdvancedNodeSearchDialog/AdvancedSearchDialog.tsx +++ b/apps/roam/src/components/AdvancedNodeSearchDialog/AdvancedSearchDialog.tsx @@ -20,6 +20,7 @@ import { type InsertTarget, } from "~/utils/advancedSearchFooterUtils"; import { DiscourseNodeSortControl } from "~/components/DiscourseNodeSortControl"; +import { getCandidateTagTitle } from "~/utils/discourseNodeSearch"; import getDiscourseNodes, { type DiscourseNode, } from "~/utils/getDiscourseNodes"; @@ -40,6 +41,7 @@ import { import { DiscourseNodeTypeFilter } from "~/components/AdvancedNodeSearchDialog/DiscourseNodeTypeFilter"; import { RenderRoamBlock, RenderRoamPage } from "~/utils/roamReactComponents"; import { AdvancedSearchFooter } from "./AdvancedSearchFooter"; +import { DisplayOptionsMenu } from "./DisplayOptionsMenu"; import { NodeTypeChipsSearchInput } from "./NodeTypeChipsSearchInput"; import { type SearchIndex, @@ -52,10 +54,39 @@ const getNodeBadgeText = (node: DiscourseNode): string => { return formatBadgeText(node.tag?.trim() || node.text); }; -const getTagStyle = (node: DiscourseNode | undefined): React.CSSProperties => { +const getTagStyle = ({ + isCandidate, + node, +}: { + isCandidate?: boolean; + node: DiscourseNode | undefined; +}): React.CSSProperties => { const color = node?.canvasSettings?.color; - if (!color) return { flexShrink: 0 }; - return { ...getNodeTagStyles(color), flexShrink: 0 }; + if (!color) { + return isCandidate + ? { backgroundColor: "transparent", border: "1px solid", flexShrink: 0 } + : { flexShrink: 0 }; + } + return { + ...getNodeTagStyles(color), + ...(isCandidate && { backgroundColor: "transparent", boxShadow: "none" }), + flexShrink: 0, + }; +}; + +// Mirrors Obsidian's "#tag · note · line" line under a candidate's title. +const getCandidateSubtitle = ({ + candidate, + nodeConfigByType, +}: { + candidate: NonNullable; + nodeConfigByType: Record; +}): string => { + const tags = candidate.nodeTypes + .map((type) => nodeConfigByType[type]?.tag) + .filter((tag): tag is string => !!tag?.trim()) + .map((tag) => `#${getCandidateTagTitle(tag)}`); + return [tags.join(" "), candidate.pageTitle].filter(Boolean).join(" · "); }; const renderHighlightedText = ( @@ -79,6 +110,7 @@ const ResultRow = ({ onClick, onMouseEnter, result, + subtitle, }: { active: boolean; keywords: string[]; @@ -86,11 +118,12 @@ const ResultRow = ({ onClick: () => void; onMouseEnter: () => void; result: SearchResult; + subtitle?: string; }) => ( ); @@ -134,6 +186,8 @@ const PreviewPane = ({ result }: { result: SearchResult | null }) => { {result.authorName || "Unknown"}
event.preventDefault()} > @@ -141,7 +195,13 @@ const PreviewPane = ({ result }: { result: SearchResult | null }) => { {isPage ? ( ) : ( - + )}
@@ -163,6 +223,8 @@ const AdvancedNodeSearchDialog = ({ const [discourseNodes, setDiscourseNodes] = useState([]); const [selectedNodeTypeIds, setSelectedNodeTypeIds] = useState([]); const [isTypeFilterPopoverOpen, setIsTypeFilterPopoverOpen] = useState(false); + const [showCandidates, setShowCandidates] = useState(false); + const [isDisplayOptionsOpen, setIsDisplayOptionsOpen] = useState(false); const resultsPanelRef = useRef(null); const inputRef = useRef(null); const [insertTarget, setInsertTarget] = useState(null); @@ -207,12 +269,15 @@ const AdvancedNodeSearchDialog = ({ setActiveIndex(0); setSort(DEFAULT_SORT_CONFIG); setSelectedNodeTypeIds([]); + setShowCandidates(false); + setIsDisplayOptionsOpen(false); setSearchIndex(null); setIndexError(false); } }, [isOpen]); useEffect(() => { + if (!isOpen) return; let cancelled = false; setIsIndexLoading(true); setIndexError(false); @@ -223,7 +288,7 @@ const AdvancedNodeSearchDialog = ({ ); setDiscourseNodes(discourseNodes); - void buildSearchIndex(discourseNodes) + void buildSearchIndex({ discourseNodes, includeCandidates: showCandidates }) .then(({ miniSearch, results: indexedResults }) => { if (cancelled) return; setSearchIndex({ miniSearch, allResults: indexedResults }); @@ -245,7 +310,7 @@ const AdvancedNodeSearchDialog = ({ return () => { cancelled = true; }; - }, [isOpen]); + }, [isOpen, showCandidates]); useEffect(() => { const timeout = setTimeout( @@ -257,7 +322,7 @@ const AdvancedNodeSearchDialog = ({ useEffect(() => { setActiveIndex(0); - }, [debouncedSearchTerm, selectedNodeTypeIds, sort]); + }, [debouncedSearchTerm, selectedNodeTypeIds, showCandidates, sort]); useEffect(() => { const panel = resultsPanelRef.current; @@ -268,7 +333,7 @@ const AdvancedNodeSearchDialog = ({ }, [activeIndex, activeResult?.uid, debouncedSearchTerm]); const onInsert = useCallback(async () => { - if (!activeResult || !insertTarget) return; + if (!activeResult || activeResult.candidate || !insertTarget) return; const pageTitle = getPageTitleByPageUid(activeResult.uid) ?? @@ -308,6 +373,7 @@ const AdvancedNodeSearchDialog = ({ results, selectedNodeTypeIds, sort, + showCandidates, }); posthog.capture("Advanced Node Search: Dock search sidebar", { @@ -332,6 +398,7 @@ const AdvancedNodeSearchDialog = ({ onClose, results, selectedNodeTypeIds, + showCandidates, sort, ]); const handleSortChange = useCallback((nextSort: SortConfig): void => { @@ -403,6 +470,7 @@ const AdvancedNodeSearchDialog = ({ (event.metaKey || event.ctrlKey) && contentState === "results" && activeResult && + !activeResult.candidate && insertTarget ) { event.preventDefault(); @@ -411,6 +479,12 @@ const AdvancedNodeSearchDialog = ({ } if (event.key === "Escape") { if (isTypeFilterPopoverOpen) return; + if (isDisplayOptionsOpen) { + event.preventDefault(); + event.stopPropagation(); + setIsDisplayOptionsOpen(false); + return; + } event.preventDefault(); onClose(); } @@ -418,6 +492,7 @@ const AdvancedNodeSearchDialog = ({ [ activeResult, contentState, + isDisplayOptionsOpen, isTypeFilterPopoverOpen, insertTarget, onClose, @@ -489,6 +564,12 @@ const AdvancedNodeSearchDialog = ({ onSortChange={handleSortChange} sort={sort} /> + +); + +const RelationsPage = ({ + pageUid, + onRowsLoaded, + onVisibilityChange, +}: { + pageUid: string; + onRowsLoaded: (uids: string[] | null) => void; + onVisibilityChange: (visible: boolean) => void; +}): React.JSX.Element => { + const { rows, loading, error, refresh } = useRelationRows({ + pageUid, + onRowsLoaded, + }); + const [filters, setFilters] = useState( + EMPTY_RELATION_FILTERS, + ); + const [search, setSearch] = useState(""); + const [uiVisible, setUiVisible] = useState(true); + const [sort, setSort] = useState("type"); + const [descending, setDescending] = useState(false); + const [page, setPage] = useState(0); + const [pageSize, setPageSize] = useState(25); + const [pageSizeInput, setPageSizeInput] = useState("25"); + const [expanded, setExpanded] = useState>(new Set()); + const viewport = useRef(null); + const options = useMemo(() => getRelationFilterOptions(rows), [rows]); + const visible = useMemo( + () => selectRelationRows({ rows, ...filters, search, sort, descending }), + [rows, filters, search, sort, descending], + ); + const lastPage = Math.max(0, Math.ceil(visible.length / pageSize) - 1); + useEffect(() => { + setPage((previous) => Math.min(previous, lastPage)); + }, [lastPage]); + const currentPage = Math.min(page, lastPage); + const start = currentPage * pageSize; + const filterCount = Object.values(filters).filter(Boolean).length; + const resetPage = (): void => { + setPage(0); + viewport.current?.scrollTo({ top: 0 }); + }; + const changeSort = (next: RelationSort): void => { + setDescending(next === sort ? !descending : next === "createdAt"); + setSort(next); + resetPage(); + }; + const clearFilters = (): void => { + setFilters(EMPTY_RELATION_FILTERS); + resetPage(); + }; + + return ( +
+
+ + Relations ({rows.length}) + + + {uiVisible && (loading ? "Updating…" : error ? "Update failed" : "")} + +
+ {uiVisible && ( + <> +
+ { + setSearch(""); + resetPage(); + }} + /> + } + onChange={(event) => { + setSearch(event.target.value); + resetPage(); + }} + /> + + {( + Object.keys(FILTER_LABELS) as (keyof RelationFilters)[] + ).map((key) => ( + + ))} +

+ Matches either end of the relation. +

+
+ } + > +
+ ); +}; + +export default RelationsPage; diff --git a/apps/roam/src/components/SortableTableHeader.tsx b/apps/roam/src/components/SortableTableHeader.tsx new file mode 100644 index 0000000000..e54354ef8c --- /dev/null +++ b/apps/roam/src/components/SortableTableHeader.tsx @@ -0,0 +1,25 @@ +import React from "react"; +import { Icon } from "@blueprintjs/core"; + +const SortableTableHeader = ({ + label, + direction, + onSort, +}: { + label: string; + direction: "ascending" | "descending" | null; + onSort: () => void; +}): React.ReactElement => ( + + + +); + +export default SortableTableHeader; diff --git a/apps/roam/src/components/mountRelationsPage.tsx b/apps/roam/src/components/mountRelationsPage.tsx new file mode 100644 index 0000000000..3169461fa3 --- /dev/null +++ b/apps/roam/src/components/mountRelationsPage.tsx @@ -0,0 +1,108 @@ +import React from "react"; +import renderWithUnmount from "roamjs-components/util/renderWithUnmount"; +import { getUidsFromId } from "roamjs-components/dom"; +import getParentUidByBlockUid from "roamjs-components/queries/getParentUidByBlockUid"; +import getBlockProps, { isJsonObject } from "~/utils/getBlockProps"; +import RelationsPage from "./RelationsPage"; + +export const initRelationsPages = (): { + mount: (args: { + heading: HTMLElement; + title: string; + pageUid: string; + }) => void; + unmount: (heading: HTMLElement) => void; + dispose: () => void; +} => { + const mounts = new Map void>(); + const mount = ({ + heading, + title, + pageUid, + }: { + heading: HTMLElement; + title: string; + pageUid: string; + }): void => { + if (mounts.has(heading)) return; + // The stored page uses singular "graph"; also recognize the plural spelling. + if (!/^roam\/js\/discourse-graphs?\/relations$/.test(title)) return; + const root = heading.closest(".roam-article, .rm-sidebar-outline"); + const titleContainer = + heading.closest(".rm-title-display-container") || heading.parentElement; + if (!root || !titleContainer || !pageUid) return; + const host = document.createElement("div"); + host.className = "dg-relations-host"; + titleContainer.after(host); + let relationUids = new Set(); + const hidden = new Set(); + let uiVisible = true; + let concealNewBlocks = true; + const hideBlock = (block: HTMLElement): void => { + if (!uiVisible || host.contains(block)) return; + const container = block.closest(".roam-block-container"); + if (!container || hidden.has(container)) return; + const uid = getUidsFromId(block.id).blockUid; + if (!uid) return; + const isRelation = + relationUids.has(uid) || + (concealNewBlocks && + getParentUidByBlockUid(uid) === pageUid && + isJsonObject(getBlockProps(uid)["discourse-graph"])); + if (!isRelation) return; + container.classList.add("dg-relation-storage-block"); + hidden.add(container); + }; + const hideAddedBlocks = (node: Node): void => { + if (!(node instanceof HTMLElement) || host.contains(node)) return; + if (node.matches(".roam-block[id]")) hideBlock(node); + node.querySelectorAll(".roam-block[id]").forEach(hideBlock); + }; + const restoreBlocks = (): void => { + hidden.forEach((block) => + block.classList.remove("dg-relation-storage-block"), + ); + hidden.clear(); + }; + const onVisibilityChange = (visible: boolean): void => { + uiVisible = visible; + restoreBlocks(); + root.querySelectorAll(".roam-block[id]").forEach(hideBlock); + }; + const onRowsLoaded = (uids: string[] | null): void => { + restoreBlocks(); + relationUids = new Set(uids || []); + // An empty successful load must still conceal the first relation added later. + // On a failed load, restore storage so it remains accessible for recovery. + concealNewBlocks = uids !== null; + root.querySelectorAll(".roam-block[id]").forEach(hideBlock); + }; + const blocksObserver = new MutationObserver((mutations) => { + mutations.forEach((mutation) => { + if (host.contains(mutation.target)) return; + mutation.addedNodes.forEach(hideAddedBlocks); + }); + }); + blocksObserver.observe(root, { childList: true, subtree: true }); + const unmount = renderWithUnmount( + , + host, + ); + mounts.set(heading, () => { + blocksObserver.disconnect(); + restoreBlocks(); + unmount(); + host.remove(); + mounts.delete(heading); + }); + }; + return { + mount, + unmount: (heading) => mounts.get(heading)?.(), + dispose: () => mounts.forEach((unmount) => unmount()), + }; +}; diff --git a/apps/roam/src/components/relationsPageData.ts b/apps/roam/src/components/relationsPageData.ts new file mode 100644 index 0000000000..6278828419 --- /dev/null +++ b/apps/roam/src/components/relationsPageData.ts @@ -0,0 +1,254 @@ +import { isJsonObject, normalizeProps, type json } from "~/utils/getBlockProps"; +import getDiscourseRelations from "~/utils/getDiscourseRelations"; +import getDiscourseNodes from "~/utils/getDiscourseNodes"; + +export const UNKNOWN_RELATION_FILTER = "__unknown__"; +export type RelationFilters = { + type: string; + nodeType: string; + status: "" | "tentative" | "confirmed"; +}; +export const EMPTY_RELATION_FILTERS: RelationFilters = { + type: "", + nodeType: "", + status: "", +}; + +export type RelationEndpoint = { + uid: string; + text: string; + isPage: boolean; + exists: boolean; + typeId: string; + typeLabel: string; +}; +export type RelationRow = { + uid: string; + type: string; + schemaUid: string; + typeKnown: boolean; + source: RelationEndpoint; + destination: RelationEndpoint; + tentative: boolean; + createdAt: number | null; + props: Record; +}; +export type RelationSort = "type" | "source" | "destination" | "createdAt"; + +export const loadRelationRows = async ( + pageUid: string, +): Promise => { + const records = (await window.roamAlphaAPI.data.async.q( + `[:find ?uid ?props ?createdAt :in $ ?pageUid :where + [?page :block/uid ?pageUid] [?page :block/children ?child] + [?child :block/uid ?uid] [?child :block/props ?props] + [(get-else $ ?child :create/time -1) ?createdAt]]`, + pageUid, + )) as [string, Record, number][]; + const relations = records.flatMap(([uid, props, createdAt]) => { + const normalized = normalizeProps(props); + const relation = isJsonObject(normalized) + ? normalized["discourse-graph"] + : undefined; + return isJsonObject(relation) ? [{ uid, props, createdAt, relation }] : []; + }); + const endpointUids = [ + ...new Set( + relations + .flatMap(({ relation }) => [ + relation.sourceUid, + relation.destinationUid, + ]) + .filter( + (uid): uid is string => typeof uid === "string" && uid.length > 0, + ), + ), + ]; + // Async Datalog pull results use unnamespaced keys, unlike roamAlphaAPI.pull. + const endpointRecords = endpointUids.length + ? ((await window.roamAlphaAPI.data.async.q( + `[:find (pull ?node [:block/uid :node/title :block/string]) + :in $ [?uid ...] :where [?node :block/uid ?uid]]`, + endpointUids, + )) as [ + { + uid: string; + title?: string; + string?: string; + }, + ][]) + : []; + const nodes = new Map(endpointRecords.map(([node]) => [node.uid, node])); + const schemas = new Map(getDiscourseRelations().map((r) => [r.id, r])); + const nodeTypes = new Map( + getDiscourseNodes().map((node) => [node.type, node.text]), + ); + nodeTypes.set("*", "Any"); + const getEndpoint = ({ + value, + typeId, + }: { + value: json | undefined; + typeId?: string; + }): RelationEndpoint => { + const uid = typeof value === "string" ? value : ""; + const node = nodes.get(uid); + // A stored relation's schema defines the node types of its two roles. + const knownType = typeId && nodeTypes.has(typeId) ? typeId : ""; + return { + uid, + text: + node?.title || + node?.string || + (node ? "Untitled block" : "Missing node"), + isPage: typeof node?.title === "string", + exists: Boolean(node), + typeId: knownType, + typeLabel: nodeTypes.get(knownType) || "Unknown type", + }; + }; + return relations.map(({ uid, props, createdAt, relation }) => { + const schemaUid = + typeof relation.hasSchema === "string" ? relation.hasSchema : ""; + const schema = schemas.get(schemaUid); + return { + uid, + props, + schemaUid, + type: schema?.label || "Unknown type", + typeKnown: Boolean(schema), + source: getEndpoint({ + value: relation.sourceUid, + typeId: schema?.source, + }), + destination: getEndpoint({ + value: relation.destinationUid, + typeId: schema?.destination, + }), + tentative: relation.tentative === "true", + createdAt: + typeof createdAt === "number" && + createdAt >= 0 && + Number.isFinite(new Date(createdAt).getTime()) + ? createdAt + : null, + }; + }); +}; + +export const selectRelationRows = ({ + rows, + type, + search, + sort, + descending, + nodeType = "", + status = "", +}: { + rows: RelationRow[]; + type: string; + search: string; + sort: RelationSort; + descending: boolean; + nodeType?: string; + status?: RelationFilters["status"]; +}): RelationRow[] => { + const query = search.trim().toLocaleLowerCase(); + const label = (row: RelationRow): string => + sort === "source" || sort === "destination" ? row[sort].text : row.type; + return rows + .filter( + (row) => + (!status || row.tentative === (status === "tentative")) && + (!type || + (type === UNKNOWN_RELATION_FILTER + ? !row.typeKnown + : row.schemaUid === type)) && + (!nodeType || + [row.source, row.destination].some( + (endpoint) => + (endpoint.typeId || UNKNOWN_RELATION_FILTER) === nodeType, + )) && + [ + row.type, + row.source.text, + row.destination.text, + row.source.typeLabel, + row.destination.typeLabel, + row.uid, + row.source.uid, + row.destination.uid, + ].some((value) => value.toLocaleLowerCase().includes(query)), + ) + .sort((a, b) => { + if (sort === "createdAt") { + // Unknown creation times stay last in either direction. + if (a.createdAt === null && b.createdAt !== null) return 1; + if (b.createdAt === null && a.createdAt !== null) return -1; + return ( + ((a.createdAt ?? 0) - (b.createdAt ?? 0) || + a.uid.localeCompare(b.uid)) * (descending ? -1 : 1) + ); + } + return ( + (label(a).localeCompare(label(b), undefined, { + numeric: true, + sensitivity: "base", + }) || a.uid.localeCompare(b.uid)) * (descending ? -1 : 1) + ); + }); +}; + +export const getRelationFilterOptions = ( + rows: RelationRow[], +): Record => { + const counts = { + type: new Map(), + nodeType: new Map(), + status: new Map([ + ["tentative", { label: "Tentative", count: 0 }], + ["confirmed", { label: "Confirmed", count: 0 }], + ]), + }; + rows.forEach((row) => { + const values = { + status: [ + { + id: row.tentative ? "tentative" : "confirmed", + label: row.tentative ? "Tentative" : "Confirmed", + }, + ], + type: [{ id: row.typeKnown ? row.schemaUid : "", label: row.type }], + nodeType: [row.source, row.destination].map((endpoint) => ({ + id: endpoint.typeId, + label: endpoint.typeLabel, + })), + }; + (Object.keys(values) as (keyof RelationFilters)[]).forEach((key) => { + // Count relations once even when both endpoints have the same node type. + new Map( + values[key].map(({ id, label }) => [ + id || UNKNOWN_RELATION_FILTER, + label, + ]), + ).forEach((label, value) => { + const previous = counts[key].get(value); + counts[key].set(value, { label, count: (previous?.count || 0) + 1 }); + }); + }); + }); + const options = ( + key: keyof RelationFilters, + ): { value: string; label: string }[] => + [...counts[key]] + .sort((a, b) => a[1].label.localeCompare(b[1].label)) + .map(([value, { label, count }]) => ({ + value, + label: `${label} (${count})`, + })); + return { + type: options("type"), + nodeType: options("nodeType"), + status: options("status"), + }; +}; diff --git a/apps/roam/src/components/results-view/ResultsView.tsx b/apps/roam/src/components/results-view/ResultsView.tsx index ec42c93421..8e4eaf8f12 100644 --- a/apps/roam/src/components/results-view/ResultsView.tsx +++ b/apps/roam/src/components/results-view/ResultsView.tsx @@ -1277,6 +1277,9 @@ const ResultsView: ResultsViewComponent = ({ { posthog.capture("Results View: Share Data Clicked", { parentUid: parentUid, diff --git a/apps/roam/src/components/settings/DiscourseRelationConfigPanel.tsx b/apps/roam/src/components/settings/DiscourseRelationConfigPanel.tsx index 73f6bf5883..9b7d27baf5 100644 --- a/apps/roam/src/components/settings/DiscourseRelationConfigPanel.tsx +++ b/apps/roam/src/components/settings/DiscourseRelationConfigPanel.tsx @@ -15,9 +15,9 @@ import { HTMLTable, ControlGroup, FocusStyleManager, - Icon, } from "@blueprintjs/core"; import type cytoscape from "cytoscape"; +import SortableTableHeader from "../SortableTableHeader"; import React, { useCallback, useEffect, @@ -1192,24 +1192,11 @@ const DiscourseRelationConfigPanel = ({ const isActive = sort?.column === column; return ( - - - + handleSort(column)} + /> ); }; diff --git a/apps/roam/src/components/useRelationRows.ts b/apps/roam/src/components/useRelationRows.ts new file mode 100644 index 0000000000..527c79dd6f --- /dev/null +++ b/apps/roam/src/components/useRelationRows.ts @@ -0,0 +1,89 @@ +import { useCallback, useEffect, useState } from "react"; +import { loadRelationRows, type RelationRow } from "./relationsPageData"; + +export const useRelationRows = ({ + pageUid, + onRowsLoaded, +}: { + pageUid: string; + onRowsLoaded: (uids: string[] | null) => void; +}): { + rows: RelationRow[]; + loading: boolean; + error: boolean; + refresh: () => void; +} => { + const [rows, setRows] = useState([]); + const [loading, setLoading] = useState(true); + const [error, setError] = useState(false); + const [version, setVersion] = useState(0); + const refresh = useCallback(() => setVersion((value) => value + 1), []); + const endpointUids = JSON.stringify( + [...new Set(rows.flatMap((row) => [row.source.uid, row.destination.uid]))] + .filter(Boolean) + .sort(), + ); + + useEffect(() => { + let timer: ReturnType; + const changed = (): void => { + clearTimeout(timer); + timer = setTimeout(refresh, 150); + }; + const pattern = "[:block/uid :node/title :block/string]"; + const entities = (JSON.parse(endpointUids) as string[]).map( + (uid) => `[:block/uid ${JSON.stringify(uid)}]`, + ); + entities.forEach((entity) => + window.roamAlphaAPI.data.addPullWatch(pattern, entity, changed), + ); + return () => { + clearTimeout(timer); + entities.forEach((entity) => + window.roamAlphaAPI.data.removePullWatch(pattern, entity, changed), + ); + }; + }, [endpointUids, pageUid, refresh]); + + useEffect(() => { + let timer: ReturnType; + const changed = (): void => { + clearTimeout(timer); + timer = setTimeout(refresh, 150); + }; + const pattern = + "[:block/uid {:block/children [:block/uid :block/props :create/time]}]"; + const entity = `[:block/uid "${pageUid}"]`; + window.roamAlphaAPI.data.addPullWatch(pattern, entity, changed); + return () => { + clearTimeout(timer); + window.roamAlphaAPI.data.removePullWatch(pattern, entity, changed); + }; + }, [pageUid, refresh]); + + useEffect(() => { + let active = true; + setLoading(true); + void loadRelationRows(pageUid) + .then((nextRows) => { + if (!active) return; + setRows(nextRows); + setError(false); + onRowsLoaded(nextRows.map((row) => row.uid)); + }) + .catch((cause: unknown) => { + if (!active) return; + setError(true); + onRowsLoaded(null); + console.error("Could not load the relations page.", cause); + }) + .finally(() => { + if (active) setLoading(false); + }); + return () => { + active = false; + }; + }, [pageUid, version, onRowsLoaded]); + + return { rows, loading, error, refresh }; +}; diff --git a/apps/roam/src/styles/discourseGraphStyles.css b/apps/roam/src/styles/discourseGraphStyles.css index a8451c8ec4..42d2efa09f 100644 --- a/apps/roam/src/styles/discourseGraphStyles.css +++ b/apps/roam/src/styles/discourseGraphStyles.css @@ -179,3 +179,241 @@ div.roamjs-discourse-drawer div.bp3-drawer { .bp3-popover-wrapper + .referenced-node-autocomplete { margin-top: 1rem; } + +.dg-relations-host { + container-type: inline-size; + margin: 1rem 0; +} +.dg-relation-storage-block { + display: none !important; +} +.dg-sortable-table-header { + display: flex; + align-items: center; + justify-content: space-between; + gap: 6px; + width: 100%; + padding: 0; + border: 0; + background: transparent; + font: inherit; + color: inherit; + text-align: left; + cursor: pointer; +} +.dg-sortable-table-header .bp3-icon { + flex-shrink: 0; +} +.bp3-html-table .dg-sortable-table-header { + color: inherit; +} +.dg-relations-heading, +.dg-relations-footer { + display: flex; + align-items: center; + justify-content: space-between; + gap: 8px; + min-height: 30px; +} +.dg-relations-heading { + margin-bottom: 8px; +} +.dg-relations-heading > span { + font-size: 12px; +} +.dg-relations-toolbar { + display: flex; + align-items: center; + gap: 6px; + margin-bottom: 12px; +} +.dg-relations-search { + flex: 1; + min-width: 0; +} +.dg-relations-toolbar > :not(.dg-relations-search) { + flex-shrink: 0; +} +.dg-relations-filter-button { + width: 116px; + white-space: nowrap; +} +.dg-relations-compact-sort { + display: none; +} +.dg-relations-filters { + width: 270px; + max-width: 85vw; + padding: 16px; +} +.dg-relations-filters p { + font-size: 12px; + margin: 0 0 8px; +} +.dg-relations-frame.bp3-card { + padding: 0; + overflow: auto; + min-height: 180px; + height: 560px; + max-height: 60vh; + scrollbar-gutter: stable; +} +.dg-relations-frame .dg-relations-table { + width: 100%; + table-layout: fixed; +} +.dg-relation-type-column { + width: 108px; +} +.dg-relation-date-column { + width: 130px; +} +.dg-relation-inspect-column { + width: 36px; +} +.dg-relations-table thead { + position: sticky; + top: 0; + z-index: 1; + border-radius: 0; + box-shadow: none; +} +.dg-relations-frame .dg-relations-table th, +.dg-relations-frame .dg-relations-table td { + padding: 10px; + vertical-align: top; +} +.dg-relations-frame .dg-relations-table th { + color: inherit; + padding-top: 12px; + padding-bottom: 12px; +} +.dg-relations-table .dg-relation-row { + margin: 0; + border-right: 0; +} +.dg-relations-frame .dg-relations-table .dg-relation-inspect { + padding: 8px 4px; +} +.dg-relations-page .dg-relation-link { + display: block; + width: 100%; + padding: 0; + min-height: 0; + text-align: left; + overflow-wrap: anywhere; + line-height: 20px; +} +.dg-relation-node-preview { + display: -webkit-box; + -webkit-line-clamp: 2; + -webkit-box-orient: vertical; + overflow: hidden; + pointer-events: none; +} +.dg-relation-type .bp3-tag { + max-width: 100%; + white-space: normal; + overflow-wrap: anywhere; +} +.dg-relation-pending { + display: block; + font-size: 11px; + margin-top: 4px; +} +.dg-relation-date { + font-size: 12px; + line-height: 20px; +} +.dg-relations-table .dg-relation-details > td { + padding-top: 0; + box-shadow: none; +} +.dg-relation-props { + padding: 10px; +} +.dg-relation-props pre { + white-space: pre-wrap; + overflow-wrap: anywhere; + max-height: 240px; + overflow-y: auto; + margin: 8px 0 0; +} +.dg-relations-footer { + padding-top: 8px; + font-size: 12px; + flex-wrap: wrap; +} +.dg-relations-table .dg-relations-empty > td { + height: 140px; + text-align: center; + vertical-align: middle; +} +@container (max-width: 680px) { + .dg-relations-compact-sort { + display: inline-block; + } + .dg-relations-table thead, + .dg-relations-table colgroup { + display: none; + } + .dg-relations-table, + .dg-relations-table tbody, + .dg-relations-table tr, + .dg-relations-table td { + display: block; + width: 100%; + } + .dg-relations-table .dg-relation-row { + display: grid; + grid-template-columns: minmax(0, 1fr) auto 28px; + grid-template-areas: "relation date inspect" "source source source" "destination destination destination"; + gap: 6px 8px; + padding: 12px; + } + .dg-relations-frame .dg-relations-table .dg-relation-row > td { + padding: 0; + border: 0; + box-shadow: none; + } + .dg-relation-source { + grid-area: source; + } + .dg-relation-type { + grid-area: relation; + align-self: center; + } + .dg-relation-destination { + grid-area: destination; + } + .dg-relation-destination .dg-relation-link { + display: flex; + gap: 6px; + } + .dg-relation-destination .dg-relation-link::before { + content: "→"; + } + .dg-relation-destination .dg-relation-node-preview { + flex: 1; + min-width: 0; + } + .dg-relation-date { + grid-area: date; + align-self: center; + } + .dg-relation-inspect { + grid-area: inspect; + } + .dg-relation-pending { + display: inline; + margin-left: 6px; + } + .dg-relations-table .dg-relation-details { + border: 0; + } + .dg-relations-table .dg-relations-empty > td { + display: flex; + align-items: center; + justify-content: center; + } +} diff --git a/apps/roam/src/utils/__tests__/candidateNodeSearch.test.ts b/apps/roam/src/utils/__tests__/candidateNodeSearch.test.ts new file mode 100644 index 0000000000..7e4792456d --- /dev/null +++ b/apps/roam/src/utils/__tests__/candidateNodeSearch.test.ts @@ -0,0 +1,320 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { DiscourseNode } from "~/utils/getDiscourseNodes"; +import { + buildCandidateBlocksByTagQuery, + getCandidateTagTitle, + stripCandidateTags, + type PulledDiscourseNode, +} from "~/utils/discourseNodeSearch"; +import { + buildSearchIndex, + searchDiscourseNodesWithMiniSearch, +} from "~/components/AdvancedNodeSearchDialog/utils"; +import { runRoamSemanticSearch } from "~/utils/discourseNodeSearchProviders"; + +// getDiscourseNodes builds its defaults at import time. +vi.hoisted(() => { + (globalThis as { window?: unknown }).window = { + roamAlphaAPI: { util: { generateUID: () => "uid" } }, + }; +}); + +const originalWindow = (globalThis as { window?: unknown }).window; + +const node = (overrides: Partial): DiscourseNode => ({ + type: "clm", + text: "Claim", + format: "[[CLM]] - {content}", + shortcut: "C", + tag: "", + backedBy: "user", + specification: [], + canvasSettings: {}, + ...overrides, +}); + +const claim = node({ tag: "#clm-candidate" }); +const evidence = node({ + type: "evd", + text: "Evidence", + format: "[[EVD]] - {content}", + tag: "#evd-candidate", +}); + +const pulledPage = (uid: string, title: string): PulledDiscourseNode => ({ + ":block/uid": uid, + ":node/title": title, + ":create/time": 1, + ":edit/time": 2, +}); + +const pulledBlock = (uid: string, text: string): PulledDiscourseNode => ({ + ":block/uid": uid, + ":block/string": text, + ":block/page": { ":node/title": "Field notes" }, + ":create/time": 3, + ":edit/time": 4, +}); + +const mockRoamQuery = (respond: (query: string) => PulledDiscourseNode[]) => { + const q = vi.fn((query: string) => + Promise.resolve(respond(query).map((result) => [result])), + ); + (globalThis as { window: unknown }).window = { + roamAlphaAPI: { data: { async: { fast: { q } } } }, + }; + return q; +}; + +const respondByQuery = (query: string): PulledDiscourseNode[] => { + if (query.includes("clm-candidate")) { + return [ + pulledBlock("b1", "Soil moisture drives yield #CLM-Candidate"), + pulledBlock("b2", "#clm-candidate #evd-candidate shared block"), + pulledBlock("b3", "#clm-candidate"), + // A block tagged with two casings of one tag page comes back twice. + pulledBlock("b4", "#clm-candidate #CLM-candidate twice"), + pulledBlock("b4", "#clm-candidate #CLM-candidate twice"), + ]; + } + if (query.includes("evd-candidate")) { + return [pulledBlock("b2", "#clm-candidate #evd-candidate shared block")]; + } + if (query.includes("CLM")) return [pulledPage("p1", "[[CLM]] - Soil")]; + return []; +}; + +afterEach(() => { + (globalThis as { window?: unknown }).window = originalWindow; +}); + +describe("candidate tag helpers", () => { + it("strips the leading hash from a configured tag", () => { + expect(getCandidateTagTitle("#clm-candidate")).toBe("clm-candidate"); + expect(getCandidateTagTitle("clm-candidate")).toBe("clm-candidate"); + expect(getCandidateTagTitle("#[[Claim Candidate]]")).toBe( + "Claim Candidate", + ); + }); + + it("queries blocks that reference the tag page, ignoring case", () => { + const query = buildCandidateBlocksByTagQuery({ + tagTitle: 'say "hi".', + pullExpression: "[:block/uid]", + }); + + expect(query).toContain('(re-pattern "(?i)^say \\"hi\\"\\\\.$")'); + expect(query).toContain("[?block :block/refs ?tag]"); + }); + + it("strips every tag form for the configured tags only", () => { + expect( + stripCandidateTags({ + text: "#clm-candidate Soil #[[evd-candidate]] drives [[clm-candidate]] #keep", + tagTitles: ["clm-candidate", "evd-candidate"], + }), + ).toBe("Soil drives #keep"); + }); + + it("strips tags in any casing", () => { + expect( + stripCandidateTags({ + text: "#CLM-Candidate Soil #[[Clm-candidate]]", + tagTitles: ["clm-candidate"], + }), + ).toBe("Soil"); + }); + + it("keeps a longer tag that continues with a non-ASCII letter", () => { + expect( + stripCandidateTags({ text: "Soil #candé", tagTitles: ["cand"] }), + ).toBe("Soil #candé"); + }); + + it("keeps a longer tag that only starts with a configured tag", () => { + expect( + stripCandidateTags({ + text: "Soil #clm-candidates", + tagTitles: ["clm-candidate"], + }), + ).toBe("Soil #clm-candidates"); + }); +}); + +describe("buildSearchIndex", () => { + it("indexes only node pages when candidates are off", async () => { + const q = mockRoamQuery(respondByQuery); + + const { results } = await buildSearchIndex({ + discourseNodes: [claim, evidence], + }); + + expect(results.map((result) => result.uid)).toEqual(["p1"]); + expect(q.mock.calls.some(([query]) => query.includes(":block/refs"))).toBe( + false, + ); + }); + + it("adds tagged blocks as candidates, one row per block", async () => { + mockRoamQuery(respondByQuery); + + const { results } = await buildSearchIndex({ + discourseNodes: [claim, evidence], + includeCandidates: true, + }); + + expect( + results.map(({ uid, type, title, candidate }) => ({ + uid, + type, + title, + candidate, + })), + ).toEqual([ + { + uid: "p1", + type: "clm", + title: "[[CLM]] - Soil", + candidate: undefined, + }, + { + uid: "b1", + type: "clm", + title: "Soil moisture drives yield", + candidate: { nodeTypes: ["clm"], pageTitle: "Field notes" }, + }, + { + uid: "b2", + type: "clm", + title: "shared block", + candidate: { nodeTypes: ["clm", "evd"], pageTitle: "Field notes" }, + }, + { + uid: "b4", + type: "clm", + title: "twice", + candidate: { nodeTypes: ["clm"], pageTitle: "Field notes" }, + }, + ]); + }); + + it("skips node types without a tag", async () => { + const q = mockRoamQuery(respondByQuery); + + await buildSearchIndex({ + discourseNodes: [node({ tag: "" })], + includeCandidates: true, + }); + + expect(q.mock.calls.some(([query]) => query.includes(":block/refs"))).toBe( + false, + ); + }); + + it("keeps node results when a candidate query fails", async () => { + mockRoamQuery((query) => { + if (query.includes(":block/refs")) throw new Error("query failed"); + return respondByQuery(query); + }); + vi.spyOn(console, "error").mockImplementation(() => undefined); + + const { results } = await buildSearchIndex({ + discourseNodes: [claim], + includeCandidates: true, + }); + + expect(results.map((result) => result.uid)).toEqual(["p1"]); + }); + + it("matches candidates on block text and applies the type filter", async () => { + mockRoamQuery(respondByQuery); + const { miniSearch, results } = await buildSearchIndex({ + discourseNodes: [claim, evidence], + includeCandidates: true, + }); + + const search = (searchTerm: string, typeFilter?: string[]) => + searchDiscourseNodesWithMiniSearch({ + miniSearch, + allResults: results, + searchTerm, + typeFilter, + }).map((entry) => entry.result.uid); + + expect(search("moisture")).toEqual(["b1"]); + expect(search("moisture", ["evd"])).toEqual([]); + expect(search("shared", ["evd"])).toEqual(["b2"]); + }); +}); + +describe("runRoamSemanticSearch", () => { + type Hit = { uid: string; type: "page" | "block" | "chunk" }; + const semanticSearch = vi.fn<(args: Record) => Hit[]>(); + const titles: Record = { + p1: "[[CLM]] - Soil", + p2: "Daily notes", + }; + + beforeEach(() => { + semanticSearch.mockReset(); + (globalThis as { window: unknown }).window = { + roamAlphaAPI: { + data: { + semanticSearchEnabled: () => true, + async: { + semanticSearch: (args: Record) => + Promise.resolve(semanticSearch(args)), + pull_many: (_pattern: string, ids: [string, string][]) => + Promise.resolve( + ids.map(([, uid]) => + titles[uid] + ? { ":block/uid": uid, ":node/title": titles[uid] } + : null, + ), + ), + }, + }, + }, + }; + }); + + const run = (candidateUids?: Set) => + runRoamSemanticSearch({ + nodeTypes: [claim], + query: "soil", + candidateUids, + }); + + it("searches pages only when there are no candidates", async () => { + semanticSearch.mockReturnValue([ + { uid: "p1", type: "page" }, + { uid: "b1", type: "block" }, + ]); + + const { filteredResults } = await run(); + + expect(semanticSearch.mock.calls[0]?.[0]).toMatchObject({ + "search-blocks": false, + "search-pages": true, + }); + expect(filteredResults.map((result) => result.uid)).toEqual(["p1"]); + }); + + it("keeps node pages and candidate blocks in hit order", async () => { + semanticSearch.mockReturnValue([ + { uid: "b1", type: "block" }, + { uid: "p2", type: "page" }, + { uid: "b9", type: "block" }, + { uid: "p1", type: "page" }, + ]); + + const { filteredResults, filteredResultCount } = await run(new Set(["b1"])); + + expect(semanticSearch.mock.calls[0]?.[0]).toMatchObject({ + "search-blocks": true, + "search-pages": true, + }); + expect(filteredResults.map((result) => result.uid)).toEqual(["b1", "p1"]); + expect(filteredResultCount).toBe(2); + }); +}); diff --git a/apps/roam/src/utils/__tests__/relationRowsLifecycle.test.ts b/apps/roam/src/utils/__tests__/relationRowsLifecycle.test.ts new file mode 100644 index 0000000000..936a800117 --- /dev/null +++ b/apps/roam/src/utils/__tests__/relationRowsLifecycle.test.ts @@ -0,0 +1,161 @@ +// @vitest-environment jsdom +import React from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { act } from "react-dom/test-utils"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { useRelationRows } from "~/components/useRelationRows"; +import { + loadRelationRows, + type RelationRow, +} from "~/components/relationsPageData"; + +vi.mock("~/components/relationsPageData", () => ({ + loadRelationRows: vi.fn(), +})); +const load = vi.mocked(loadRelationRows); +type PullWatch = (pattern: string, entity: string, changed: () => void) => void; +const addPullWatch = vi.fn(); +const removePullWatch = vi.fn(); +const onRowsLoaded = vi.fn(); +let root: Root; +let result: ReturnType; +const Harness = ({ pageUid = "storage" }: { pageUid?: string }): null => { + result = useRelationRows({ pageUid, onRowsLoaded }); + return null; +}; +const render = async (pageUid = "storage"): Promise => { + await act(async () => { + root.render(React.createElement(Harness, { pageUid })); + await Promise.resolve(); + }); +}; +beforeEach(() => { + vi.resetAllMocks(); + vi.useFakeTimers(); + Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); + Object.assign(window, { + roamAlphaAPI: { data: { addPullWatch, removePullWatch } }, + }); + load.mockResolvedValue([]); + root = createRoot(document.createElement("div")); +}); +afterEach(() => { + act(() => { + root.unmount(); + }); + vi.useRealTimers(); + vi.restoreAllMocks(); +}); +describe("relations page subscription", () => { + it("coalesces relation edits and releases its watch and pending refresh on unmount", async () => { + await render(); + const [pattern, entity, changed] = addPullWatch.mock.calls[0]; + changed(); + changed(); + changed(); + await act(async () => { + await vi.advanceTimersByTimeAsync(150); + }); + expect(load).toHaveBeenCalledTimes(2); + changed(); + act(() => { + root.unmount(); + }); + expect(removePullWatch).toHaveBeenCalledWith(pattern, entity, changed); + await vi.advanceTimersByTimeAsync(150); + expect(load).toHaveBeenCalledTimes(2); + }); + it("ignores an old load after switching pages", async () => { + let resolveOld: (rows: RelationRow[]) => void = () => {}; + load.mockImplementationOnce( + () => + new Promise((resolve) => { + resolveOld = resolve; + }), + ); + await render("old-page"); + await render("new-page"); + const fresh = result.rows; + await act(async () => { + resolveOld([{ uid: "old-relation" } as RelationRow]); + await Promise.resolve(); + }); + expect(result.rows).toBe(fresh); + expect(onRowsLoaded).toHaveBeenCalledTimes(1); + expect(removePullWatch.mock.calls[0][1]).toBe('[:block/uid "old-page"]'); + }); + it("keeps loaded rows on failure, restores storage, then recovers on refresh", async () => { + vi.spyOn(console, "error").mockImplementation(() => {}); + const rows = [ + { + uid: "relation", + source: { uid: "a" }, + destination: { uid: "b" }, + } as RelationRow, + ]; + load.mockResolvedValueOnce(rows); + await render(); + load.mockRejectedValueOnce(new Error("offline")); + await act(async () => { + result.refresh(); + await Promise.resolve(); + }); + expect(result.error).toBe(true); + expect(result.rows).toBe(rows); + expect(onRowsLoaded).toHaveBeenLastCalledWith(null); + await act(async () => { + result.refresh(); + await Promise.resolve(); + }); + expect(result.error).toBe(false); + expect(result.rows).toEqual([]); + expect(onRowsLoaded).toHaveBeenLastCalledWith([]); + }); + it("refreshes endpoint edits and deletions, deduplicates watches, and releases replaced endpoints", async () => { + const rows = [ + { uid: "one", source: { uid: "a" }, destination: { uid: "b" } }, + { uid: "two", source: { uid: "a" }, destination: { uid: "a" } }, + ] as RelationRow[]; + load.mockResolvedValue(rows); + await render(); + expect(addPullWatch.mock.calls.map(([, entity]) => entity)).toEqual([ + '[:block/uid "storage"]', + '[:block/uid "a"]', + '[:block/uid "b"]', + ]); + const [pattern, entity, changed] = addPullWatch.mock.calls[1]; + expect(pattern).toContain(":node/title"); + expect(pattern).toContain(":block/string"); + changed(); + changed(); + await act(async () => { + await vi.advanceTimersByTimeAsync(150); + }); + expect(load).toHaveBeenCalledTimes(2); + expect(addPullWatch).toHaveBeenCalledTimes(3); + load.mockResolvedValue([ + { ...rows[0], destination: { uid: "c" } } as RelationRow, + ]); + changed(); + await act(async () => { + await vi.advanceTimersByTimeAsync(150); + }); + expect(removePullWatch).toHaveBeenCalledWith(pattern, entity, changed); + expect(removePullWatch).toHaveBeenCalledWith( + pattern, + '[:block/uid "b"]', + changed, + ); + expect(addPullWatch.mock.calls.at(-1)?.[1]).toBe('[:block/uid "c"]'); + const latestChanged = addPullWatch.mock.calls.at(-1)![2]; + latestChanged(); + act(() => root.unmount()); + await vi.advanceTimersByTimeAsync(150); + expect(load).toHaveBeenCalledTimes(3); + expect(removePullWatch).toHaveBeenCalledWith( + pattern, + '[:block/uid "c"]', + latestChanged, + ); + }); +}); diff --git a/apps/roam/src/utils/__tests__/relationsPage.test.ts b/apps/roam/src/utils/__tests__/relationsPage.test.ts new file mode 100644 index 0000000000..0fc6691132 --- /dev/null +++ b/apps/roam/src/utils/__tests__/relationsPage.test.ts @@ -0,0 +1,446 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { + loadRelationRows, + getRelationFilterOptions, + UNKNOWN_RELATION_FILTER, + selectRelationRows, + type RelationRow, +} from "~/components/relationsPageData"; + +vi.mock("~/utils/getDiscourseRelations", () => ({ + default: () => [ + { + id: "supports", + label: "Supports", + source: "evidence", + destination: "claim", + }, + { + id: "challenges", + label: "Challenges", + source: "claim", + destination: "claim", + }, + { id: "generic", label: "Relates to", source: "*", destination: "*" }, + ], +})); +vi.mock("~/utils/getDiscourseNodes", () => ({ + default: () => [ + { type: "evidence", text: "Evidence" }, + { type: "claim", text: "Claim" }, + ], +})); +const q = vi.fn(); +const pull = vi.fn(); +beforeEach(() => { + vi.resetAllMocks(); + vi.stubGlobal("window", { roamAlphaAPI: { data: { async: { q } }, pull } }); +}); + +describe("relations page data", () => { + it("keeps the supported Any role distinct from unknown node types", async () => { + q.mockResolvedValueOnce([ + ["generic-relation", { "discourse-graph": { hasSchema: "generic" } }, 0], + ]); + const rows = await loadRelationRows("storage"); + expect(rows[0].source).toMatchObject({ typeId: "*", typeLabel: "Any" }); + expect(rows[0].destination).toMatchObject({ + typeId: "*", + typeLabel: "Any", + }); + expect(getRelationFilterOptions(rows).nodeType).toEqual([ + { value: "*", label: "Any (1)" }, + ]); + }); + it("resolves page and block endpoints, preserves raw props, and ignores other blocks", async () => { + const props = { + ":discourse-graph": { + ":hasSchema": "supports", + ":sourceUid": "page", + ":destinationUid": "block", + ":tentative": "true", + }, + ":extra": "keep", + }; + q.mockResolvedValueOnce([ + ["rel", props, 1704067200000], + ["note", { unrelated: true }], + ]).mockResolvedValueOnce([ + [{ uid: "page", title: "A claim" }], + [{ uid: "block", string: "Evidence" }], + ]); + const rows = await loadRelationRows("storage-page"); + expect(q.mock.calls[0][1]).toBe("storage-page"); + expect(q.mock.calls[1][1]).toEqual(["page", "block"]); + expect(pull).not.toHaveBeenCalled(); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + type: "Supports", + tentative: true, + createdAt: 1704067200000, + props, + source: { + text: "A claim", + isPage: true, + typeId: "evidence", + typeLabel: "Evidence", + }, + destination: { + text: "Evidence", + isPage: false, + typeId: "claim", + typeLabel: "Claim", + }, + }); + expect(rows[0].props).toBe(props); + }); + + it("keeps malformed relations inspectable and shares endpoint lookups", async () => { + q.mockResolvedValueOnce([ + [ + "one", + { + "discourse-graph": { + sourceUid: "deleted", + destinationUid: "deleted", + hasSchema: "missing", + }, + }, + ], + ["two", { "discourse-graph": { sourceUid: 42 } }], + ["invalid", { "discourse-graph": null }], + ]).mockResolvedValueOnce([]); + const rows = await loadRelationRows("storage"); + expect(rows).toHaveLength(2); + expect(rows[0]).toMatchObject({ + type: "Unknown type", + source: { text: "Missing node", exists: false }, + }); + expect(rows[1].source.uid).toBe(""); + expect(rows.every((row) => row.createdAt === null)).toBe(true); + expect(q).toHaveBeenCalledTimes(2); + expect(q.mock.calls[1][1]).toEqual(["deleted"]); + expect(pull).not.toHaveBeenCalled(); + }); + + it("loads thousands of endpoints with one asynchronous batch instead of synchronous reads", async () => { + const records = Array.from({ length: 2000 }, (_, index) => [ + `relation-${index}`, + { + "discourse-graph": { + sourceUid: `source-${index}`, + destinationUid: "shared", + }, + }, + 1704067200000, + ]); + q.mockResolvedValueOnce(records).mockImplementationOnce( + (_query, uids: string[]) => + Promise.resolve(uids.map((uid) => [{ uid: uid, title: uid }])), + ); + const rows = await loadRelationRows("storage"); + expect(rows).toHaveLength(2000); + expect(rows[1999].source).toMatchObject({ + text: "source-1999", + exists: true, + }); + expect(q).toHaveBeenCalledTimes(2); + expect(q.mock.calls[1][1]).toHaveLength(2001); + expect(pull).not.toHaveBeenCalled(); + }); + + it("skips endpoint loading when no valid endpoint UIDs exist", async () => { + q.mockResolvedValueOnce([ + ["rel", { "discourse-graph": { sourceUid: "", destinationUid: 42 } }], + ]); + const rows = await loadRelationRows("storage"); + expect(rows[0].source.exists).toBe(false); + expect(rows[0].destination.exists).toBe(false); + expect(q).toHaveBeenCalledTimes(1); + }); + + it("propagates a failed endpoint batch instead of marking every node missing", async () => { + q.mockResolvedValueOnce([ + ["rel", { "discourse-graph": { sourceUid: "page" } }], + ]).mockRejectedValueOnce(new Error("endpoint read failed")); + await expect(loadRelationRows("storage")).rejects.toThrow( + "endpoint read failed", + ); + }); + + it("propagates read failures instead of presenting an empty graph", async () => { + q.mockRejectedValue(new Error("read failed")); + await expect(loadRelationRows("storage")).rejects.toThrow("read failed"); + }); + it.each([-1, NaN, Infinity, 9e15, "1704067200000"])( + "keeps a relation with an invalid creation time (%s) inspectable", + async (createdAt) => { + q.mockResolvedValue([["rel", { "discourse-graph": {} }, createdAt]]); + expect((await loadRelationRows("storage"))[0].createdAt).toBeNull(); + }, + ); +}); + +const row = ({ + uid, + type, + source, + destination, +}: { + uid: string; + type: string; + source: string; + destination: string; +}): RelationRow => ({ + uid, + type, + schemaUid: type.toLowerCase(), + typeKnown: true, + tentative: false, + props: {}, + createdAt: null, + source: { + uid: `source-${uid}`, + text: source, + exists: true, + isPage: true, + typeId: "evidence", + typeLabel: "Evidence", + }, + destination: { + uid: `dest-${uid}`, + text: destination, + exists: true, + isPage: true, + typeId: "claim", + typeLabel: "Claim", + }, +}); +const rows = [ + row({ uid: "b", type: "Supports", source: "Study 10", destination: "Alpha" }), + row({ uid: "a", type: "Challenges", source: "Study 2", destination: "Beta" }), + row({ uid: "c", type: "Supports", source: "Study 1", destination: "Gamma" }), +]; +describe("relations page filtering and sorting", () => { + it("filters tentative and confirmed relations alongside type and search", () => { + const mixed = [ + rows[0], + { ...rows[1], tentative: true }, + { ...rows[2], tentative: true }, + ]; + const select = { + rows: mixed, + type: "", + search: "", + sort: "type" as const, + descending: false, + }; + expect( + selectRelationRows({ ...select, status: "tentative" }).map((r) => r.uid), + ).toEqual(["a", "c"]); + expect( + selectRelationRows({ ...select, status: "confirmed" }).map((r) => r.uid), + ).toEqual(["b"]); + expect(selectRelationRows({ ...select, status: "" })).toHaveLength(3); + expect( + selectRelationRows({ + ...select, + status: "tentative", + type: "supports", + nodeType: "evidence", + search: "Gamma", + }).map((r) => r.uid), + ).toEqual(["c"]); + expect( + selectRelationRows({ ...select, status: "confirmed", search: "Gamma" }), + ).toEqual([]); + expect(getRelationFilterOptions(mixed).status).toEqual([ + { value: "confirmed", label: "Confirmed (1)" }, + { value: "tentative", label: "Tentative (2)" }, + ]); + expect(getRelationFilterOptions([]).status).toEqual([ + { value: "confirmed", label: "Confirmed (0)" }, + { value: "tentative", label: "Tentative (0)" }, + ]); + }); + it("matches node types at either end and counts each relation once", () => { + const mixed = [ + rows[0], + { + ...rows[1], + source: { ...rows[1].source, typeId: "claim", typeLabel: "Claim" }, + }, + { + ...rows[2], + destination: { + ...rows[2].destination, + typeId: "question", + typeLabel: "Question", + }, + }, + ]; + const select = { + rows: mixed, + type: "", + search: "", + sort: "type" as const, + descending: false, + }; + expect( + selectRelationRows({ ...select, nodeType: "claim" }).map((r) => r.uid), + ).toEqual(["a", "b"]); + expect( + selectRelationRows({ ...select, nodeType: "question" }).map((r) => r.uid), + ).toEqual(["c"]); + expect( + selectRelationRows({ + ...select, + type: "supports", + nodeType: "claim", + search: "study", + }).map((r) => r.uid), + ).toEqual(["b"]); + expect( + selectRelationRows({ + ...select, + nodeType: "missing-type", + }), + ).toEqual([]); + const reversed = mixed.map((row) => ({ + ...row, + source: row.destination, + destination: row.source, + })); + expect( + selectRelationRows({ ...select, rows: reversed, nodeType: "claim" }).map( + (r) => r.uid, + ), + ).toEqual(["a", "b"]); + expect(getRelationFilterOptions(mixed).nodeType).toEqual([ + { value: "claim", label: "Claim (2)" }, + { value: "evidence", label: "Evidence (2)" }, + { value: "question", label: "Question (1)" }, + ]); + expect(getRelationFilterOptions(reversed).nodeType).toEqual( + getRelationFilterOptions(mixed).nodeType, + ); + }); + it("groups all unknown schemas into one selectable filter, including missing schema IDs", () => { + const unknown = ["deleted-schema", ""].map((schemaUid, index) => ({ + ...rows[index], + schemaUid, + type: "Unknown type", + typeKnown: false, + source: { ...rows[index].source, typeId: "", typeLabel: "Unknown type" }, + })); + const mixed = [...unknown, rows[2]]; + const options = getRelationFilterOptions(mixed); + expect(options.type).toEqual([ + { value: "supports", label: "Supports (1)" }, + { value: UNKNOWN_RELATION_FILTER, label: "Unknown type (2)" }, + ]); + expect( + selectRelationRows({ + rows: mixed, + type: UNKNOWN_RELATION_FILTER, + nodeType: UNKNOWN_RELATION_FILTER, + search: "", + sort: "type", + descending: false, + }), + ).toHaveLength(2); + }); + it("keeps identically named node types separate by their stable IDs", () => { + const mixed = [ + rows[0], + { ...rows[1], source: { ...rows[1].source, typeId: "other-evidence" } }, + ]; + expect(getRelationFilterOptions(mixed).nodeType).toEqual([ + { value: "claim", label: "Claim (2)" }, + { value: "evidence", label: "Evidence (1)" }, + { value: "other-evidence", label: "Evidence (1)" }, + ]); + expect( + selectRelationRows({ + rows: mixed, + type: "", + nodeType: "other-evidence", + search: "evidence", + sort: "type", + descending: false, + }).map((r) => r.uid), + ).toEqual(["a"]); + }); + it.each([false, true])( + "sorts creation timestamps chronologically (descending: %s), with unknown dates last", + (descending) => { + const dated = [ + { ...rows[0], uid: "new", createdAt: Date.UTC(2026, 0, 1) }, + { ...rows[0], uid: "unknown", createdAt: null }, + { ...rows[0], uid: "old", createdAt: Date.UTC(2025, 11, 31) }, + { ...rows[0], uid: "epoch", createdAt: 0 }, + ]; + expect( + selectRelationRows({ + rows: dated, + type: "", + search: "", + sort: "createdAt", + descending, + }).map((row) => row.uid), + ).toEqual( + descending + ? ["new", "old", "epoch", "unknown"] + : ["epoch", "old", "new", "unknown"], + ); + }, + ); + it("combines type and case-insensitive search including endpoint UIDs", () => { + expect( + selectRelationRows({ + rows, + type: "supports", + search: " STUDY 1 ", + sort: "type", + descending: false, + }).map((r) => r.uid), + ).toEqual(["b", "c"]); + expect( + selectRelationRows({ + rows, + type: "supports", + search: "dest-a", + sort: "type", + descending: false, + }), + ).toEqual([]); + expect( + selectRelationRows({ + rows, + type: "", + search: "dest-a", + sort: "type", + descending: false, + }).map((r) => r.uid), + ).toEqual(["a"]); + }); + it.each([ + ["type", false, ["a", "b", "c"]], + ["source", false, ["c", "a", "b"]], + ["destination", true, ["c", "a", "b"]], + ] as const)( + "sorts by %s without modifying the loaded data", + (sort, descending, expected) => { + expect( + selectRelationRows({ + rows, + type: "", + search: "", + sort, + descending, + }).map((r) => r.uid), + ).toEqual(expected); + expect(rows.map((r) => r.uid)).toEqual(["b", "a", "c"]); + }, + ); +}); diff --git a/apps/roam/src/utils/__tests__/relationsPageControls.test.ts b/apps/roam/src/utils/__tests__/relationsPageControls.test.ts new file mode 100644 index 0000000000..1dda530cd8 --- /dev/null +++ b/apps/roam/src/utils/__tests__/relationsPageControls.test.ts @@ -0,0 +1,92 @@ +// @vitest-environment jsdom +import React from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { act, Simulate } from "react-dom/test-utils"; +import { afterEach, beforeEach, expect, it, vi } from "vitest"; +import RelationsPage from "~/components/RelationsPage"; +import { type RelationRow } from "~/components/relationsPageData"; + +const state = vi.hoisted(() => ({ rows: [] as RelationRow[] })); +vi.mock("~/utils/getDiscourseNodes", () => ({ default: () => [] })); +vi.mock("~/utils/getDiscourseRelations", () => ({ default: () => [] })); +vi.mock("~/components/useRelationRows", () => ({ + useRelationRows: () => ({ + rows: state.rows, + loading: false, + error: false, + refresh: vi.fn(), + }), +})); +vi.mock("~/utils/roamReactComponents", () => ({ + RenderRoamBlockString: ({ string }: { string: string }) => string, +})); +let root: Root; +let container: HTMLDivElement; +const render = (): void => { + act(() => + root.render( + React.createElement(RelationsPage, { + pageUid: "storage", + onRowsLoaded: vi.fn(), + onVisibilityChange: vi.fn(), + }), + ), + ); +}; +beforeEach(() => { + Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); + container = document.createElement("div"); + root = createRoot(container); + HTMLElement.prototype.scrollTo = vi.fn(); + const endpoint = { + uid: "a", + text: "Endpoint", + exists: true, + isPage: false, + typeId: "type", + typeLabel: "Type", + }; + state.rows = Array.from({ length: 125 }, (_, i) => ({ + uid: String(i), + type: "Informs", + schemaUid: "schema", + typeKnown: true, + source: endpoint, + destination: endpoint, + tentative: false, + createdAt: 1, + props: {}, + })); +}); +afterEach(() => act(() => root.unmount())); + +it("stays on the clamped page after deleted relations are restored", () => { + render(); + const next = container.querySelector( + '[aria-label="Next relations"]', + )!; + act(() => Simulate.click(next)); + expect(container.textContent).toContain("26–50 of 125"); + const all = state.rows; + state.rows = all.slice(0, 20); + render(); + expect(container.textContent).toContain("1–20 of 20"); + state.rows = all; + render(); + expect(container.textContent).toContain("1–25 of 125"); +}); + +it("caps rendering at 100 results and restores the capped input on blur", () => { + render(); + const input = container.querySelector( + '[aria-label="Results per page"]', + )!; + expect(input.max).toBe("100"); + act(() => { + input.value = "1000"; + Simulate.change(input); + }); + expect(container.querySelectorAll(".dg-relation-row")).toHaveLength(100); + act(() => Simulate.blur(input)); + expect(input.value).toBe("100"); +}); diff --git a/apps/roam/src/utils/__tests__/relationsPageMount.test.ts b/apps/roam/src/utils/__tests__/relationsPageMount.test.ts new file mode 100644 index 0000000000..97847d4cfd --- /dev/null +++ b/apps/roam/src/utils/__tests__/relationsPageMount.test.ts @@ -0,0 +1,149 @@ +// @vitest-environment jsdom +import type React from "react"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { initRelationsPages } from "~/components/mountRelationsPage"; + +const mocks = vi.hoisted(() => ({ + render: vi.fn(), + unmount: vi.fn(), + props: vi.fn(), + parent: vi.fn(), +})); +vi.mock("roamjs-components/util/renderWithUnmount", () => ({ + default: mocks.render, +})); +vi.mock("roamjs-components/dom", () => ({ + getUidsFromId: (id: string) => ({ blockUid: id }), +})); +vi.mock("roamjs-components/queries/getParentUidByBlockUid", () => ({ + default: mocks.parent, +})); +vi.mock("~/utils/getBlockProps", () => ({ + default: mocks.props, + isJsonObject: (value: unknown) => + Boolean(value && typeof value === "object" && !Array.isArray(value)), +})); +vi.mock("~/components/RelationsPage", () => ({ default: () => null })); +let pages: ReturnType; +let article: HTMLElement; +let heading: HTMLElement; +let loaded: (uids: string[] | null) => void; +let setVisible: (visible: boolean) => void; +const addBlock = ({ + uid, + parent = article, +}: { + uid: string; + parent?: HTMLElement; +}): HTMLElement => { + const container = document.createElement("div"); + container.className = "roam-block-container"; + const block = document.createElement("div"); + block.className = "roam-block"; + block.id = uid; + container.append(block); + parent.append(container); + return container; +}; +beforeEach(() => { + vi.resetAllMocks(); + document.body.innerHTML = + '

Relations

'; + article = document.querySelector("article")!; + heading = document.querySelector("h1")!; + mocks.render.mockImplementation( + ( + element: React.ReactElement<{ + onRowsLoaded: typeof loaded; + onVisibilityChange: typeof setVisible; + }>, + ) => { + loaded = element.props.onRowsLoaded; + setVisible = element.props.onVisibilityChange; + return mocks.unmount; + }, + ); + mocks.props.mockReturnValue({ "discourse-graph": {} }); + mocks.parent.mockReturnValue("storage"); + pages = initRelationsPages(); + pages.mount({ + heading, + title: "roam/js/discourse-graph/relations", + pageUid: "storage", + }); +}); +afterEach(() => { + pages.dispose(); + document.body.innerHTML = ""; +}); +describe("relations page mounting", () => { + it("restores storage while hidden, including new blocks and refreshed rows, then conceals it again", async () => { + const direct = addBlock({ uid: "direct" }); + loaded(["direct"]); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(true); + setVisible(false); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(false); + const added = addBlock({ uid: "added" }); + await Promise.resolve(); + expect(added.classList.contains("dg-relation-storage-block")).toBe(false); + loaded(["direct", "added"]); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(false); + expect(added.classList.contains("dg-relation-storage-block")).toBe(false); + setVisible(true); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(true); + expect(added.classList.contains("dg-relation-storage-block")).toBe(true); + setVisible(false); + loaded(null); + setVisible(true); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(false); + expect(added.classList.contains("dg-relation-storage-block")).toBe(false); + }); + it("conceals the first live relation on an empty page but preserves nested relations and notes", async () => { + loaded([]); + mocks.parent.mockImplementation((uid: string) => + uid === "nested" ? "note" : "storage", + ); + mocks.props.mockImplementation((uid: string) => + uid === "note" ? {} : { "discourse-graph": {} }, + ); + const note = addBlock({ uid: "note" }); + const nested = addBlock({ uid: "nested", parent: note }); + const direct = addBlock({ uid: "direct" }); + await Promise.resolve(); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(true); + expect(nested.classList.contains("dg-relation-storage-block")).toBe(false); + expect(note.classList.contains("dg-relation-storage-block")).toBe(false); + }); + it("ignores UI mutations and restores storage on failure or unmount", async () => { + const direct = addBlock({ uid: "direct" }); + loaded(["direct"]); + await Promise.resolve(); + mocks.parent.mockClear(); + mocks.props.mockClear(); + addBlock({ + uid: "rendered-preview", + parent: article.querySelector(".dg-relations-host")!, + }); + await Promise.resolve(); + expect(mocks.parent).not.toHaveBeenCalled(); + expect(mocks.props).not.toHaveBeenCalled(); + loaded(null); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(false); + loaded(["direct"]); + pages.unmount(heading); + expect(direct.classList.contains("dg-relation-storage-block")).toBe(false); + expect(article.querySelector(".dg-relations-host")).toBeNull(); + expect(mocks.unmount).toHaveBeenCalledTimes(1); + addBlock({ uid: "after-unmount" }); + await Promise.resolve(); + expect(mocks.props).not.toHaveBeenCalled(); + }); + it("does not mount twice for the same heading", () => { + pages.mount({ + heading, + title: "roam/js/discourse-graph/relations", + pageUid: "storage", + }); + expect(mocks.render).toHaveBeenCalledTimes(1); + }); +}); diff --git a/apps/roam/src/utils/__tests__/resultsViewShareData.test.ts b/apps/roam/src/utils/__tests__/resultsViewShareData.test.ts new file mode 100644 index 0000000000..1fd2ebc1e1 --- /dev/null +++ b/apps/roam/src/utils/__tests__/resultsViewShareData.test.ts @@ -0,0 +1,167 @@ +// @vitest-environment jsdom +import React from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { act } from "react-dom/test-utils"; +import { Dialog, Popover } from "@blueprintjs/core"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import ResultsView from "~/components/results-view/ResultsView"; + +vi.mock("~/components/Export", () => ({ + default: ({ isOpen, onClose }: { isOpen: boolean; onClose: () => void }) => + React.createElement( + Dialog, + { + isOpen, + title: "Share Query Results", + autoFocus: false, + enforceFocus: false, + }, + React.createElement("input", { "aria-label": "Filename" }), + React.createElement("button", { onClick: onClose }, "Cancel"), + ), +})); +vi.mock("~/components/results-view/Charts", () => ({ default: () => null })); +vi.mock("~/components/results-view/Timeline", () => ({ default: () => null })); +vi.mock("~/components/results-view/Kanban", () => ({ default: () => null })); +vi.mock("~/components/results-view/ResultsTable", () => ({ + default: () => null, +})); +vi.mock("~/components/results-view/Inputs", () => ({ Inputs: () => null })); +vi.mock("~/utils/parseQuery", () => ({ default: vi.fn() })); +vi.mock("~/utils/fireQuery", () => ({ getDatalogQuery: vi.fn() })); +vi.mock("posthog-js", () => ({ default: { capture: vi.fn() } })); +vi.mock("roamjs-components/components/ExtensionApiContext", () => ({ + useExtensionAPI: () => ({ settings: { get: () => undefined } }), +})); +vi.mock("~/utils/parseResultSettings", () => ({ + default: () => ({ + resultNodeUid: "results", + activeSort: [], + filters: {}, + columnFilters: [], + random: 0, + page: 1, + pageSize: 10, + views: [{ uid: "text", mode: "link" }], + searchFilter: "", + showSearchFilter: false, + showInterface: true, + showInputs: false, + showAlias: false, + alias: "", + layout: { mode: "table" }, + }), +})); +vi.mock("~/utils/postProcessResults", () => ({ + default: (results: unknown[]) => ({ + allProcessedResults: results, + paginatedResults: results, + }), +})); + +let container: HTMLDivElement; +let root: Root; + +beforeEach(() => { + vi.stubGlobal("IS_REACT_ACT_ENVIRONMENT", true); + vi.useFakeTimers(); + container = document.createElement("div"); + document.body.appendChild(container); + root = createRoot(container); +}); + +afterEach(() => { + act(() => root.unmount()); + container.remove(); + vi.useRealTimers(); + vi.unstubAllGlobals(); +}); + +const settle = async (): Promise => { + await act(async () => { + await vi.advanceTimersByTimeAsync(400); + }); +}; + +const click = async (element: HTMLElement | null): Promise => { + expect(element).not.toBeNull(); + act(() => element!.click()); + await settle(); +}; + +describe("ResultsView Share Data", () => { + it.each(["page reference", "page-level context", "regular query"])( + "keeps the dialog open and usable from %s", + async (context) => { + const view = React.createElement(ResultsView, { + parentUid: "query", + results: [{ uid: "result", text: "A result" }], + columns: [{ key: "text", uid: "text", selection: "text" }], + onRefresh: vi.fn(), + simplified: context !== "regular query", + preventSavingSettings: context !== "regular query", + }); + act(() => { + root.render( + context === "page reference" + ? React.createElement(Popover, { + autoFocus: false, + target: React.createElement( + "button", + null, + "Discourse context", + ), + content: view, + }) + : view, + ); + }); + if (context === "page reference") { + await click(container.querySelector("button")); + } + await click( + document + .querySelector( + ".roamjs-query-results-view .bp3-icon-more", + ) + ?.closest("button") ?? null, + ); + const share = Array.from( + document.querySelectorAll(".bp3-menu-item"), + ).find((item) => item.textContent === "Share Data"); + await click(share ?? null); + + expect( + document.querySelector(".roamjs-query-results-view"), + ).not.toBeNull(); + expect(document.querySelector(".bp3-dialog")?.textContent).toContain( + "Share Query Results", + ); + const input = document.querySelector( + 'input[aria-label="Filename"]', + ); + expect(input).not.toBeNull(); + act(() => input!.focus()); + expect(document.activeElement).toBe(input); + expect( + Array.from(document.querySelectorAll(".bp3-menu-item")).some( + (item) => item.textContent === "Share Data", + ), + ).toBe(false); + + await click( + Array.from( + document.querySelectorAll(".bp3-dialog button"), + ).find((button) => button.textContent === "Cancel") ?? null, + ); + expect(document.querySelector(".bp3-dialog")).toBeNull(); + expect( + document.querySelector(".roamjs-query-results-view"), + ).not.toBeNull(); + if (context === "page reference") { + await click(container.querySelector("button")); + expect(document.querySelector(".roamjs-query-results-view")).toBeNull(); + } + }, + ); +}); diff --git a/apps/roam/src/utils/__tests__/searchDiscourseNodes.test.ts b/apps/roam/src/utils/__tests__/searchDiscourseNodes.test.ts index 9163b806e2..1d00059b58 100644 --- a/apps/roam/src/utils/__tests__/searchDiscourseNodes.test.ts +++ b/apps/roam/src/utils/__tests__/searchDiscourseNodes.test.ts @@ -116,6 +116,26 @@ describe("searchDiscourseNodes", () => { expect(results.map((r) => r.result.uid)).toEqual(["s0", "s1", "m1", "m2"]); }); + it("passes candidate uids to the semantic provider", async () => { + runRoamSemanticSearch.mockResolvedValue({ + filteredResults: [], + filteredResultCount: 0, + }); + const candidateUids = new Set(["b1"]); + + await searchDiscourseNodes({ + nodeTypes: [], + query: "governance", + resultsByUid: new Map(), + runMiniSearch: () => [], + candidateUids, + }); + + expect(runRoamSemanticSearch).toHaveBeenCalledWith( + expect.objectContaining({ candidateUids }), + ); + }); + it("falls back to MiniSearch when the semantic provider throws", async () => { runRoamSemanticSearch.mockRejectedValue(new Error("semantic unavailable")); diff --git a/apps/roam/src/utils/discourseNodeSearch.ts b/apps/roam/src/utils/discourseNodeSearch.ts index 37fd2fde43..d7cf78ef49 100644 --- a/apps/roam/src/utils/discourseNodeSearch.ts +++ b/apps/roam/src/utils/discourseNodeSearch.ts @@ -14,6 +14,8 @@ export const BASIC_DISCOURSE_NODE_PULL = export const DISCOURSE_NODE_SEARCH_METADATA_PULL = `[:block/string :node/title :block/uid :create/time :edit/time {:create/user [:user/display-name :user/email]} {:edit/user [:user/display-name :user/email]}]`; +export const CANDIDATE_BLOCK_SEARCH_PULL = `[:block/string :block/uid :create/time :edit/time {:create/user [:user/display-name :user/email]} {:edit/user [:user/display-name :user/email]} {:block/page [:node/title]}]`; + /* eslint-disable @typescript-eslint/naming-convention */ type PulledDiscourseUser = { ":user/display-name"?: string; @@ -35,6 +37,7 @@ export type PulledDiscourseNode = { ":edit/time"?: string | number; ":create/user"?: PulledDiscourseUser; ":edit/user"?: PulledDiscourseUser; + ":block/page"?: { ":node/title"?: string }; }; /* eslint-enable @typescript-eslint/naming-convention */ @@ -109,6 +112,79 @@ export const queryDiscourseNodesByFormat = async ({ return queryResults.map(([result]) => result); }; +export const hasCandidateTag = ( + node: DiscourseNode, +): node is DiscourseNode & { tag: string } => !!node.tag?.trim(); + +// `#tag` and `#[[tag]]` both name the tag page `tag`. +export const getCandidateTagTitle = (tag: string): string => + tag + .trim() + .replace(/^#/, "") + .replace(/^\[\[(.*)\]\]$/, "$1"); + +const escapeRegExp = (value: string): string => + value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + +export const buildCandidateBlocksByTagQuery = ({ + tagTitle, + pullExpression, +}: { + tagTitle: string; + pullExpression: string; +}): string => { + // Case-insensitive, like the plugin's other node tag matching (getCleanTagText). + const titleRegex = new RegExp(`^${escapeRegExp(tagTitle)}$`); + return `[ + :find + (pull ?block ${pullExpression}) + :where + [(re-pattern "(?i)${escapeRegexPatternForDatalog(titleRegex)}") ?tag-title-regex] + [?tag :node/title ?tag-title] + [(re-find ?tag-title-regex ?tag-title)] + [?block :block/refs ?tag] +]`; +}; + +export const queryCandidateBlocksByTag = async ({ + node, + pullExpression = BASIC_DISCOURSE_NODE_PULL, +}: { + node: DiscourseNode & { tag: string }; + pullExpression?: string; +}): Promise => { + const query = buildCandidateBlocksByTagQuery({ + tagTitle: getCandidateTagTitle(node.tag), + pullExpression, + }); + const queryResults = (await window.roamAlphaAPI.data.async.fast.q(query)) as [ + PulledDiscourseNode, + ][]; + + return queryResults.map(([result]) => result); +}; + +export const stripCandidateTags = ({ + text, + tagTitles, +}: { + text: string; + tagTitles: string[]; +}): string => + tagTitles + .reduce((stripped, tagTitle) => { + const escaped = escapeRegExp(tagTitle); + return stripped.replace( + new RegExp( + `#?\\[\\[${escaped}\\]\\]|#${escaped}(?![\\p{L}\\p{N}_-])`, + "giu", + ), + "", + ); + }, text) + .replace(/\s+/g, " ") + .trim(); + export const queryDiscourseNodesByFormatSync = ({ node, pullExpression = BASIC_DISCOURSE_NODE_PULL, diff --git a/apps/roam/src/utils/discourseNodeSearchProviders.ts b/apps/roam/src/utils/discourseNodeSearchProviders.ts index ab892e08db..b795a74a2b 100644 --- a/apps/roam/src/utils/discourseNodeSearchProviders.ts +++ b/apps/roam/src/utils/discourseNodeSearchProviders.ts @@ -252,13 +252,17 @@ export const searchSemanticNodeTitles = async ({ export const runRoamSemanticSearch = async ({ nodeTypes, query, + candidateUids, }: { nodeTypes: DiscourseNode[]; query: string; + // Block hits are kept only when listed here, so this also carries the type filter. + candidateUids?: Set; }): Promise => { const trimmedQuery = query.trim(); const matchers = getDiscourseNodeFormatMatchers(nodeTypes); - if (!trimmedQuery || !matchers.length) { + const includeBlocks = !!candidateUids?.size; + if (!trimmedQuery || (!matchers.length && !includeBlocks)) { return { rawResults: [], rawResultCount: 0, @@ -274,7 +278,7 @@ export const runRoamSemanticSearch = async ({ const hits = (await window.roamAlphaAPI.data.async.semanticSearch({ "search-str": trimmedQuery, "hide-code-blocks": false, - "search-blocks": false, + "search-blocks": includeBlocks, "search-pages": true, k: ROAM_SEMANTIC_SEARCH_RESULT_LIMIT, })) as RoamSemanticSearchHit[]; @@ -301,9 +305,29 @@ export const runRoamSemanticSearch = async ({ }); }); - const filteredResults = filterResultsToDiscourseNodeTitles({ - matchers, - results: rawResults, + const nodeResultsByUid = new Map( + filterResultsToDiscourseNodeTitles({ matchers, results: rawResults }).map( + (result) => [result.uid, result], + ), + ); + const seenUids = new Set(); + const filteredResults = hits.flatMap((hit): AdminSearchResultItem[] => { + const uid = hit.uid || ""; + if (seenUids.has(uid)) return []; + const result = + hit.type === "page" + ? nodeResultsByUid.get(uid) + : candidateUids?.has(uid) + ? toResultItem({ + uid, + text: hit.text || hit.string || hit.content || uid, + score: hit.score, + source: hit.type, + }) + : undefined; + if (!result) return []; + seenUids.add(uid); + return [result]; }); return { @@ -313,8 +337,8 @@ export const runRoamSemanticSearch = async ({ filteredResultCount: filteredResults.length, note: `.semanticSearch returned ${pageHits.length}/${ROAM_SEMANTIC_SEARCH_RESULT_LIMIT} page hits; ` + - `${hits.length - pageHits.length} non-page hits were discarded; ` + - `${filteredResults.length} matched discourse node page-title formats.`, + `${hits.length - pageHits.length} non-page hits; ` + + `${filteredResults.length} matched discourse node page-title formats or candidate blocks.`, }; }; diff --git a/apps/roam/src/utils/discourseNodeSearchTypes.ts b/apps/roam/src/utils/discourseNodeSearchTypes.ts index 516eee4721..4b54e9b711 100644 --- a/apps/roam/src/utils/discourseNodeSearchTypes.ts +++ b/apps/roam/src/utils/discourseNodeSearchTypes.ts @@ -12,6 +12,8 @@ export type SearchResult = { createdAt: string; lastModified: string; authorName: string; + // Set for a block tagged with a node type's candidate tag, not a node page. + candidate?: { nodeTypes: string[]; pageTitle: string }; }; // score and source are carried for diagnostics and future rank fusion; ordering diff --git a/apps/roam/src/utils/initializeObserversAndListeners.ts b/apps/roam/src/utils/initializeObserversAndListeners.ts index b80536ac3e..472c0e942c 100644 --- a/apps/roam/src/utils/initializeObserversAndListeners.ts +++ b/apps/roam/src/utils/initializeObserversAndListeners.ts @@ -50,6 +50,7 @@ import { renderPublishNodeTitleButton } from "~/components/PublishNodeTitleButto import { renderRefreshImportedNodeTitleButton } from "~/components/RefreshImportedNodeTitleButton"; import { readImportedSourceIdentity } from "~/utils/importedSourceIdentity"; import { renderCanvasEmbed } from "~/components/canvas/CanvasEmbed"; +import { initRelationsPages } from "~/components/mountRelationsPage"; import getPageUidByPageTitle from "roamjs-components/queries/getPageUidByPageTitle"; import getPageTitleByPageUid from "roamjs-components/queries/getPageTitleByPageUid"; import findDiscourseNode from "./findDiscourseNode"; @@ -105,6 +106,7 @@ export const initObservers = ({ }; cleanups: Array<() => void>; } => { + const relationsPages = initRelationsPages(); const pageTitleObserver = createHTMLObserver({ tag: "H1", className: "rm-title-display", @@ -112,6 +114,8 @@ export const initObservers = ({ const h1 = e as HTMLHeadingElement; const { title, uid } = getTitleAndUidFromHeader(h1); + relationsPages.mount({ heading: h1, title, pageUid: uid }); + const settings = bulkReadSettings(); const props = { title, h1, onloadArgs }; @@ -158,6 +162,7 @@ export const initObservers = ({ renderTldrawCanvasInSidebar(props); } }, + removeCallback: relationsPages.unmount, }); const queryBlockObserver = createButtonObserver({ @@ -478,6 +483,11 @@ export const initObservers = ({ discourseNodeSearchTriggerListener, nodeCreationPopoverListener, }, - cleanups: [unsubGlobalTrigger, unsubPersonalTrigger, unsubSearchTrigger], + cleanups: [ + unsubGlobalTrigger, + unsubPersonalTrigger, + unsubSearchTrigger, + relationsPages.dispose, + ], }; }; diff --git a/apps/roam/src/utils/searchDiscourseNodes.ts b/apps/roam/src/utils/searchDiscourseNodes.ts index 9be5bacd15..22e5cb2a7c 100644 --- a/apps/roam/src/utils/searchDiscourseNodes.ts +++ b/apps/roam/src/utils/searchDiscourseNodes.ts @@ -34,11 +34,13 @@ export const searchDiscourseNodes = async ({ query, resultsByUid, runMiniSearch, + candidateUids, }: { nodeTypes: DiscourseNode[]; query: string; resultsByUid: Map; runMiniSearch: () => ScoredSearchResult[]; + candidateUids?: Set; }): Promise => { const trimmedQuery = query.trim(); if (!trimmedQuery) return []; @@ -52,6 +54,7 @@ export const searchDiscourseNodes = async ({ const providerResult = await runRoamSemanticSearch({ nodeTypes, query: trimmedQuery, + candidateUids, }); const semanticResults = providerResult.filteredResults.map((item) => toScoredSearchResultFromSemantic({ diff --git a/apps/website/app/(home)/layout.tsx b/apps/website/app/(home)/layout.tsx index 0de0a16d4b..cd7f2125e0 100644 --- a/apps/website/app/(home)/layout.tsx +++ b/apps/website/app/(home)/layout.tsx @@ -60,7 +60,7 @@ const HomeLayout = async ({ return (
-
-
+
+
diff --git a/apps/website/app/(home)/page.tsx b/apps/website/app/(home)/page.tsx index de0ac435ed..e1094706e1 100644 --- a/apps/website/app/(home)/page.tsx +++ b/apps/website/app/(home)/page.tsx @@ -10,54 +10,25 @@ import { GitBranch, Mail, MessageCircle, - Network, Puzzle, - Sparkles, } from "lucide-react"; import { getAllBlogs } from "~/(home)/blog/readBlogs"; +import { FoundationsSection } from "~/components/FoundationsSection"; +import { GraphBackgroundAnimation } from "~/components/GraphBackgroundAnimation"; import { Logo } from "~/components/Logo"; import { PlatformBadge } from "~/components/PlatformBadge"; +import { SectionHeader } from "~/components/SectionHeader"; import { TeamPerson } from "~/components/TeamPerson"; -import { TEAM_MEMBERS } from "~/data/constants"; +import { MATSU_LAB_URL, TEAM_MEMBERS } from "~/data/constants"; import { STATIC_NEWS_ITEMS } from "~/data/news"; import type { NewsItem } from "~/types/news"; +import { getGraphColors } from "~/utils/brandColors"; import { formatDisplayDate } from "~/utils/formatDate"; import { sortByDateDesc } from "~/utils/sortByDate"; const SLACK_URL = "https://join.slack.com/t/discoursegraphs/shared_invite/zt-37xklatti-cpEjgPQC0YyKYQWPNgAkEg"; -const FEATURE_CARDS = [ - { - alt: "Structured scientific infrastructure diagram", - description: - "Map claims, evidence, questions, and projects as modular pieces that can be reused across people, tools, and contexts.", - image: "/section1.webp", - title: "Make research ideas composable", - }, - { - alt: "Discourse graph coordination layer", - description: - "Turn high-signal findings into shareable graph objects that support attribution, reuse, and decentralized collaboration.", - image: "/section4.webp", - title: "Liberate findings from static files", - }, - { - alt: "Shareable findings workflow", - description: - "Exchange work in a form that is easier to find, update, and build on than documents locked into one hierarchy.", - image: "/section3.webp", - title: "Synthesize and update continuously", - }, - { - alt: "Knowledge synthesis workflow", - description: - "Separate observations from interpretations so researchers can compare, remix, and update shared knowledge without flattening disagreement.", - image: "/section2.webp", - title: "Communicate with better structure", - }, -]; - const LAB_BENEFITS = [ "Identify knowledge gaps and starter projects for new researchers", "Onboard collaborators into active research faster", @@ -200,34 +171,6 @@ const SUPPORTERS: Supporter[] = [ }, ]; -type SectionHeaderProps = { - description?: string; - eyebrow: string; - isWide?: boolean; - title: string; -}; - -const SectionHeader = ({ - description, - eyebrow, - isWide = false, - title, -}: SectionHeaderProps): ReactElement => ( -
-

- {eyebrow} -

-

- {title} -

- {description && ( -

- {description} -

- )} -
-); - const ArrowLink = ({ children, href, @@ -246,6 +189,7 @@ const ArrowLink = ({ const Home = async (): Promise => { const blogs = await getAllBlogs(); + const graphColors = getGraphColors(); const blogNewsItems: NewsItem[] = blogs.map((blog) => ({ date: blog.date, @@ -261,26 +205,20 @@ const Home = async (): Promise => { return (
-
- Discourse Graph network visualization -
+
+
-
-

+

+

Open infrastructure for collaborative knowledge synthesis

-

+

Create knowledge. Make connections.

-

+

A tool and ecosystem for turning research claims, evidence, and questions into modular graphs that teams can share, update, and build on. @@ -297,14 +235,14 @@ const Home = async (): Promise => { Explore plugins

-
- -
- - - - - - - - -
-
- -
- {FEATURE_CARDS.map((feature) => ( - -
- {feature.alt} -
- -

- {feature.title} -

-

- {feature.description} -

- - - ))} -
- -
-
-

- Tool choice -

-

- Client-agnostic and researcher-aligned -

-
-

- Discourse Graphs are a decentralized knowledge exchange - protocol designed to be implemented and owned by researchers - rather than publishers. -

-

- The model can be implemented in networked notebook software - like{" "} - Roam Research,{" "} - Obsidian, and other - tools researchers already use. -

-

- Discourse Graphs are like GitHub for scientific - communication. -

-
-
-
-
- Client-agnostic discourse graph workflow -
-
- Researcher-aligned discourse graph workflow -
-
+ +
+

+ Scientific inquiry has been artificially constrained by the + medium it's transferred in and the length of the + publication process. +

+

+ Discourse graphs are breaking free of these constraints, making + science more accessible and collaborative by building{" "} + + modular, interoperable scientific knowledge systems + + . +

+

+ This new method is available to use now in plugins for Roam + Research and Obsidian and a tool-agnostic protocol is in + development to enable collaboration across platforms. +

+ +
=> { />
- Snapshot of MATSU lab{" "} + Snapshot of MATSU lab{" "} Discourse Graph
@@ -750,7 +605,7 @@ const Home = async (): Promise => {
-
+
=> {

Help build better infrastructure for collaborative research

-

+

We are building user-friendly Discourse Graph plugins in tools for thought and would like to hear from researchers, developers, labs, and funders. @@ -771,24 +626,24 @@ const Home = async (): Promise => {

-