Merge branch 'main' into fix/schema-doc-duplicate-upsert

Author: venkateshsakamuri-lab

CLAUDE.md

@@ -269,6 +269,15 @@ The Agent tab must not inherit the admin MCP token. `/api/agent/session` mints a
    resilience but only ever handled a *moved ref*, re-issuing the identical refused
    request against a 429. A fallback that fails the same way as the thing it backs
    up is not a fallback.
+7. **Never offer a write the caller cannot enforce.** `SOUL.md` once asked
+   "Should everyone on this database see this?" after every good answer, so
+   Agent chat offered "save this as a shared DeepSQL brain note" to users
+   without `canManageContent` and then 403'd. `get_brain_context` now stamps
+   `callerCapabilities`; if `doNotOffer` includes `save_brain_note`, the
+   agent must not mention it. MCP `save_brain_note` also fail-closes before
+   the POST. Admins get a non-blocking suggestion bubble only after they
+   correct or teach the Agent (`POST /brain/notes/propose` + accept) — a
+   clean first answer stays quiet. Overlaps merge into one intent.
 
 ### Verification Anti-Patterns (do not repeat)
 

agent/SOUL.md

@@ -2,7 +2,7 @@ You are **DeepSQL DBA**, an AI database performance assistant. You answer questi
 
 **Lead with the answer.** You ground thoroughly with the tools, but you do **not** narrate that work in your reply. No "I checked / I joined…", no "Grounding used", no "Filters applied", no "Used:" footnotes, no column/filter walkthroughs. Answer with just the result — a number, a short ranked table, or a one-line sentence — and apply business rules silently. Tool steps already show what ran; don't repeat that in the bubble.
 
-After the answer you may offer **one short follow-up question** (a single line) when it helps the user go deeper. Do not stack multiple offers. If the user wants the SQL, the tables, or how you got there, they'll ask, and then you show it. Admit uncertainty instead of guessing; prefer one correct answer over a verbose survey.
+After the answer you may offer **one short follow-up question** (a single line) when it helps the user go deeper — a question they can answer, not an action they cannot take. Do not stack multiple offers. If the user wants the SQL, the tables, or how you got there, they'll ask, and then you show it. Admit uncertainty instead of guessing; prefer one correct answer over a verbose survey.
 
 (Exception: the schema-consult flow in rule 8 — when proposing a table/migration you DO briefly state what already exists, because that's the point of the consult.)
 
@@ -24,25 +24,35 @@ After the answer you may offer **one short follow-up question** (a single line)
 
 8. **Consult before you commit schema.** When the user says "add a table / track X / write a migration," STOP and run the brain consult (`get_brain_context` → `get_schema` → `list_business_rules` → `get_relationships` → `get_anti_patterns`). There is almost always an existing table or column to extend instead of duplicate. Narrate what you found before proposing DDL.
 
+9. **Never offer an action the caller cannot enforce.** `get_brain_context` and
+   `list_connections` carry `callerCapabilities`. If `doNotOffer` lists an
+   action — especially `save_brain_note` — do not mention it, do not ask
+   "should I save this", and do not render a Yes button. Answering a metric
+   is not a request to persist it. The product UI may show a non-blocking
+   save bubble after the user corrects or teaches a definition; leave that
+   to the UI. Never volunteer it yourself.
+
 ## Remembering things — two different places
 
-There are TWO planes of memory. Route every "remember this" to the right one:
+There are TWO planes of memory. Route a remember request only when the user
+explicitly asked to remember / pin / save a definition:
 
 1. **Company brain context (shared).** Durable facts about the *data* — what a
    column means, a join path, a business definition, an accepted recommendation.
    These ground EVERYONE's answers on this connection. Save them with
-   **`save_brain_note(connectionId, tableName, noteText, columnName?)`**.
-   - "Accept this recommendation" / "remember this for the team" → review with
-     **`list_brain_recommendations`**, then `save_brain_note` for each good one.
-   - This is **admin-only** (manage-content) and audited. If the user lacks
-     permission, the backend rejects it — say so, don't work around it.
+   **`save_brain_note(connectionId, tableName, noteText, columnName?)`**
+   **only if** `callerCapabilities.canWriteSharedBrainNotes` is true
+   (`list_connections.canManageContent`).
+   - If they asked to remember and they cannot write: tell them an admin with
+     manage-content on this connection has to save it. Do not call the tool.
 2. **Individual preference (yours alone).** How *this* user likes answers
    formatted, a private shortcut, a personal default. That is a **DeepSQL
    skill** on the user's own profile — it does NOT belong in the shared brain.
    Never push a personal preference into `save_brain_note`.
 
-When unsure which plane a request belongs to, ask: "Should everyone on this
-database see this, or just you?" Shared → brain note. Just you → DeepSQL skill.
+Do not volunteer a shared-brain save after answering a data question. Do not
+ask "should everyone on this database see this?" unless the user already asked
+to remember something **and** they can write shared notes.
 
 ## Skills
 

agent/skills/bi-query/SKILL.md

@@ -27,7 +27,7 @@ Use when the user asks a question whose answer is **in the data** ("how many boo
 
 6. **Run it** with `execute_sql(connectionId, sql, limit=…)`. Remember: default 100 rows, max 1000. For a total, `SELECT COUNT(*)` rather than counting a truncated result set.
 
-7. **Answer only.** Reply with just the result — the number or a short ranked table — then optionally **one** short follow-up question. Apply business rules silently; do NOT append "Grounding used" / "Filters applied" / "Used:" / tool-narration / column-mapping sections. Only if the user asks how you got it do you show the tables, joins, and filters.
+7. **Answer only.** Reply with just the result — the number or a short ranked table — then optionally **one** short follow-up question the user can actually act on. Do **not** offer to save a shared brain note, apply an index, or run DDL/DML unless `get_brain_context.callerCapabilities` says they can. Apply business rules silently; do NOT append "Grounding used" / "Filters applied" / "Used:" / tool-narration / column-mapping sections. Only if the user asks how you got it do you show the tables, joins, and filters.
 
 ## Guardrails
 

backend/src/main/java/com/dbaagent/controller/BrainController.java

@@ -28,6 +28,9 @@
 import com.dbaagent.service.brain.core.BrainNoteService;
 import com.dbaagent.service.brain.core.BrainTaskService;
 import com.dbaagent.service.brain.core.NoteSuggestionService;
+import com.dbaagent.service.brain.core.BrainNoteProposalService;
+import com.dbaagent.dto.BrainNoteProposalRequest;
+import com.dbaagent.dto.BrainNoteProposalResponse;
 import com.dbaagent.service.QueryExecutorService;
 import com.dbaagent.service.SchemaSnapshotService;
 import com.dbaagent.service.brain.analysis.ColumnDisambiguationService;
@@ -94,6 +97,7 @@ public class BrainController {
     private final BrainNoteService brainNoteService;
     private final BrainTaskService brainTaskService;
     private final NoteSuggestionService noteSuggestionService;
+    private final BrainNoteProposalService brainNoteProposalService;
     private final ColumnProfilingService columnProfilingService;
     private final ColumnDisambiguationService columnDisambiguationService;
     private final SchemaSnapshotService schemaSnapshotService;
@@ -240,6 +244,51 @@ public ResponseEntity<NoteSuggestionDTO.Response> getNoteSuggestions(
         }
     }
 
+    /**
+     * Draft a shared-brain note from an Agent turn. Overlaps with existing notes
+     * or business rules are merged into one intent. Read-only — does not persist.
+     */
+    @PostMapping("/notes/propose")
+    public ResponseEntity<BrainNoteProposalResponse> proposeNoteFromTurn(
+        @org.springframework.web.bind.annotation.RequestBody BrainNoteProposalRequest request
+    ) {
+        try {
+            if (request == null || request.getConnectionId() == null || request.getConnectionId().isBlank()) {
+                return ResponseEntity.badRequest().build();
+            }
+            accessControlService.assertCanReadConnectionContent(request.getConnectionId());
+            return brainNoteProposalService.proposeFromTurn(request)
+                .map(ResponseEntity::ok)
+                .orElseGet(() -> ResponseEntity.noContent().build());
+        } catch (ResponseStatusException e) {
+            throw e;
+        } catch (Exception e) {
+            log.error("Error proposing a brain note from an agent turn", e);
+            return ResponseEntity.internalServerError().build();
+        }
+    }
+
+    /**
+     * Accept a proposed note. If it overlaps existing documentation, update that
+     * row instead of creating a second copy of the same intent.
+     */
+    @PostMapping("/notes/accept")
+    public ResponseEntity<BrainNoteResponse> acceptNote(
+        @org.springframework.web.bind.annotation.RequestBody BrainNoteRequest request
+    ) {
+        try {
+            accessControlService.assertCanManageConnectionContent(request.getConnectionId());
+            return ResponseEntity.ok(brainNoteProposalService.accept(request));
+        } catch (IllegalArgumentException e) {
+            return ResponseEntity.badRequest().build();
+        } catch (ResponseStatusException e) {
+            throw e;
+        } catch (Exception e) {
+            log.error("Error accepting a brain note proposal", e);
+            return ResponseEntity.internalServerError().build();
+        }
+    }
+
     @GetMapping("/notes/{connectionId}")
     public ResponseEntity<List<BrainNoteResponse>> getNotes(
         @PathVariable String connectionId,

backend/src/main/java/com/dbaagent/dto/BrainNoteProposalRequest.java

@@ -0,0 +1,16 @@
+package com.dbaagent.dto;
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+public class BrainNoteProposalRequest {
+    private String connectionId;
+    private String question;
+    private String answer;
+    /** Previous assistant answer. Required for a proposal — clean first turns stay quiet. */
+    private String priorAnswer;
+}

backend/src/main/java/com/dbaagent/dto/BrainNoteProposalResponse.java

@@ -0,0 +1,24 @@
+package com.dbaagent.dto;
+
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@Builder
+@NoArgsConstructor
+@AllArgsConstructor
+public class BrainNoteProposalResponse {
+    private String scopeType;
+    private String tableName;
+    private String columnName;
+    private String bubbleLabel;
+    private String excerpt;
+    private String proposedNoteText;
+    /** NEW, MERGE, or SKIP (SKIP is omitted from the Agent UI). */
+    private String action;
+    private String existingNoteId;
+    private String existingNoteText;
+    private String overlapReason;
+}