fix(server): a ledger of another version says where it is, what to do and where to ask

packages/ledger/README.md

@@ -267,7 +267,7 @@ The ledger keeps content whole beside its messages rather than in them, since ev
 
 `content.stage(playbook, run, content)`, `content.staged(playbook, run, sha256)` and `content.unstage(playbook, run)` keep, read and remove content staged under a run, by its id, with the rules of a put and a get for the uploads, the heads and the digest, so a server reads what another staged on the same database. `unstage` records the run in `unstaged_runs` and then, in one transaction, removes the run's heads and then its chunks. The record stays, so `unstaged_runs` grows by one small row, the playbook key and the run id, for each run whose staging ended, as many as the runs that stage content; no capability of this repository stages content yet. A stage under a run already unstaged is refused with `StagingEnded` and leaves its stream unread, and a stage looks again once its head is written; if the run was unstaged meanwhile it deletes its own head and chunks and is refused the same way. Since the unstage records the run before it deletes and the stage writes its head before it looks, either the stage sees the record or the unstage's deletes see the head, so a stage that races an unstage leaves nothing behind. A read of a staged content whose run is unstaged while it is read fails at the first chunk it finds gone, as a defect, in words that say so: `The content staged under <digest> has no chunk <n>, as when its run is unstaged while it is read`.
 
-A database whose content tables have another shape than these, a chunk of text or a head without a media type, is refused when the ledger opens, with words that name the columns found and the columns kept and say to delete the local database, since nothing stored before launch is carried over.
+A database whose content tables have another shape than these, a chunk of text or a head without a media type, is refused when the ledger opens, as a `LedgerShapeRefused` of `@on.auto/ledger` that names the table, what it holds, the columns found and the columns kept, in words that say to delete the local database, since nothing stored before launch is carried over. The server, which knows where the ledger is kept, answers it with words of its own: where the ledger is, what to do, and where to ask.
 
 `measure/recorded-content.ts` runs `measuredContent` of `measure/content-measurement.ts` on each store. It puts the answers the tests keep, `answerOfAbout` of `@on.auto/operations/testing`, of 100 bytes to 20 MB of JSON, three times each, the second time as it is and the third with its digest expected, and reads each back and parses it; it puts the 64 MiB of bytes the tests generate, `generatedStream`, as a stream the same three ways and streams them back, their digest checked. The content is made before each put is timed, and SQLite is opened as the server opens it, with a page cache of 8 MiB and no memory map. It then puts and gets the 64 MiB once more for another playbook, made as the stream is pulled, and samples two figures of the process every 5 ms: the heap in use and the array buffers after a full collection, `heapUsed + arrayBuffers` of `process.memoryUsage()` after `gc()`, and the resident set, `rss`, each reported as the most it rose above its figure before the put or the get. The routine takes any `RecordedContent`, and is the subpath `@on.auto/ledger/content-measurement`, so the object store of `@on.auto/object-storage` is measured by the same steps, as its README records:
 
@@ -312,7 +312,7 @@ The ledger keeps the rows of the secret store of [`@on.auto/secrets`](../secrets
 | `wrapped_key`   | the value's data key as the custodian wrapped it, or `NULL` for a keeper that wraps none |
 | `kid`           | the kid of the custodian's key that wrapped it, or `NULL` for a keeper that wraps none   |
 
-A database whose table of secrets has another shape than this is refused when the ledger opens, in the words that refuse an older shape of the content tables, `requireTheShapeKept` of `src/content/content-tables.ts`, naming the columns found and the columns kept and saying to delete the local database.
+A database whose table of secrets has another shape than this is refused when the ledger opens, as an older shape of the content tables is, by `requireTheShapeKept` of `src/content/content-tables.ts`, with the same `LedgerShapeRefused`.
 
 Each operation of the port is one statement, every value bound: `keep` an `INSERT`; `read` a `SELECT` that answers `claimed_until` only while the claim is open; `replace` and `claim` one conditional `UPDATE` each with `RETURNING`, under the version read, the claim's also under no claim being open; `endingsOf` one `SELECT` of the ids named, and nothing asked for no ids; `forget` a `DELETE` that returns the holding it deleted; `heldBy` a `SELECT` of a keeper's rows, under a kid where one is named and after an id where one is named, `ORDER BY id` with the `LIMIT` asked; and `reheld` a conditional `UPDATE` of the four columns of the holding, only while the row's version, `kept_by` and `held` are the ones read and its `wrapped_key` and `kid` too, compared through `COALESCE(…, '')` so that a holding that has none compares alike on both stores. Every time the rows write or compare, `kept_at`, the claim's end and the moment a claim is judged open, is read from Effect's `Clock`, the clock the times of the facts are read from, so a test drives it with `TestClock`. A failure of the database is a defect, as it is for the ledger.
 

packages/ledger/src/content/content-tables.ts

@@ -5,6 +5,7 @@ import type { StatementExecutor } from '../event-store.ts';
 import { inTurn } from '../projections/inline-projection.ts';
 import { createMissingIndexes, type PlaybookIndex } from '../recorded/missing-indexes.ts';
 import type { ContentDialect } from './content-dialects.ts';
+import { ledgerShapeRefused } from './ledger-shape-refused.ts';
 
 export interface Shelf {
   readonly chunks: string;
@@ -115,10 +116,6 @@ export async function namesFound(execute: StatementExecutor, sql: SQL): Promise<
   return Schema.decodeUnknownSync(NameRows)(rows).map(({ name }) => name);
 }
 
-function listed(columns: readonly string[]): string {
-  return columns.toSorted().join(', ');
-}
-
 export interface KeptTable {
   readonly name: string;
   readonly columns: readonly string[];
@@ -130,11 +127,10 @@ export async function requireTheShapeKept(
   dialect: ContentDialect,
   { name, columns, holds }: KeptTable,
 ): Promise<void> {
-  const found = await namesFound(execute, dialect.columnsOf(name));
-  if (listed(found) !== listed(columns)) {
-    throw new Error(
-      `The ledger's database keeps ${holds} in a shape this server no longer reads: ${name} has the columns ${listed(found)}, where this server keeps ${listed(columns)}. Nothing stored before launch is carried over, so delete the local database and start the server again`,
-    );
+  const found = (await namesFound(execute, dialect.columnsOf(name))).toSorted();
+  const kept = columns.toSorted();
+  if (found.join() !== kept.join()) {
+    throw ledgerShapeRefused({ table: name, holds, found, kept });
   }
 }
 

packages/ledger/src/content/ledger-shape-refused.ts

@@ -0,0 +1,22 @@
+import { Data } from 'effect';
+
+export interface RefusedShape {
+  readonly table: string;
+  readonly holds: string;
+  readonly found: readonly string[];
+  readonly kept: readonly string[];
+}
+
+export class LedgerShapeRefused extends Data.TaggedError('ledger_shape_refused')<
+  RefusedShape & { readonly message: string }
+> {}
+
+export function ledgerShapeRefused({ table, holds, found, kept }: RefusedShape): LedgerShapeRefused {
+  return new LedgerShapeRefused({
+    table,
+    holds,
+    found,
+    kept,
+    message: `The ledger's database keeps ${holds} in a shape this server no longer reads: ${table} has the columns ${found.join(', ')}, where this server keeps ${kept.join(', ')}. Nothing stored before launch is carried over, so delete the local database and start the server again`,
+  });
+}

packages/ledger/src/index.ts

@@ -20,6 +20,7 @@ export type {
   StoredPlace,
   StoredRecord,
 } from './event-store.ts';
+export { LedgerShapeRefused, type RefusedShape } from './content/ledger-shape-refused.ts';
 export { cursorOf as recordedCursorOf } from './recorded/cursor.ts';
 export {
   decisionLoop,

packages/ledger/src/secrets/kept-secrets-behaviour.ts

@@ -3,6 +3,7 @@ import { rowsBehaviour, storeBehaviour, type SecretsEntry } from '@on.auto/secre
 import type { Effect } from 'effect';
 import { describe, expect, it } from 'vitest';
 
+import { LedgerShapeRefused } from '../content/ledger-shape-refused.ts';
 import { aLedger, type LedgerEntry } from '../testing/ledger-entry.ts';
 
 export interface KeptSecretsEntry extends LedgerEntry {
@@ -25,16 +26,24 @@ function secretsEntryOf(entry: KeptSecretsEntry): SecretsEntry {
 }
 
 function olderShapeBehaviour(entry: KeptSecretsEntry): void {
-  it('refuses a database whose table of secrets has an older shape when the ledger opens, with words that say to delete it', async () => {
+  it('refuses a database whose table of secrets has an older shape when the ledger opens, naming the columns found and kept, with words that say to delete it', async () => {
     const database = await entry.aDatabase();
     await entry.queried(
       database,
       'CREATE TABLE kept_secrets (id TEXT PRIMARY KEY, version INTEGER NOT NULL, kept_at TEXT NOT NULL, kept_by TEXT NOT NULL, held TEXT NOT NULL)',
     );
 
-    await expect(aLedger(entry, database)).rejects.toThrow(
-      "The ledger's database keeps secrets in a shape this server no longer reads: kept_secrets has the columns held, id, kept_at, kept_by, version, where this server keeps claimed_until, ends_with, held, id, kept_at, kept_by, kid, version, wrapped_key. Nothing stored before launch is carried over, so delete the local database and start the server again",
-    );
+    const refused = await aLedger(entry, database).catch((error: unknown) => error);
+
+    expect(refused).toBeInstanceOf(LedgerShapeRefused);
+    expect(refused).toMatchObject({
+      table: 'kept_secrets',
+      holds: 'secrets',
+      found: ['held', 'id', 'kept_at', 'kept_by', 'version'],
+      kept: ['claimed_until', 'ends_with', 'held', 'id', 'kept_at', 'kept_by', 'kid', 'version', 'wrapped_key'],
+      message:
+        "The ledger's database keeps secrets in a shape this server no longer reads: kept_secrets has the columns held, id, kept_at, kept_by, version, where this server keeps claimed_until, ends_with, held, id, kept_at, kept_by, kid, version, wrapped_key. Nothing stored before launch is carried over, so delete the local database and start the server again",
+    });
   });
 }
 

packages/server/src/command-line/failure-notice.test.ts

@@ -0,0 +1,119 @@
+import { stripVTControlCharacters } from 'node:util';
+
+import { Redacted } from 'effect';
+import { describe, expect, it } from 'vitest';
+
+import { ledgerOfAnotherVersion } from '../database/ledger-of-another-version.ts';
+import type { ColorStream } from '../logging/terminal.ts';
+import { startupFailureNoticeFor } from './failure-notice.ts';
+
+const secretsOfAnOlderShape = {
+  table: 'kept_secrets',
+  holds: 'secrets',
+  found: ['held', 'id', 'kept_at', 'kept_by', 'version'],
+  kept: ['claimed_until', 'ends_with', 'held', 'id', 'kept_at', 'kept_by', 'kid', 'version', 'wrapped_key'],
+};
+
+const ledgerFileOfAnotherVersion = ledgerOfAnotherVersion(secretsOfAnOlderShape, {
+  ledger: { store: 'sqlite', file: '/Users/ada/.auto-runtime/ledger.db' },
+  distribution: 'npx',
+});
+
+const servicesFailed = new Error("The server's services could not start", { cause: ledgerFileOfAnotherVersion });
+
+const notATerminal: ColorStream = {};
+
+const aTerminalWithColors: ColorStream = { isTTY: true, getColorDepth: () => 24 };
+
+describe('the notice that a server could not start on a ledger file of another version', () => {
+  it('says where the ledger is and why it is refused, what to do with the command that does it, where to ask, and the columns last', () => {
+    expect(startupFailureNoticeFor(notATerminal)(servicesFailed)?.split('\n')).toEqual([
+      '  ✖ Auto Runtime could not start',
+      '',
+      '  The ledger is kept in the file /Users/ada/.auto-runtime/ledger.db, and was',
+      '  made by another version of Auto Runtime. This version keeps secrets in a',
+      '  different shape and cannot read the secrets this ledger holds.',
+      '',
+      '  What you can do',
+      '    Delete the ledger and start Auto Runtime again. Everything in the ledger',
+      '    goes with it, every playbook, definition, run and connection among it.',
+      '',
+      '      rm -f /Users/ada/.auto-runtime/ledger.db /Users/ada/.auto-runtime/ledger.db-wal /Users/ada/.auto-runtime/ledger.db-shm',
+      '',
+      '  Need a hand? Ask us on Discord: https://discord.gg/B8BKcKMRm8',
+      '',
+      '  kept_secrets has the columns held, id, kept_at, kept_by, version, where',
+      '  this version keeps claimed_until, ends_with, held, id, kept_at, kept_by,',
+      '  kid, version, wrapped_key.',
+    ]);
+  });
+
+  it('marks the failure in red, sets the command apart in cyan, links Discord and dims the columns where stderr is a terminal with colors', () => {
+    const notice = String(startupFailureNoticeFor(aTerminalWithColors)(ledgerFileOfAnotherVersion));
+
+    expect(notice).toContain('\u001B[31m✖\u001B[39m');
+    expect(notice).toContain(
+      '\u001B[36mrm -f /Users/ada/.auto-runtime/ledger.db /Users/ada/.auto-runtime/ledger.db-wal /Users/ada/.auto-runtime/ledger.db-shm\u001B[39m',
+    );
+    expect(notice).toContain('\u001B]8;;https://discord.gg/B8BKcKMRm8\u001B\\');
+    expect(notice).toContain('\u001B[2m  kid, version, wrapped_key.\u001B[22m');
+    expect(stripVTControlCharacters(notice)).toBe(startupFailureNoticeFor(notATerminal)(servicesFailed));
+  });
+
+  it('leaves any other failure to the line that names it', () => {
+    expect([
+      startupFailureNoticeFor(aTerminalWithColors)(new Error('listen EADDRINUSE: address already in use :::5555')),
+      startupFailureNoticeFor(aTerminalWithColors)('a failure that is no error'),
+    ]).toEqual([undefined, undefined]);
+  });
+});
+
+describe('the notice that a server in the image could not start on a ledger file of another version', () => {
+  it('says which files to delete from the volume, or to remove the volume, with no command', () => {
+    const inTheImage = ledgerOfAnotherVersion(secretsOfAnOlderShape, {
+      ledger: { store: 'sqlite', file: '/app/data/ledger.db' },
+      distribution: 'docker',
+    });
+
+    expect(startupFailureNoticeFor(notATerminal)(inTheImage)?.split('\n').slice(6, 13)).toEqual([
+      '  What you can do',
+      '    Delete ledger.db, ledger.db-wal and ledger.db-shm from the volume',
+      "    mounted at /data, the ledger's folder /app/data in the image, or remove",
+      '    the volume, and start Auto Runtime again. Everything in the ledger goes',
+      '    with it, every playbook, definition, run and connection among it.',
+      '',
+      '  Need a hand? Ask us on Discord: https://discord.gg/B8BKcKMRm8',
+    ]);
+  });
+});
+
+describe('the notice that a server could not start on a ledger in PostgreSQL of another version', () => {
+  it('says to point DATABASE_URL at an empty database, with no command', () => {
+    const inPostgreSQL = ledgerOfAnotherVersion(secretsOfAnOlderShape, {
+      ledger: {
+        store: 'postgresql',
+        url: Redacted.make('postgresql://playbooks:a-secret-password@db.example.com/auto_runtime'),
+        host: 'db.example.com:5432',
+        database: 'auto_runtime',
+        credentials: { kind: 'url' },
+      },
+      distribution: 'npx',
+    });
+
+    expect(startupFailureNoticeFor(notATerminal)(inPostgreSQL)?.split('\n').slice(2, 15)).toEqual([
+      '  The ledger is kept in PostgreSQL, in the database auto_runtime on',
+      '  db.example.com:5432, and was made by another version of Auto Runtime. This',
+      '  version keeps secrets in a different shape and cannot read the secrets',
+      '  this ledger holds.',
+      '',
+      '  What you can do',
+      '    Point DATABASE_URL at an empty database, or empty this one, and start',
+      '    Auto Runtime again. The ledger then starts with no playbooks,',
+      '    definitions or runs.',
+      '',
+      '  Need a hand? Ask us on Discord: https://discord.gg/B8BKcKMRm8',
+      '',
+      '  kept_secrets has the columns held, id, kept_at, kept_by, version, where',
+    ]);
+  });
+});

packages/server/src/command-line/failure-notice.ts

@@ -0,0 +1,56 @@
+import {
+  columnsCompared,
+  helpOfferedAt,
+  ledgerOfAnotherVersionIn,
+  type LedgerOfAnotherVersion,
+} from '../database/ledger-of-another-version.ts';
+import { colorDepthOf, linkerFor, painterFor, type ColorDepth, type ColorStream } from '../logging/terminal.ts';
+import { discordInvite } from './local-banner.ts';
+
+const noticeWidth = 76;
+
+function wrapped(text: string, indent: string): readonly string[] {
+  const width = noticeWidth - indent.length;
+  const lines: string[] = [];
+  for (const word of text.split(' ')) {
+    const last = lines.pop();
+    if (last === undefined) {
+      lines.push(word);
+    } else if (last.length + 1 + word.length <= width) {
+      lines.push(`${last} ${word}`);
+    } else {
+      lines.push(last, word);
+    }
+  }
+  return lines.map((line) => `${indent}${line}`);
+}
+
+type Refusal = Pick<LedgerOfAnotherVersion, 'happened' | 'remedy' | 'table' | 'holds' | 'found' | 'kept'>;
+
+function ledgerNotice(refusal: Refusal, depth: ColorDepth): string {
+  const paint = painterFor(depth);
+  const link = linkerFor(depth);
+  const { says, command } = refusal.remedy;
+  const shown = command === undefined ? [] : ['', `      ${paint('cyan', command)}`];
+  return [
+    `  ${paint('red', '✖')} ${paint('bold', 'Auto Runtime could not start')}`,
+    '',
+    ...wrapped(refusal.happened, '  '),
+    '',
+    `  ${paint('bold', 'What you can do')}`,
+    ...wrapped(says, '    '),
+    ...shown,
+    '',
+    `  ${helpOfferedAt(paint('cyan', link(discordInvite, discordInvite)))}`,
+    '',
+    ...wrapped(columnsCompared(refusal), '  ').map((line) => paint('dim', line)),
+  ].join('\n');
+}
+
+export function startupFailureNoticeFor(stderr: ColorStream): (failure: unknown) => string | undefined {
+  const depth = colorDepthOf(stderr);
+  return (failure) => {
+    const refusal = ledgerOfAnotherVersionIn(failure);
+    return refusal === undefined ? undefined : ledgerNotice(refusal, depth);
+  };
+}

packages/server/src/command-line/local-banner.ts

@@ -20,6 +20,8 @@ const tagline = 'Runs the playbooks your agent builds';
 
 const documentation = 'https://on.auto/docs/';
 
+export const discordInvite = 'https://discord.gg/B8BKcKMRm8';
+
 function agentCommandsFor({ version, port, mode }: Ready, mcp: string): readonly (readonly [string, string])[] {
   if (mode !== 'local') {
     return [

packages/server/src/composition/composition-root.ts

@@ -1,7 +1,7 @@
 import type { DispatcherServices } from '@on.auto/operations';
 
 import { storesAndConnectionsOf } from '../connections/served-connections.ts';
-import { ledgerDatabaseOf } from '../database/ledger-database.ts';
+import { ledgerDatabaseOf, withARefusedShapeExplained } from '../database/ledger-database.ts';
 import { defaultServerOptions, type ServerOptions } from '../lifecycle/lifecycle.ts';
 import { serveWorkflows, type WorkflowParts } from '../workflows/workflows.ts';
 import { applicationLayer } from './application-layer.ts';
@@ -22,7 +22,9 @@ export function compositionRootWith(
     composed: (settings, telemetry) => {
       const database = ledgerDatabaseOf(settings, telemetry.appended);
       return {
-        services: applicationLayer(ledgerLayerFor(settings, database, wiring.projections.appends)),
+        services: applicationLayer(
+          withARefusedShapeExplained(ledgerLayerFor(settings, database, wiring.projections.appends), settings),
+        ),
         serve: async (runtime, ownOrigin) => {
           const { ledger, workflows, identityIssuer } = settings;
           const { stores, retryStateKeys, consent, connections, source, offering, clientMetadata } =