From 32a077448a51a8b0910693d66da85995c3ad18a0 Mon Sep 17 00:00:00 2001 From: Nate Moore Date: Sat, 22 Aug 2026 23:35:14 -0400 Subject: [PATCH 1/3] fix: preserve grapheme clusters (combining marks) in cell emission MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Cell struct stored only a single uint32_t codepoint, so render_text silently dropped every combining mark (wcwidth ≤ 0): the emitted ANSI stream contained bare base codepoints with no continuations. Root cause: kitty-graphics Unicode placeholder cells require two combining diacritics (row/col index from the rowcolumn-diacritics table) to follow the base U+10EEEE codepoint. Because those marks were dropped, every placeholder cell emitted as row 0 — producing the N-repeated-top-band banding artifact in multi-row placements. The same drop affected any combining-mark or ZWJ content: accented characters (e + U+0301), flag pairs, skin-tone modifiers. Fix: - Cell gains `uint32_t combining[8]` (zero-terminated). Marks beyond 8 are silently truncated from the end; the first marks always survive. - cells_fill uses a designated-init template so combining[] is zeroed on every back-buffer reset, and setcell clears it on every base write. - render_text tracks the last-written column; zero-width codepoints go to append_combining() instead of being discarded. - present_cups / present_lines emit combining[] bytes immediately after the base character, before any cursor repositioning. - OUT_BYTES_PER_CELL bumped 64→128 to cover base + 8 combining marks. - cell_cmp checks combining[] so a changed mark triggers a diff/rediff. - Spec updated: §8.3.3 normative preservation requirement, §13 cluster cell representation with truncation semantics, §13 measurement note. Four new tests: kitty placeholder (U+10EEEE + 2 marks), combining accent (e + U+0301), ZWJ family emoji (ZWJ preserved per-cell; following emoji start new cells — inherent cell-model constraint, documented), and truncation-from-end pinned at 8 marks. --- specs/renderer-spec.md | 31 ++++++++++-- src/cell.c | 17 ++++--- src/cell.h | 9 +++- src/clayterm.c | 29 ++++++++++- test/term.test.ts | 108 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 180 insertions(+), 14 deletions(-) diff --git a/specs/renderer-spec.md b/specs/renderer-spec.md index 398e78a..b1e544c 100644 --- a/specs/renderer-spec.md +++ b/specs/renderer-spec.md @@ -448,6 +448,15 @@ NOT override the background already present in each glyph cell; element backgrounds established by `open({ bg })` remain in effect, and the terminal default remains in effect where no element background applies. +**Grapheme cluster preservation.** When `content` contains grapheme clusters — a +base codepoint followed by one or more combining marks (Unicode codepoints with +`wcwidth` ≤ 0) — the renderer MUST preserve and emit the full cluster. Combining +marks MUST NOT be silently dropped. Combining marks do not advance the cursor +position; they are attached to the preceding base codepoint's cell and emitted +immediately after the base codepoint's bytes in the output stream. A cell +occupied by a grapheme cluster with combining marks MUST be treated as changed +(and thus emitted) when any mark in the cluster changes between frames. + The set of styling properties accepted by `props` is part of the current implementation surface and may be extended. @@ -927,11 +936,23 @@ elements; clip regions; and scroll containers. **Text measurement.** Text width measurement uses `wcwidth`-based character width computation, supporting ASCII, CJK wide characters, and other Unicode -codepoints. - -**Cell representation.** Each cell in the buffer stores a Unicode codepoint, a -foreground color (packed ARGB with attribute flags in the high byte), and a -background color. +codepoints. Combining marks (codepoints with `wcwidth` ≤ 0) contribute zero to +measured width; they attach to the preceding base codepoint's cell and are not +counted as separate cells in layout. Measurement and rendering MUST agree: if +the measurer ignores a combining mark for width purposes, the renderer MUST +still attach and emit it. + +**Cell representation.** Each cell in the buffer stores a grapheme cluster — a +base Unicode codepoint plus up to 8 combining-mark codepoints — together with a +foreground color (packed ARGB with attribute flags in the high byte) and a +background color. The combining-mark slots are zero-terminated; a cell with no +combining marks stores zero in every slot. When a text string produces more than +8 combining marks for a single base codepoint, the excess marks are silently +truncated from the end (marks 1–8 are kept, marks 9+ are discarded), ensuring +that the first and most semantically significant marks always survive. Cell +comparison for diffing considers combining marks: a cell is considered changed +when any combining mark differs from the front buffer, not only when the base +codepoint or color attributes differ. **Border junction resolution.** When bordered elements share edges, the renderer accumulates per-cell direction bitmasks and resolves them to correct box-drawing diff --git a/src/cell.c b/src/cell.c index 476a33b..4a5e14f 100644 --- a/src/cell.c +++ b/src/cell.c @@ -4,13 +4,18 @@ void cells_fill(Cell *buf, int w, int h, uint32_t ch, uint32_t fg, uint32_t bg) { - for (int i = 0; i < w * h; i++) { - buf[i].ch = ch; - buf[i].fg = fg; - buf[i].bg = bg; - } + /* Designated init zeros unspecified fields (including combining[]). */ + Cell tmpl = {.ch = ch, .fg = fg, .bg = bg}; + for (int i = 0; i < w * h; i++) + buf[i] = tmpl; } int cell_cmp(Cell *a, Cell *b) { - return a->ch != b->ch || a->fg != b->fg || a->bg != b->bg; + if (a->ch != b->ch || a->fg != b->fg || a->bg != b->bg) + return 1; + for (int i = 0; i < CELL_MAX_COMBINING; i++) { + if (a->combining[i] != b->combining[i]) + return 1; + } + return 0; } diff --git a/src/cell.h b/src/cell.h index 5a1db90..e6d9aca 100644 --- a/src/cell.h +++ b/src/cell.h @@ -5,10 +5,15 @@ #include +/* Maximum combining marks stored per cell. Marks beyond this limit are + * silently truncated from the end; the first CELL_MAX_COMBINING are kept. */ +#define CELL_MAX_COMBINING 8 + typedef struct { uint32_t ch; - uint32_t fg; /* 0xAARRGGBB — upper byte: attribute flags */ - uint32_t bg; /* 0xAARRGGBB — upper byte: attribute flags */ + uint32_t fg; /* 0xAARRGGBB — upper byte: attribute flags */ + uint32_t bg; /* 0xAARRGGBB — upper byte: attribute flags */ + uint32_t combining[CELL_MAX_COMBINING]; /* zero-terminated combining-mark codepoints */ } Cell; /* Attribute flags (packed into high byte of fg) */ diff --git a/src/clayterm.c b/src/clayterm.c index 9464669..4a23273 100644 --- a/src/clayterm.c +++ b/src/clayterm.c @@ -92,7 +92,9 @@ struct Clayterm { * Output buffer is sized at 64 bytes per cell — enough for worst-case * full-screen redraws with truecolor SGR sequences on every cell. */ -#define OUT_BYTES_PER_CELL 64 +/* 128 bytes per cell: ~84 bytes worst-case for CUP + SGR sequences, plus up + * to 4 (base SMP char) + 8×4 (combining marks) = 36 bytes of cluster text. */ +#define OUT_BYTES_PER_CELL 128 /* ── Cell buffer ops ──────────────────────────────────────────────── */ @@ -116,6 +118,22 @@ static void setcell(struct Clayterm *ct, int x, int y, uint32_t ch, uint32_t fg, if (!(bg & ATTR_DEFAULT)) { c->bg = bg; } + for (int i = 0; i < CELL_MAX_COMBINING; i++) + c->combining[i] = 0; +} + +/* Append a combining-mark codepoint to the cell at (x, y) in the back buffer. + * Marks beyond CELL_MAX_COMBINING are silently dropped (truncation from end). */ +static void append_combining(struct Clayterm *ct, int x, int y, uint32_t cp) { + if (x < 0 || x >= ct->w || y < 0 || y >= ct->h) + return; + Cell *c = cell_at(ct, ct->back, x, y); + for (int i = 0; i < CELL_MAX_COMBINING; i++) { + if (c->combining[i] == 0) { + c->combining[i] = cp; + return; + } + } } /* ── Escape sequence generation ───────────────────────────────────── */ @@ -221,6 +239,8 @@ static void present_cups(struct Clayterm *ct, int row) { emit_ch(ct, i, y, row, ' '); } else { emit_ch(ct, x, y, row, back->ch); + for (int ci = 0; ci < CELL_MAX_COMBINING && back->combining[ci]; ci++) + buf_char(&ct->out, back->combining[ci]); /* mark trailing cells of wide char as invalid in front * so they'll diff when overwritten by narrow chars */ for (int i = 1; i < w; i++) { @@ -268,6 +288,8 @@ static void present_lines(struct Clayterm *ct) { if (!iswprint(ch)) ch = 0xfffd; buf_char(&ct->out, ch); + for (int ci = 0; ci < CELL_MAX_COMBINING && back->combining[ci]; ci++) + buf_char(&ct->out, back->combining[ci]); for (int i = 1; i < w; i++) { Cell *fw = cell_at(ct, ct->front, x + i, y); fw->ch = 0xffffffff; @@ -313,6 +335,7 @@ static void render_text(struct Clayterm *ct, int x0, int y0, const char *p = t->stringContents.chars; int rem = t->stringContents.length; int x = x0; + int last_x = -1; /* column of the most-recently written base cell */ while (rem > 0) { uint32_t cp; @@ -326,7 +349,11 @@ static void render_text(struct Clayterm *ct, int x0, int y0, cw = 1; if (cw > 0) { setcell(ct, x, y0, cp, fg, bg); + last_x = x; x += cw; + } else if (last_x >= 0) { + /* combining mark: attach to the preceding base cell */ + append_combining(ct, last_x, y0, cp); } p += n; rem -= n; diff --git a/test/term.test.ts b/test/term.test.ts index 121ece2..07f4294 100644 --- a/test/term.test.ts +++ b/test/term.test.ts @@ -337,6 +337,114 @@ describe("term", () => { }); }); + describe("grapheme clusters", () => { + /* Helper: count occurrences of a substring in a string */ + function countOf(haystack: string, needle: string): number { + let n = 0; + let pos = 0; + while ((pos = haystack.indexOf(needle, pos)) !== -1) { + n++; + pos += needle.length; + } + return n; + } + + it("preserves kitty-graphics placeholder cluster (base + 2 combining marks)", async () => { + /* U+10EEEE = kitty placeholder; U+0305 = row-index diacritic; + * U+030D = col-index diacritic. Both marks must appear in the emitted + * bytes immediately after the base codepoint. */ + let term2 = await createTerm({ width: 40, height: 4 }); + let out = decode( + term2 + .render( + [ + open("root", { + layout: { width: grow(), height: grow(), direction: "ttb" }, + }), + text("\u{10EEEE}\u{0305}\u{030D}"), + close(), + ], + { mode: "line" }, + ) + .output, + ); + + expect(out).toContain("\u{10EEEE}\u{0305}\u{030D}"); + }); + + it("preserves combining accent (e + U+0301)", async () => { + let term2 = await createTerm({ width: 40, height: 4 }); + let out = decode( + term2 + .render( + [ + open("root", { + layout: { width: grow(), height: grow(), direction: "ttb" }, + }), + text("e\u{0301}"), + close(), + ], + { mode: "line" }, + ) + .output, + ); + + /* The combining mark must follow the base immediately in the output. */ + expect(out).toContain("e\u{0301}"); + }); + + it("preserves ZWJ in output (ZWJ is per-cell combining; following emoji start new cells)", async () => { + /* 👨‍👩‍👧‍👦 = 👨 ZWJ 👩 ZWJ 👧 ZWJ 👦. + * ZWJ (U+200D, wcwidth 0) attaches to the preceding base emoji cell. + * The following emoji have positive wcwidth and start new cells, so the + * family sequence is split across cells in the cell-based model. The + * invariant tested here: ZWJ bytes MUST NOT be dropped from the output. */ + let term2 = await createTerm({ width: 40, height: 4 }); + let out = decode( + term2 + .render( + [ + open("root", { + layout: { width: grow(), height: grow(), direction: "ttb" }, + }), + text("👨\u{200D}👩\u{200D}👧\u{200D}👦"), + close(), + ], + { mode: "line" }, + ) + .output, + ); + + expect(countOf(out, "\u{200D}")).toBe(3); + expect(out).toContain("👨"); + expect(out).toContain("👩"); + }); + + it("truncates excess combining marks from the end (first 8 survive)", async () => { + /* A base char followed by 9 combining graves (U+0300). + * The 9th mark exceeds CELL_MAX_COMBINING=8 and is silently dropped; + * the first 8 must appear in the output. */ + let term2 = await createTerm({ width: 40, height: 4 }); + let grave = "\u{0300}"; + let out = decode( + term2 + .render( + [ + open("root", { + layout: { width: grow(), height: grow(), direction: "ttb" }, + }), + text("a" + grave.repeat(9)), + close(), + ], + { mode: "line" }, + ) + .output, + ); + + expect(countOf(out, grave)).toBe(8); + }); + }); + describe("row offset", () => { it("renders two frames at the offset position", async () => { let term = await createTerm({ width: 20, height: 5 }); From 902fabe8c2e1a629b3b10f4db4a5c08535b7bd53 Mon Sep 17 00:00:00 2001 From: Nate Moore Date: Sun, 23 Aug 2026 07:53:13 -0400 Subject: [PATCH 2/3] style: deno fmt --- test/term.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/term.test.ts b/test/term.test.ts index 4cde09e..5a0858f 100644 --- a/test/term.test.ts +++ b/test/term.test.ts @@ -445,7 +445,7 @@ describe("term", () => { expect(countOf(out, grave)).toBe(8); }); }); - + describe("caret placement", () => { // These tests use `print()`, which marks the terminal's final // cursor position by appending U+0332 COMBINING LOW LINE to that From 07a84e91282846900cc2be167484f7b143cbb6a3 Mon Sep 17 00:00:00 2001 From: Nate Moore Date: Sun, 23 Aug 2026 07:58:39 -0400 Subject: [PATCH 3/3] style: clang-format --- src/cell.h | 7 ++++--- src/clayterm.c | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/cell.h b/src/cell.h index e6d9aca..1755b2c 100644 --- a/src/cell.h +++ b/src/cell.h @@ -11,9 +11,10 @@ typedef struct { uint32_t ch; - uint32_t fg; /* 0xAARRGGBB — upper byte: attribute flags */ - uint32_t bg; /* 0xAARRGGBB — upper byte: attribute flags */ - uint32_t combining[CELL_MAX_COMBINING]; /* zero-terminated combining-mark codepoints */ + uint32_t fg; /* 0xAARRGGBB — upper byte: attribute flags */ + uint32_t bg; /* 0xAARRGGBB — upper byte: attribute flags */ + uint32_t combining[CELL_MAX_COMBINING]; /* zero-terminated combining-mark + codepoints */ } Cell; /* Attribute flags (packed into high byte of fg) */ diff --git a/src/clayterm.c b/src/clayterm.c index fdf5980..ae86548 100644 --- a/src/clayterm.c +++ b/src/clayterm.c @@ -140,7 +140,8 @@ static void setcell(struct Clayterm *ct, int x, int y, uint32_t ch, uint32_t fg, } /* Append a combining-mark codepoint to the cell at (x, y) in the back buffer. - * Marks beyond CELL_MAX_COMBINING are silently dropped (truncation from end). */ + * Marks beyond CELL_MAX_COMBINING are silently dropped (truncation from end). + */ static void append_combining(struct Clayterm *ct, int x, int y, uint32_t cp) { if (x < 0 || x >= ct->w || y < 0 || y >= ct->h) return;