fix: guard empty ObjectScript scanner state

internal/cbm/vendored/grammars/MANIFEST.md

@@ -50,7 +50,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
   newlines; the new files keep upstream's bytes exactly. Per-file SHA-256
   values are pinned in `scripts/vendored-checksums.txt`. Regression coverage:
   the `cs_*_issue1748` tests in `tests/test_parse_coverage.c`.
-- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c`/`scanner.c` are otherwise byte-for-byte upstream — on re-vendor, re-apply only that single include rename. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
+- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c` files remain byte-for-byte upstream. Each `scanner.c` also guards empty-state deserialization with `if (length > 0)` before `memcpy`, because Tree-sitter can pass a NULL buffer with zero length. On re-vendor, re-apply the include rename and this guard unless upstream already carries them. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
 - **mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored.
 - **arkts** (added 2026-08-26): **first-party derivative** — a fork of [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `typescript` dialect pinned @ `75b3874edb2d` (v0.23.2, the same commit our vendored typescript/tsx come from), on the tree-sitter-javascript base @ `3a837b6f3658` (v0.23.1, the version upstream's own package-lock pins), extended with ArkTS/ArkUI syntax (`@Component struct` declarations, UI-DSL trailing-closure calls with post-block attribute chains, decorated function declarations, `import lazy`, `@Extend`/`@Styles` leading-dot attribute chains, anonymous `stateStyles` style blocks). Grammar source + corpus tests live in `tools/tree-sitter-arkts/`; regenerate with `npx tree-sitter-cli@0.25.10 generate` (ABI 15). **External scanner:** `scanner.c` is a symbol-rename trampoline; `_common_scanner.h` is byte-identical (same SHA-256) to the reviewed `typescript/_common_scanner.h` already shipped. **LICENSE:** tree-sitter-typescript's MIT text **verbatim and byte-identical** (© 2017 Max Brunsfeld) — which is what MIT requires of a derivative work, and what lets the provenance audit byte-verify it against upstream rather than take a note on trust. The fork is registered in the audit's `FORKS` map. Our own copyright for the ArkTS additions, and the tree-sitter-javascript attribution (© 2014 Max Brunsfeld), live with the SOURCE in `tools/tree-sitter-arkts/grammar.js` and in `THIRD_PARTY.md`, which is what ships in the release archives.
 - **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&amp;`/`&#38;` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx tree-sitter-cli@0.25.10 generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
@@ -109,6 +109,8 @@ row instead.
 | crystal    | `crystal/scanner.c`, serialize    | guard `memcpy(&buffer[offset], state->literals.contents, literal_content_size)` with `if (literal_content_size > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source on the empty-state serialize round-trip (formal UB, harmless) |
 | rescript   | `rescript/scanner.c`, deserialize | guard `memcpy(state, buffer, n_bytes)` with `if (n_bytes > 0)` | UBSan: zero-length `memcpy` with a NULL `buffer` / `n_bytes == 0` on empty-state deserialize (formal UB, harmless). The sibling serialize copies a fixed `sizeof(ScannerState)` (always > 0, non-NULL src) and needs no guard. |
 | rescript   | `rescript/scanner.c`, scan, `TEMPLATE_CHARS` branch | `if (valid_symbols[TEMPLATE_CHARS])` → `if (valid_symbols[TEMPLATE_CHARS] && !valid_symbols[NEWLINE])` | **Quadratic lexing in error recovery (#2176).** The only parse state whose external lex state enables both tokens is tree-sitter's ERROR state (`ts_lex_modes[0]`, external state 1; the in-template state 7 has no `_newline`), where every external token is marked valid and the lexer retries at each byte. From every byte the template loop ran to the next `` ` ``, `$`, `\` or NUL — the end of file when there is none — and returned false, so an unparseable stretch cost O(n²) inside lexing, where the parse budget's progress callback never runs. 50 KB of `0xFF` took 5.3 s, 100 KB 30 s; after: 3 ms, linear (lexer bytes pulled for 4 KB → 8 KB of `0xFF`: 8.8 M → 35.3 M before, 8.7 K → 17.4 K after, `complexity_rescript_error_recovery_lexing_is_linear`). Valid files parse to byte-identical trees; broken files recover more (a `let` after an unterminated expression is kept instead of swallowed). Upstream pin `43c2f1f35024` carries the unguarded branch |
+| objectscript_routine | `objectscript_routine/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
+| objectscript_udl | `objectscript_udl/scanner.c`, deserialize | guard `memcpy(payload, buffer, length)` with `if (length > 0)` | UBSan: Tree-sitter passes a NULL buffer and zero length for empty scanner state; skip the zero-length copy |
 | purescript | `purescript/scanner.c`, serialize | guard `memcpy(buffer, indents->data, to_copy)` with `if (to_copy > 0)` | UBSan: zero-length `memcpy` with a NULL/0-size source when the indent vector is empty (formal UB, harmless) |
 | plsql      | `plsql/parser.c`, include         | `#include <tree_sitter/parser.h>` → `#include "tree_sitter/parser.h"` | The older ABI-14 generator emits angle brackets; every other vendored grammar uses the quoted form, which resolves the per-grammar `tree_sitter/` header from the including file's directory |
 | properties | `properties/scanner.c`, whole file | move the `reached_eof` flag out of a file-scope `static bool` and into a per-parser payload allocated by `..._external_scanner_create()` | **Data race, and the only one of its kind in 103 vendored scanners.** That flag is how the scanner refuses to emit a second end-of-input `FAKE_EOL`, and upstream keeps it in ONE process-wide object. With several worker threads indexing `.properties` files at once, whichever reached EOF first set the flag, so another thread's `!reached_eof` was false and its parse never received the `FAKE_EOL` it needed to finish — it sat at end-of-input asking for a token that would never arrive. Measured 2026-09-19: ~106 M parse operations on a 253-byte file before the per-file budget stopped it, after which the file was dropped from the graph and java/kotlin indexes differed run to run. Interleaved A/B under identical load, 40 runs each: unpatched 6 stalls, patched 0. The serialize/deserialize wire format is unchanged (the state is still the returned length). Upstream `6310671b24d4` still carries the static, so a re-vendor must re-apply this |

scripts/vendored-checksums.txt

@@ -5,7 +5,7 @@ c5cfb43042b6b72045f4ba997834d0a7786d2793d91680868b5815b39f14fc78  internal/cbm/v
 b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea  internal/cbm/vendored/common/tree_sitter/alloc.h
 31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c  internal/cbm/vendored/common/tree_sitter/array.h
 180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2  internal/cbm/vendored/common/tree_sitter/parser.h
-0cd77186d628637e5559fbf2b9dc678a293a69a68ae23778ba21ad42dbf2db8e  internal/cbm/vendored/grammars/MANIFEST.md
+38172147795b4729640a3d9ef61ac94719e68ab5e3f9aaca792aa1c8c36d2940  internal/cbm/vendored/grammars/MANIFEST.md
 ad8425038de519f8c4e3e9339feebf99dfad8a6002dcf79227d91402780a32cc  internal/cbm/vendored/grammars/ada/LICENSE
 02805ec13939b749c891567be36bf024b09034e04c80683a1ae667272458d549  internal/cbm/vendored/grammars/ada/parser.c
 115a75d000bef9c70c4de7dfbb7f2a80b90cbb14ba265056dc7abb1ef3b9b6db  internal/cbm/vendored/grammars/ada/tree_sitter/alloc.h
@@ -525,14 +525,14 @@ c5fa4d1d5820cc59f39cb3cb9a9a171e5144b2f63d285cf3450be0a8e549ede8  internal/cbm/v
 85b458dc150c4d17873dd5e73d01cfa23c797997a09d841884e08378242e929c  internal/cbm/vendored/grammars/objectscript_routine/LICENSE
 f3a047fba6d238da258c0370878e703504b347aad13add09a021935daa891bd3  internal/cbm/vendored/grammars/objectscript_routine/objectscript_common.h
 fb9320a77f7c3c110540e9a569c2a8dd7ff74784ae05aac962f4bcbab1343640  internal/cbm/vendored/grammars/objectscript_routine/parser.c
-2fde85097c7df0180b7a204eeae97c32a541f1fcc1b44a3837a57f26ac9966fa  internal/cbm/vendored/grammars/objectscript_routine/scanner.c
+1da66e46ba02fcf2f9ca42b2a80c8658e150ae7b5154d81b8968117b959b5af7  internal/cbm/vendored/grammars/objectscript_routine/scanner.c
 b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea  internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/alloc.h
 31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c  internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/array.h
 180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2  internal/cbm/vendored/grammars/objectscript_routine/tree_sitter/parser.h
 85b458dc150c4d17873dd5e73d01cfa23c797997a09d841884e08378242e929c  internal/cbm/vendored/grammars/objectscript_udl/LICENSE
 f3a047fba6d238da258c0370878e703504b347aad13add09a021935daa891bd3  internal/cbm/vendored/grammars/objectscript_udl/objectscript_common.h
 2003ffb1a07ec30a2f7176c65aae958b804074af8356a81a3d42417a240f2e15  internal/cbm/vendored/grammars/objectscript_udl/parser.c
-2e80c4939726948b0ebf8af56103a9c0278a038eeb5be0b769c59d2b4e2d36b1  internal/cbm/vendored/grammars/objectscript_udl/scanner.c
+537be452c5079531e08edc302b65c63f45956510f6ee96ec88c1d7531d203f1e  internal/cbm/vendored/grammars/objectscript_udl/scanner.c
 b29c1c9fb7cc82f58c84b376df1297d6e2737a1d655fd356db0859e3c29c2fea  internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/alloc.h
 31e60a1bff6f715afacce03b5b70efe42b58371b4f9595dd4af52a577ff9608c  internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/array.h
 180b893c8734778fd32f372dfbc27bd6ad1cd2221f26150b31256ff6716320d2  internal/cbm/vendored/grammars/objectscript_udl/tree_sitter/parser.h

tests/test_grammar_regression.c

@@ -16,6 +16,72 @@
 #include "test_framework.h"
 #include "cbm.h"
 #include "grammar_cases.h"
+#include <string.h>
+
+typedef void *(*grammar_scanner_create_fn)(void);
+typedef unsigned (*grammar_scanner_serialize_fn)(void *payload, char *buffer);
+typedef void (*grammar_scanner_deserialize_fn)(void *payload, const char *buffer, unsigned length);
+typedef void (*grammar_scanner_destroy_fn)(void *payload);
+
+extern void *tree_sitter_objectscript_routine_external_scanner_create(void);
+extern unsigned tree_sitter_objectscript_routine_external_scanner_serialize(void *payload,
+                                                                            char *buffer);
+extern void tree_sitter_objectscript_routine_external_scanner_deserialize(void *payload,
+                                                                          const char *buffer,
+                                                                          unsigned length);
+extern void tree_sitter_objectscript_routine_external_scanner_destroy(void *payload);
+extern void *tree_sitter_objectscript_udl_external_scanner_create(void);
+extern unsigned tree_sitter_objectscript_udl_external_scanner_serialize(void *payload,
+                                                                        char *buffer);
+extern void tree_sitter_objectscript_udl_external_scanner_deserialize(void *payload,
+                                                                      const char *buffer,
+                                                                      unsigned length);
+extern void tree_sitter_objectscript_udl_external_scanner_destroy(void *payload);
+
+static bool grammar_scanner_empty_deserialize_and_roundtrip(
+    grammar_scanner_create_fn create, grammar_scanner_serialize_fn serialize,
+    grammar_scanner_deserialize_fn deserialize, grammar_scanner_destroy_fn destroy) {
+    char reference_state[4096];
+    char empty_state[4096];
+    char roundtrip_state[4096];
+    void *reference = create();
+    void *empty = create();
+    void *roundtrip = create();
+    bool passed = false;
+
+    if (!reference || !empty || !roundtrip) {
+        goto done;
+    }
+
+    unsigned reference_length = serialize(reference, reference_state);
+    if (reference_length == 0 || reference_length > sizeof(reference_state)) {
+        goto done;
+    }
+
+    deserialize(empty, NULL, 0);
+    unsigned empty_length = serialize(empty, empty_state);
+    if (reference_length != empty_length ||
+        memcmp(reference_state, empty_state, reference_length) != 0) {
+        goto done;
+    }
+
+    deserialize(roundtrip, reference_state, reference_length);
+    unsigned roundtrip_length = serialize(roundtrip, roundtrip_state);
+    passed = reference_length == roundtrip_length &&
+             memcmp(reference_state, roundtrip_state, reference_length) == 0;
+
+done:
+    if (reference) {
+        destroy(reference);
+    }
+    if (empty) {
+        destroy(empty);
+    }
+    if (roundtrip) {
+        destroy(roundtrip);
+    }
+    return passed;
+}
 
 static int reg_has_def_any(CBMFileResult *r, const char *name) {
     for (int i = 0; i < r->defs.count; i++) {
@@ -416,6 +482,26 @@ TEST(grammar_regression_all) {
     PASS();
 }
 
+TEST(grammar_objectscript_routine_scanner_empty_deserialize_and_roundtrip) {
+    ASSERT_TRUE(grammar_scanner_empty_deserialize_and_roundtrip(
+        tree_sitter_objectscript_routine_external_scanner_create,
+        tree_sitter_objectscript_routine_external_scanner_serialize,
+        tree_sitter_objectscript_routine_external_scanner_deserialize,
+        tree_sitter_objectscript_routine_external_scanner_destroy));
+    PASS();
+}
+
+TEST(grammar_objectscript_udl_scanner_empty_deserialize_and_roundtrip) {
+    ASSERT_TRUE(grammar_scanner_empty_deserialize_and_roundtrip(
+        tree_sitter_objectscript_udl_external_scanner_create,
+        tree_sitter_objectscript_udl_external_scanner_serialize,
+        tree_sitter_objectscript_udl_external_scanner_deserialize,
+        tree_sitter_objectscript_udl_external_scanner_destroy));
+    PASS();
+}
+
 void suite_grammar_regression(void) {
+    RUN_TEST(grammar_objectscript_routine_scanner_empty_deserialize_and_roundtrip);
+    RUN_TEST(grammar_objectscript_udl_scanner_empty_deserialize_and_roundtrip);
     RUN_TEST(grammar_regression_all);
 }