Simplify the streamed Silk lexer with current language features

[julia-script]

Stacked on #371 (julia/self-hosted-stream-lexer).

The streamed Silk lexer still used verbose bootstrap-era byte checks, cursor arithmetic, and duration flags. This refactor uses short-circuit predicates, contextually typed integer literals, an exhaustively matched literal-category enum, direct destructuring, and a duration-unit bitset. It removes the duplicated compound-punctuation precheck and shortens the lexer by 268 lines while preserving its output.

Remove the redundant usize alias in the entry point and import the token writer explicitly, avoiding the collision between the Token type and its module namespace.

Also simplify the TypeScript bootstrap lexer with exhaustive duration-diagnostic matching, keyword length buckets, and shared punctuation classification. Extend its malformed-duration test to exercise the separator diagnostic and regenerate the compiler integrity digest.

Validation:

• pnpm check passed, including the full compiler suite and shared native acceptance corpus.
• pnpm release:candidate passed all 10 checks.
• Silk analysis, canonical formatting, and native build passed.
• Native output matched the original executable on all six experiment fixtures plus generated duration, byte-pair, and mixed-byte inputs.
• The TypeScript lexer matched its original implementation on 70,536 generated inputs; its existing benchmark measured 8.18 MiB/s versus 7.46 MiB/s before (single-run measurements).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
silk-effect-docs	Ready	Preview	Sep 7, 2026 5:15pm UTC