Add streamed self-hosted lexer experiment

[julia-script]

What changed

Add an experimental self-hosted lexer under compiler/. It reads one normalized relative Silk source path, borrows the file bytes, and exposes tokenization through Stream<Token, never ? never>.take without retaining a token vector.

The lexer follows the current bootstrap compiler for keywords, trivia, longest-match punctuation including ..., lifetimes, numeric and duration literals, static and character literals, invalid regions, focused LEX0001–LEX0012 diagnostics, and one-shot EOF behavior. The CLI prints each token and diagnostic directly to stdout.

The application edge uses the current source-owned OsFileSystem and StdoutWriter providers. With the JUL-152 repair now on main, the host-input, filesystem, and writer operations remain in one sequential program body without the earlier helper-splitting workaround.

For example:

silk run --manifest-path compiler/silk.toml -- compiler/fixtures/empty.silk

prints:

EndOfFile 0..0

The project includes focused fixtures, usage documentation, and the compiler limitations found while writing real Silk code. Follow-up work is tracked in JUL-153, JUL-154, and JUL-155; JUL-152 is recorded as resolved on current main.

Validation

• Rebased onto dd4510fa
• silk check --manifest-path compiler/silk.toml
• Native build of the 347-symbol lexer
• Exact differential comparison with the current bootstrap lexer across all six fixtures: token kinds, byte spans, diagnostic codes, reason tags, and diagnostic spans
• Lossless span coverage and one zero-width EOF per fixture
• Missing, extra, absolute, dot-component, parent-component, missing-file, and unreadable-file argument/path cases rejected
• pnpm check

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
silk-effect-docs	Ready	Preview	Sep 7, 2026 3:31am UTC