Skip to content

DDIR: join fragments as a separate, opt-in step from surface text to plain DDP (spike) - #1090

Closed
frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-join-fragments-surface
Closed

frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-join-fragments-surface

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

The join fragments of #1085, held apart from the core language as a separate, opt-in step from surface text to plain DDP. A draft, for comparison with #1085; both are spikes.

How it differs from #1085

#1085 This draft
Core lexer, parser, syntax tree, lowering, printers *, [, ] tokens, the name( rule, Expr::Fragment, a lowering call, a printer arm (+44 lines) master-next's, except lambda_in becomes crate-visible (+1/−1)
Fragment code parse/pipe/fragment.rs, 480 lines, extending the core parser src/surface.rs, 672 lines: its own scanner, a small printer for the generated text, surface positions
Tests 26 tests, 477 lines 8 tests, 215 lines
Tools always on ddir-plan --fragments, server load … --fragments; everything else reads plain DDP
Total against master-next +1,001/−4 +933/−24

How it works

  • surface::expand(text) replaces each fragment's source range with plain DDP and leaves every other byte unchanged; comments inside a fragment are copied to the start of its replacement.
  • The core parser then reads the result. No fragment token, parser rule, syntax-tree node or lowering hook exists in the core.
  • Errors keep their messages, prefixed with the line and column in the surface text.
  • The expanded text is ordinary DDP an author can read, save and load without the flag.

The expansion is #1085's: written join order, key-slot reads, direct first joins, each test as a plain filter at its earliest join, variants through enlist unweave, and constant later keys rejected.

Checks

Not intended to land as is.

Implemented by GPT-6 Sol (Codex); reviewed, tested and opened by Claude Code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL

…plain DDP (spike)

The same join fragments as TimelyDataflow#1085, held apart from the core language.
`surface::expand` finds each fragment in the text and replaces its source range with plain DDP, leaving every other byte unchanged; the core parser then reads the result.
The core lexer, parser, syntax tree, lowering and printers are master-next's, except that `lambda_in` becomes crate-visible so the scanner finds lambda boundaries as the lexer does.

`ddir-plan --fragments` and the server's `load … --fragments` opt in; everything else reads plain DDP, and saved expanded text loads without the flag.
Errors keep their messages, prefixed with the line and column in the surface text.
The expansion is TimelyDataflow#1085's: written join order, key-slot reads, direct first joins, each test as a plain filter at its earliest join, variants through `enlist unweave`, and constant later keys rejected.

Eight tests keep the essentials; the emitted plans match TimelyDataflow#1085's byte for byte.

Co-Authored-By: GPT-6 Sol (Codex) <noreply@openai.com>
Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant