Skip to content

Sync latest liquid-html-parser improvements - #1372

Draft
PhilippeCollin wants to merge 22 commits into
mainfrom
liquid-html-parser-sync
Draft

PhilippeCollin wants to merge 22 commits into
mainfrom
liquid-html-parser-sync

Conversation

@PhilippeCollin

@PhilippeCollin PhilippeCollin commented Oct 30, 2026 •

Copy link
Copy Markdown

What are you adding in this PR?

This PR makes liquid-html-parser about 2.6× faster on real themes, and brings it closer to how Liquid and browsers read a template.

Shopify keeps a second copy of the parser in another codebase, which picked up both the speedups and a set of fixes. This PR ports them, adapted to this repo's AST, specs and consumers (prettier, theme-check, the language server). Each commit is one change and reads on its own. The changesets describe each change for the changelog.

Faster parsing

The first five commits are #1329, #1333, #1330 and #1331, adapted to the current main, where the tokenizer loop has since moved into createTokenizer. Later commits build on them. Those PRs have the full profiling, the per-change benchmarks and the differential checks. They can close if this one lands, or land first and this branch drops them.

Commits Change PR
34dae62, f11f802 Tokenize plain text runs in one step #1329
5b634d9 Check character codes before regexes in the tokenizers #1333
320fa1f Trim text around Liquid tags in linear time #1330
a73b174 Parse Object.prototype names as ordinary names #1331

Measured on this branch with Apple M3 Pro and Node 24.22. The input is the parser's 429 fixture files (Dawn, Horizon and the base theme, 3.4 MB). Builds ran in alternating processes, 5 rounds × 30 passes. Each value is the median.

All 429 files main after a73b174 this PR Speedup vs main
toLiquidHtmlAST 112.7 ms 37.2 ms 43.2 ms 2.6×
toLiquidAST 113.3 ms 36.2 ms 42.2 ms 2.7×
toTolerantLiquidHtmlAST (prettier's parse step) 112.2 ms 36.6 ms 43.5 ms 2.6×

The later ports cost about 6 ms of that back. A per-commit run puts about 4 ms on 56e78a4, which adds the tokenizer's HTML comment mode and makes raw tag bodies opaque to it. The other commits are within noise. Every build parses all 429 files without an error.

Liquid and HTML fixes, and who wrote them

Commits Change Original work by
8445f5a, df9074a, e0a1152, 0d65a38 Legacy {% render 'x' with key: value %} and {% include %} arguments Alok Swamy
c57e318 reversed anywhere among for and tablerow arguments Guilherme Carreiro
5492ff9, 4ce802f One BlockArgument type for {% block %} and {% section %} arguments Natacha Levesque, Charles-P. Clermont
f2cfbee Tests for parse error locations at the end of input Iain Wilson
4e85163, 7244211 Each raw tag ends its body by the rule of the Liquid class that parses it River, with Martin Morissette and Mbarak Bujra
7368046, 11cc0f9 pairDanglingCloseTags for elements opened in each branch of a conditional River
c80848c Tests that block and section markup keep the full source Claudéric Demers
b7c9102 <script> and <style> bodies are raw text River, with Martin Morissette
56e78a4 An HTML comment mode in the tokenizer River, with Mbarak Bujra
dcd9e3f Parse the Liquid inside HTML comments River, with Martin Morissette and Mbarak Bujra

Behaviour changes worth a look

  • Liquid in HTML comments is parsed. HtmlComment has children and blockStartPosition/blockEndPosition. A block can open on one side of a comment delimiter and close on the other, so {% if a %}<!-- {% endif %} --> parses. A comment that crosses a block boundary is split into several HtmlComment nodes, and a comment that never closes ends at the end of the file. opaqueHtmlComments: true keeps the old opaque body.
  • Checks see the Liquid in comments. An end tag in a comment that closes no block is now a syntax error, and checks such as UndefinedObject and MissingTemplate report Liquid in comments. Liquid renders that Liquid, so this matches the output, but themes with broken Liquid commented out will get new offenses.
  • HTML comments tokenize as in a browser. Only Liquid and the close count inside a comment. --!> closes one, <!--> and <!---> are empty comments, and --> outside a comment is text.
  • <script> and <style> bodies are raw text. A < in them no longer opens an element, and a </script> inside Liquid or a {% comment %} doesn't close the script. <script>a<script>b</script>c</script> now fails with "Attempting to close HtmlElement 'script' before it was opened", so prettier won't format such a file and theme-check reports it. Nested <svg> still pairs.
  • Raw tags close where Liquid closes them. {% endcomment xyz %}, {% endraw xyz %} and {% enddoc foo %} close their tags. A nested {% doc %} is rejected, and {% style %} nests. A comment whose body Liquid's tokenizer runs past the {% endcomment %} now fails, as it does in Liquid. The same rules apply inside {% liquid %}.
  • Error reports moved for two raw tag cases. In liquid-html-syntax-error.yml, LiquidSyntaxError now reports the later unclosed tag, or the comment's open tag, with the parser's message (7244211).
  • {% for x in c limit: 5 reversed %} parses instead of falling back to string markup.
  • BlockArgument replaces the LiquidNamedArgument | BlockArrayArgument union for block and section arguments. BlockArrayArgument is still exported, as a subtype.
  • pairDanglingCloseTags is opt-in and off by default.
  • Prettier prints a split or unclosed comment as written. A split piece that holds only whitespace prints as an empty line.
  • The language server completes inside HTML comments. After an error inside a comment, tolerant recovery resumes at the comment's -->, so the markup after it stays HTML.

Open questions

  1. A nested <script> is now a parse error, so prettier refuses the file. Is that acceptable, or should the parser keep pairing nested scripts?
  2. The new offenses for Liquid in HTML comments: should any check skip comments? Liquid renders that Liquid, so I lean no.

Not ported

  • The other copy ends an <svg> at its first </svg>. This tokenizer counts nested SVGs, and I kept that.
  • Hybrid tags: this repo removed them on purpose in af297ba, so section stays a plain tag.
  • Empty filter argument lists ({{ 'key' | t: }}) and whitespace around = in attributes: this parser already handles both.
  • The end-of-input error fix: already fixed here, so only the tests came over.
  • Dead code removal, benchmarks, an unused isBareBlockArgumentName helper, and test helpers whose tests are specs here.

What's next? Any followup issues?

Decide the open questions above. If #1329, #1330, #1331 and #1333 merge first, rebase this branch to drop them.

Tophatting

  • pnpm test: 8110 passed, 5 skipped, 0 failed. pnpm type-check is clean.
  • Parser specs were re-recorded with UPDATE_SPECS=1 where the AST changed. New cases in specs/parser/html.yml cover split, unclosed and reopened comments.
  • The new prettier fixtures for split and unclosed comments pass with TEST_IDEMPOTENCE=true.

Before you deploy

  • I included a minor bump changeset (liquid-html-parser gains HtmlComment.children, BlockArgument and pairDanglingCloseTags)
  • I included patch bump changesets for the fixes and the consumers
  • My feature is backward compatible: no. HtmlComment gains children, and the stricter parsing above rejects some templates it used to accept.

🤖 Generated with Claude Code

PhilippeCollin and others added 22 commits October 30, 2026 10:55
The document tokenizer advanced one character at a time and ran every
mode's startsWith checks on each one, even inside plain text. Most
characters cannot start a token in the current mode, so jump straight to
the next one that can and let the main loop re-check it there.

Liquid tag and output bodies use indexOf for their closing delimiter.
The `<` tag-start check uses char codes instead of a regex.

Tokens and ASTs are unchanged.
The existing tests only had quotes directly after `=`, where the main
loop checks them without the fast path, so a helper missing a stop
character could pass CI. The oracle suites that might catch it are
skipped there because the theme fixtures are gitignored.

Route the five text branches through one `nextTextCandidate` dispatcher,
so a test-only `tokenizeWithoutFastPath` can run the same loop one
character at a time. Tests now:
- put each token start in the middle of plain text in every mode
- compare `tokenize` with `tokenizeWithoutFastPath` on every string of
  up to 4 token-relevant characters, in every entry state
- `scanLiquidOpen` returns early unless the character is `{`, instead
  of four failed `startsWith` calls at every `<`, `-`, quote, `/`, `>`
  and `=` the text fast path stops at.
- The Liquid expression tokenizer tests digits with char codes instead
  of evaluating `/\d/` literals per token, and only runs the whitespace
  regex when the character could be whitespace.

Each check is a pre-filter in front of the existing `startsWith` or
regex, so letting extra characters through can't change tokens; only
rejecting a real `{` or digit could, and the suite catches that.
`value.replace(/\s+$/, '')` retries the match at every position of each
whitespace run, so its cost grows with the square of the run length.
It was the most expensive single step when parsing Dawn and Horizon.

`trimStart()`/`trimEnd()` strip exactly the characters `\s` matches
(both use the spec's WhiteSpace and LineTerminator sets), in linear
time. A test pins that set and the long-run case.
`isLiteralKeyword` used `s in LiquidLiteralValues`, and `tagForName`
read `this.builtins[name]` from a plain object. Both also see inherited
properties, so `{{ constructor }}` became a LiquidLiteral whose value
was the Object function, and `{% constructor %}` / `{% __proto__ %}`
resolved to a non-tag "definition" and threw.

Recognize literal keywords with a switch (also faster than `in` on this
hot path) and store builtin tags in a Map. A test checks every
LiquidLiteralValues key is still recognized.
Shopify Liquid treats `{% render 'x' with key: value %}` as both `with key` and a
named argument, so the parser now leaves `key:` for the argument loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruby accepts `for x in c limit:5 reversed` as well as `for x in c reversed limit:5`.
A trailing `reversed` used to throw and degrade the tag to string markup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block and section markup now share a single BlockArgument type, whose value is
a scalar or a single-level array literal, instead of a LiquidNamedArgument |
BlockArrayArgument union. Filters in these arguments are still rejected, and
the tag falls back to string markup.

Co-Authored-By: Charles-P. Clermont <charles@cpclermont.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liquid-html-parser already clamps out-of-range indices when it builds a
LiquidHTMLASTParsingError; these tests pin that an unclosed `{{` at the end of
input is reported as a parse error with a real location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each raw tag now carries the rule of the Liquid class that parses it:

- comment nests, skips raw bodies, and closes on a tag named endcomment
  whatever follows the name, as Comment#parse_body does;
- raw, javascript, schema and stylesheet close at the last tag of their first
  end<name> token, as Liquid::Raw does;
- doc reads like raw but rejects a nested doc, as Liquid::Doc does;
- style is a Liquid::Block, so it nests and skips raw bodies inside it.

The same rules apply to raw tags inside {% liquid %}, whose lines are named
only after ASCII indentation. Templates that Liquid reports as never closed
now fail to parse here too.

Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block and section arguments are now one BlockArgument type, so the completion
params narrow on the argument's value rather than on the argument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser now closes {% raw %} and {% doc %} on an end tag with extra strings,
as Liquid does, so a later unclosed tag is the one reported. It also rejects a
comment whose body Liquid's tokenizer runs past the endcomment, so
LiquidSyntaxError reports the parser's error at the comment's open tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Themes often open an element in each branch of a conditional and close it
once after the conditional:

  {% if a %}<div class="x">{% else %}<div class="y">{% endif %}
    {{ content_for_layout }}
  </div>

Liquid writes this as text, so Shopify renders it, but the parser matched the
close tag against the enclosing element and threw a few tags later.

With the new pairDanglingCloseTags option, off by default, such a close tag
becomes an HtmlDanglingMarkerClose when an earlier if, unless or case in the
same scope leaves an element of that name open on every path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branch-opened elements are now tracked as a stack per scope, so a crossed close
like <section><div> … </section> is no longer converted, and an unmatched
dangling close leaves no debt against elements opened later.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A <script> or <style> body is raw text: a < followed by a letter, ! or { in
it is JavaScript or CSS, not an HTML tag. When the token stream doesn't
close the element where the source does, the body is tokenized again in a
raw-text mode that only reads Liquid delimiters, and the rest of the
document after the close tag is tokenized again if the old stream ran past
it.

The close tag is found in the source, skipping Liquid delimiters and the
bodies of Liquid raw tags such as {% comment %}, so a </script> inside
either doesn't close the element. A raw tag inside the body that reslices
the stream stays within the body.

Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After <!-- the tokenizer recognizes only Liquid delimiters and the
comment's close, so markup in a comment (a <b, an unbalanced quote, a
nested <!--) can't open a tag mode that swallows the -->. The close is -->
or --!>, plus the HTML spec's empty comments <!--> and <!--->. Outside a
comment, --> is text. The parser finds the close by its token, which
replaces the source scan for --> that conditional comments needed.

Liquid raw tag bodies are opaque to the tokenizer too: given the
registry's body rules, it ends a raw body where the parser does,
tokenizes it on its own, and resumes after the end tag in the mode the
tag sat in. So {% comment %}<!-- old {% endcomment %} can't turn the rest
of the file into comment text, and a <b or a quote in a raw body no
longer leaks out.

Outside SVG, <script> and <style> bodies are raw text on the first pass.
Inside an <svg>, up to the </svg> that balances it, <!-- opens no mode:
the parser keeps an svg body raw, and SVG holds CDATA and foreignObject
scripts where <!-- isn't a comment.

Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Liquid treats <!-- and --> as text, so the Liquid inside an HTML comment
renders, and its tags still open, branch and close blocks. The parser
read a comment's body as opaque, so {% if a %}<!-- {% endif %} --> failed
with an unclosed if, and checks never saw the Liquid in a comment.

HtmlComment now has children (text and Liquid, never markup) and
blockStartPosition/blockEndPosition for its delimiters:
- A comment that crosses a Liquid block boundary is split there, and the
  part on the split side has an empty delimiter range.
- A comment whose --> sits in a block opened inside it ends with that
  block.
- A comment that never closes ends at the end of the source, as in a
  browser.
- An end tag in a comment that matches no open block fails, as in Liquid.
- opaqueHtmlComments keeps comment bodies opaque, for HTML that is data
  rather than a template.

Consumers:
- Prettier prints a split or unclosed comment as written, trimming the
  whitespace on its delimiter-less side so formatting stays idempotent.
- Completions descend into a comment's children.
- After an error inside a comment, tolerant recovery resumes at the
  comment's --> instead of skipping it, so the markup after it stays
  HTML.

Co-Authored-By: River <river@shopify.com>
Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PhilippeCollin PhilippeCollin changed the title Port liquid-html-parser fixes for HTML comments, raw tags and script bodies Speed up liquid-html-parser and port its HTML comment, raw tag and script fixes Oct 30, 2026
@PhilippeCollin PhilippeCollin changed the title Speed up liquid-html-parser and port its HTML comment, raw tag and script fixes Sync latest liquid-html-parser improvements Oct 30, 2026
@PhilippeCollin
PhilippeCollin force-pushed the liquid-html-parser-sync branch from 0fcede9 to 6750681 Compare October 30, 2026 20:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants