Repository navigation
Sync latest liquid-html-parser improvements - #1372
Draft
PhilippeCollin wants to merge 22 commits into
Draft
PhilippeCollin wants to merge 22 commits into
PhilippeCollin wants to merge 22 commits into
Conversation
The document tokenizer advanced one character at a time and ran every mode's startsWith checks on each one, even inside plain text. Most characters cannot start a token in the current mode, so jump straight to the next one that can and let the main loop re-check it there. Liquid tag and output bodies use indexOf for their closing delimiter. The `<` tag-start check uses char codes instead of a regex. Tokens and ASTs are unchanged.
The existing tests only had quotes directly after `=`, where the main loop checks them without the fast path, so a helper missing a stop character could pass CI. The oracle suites that might catch it are skipped there because the theme fixtures are gitignored. Route the five text branches through one `nextTextCandidate` dispatcher, so a test-only `tokenizeWithoutFastPath` can run the same loop one character at a time. Tests now: - put each token start in the middle of plain text in every mode - compare `tokenize` with `tokenizeWithoutFastPath` on every string of up to 4 token-relevant characters, in every entry state
- `scanLiquidOpen` returns early unless the character is `{`, instead
of four failed `startsWith` calls at every `<`, `-`, quote, `/`, `>`
and `=` the text fast path stops at.
- The Liquid expression tokenizer tests digits with char codes instead
of evaluating `/\d/` literals per token, and only runs the whitespace
regex when the character could be whitespace.
Each check is a pre-filter in front of the existing `startsWith` or
regex, so letting extra characters through can't change tokens; only
rejecting a real `{` or digit could, and the suite catches that.
`value.replace(/\s+$/, '')` retries the match at every position of each whitespace run, so its cost grows with the square of the run length. It was the most expensive single step when parsing Dawn and Horizon. `trimStart()`/`trimEnd()` strip exactly the characters `\s` matches (both use the spec's WhiteSpace and LineTerminator sets), in linear time. A test pins that set and the long-run case.
`isLiteralKeyword` used `s in LiquidLiteralValues`, and `tagForName`
read `this.builtins[name]` from a plain object. Both also see inherited
properties, so `{{ constructor }}` became a LiquidLiteral whose value
was the Object function, and `{% constructor %}` / `{% __proto__ %}`
resolved to a non-tag "definition" and threw.
Recognize literal keywords with a switch (also faster than `in` on this
hot path) and store builtin tags in a Map. A test checks every
LiquidLiteralValues key is still recognized.
Shopify Liquid treats `{% render 'x' with key: value %}` as both `with key` and a
named argument, so the parser now leaves `key:` for the argument loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruby accepts `for x in c limit:5 reversed` as well as `for x in c reversed limit:5`. A trailing `reversed` used to throw and degrade the tag to string markup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block and section markup now share a single BlockArgument type, whose value is a scalar or a single-level array literal, instead of a LiquidNamedArgument | BlockArrayArgument union. Filters in these arguments are still rejected, and the tag falls back to string markup. Co-Authored-By: Charles-P. Clermont <charles@cpclermont.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liquid-html-parser already clamps out-of-range indices when it builds a
LiquidHTMLASTParsingError; these tests pin that an unclosed `{{` at the end of
input is reported as a parse error with a real location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each raw tag now carries the rule of the Liquid class that parses it:
- comment nests, skips raw bodies, and closes on a tag named endcomment
whatever follows the name, as Comment#parse_body does;
- raw, javascript, schema and stylesheet close at the last tag of their first
end<name> token, as Liquid::Raw does;
- doc reads like raw but rejects a nested doc, as Liquid::Doc does;
- style is a Liquid::Block, so it nests and skips raw bodies inside it.
The same rules apply to raw tags inside {% liquid %}, whose lines are named
only after ASCII indentation. Templates that Liquid reports as never closed
now fail to parse here too.
Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block and section arguments are now one BlockArgument type, so the completion params narrow on the argument's value rather than on the argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parser now closes {% raw %} and {% doc %} on an end tag with extra strings,
as Liquid does, so a later unclosed tag is the one reported. It also rejects a
comment whose body Liquid's tokenizer runs past the endcomment, so
LiquidSyntaxError reports the parser's error at the comment's open tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Themes often open an element in each branch of a conditional and close it
once after the conditional:
{% if a %}<div class="x">{% else %}<div class="y">{% endif %}
{{ content_for_layout }}
</div>
Liquid writes this as text, so Shopify renders it, but the parser matched the
close tag against the enclosing element and threw a few tags later.
With the new pairDanglingCloseTags option, off by default, such a close tag
becomes an HtmlDanglingMarkerClose when an earlier if, unless or case in the
same scope leaves an element of that name open on every path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branch-opened elements are now tracked as a stack per scope, so a crossed close like <section><div> … </section> is no longer converted, and an unmatched dangling close leaves no debt against elements opened later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A <script> or <style> body is raw text: a < followed by a letter, ! or { in
it is JavaScript or CSS, not an HTML tag. When the token stream doesn't
close the element where the source does, the body is tokenized again in a
raw-text mode that only reads Liquid delimiters, and the rest of the
document after the close tag is tokenized again if the old stream ran past
it.
The close tag is found in the source, skipping Liquid delimiters and the
bodies of Liquid raw tags such as {% comment %}, so a </script> inside
either doesn't close the element. A raw tag inside the body that reslices
the stream stays within the body.
Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After <!-- the tokenizer recognizes only Liquid delimiters and the
comment's close, so markup in a comment (a <b, an unbalanced quote, a
nested <!--) can't open a tag mode that swallows the -->. The close is -->
or --!>, plus the HTML spec's empty comments <!--> and <!--->. Outside a
comment, --> is text. The parser finds the close by its token, which
replaces the source scan for --> that conditional comments needed.
Liquid raw tag bodies are opaque to the tokenizer too: given the
registry's body rules, it ends a raw body where the parser does,
tokenizes it on its own, and resumes after the end tag in the mode the
tag sat in. So {% comment %}<!-- old {% endcomment %} can't turn the rest
of the file into comment text, and a <b or a quote in a raw body no
longer leaks out.
Outside SVG, <script> and <style> bodies are raw text on the first pass.
Inside an <svg>, up to the </svg> that balances it, <!-- opens no mode:
the parser keeps an svg body raw, and SVG holds CDATA and foreignObject
scripts where <!-- isn't a comment.
Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Liquid treats <!-- and --> as text, so the Liquid inside an HTML comment
renders, and its tags still open, branch and close blocks. The parser
read a comment's body as opaque, so {% if a %}<!-- {% endif %} --> failed
with an unclosed if, and checks never saw the Liquid in a comment.
HtmlComment now has children (text and Liquid, never markup) and
blockStartPosition/blockEndPosition for its delimiters:
- A comment that crosses a Liquid block boundary is split there, and the
part on the split side has an empty delimiter range.
- A comment whose --> sits in a block opened inside it ends with that
block.
- A comment that never closes ends at the end of the source, as in a
browser.
- An end tag in a comment that matches no open block fails, as in Liquid.
- opaqueHtmlComments keeps comment bodies opaque, for HTML that is data
rather than a template.
Consumers:
- Prettier prints a split or unclosed comment as written, trimming the
whitespace on its delimiter-less side so formatting stays idempotent.
- Completions descend into a comment's children.
- After an error inside a comment, tolerant recovery resumes at the
comment's --> instead of skipping it, so the markup after it stays
HTML.
Co-Authored-By: River <river@shopify.com>
Co-Authored-By: Martin Morissette <martin.morissette@shopify.com>
Co-Authored-By: Mbarak Bujra <mbarak.bujra@shopify.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PhilippeCollin
force-pushed
the
liquid-html-parser-sync
branch
from
October 30, 2026 20:06
0fcede9 to
6750681
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are you adding in this PR?
This PR makes
liquid-html-parserabout 2.6× faster on real themes, and brings it closer to how Liquid and browsers read a template.Shopify keeps a second copy of the parser in another codebase, which picked up both the speedups and a set of fixes. This PR ports them, adapted to this repo's AST, specs and consumers (prettier, theme-check, the language server). Each commit is one change and reads on its own. The changesets describe each change for the changelog.
Faster parsing
The first five commits are #1329, #1333, #1330 and #1331, adapted to the current
main, where the tokenizer loop has since moved intocreateTokenizer. Later commits build on them. Those PRs have the full profiling, the per-change benchmarks and the differential checks. They can close if this one lands, or land first and this branch drops them.Object.prototypenames as ordinary namesMeasured on this branch with Apple M3 Pro and Node 24.22. The input is the parser's 429 fixture files (Dawn, Horizon and the base theme, 3.4 MB). Builds ran in alternating processes, 5 rounds × 30 passes. Each value is the median.
mainmaintoLiquidHtmlASTtoLiquidASTtoTolerantLiquidHtmlAST(prettier's parse step)The later ports cost about 6 ms of that back. A per-commit run puts about 4 ms on 56e78a4, which adds the tokenizer's HTML comment mode and makes raw tag bodies opaque to it. The other commits are within noise. Every build parses all 429 files without an error.
Liquid and HTML fixes, and who wrote them
{% render 'x' with key: value %}and{% include %}argumentsreversedanywhere amongforandtablerowargumentsBlockArgumenttype for{% block %}and{% section %}argumentspairDanglingCloseTagsfor elements opened in each branch of a conditional<script>and<style>bodies are raw textBehaviour changes worth a look
HtmlCommenthaschildrenandblockStartPosition/blockEndPosition. A block can open on one side of a comment delimiter and close on the other, so{% if a %}<!-- {% endif %} -->parses. A comment that crosses a block boundary is split into severalHtmlCommentnodes, and a comment that never closes ends at the end of the file.opaqueHtmlComments: truekeeps the old opaque body.UndefinedObjectandMissingTemplatereport Liquid in comments. Liquid renders that Liquid, so this matches the output, but themes with broken Liquid commented out will get new offenses.--!>closes one,<!-->and<!--->are empty comments, and-->outside a comment is text.<script>and<style>bodies are raw text. A<in them no longer opens an element, and a</script>inside Liquid or a{% comment %}doesn't close the script.<script>a<script>b</script>c</script>now fails with "Attempting to close HtmlElement 'script' before it was opened", so prettier won't format such a file and theme-check reports it. Nested<svg>still pairs.{% endcomment xyz %},{% endraw xyz %}and{% enddoc foo %}close their tags. A nested{% doc %}is rejected, and{% style %}nests. A comment whose body Liquid's tokenizer runs past the{% endcomment %}now fails, as it does in Liquid. The same rules apply inside{% liquid %}.liquid-html-syntax-error.yml,LiquidSyntaxErrornow reports the later unclosed tag, or the comment's open tag, with the parser's message (7244211).{% for x in c limit: 5 reversed %}parses instead of falling back to string markup.BlockArgumentreplaces theLiquidNamedArgument | BlockArrayArgumentunion for block and section arguments.BlockArrayArgumentis still exported, as a subtype.pairDanglingCloseTagsis opt-in and off by default.-->, so the markup after it stays HTML.Open questions
<script>is now a parse error, so prettier refuses the file. Is that acceptable, or should the parser keep pairing nested scripts?Not ported
<svg>at its first</svg>. This tokenizer counts nested SVGs, and I kept that.sectionstays a plain tag.{{ 'key' | t: }}) and whitespace around=in attributes: this parser already handles both.isBareBlockArgumentNamehelper, and test helpers whose tests are specs here.What's next? Any followup issues?
Decide the open questions above. If #1329, #1330, #1331 and #1333 merge first, rebase this branch to drop them.
Tophatting
pnpm test: 8110 passed, 5 skipped, 0 failed.pnpm type-checkis clean.UPDATE_SPECS=1where the AST changed. New cases inspecs/parser/html.ymlcover split, unclosed and reopened comments.TEST_IDEMPOTENCE=true.Before you deploy
changeset(liquid-html-parsergainsHtmlComment.children,BlockArgumentandpairDanglingCloseTags)changesets for the fixes and the consumersHtmlCommentgains children, and the stricter parsing above rejects some templates it used to accept.🤖 Generated with Claude Code