Skip to content

perf(tokenizer): scan quoted attribute values in one loop - #2567

Open
g4lb wants to merge 1 commit into
fb55:masterfrom
g4lb:perf-attribute-value-scan
Open

g4lb wants to merge 1 commit into
fb55:masterfrom
g4lb:perf-attribute-value-scan

Conversation

@g4lb

@g4lb g4lb commented Nov 7, 2026 •

Copy link
Copy Markdown

With decodeEntities: false, the tokenizer skips to the closing quote of an attribute value. With the default decodeEntities: true, each character of the value goes through the parse loop. Now this path also skips, to the next quote or &. The output does not change.

parseDocument on the built dist:

Input Before (Node 22) After (Node 22) Node 22 Node 20
Wikipedia "HTML" page, 809 KB 6.20 ms 4.89 ms -21.2% -12.0%
GitHub repository page, 333 KB 1.92 ms 1.56 ms -19.0% -12.3%
MDN <table> page, 248 KB 1.58 ms 1.42 ms -10.2% -5.3%
<a href="…"> with a 50,000-character value 194 µs 35.7 µs -81.5% -75.9%

I used 16 interleaved rounds and a Mann-Whitney test (Apple M5, Node 22.23.1 and 20.20.2). I did not measure other engines.

Slowdowns, in 57 benchmarks:

  • Attribute values with an entity after each character: +4.0% on Node 22, +1.1% on Node 20. Values with only entities: +3.1% on Node 22, -0.2% on Node 20.
  • The Markdown syntax page, 32 KB: +1.9% on Node 20. It is 1.6% faster on Node 22.
  • All other benchmarks that are not faster show no significant change. The old code against itself shows no significant change in any benchmark.

This PR does not change fastForwardTo, so it does not overlap with #2525.

Checks:

  • npm test passes on Node 22, and vitest passes on Node 20.
  • Old and new outputs matched on 3,040,000 results: events, positions, DOM, and tokenizer callbacks, with chunked input.

View guided diff

Summary by CodeRabbit

  • Bug Fixes
    • Quoted attribute values are now parsed through to the next quote, character reference, or end of available input, improving handling of values split across input buffers.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-07T06:28:42.762553Z 36895d9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The tokenizer now scans ordinary characters in quoted attribute values until it reaches the matching quote, an ampersand, or the buffer end. Ampersands continue to start entity decoding.

Changes: Attribute Value Scanning

Layer / File(s) Summary
Scan quoted attribute values
src/Tokenizer.ts
handleInAttributeValue advances across ordinary characters to the matching quote, an ampersand, or the buffer end. Ampersands continue to start entity decoding.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Suggested reviewers: fb55, spokodev

Merge Risk: Minimal · up to 36895

Quoted attribute scanning is optimized without an identified merge-blocking issue.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the tokenizer performance change: quoted attribute values are now scanned in one loop. The raw summary confirms that handleInAttributeValue in src/Tokenizer.ts scans decoded quoted att…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the tokenizer performance change: quoted attribute values are now scanned in one loop. The raw summary confirms that handleInAttributeValue in src/Tokenizer.ts scans decoded quoted attribute values directly to the next quote, ampersand, or buffer end. The objectives describe the same change. The title is specific, concise, and follows the conventional commit style, so it is a clear match for the main change.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit scans the quoted text,
Past plain words, quick and light.
At each ampersand, it pauses,
Where entity decoding starts.
Then onward, toward the closing quote.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: fdd5798c-f708-4f23-8320-94805c9d87f1
Commits

Reviewing files that changed from the base of the PR and between 5696889 and 36895d9.

Files selected for processing: 1

  • src/Tokenizer.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

View guided diff | Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] This PR appears safe to merge.

Summary

Speeds up quoted attribute values when decodeEntities is enabled.

  • Quoted attribute values now scan ahead to the next quote or ampersand.

Reviews (1) · Last reviewed commit: "perf(tokenizer): scan quoted attribute v..." · Reviewed by Greptile

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant