You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci(bench): measure the incremental benchmark as a production bundle - #2379
The incremental update benchmark now measures TanStack DB as a production app runs it. Before this change, the benchmark imported db's source through tsx. It therefore measured development-mode code that every production build removes: process.env.NODE_ENV checks, development-only branches, and private members without their mangled names.
A new runner, scripts/bench/production.mjs, bundles a benchmark with esbuild against the built dist of db and db-ivm. The bundle is minified, and process.env.NODE_ENV is set to production. These are the settings that the size scenarios in scripts/size-compare.mjs use.
The runner refuses a dist that is older than its package's sources. It also fails if the bundle includes any module from a package's src, so a benchmark cannot measure a development copy of a package beside its dist.
The bundle has an inline source map. A failed assertion in the benchmark names its line in incremental-update.ts.
The benchmark keeps its import of db's source entry for its types. The runner redirects that import to the built entry, so the root tsc run does not change.
CI builds db-ivm and db on both sides, then runs the bundle. This adds about 7 seconds to each side. The base side takes the runner and its freshness check from the PR head, so both sides run the same benchmark code.
The README states what the benchmark measures. It also states that the 2026-11-05 runner-variation data came from source-mode runs.
The maintainer asked for this change on 2026-11-09. The trigger was #2378: it flagged 5 writes in 100-row configurations. Those writes ran development-only state probes that a production bundle removes.
Verification
The bundle's inputs are 118 files from packages/db/dist and 28 files from packages/db-ivm/dist. It has no inputs from src, and the output contains no NODE_ENV reads.
A short run of the bundled benchmark completes with all cases.
With a second import of db's source added to the benchmark, the runner fails and names the source files. Before this check, that run completed with two copies of db in the bundle.
With an error thrown at the top of the benchmark, the stack trace names scripts/bench/incremental-update.ts:1:7.
node --test scripts/bench/inputs-changed.test.mjs scripts/bench/compare-incremental-update.test.mjs: 22 of 22 pass.
One source-mode run and one production run on the same machine: the production writes had a geometric mean of 0.90× and cold hydration 0.97×. This is one run of each, so it shows only the direction.
Limits
Node servers that load db from node_modules without a bundler run code that this benchmark does not measure.
The CI gate thresholds were calibrated on source-mode runs. Production-bundle runs on CI have no separate calibration yet.
✅ Checklist
I have tested this code locally with pnpm test. I ran the benchmark's own tests and the bundled benchmark.
🚀 Release Impact
This change affects published code, and I have generated a changeset.
This change is docs/CI/dev-only (no release).
This pull request and its description were written by Isaac.
Summary by CodeRabbit
Chores
Benchmark workflows now build both packages and run measurements using production bundles. Base and candidate runs use the same benchmark scripts and freshness checks.
Runs require fresh package builds and support passing arguments to the benchmark.
Documentation
Updated benchmark instructions to cover the production runner and clarify how development-mode timings differ from production measurements.
The benchmark imported db's source through tsx, so it measured
development-mode code: unmangled private members, NODE_ENV checks and
development-only branches that every production app strips. A new
runner, scripts/bench/production.mjs, bundles the benchmark against the
built db and db-ivm dist with esbuild, minified, with
process.env.NODE_ENV set to production, as the size scenarios do. It
refuses a dist older than its package's sources. CI builds both
packages on each side and runs the bundle.
Co-authored-by: Isaac <no-reply@databricks.com>
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
No actionable comments were generated in the recent review.
Walkthrough
The benchmark runner now checks built package freshness, bundles the benchmark in production mode, and executes it. The workflow uses the runner for candidate and base benchmarks. The README documents the runner and clarifies the mode used for previously cited CI timings.
Changes: Production benchmark execution
Layer / File(s)
Summary
Bundle and run benchmark scripts/bench/production.mjs
The runner checks package distribution freshness, bundles the benchmark as minified Node ESM with production mode and built-package aliases, rejects bundles containing package source modules, and imports the bundle with forwarded arguments.
Workflow and documentation integration .github/workflows/benchmark.yml, scripts/bench/README.md
The candidate and base runs build both packages and use the runner. The README describes the production bundle and states that cited CI timings measured development-mode source.
sequenceDiagram
participant BenchmarkWorkflow
participant ProductionRunner
participant Esbuild
participant BuiltPackages
participant BenchmarkBundle
BenchmarkWorkflow->>ProductionRunner: pass benchmark path and arguments
ProductionRunner->>BuiltPackages: check distribution freshness
ProductionRunner->>Esbuild: bundle with production settings and package aliases
Esbuild->>BuiltPackages: resolve db and db-ivm imports
Esbuild->>BenchmarkBundle: write minified Node ESM bundle
ProductionRunner->>BenchmarkBundle: import bundle with forwarded arguments
Loading
Possibly related PRs
TanStack/db#2356: Changes the same benchmark and CI workflow with a nested workload, repeated reports, and regression gating.
TanStack/db#2171: Changes the transaction settlement contract and updates the incremental benchmark to observe settlement.
Suggested reviewers:kevin-dp
Merge Risk: Minimal · up to df5a7
The benchmark now uses built production bundles. No concrete merge-blocking risk remains.
Pre-merge checks: 4 · 1
Failed checks: 1 warning
Check name
Status
Explanation
Resolution
Description check
The description follows the repository template. It has the Changes, Oracle Coverage, Checklist, and Release Impact sections. The Changes section states what changed and why: the benchmark now measure…
Complete the "Oracle Coverage" section. For a non-bug change without a practical oracle, name the relevant coverage-map entry and explain why the owner cannot be extended, no independent oracle can be built, and no receiving witness can be …
Passed checks: 4 passed
Check name
Status
Explanation
Title check
The title "ci(bench): measure the incremental benchmark as a production bundle" matches the main change. The PR adds scripts/bench/production.mjs, which bundles the benchmark against built dist output…
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title "ci(bench): measure the incremental benchmark as a production bundle" matches the main change. The PR adds scripts/bench/production.mjs, which bundles the benchmark against built dist output with minification and NODE_ENV=production, and the CI workflow now runs that bundle. The title is short, specific, and uses a conventional commit prefix.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.)
Full details: Description check
Explanation
The description follows the repository template. It has the Changes, Oracle Coverage, Checklist, and Release Impact sections. The Changes section states what changed and why: the benchmark now measures a production bundle, not development-mode source imported through tsx. It lists the new runner, the freshness and source-inclusion guards, the CI changes with their cost, and the README update. The author also gives verification evidence and limits. The Oracle Coverage section is not filled in. It does not name a coverage-map owner or summarize oracle evidence, and it does not explain why no practical oracle exists. This section applies to behavior changes, so the gap is significant. The template asks for different content depending on whether the change is a bug fix or a non-bug change. The description also leaves the changeset checkbox unchecked and checks the docs/CI/dev-only box, which is consistent with the stated release impact.
Resolution
Complete the "Oracle Coverage" section. For a non-bug change without a practical oracle, name the relevant coverage-map entry and explain why the owner cannot be extended, no independent oracle can be built, and no receiving witness can be added. Also confirm that the maintainer-requested benchmark change has no observable behavior change that needs oracle evidence. If it has none, state that explicitly so the reviewer can verify it.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: f869f7ba-f4ab-4226-93eb-198554587e02
Commits
Reviewing files that changed from the base of the PR and between 7e1f6d4 and df5a78a.
Files selected for processing: 2
.github/workflows/benchmark.yml
scripts/bench/production.mjs
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review.
Comparing 8ebff4e74 (this PR) against c6a85dc0c (base). Write times are per-write medians over 100 iterations (20 warmup writes). Timings use the median of 3 fresh fixture repetition(s).
Overall median write time vs base: 1.08× · cold hydrate time: 1.06× (geometric mean of per-case ratios; lower is faster).
Writes: 1 regression(s), 0 improvement(s) (threshold: ±20% and >0.05ms).
Cold hydrate: 0 regression(s), 1 improvement(s) (threshold: ±50% and >5ms).
CI regression gate: PASS. Query geometric means: >+30% writes or >+50% cold. A case also fails when every repetition exceeds the entire base range by >+50% and the metric's absolute floor.
Per-case flags are noisy on shared runners. Read the geometric means first.
Writes
Query
Δ median write time (geomean)
Best case
Worst case
Flags
list: newest 50 open
+2.4%
-38.5%
+57.6%
—
list + author
+10.3%
-0.1%
+55.0%
—
list + comment count
+10.3%
-3.4%
+30.6%
—
list + 3 recent comments
+5.9%
-2.8%
+26.1%
—
issue detail + comments
+6.0%
-11.3%
+67.1%
—
open issues + documents
+21.2%
-2.8%
+113.4%
1 🔴
Cold hydrate
Query
Δ cold hydrate time (geomean)
Best case
Worst case
Flags
list: newest 50 open
+6.0%
-30.8%
+56.6%
—
list + author
-5.0%
-78.9%
+69.2%
1 🟢
list + comment count
+14.6%
-22.5%
+168.4%
—
list + 3 recent comments
-0.8%
-35.5%
+16.6%
—
issue detail + comments
+13.7%
-5.5%
+39.2%
—
open issues + documents
+11.6%
-4.7%
+40.6%
—
Each row aggregates that query's scale/index/write-mode configurations; per-configuration tables below.
Runner: node v24.8.0, linux 7.0.0-1017-azure, AMD EPYC 9V45 96-Core Processor. Timings on shared CI runners are noisy; treat small deltas as indicative only.
Review fixes for the production benchmark runner:
- The runner fails when esbuild's metafile lists any packages/*/src
input. Before, a second import of db source bundled a development
copy of db beside its dist without notice (a planted import of
packages/db/src/collection/index.js ran to exit 0; it now fails and
names the source files).
- The bundle carries an inline source map, and the runner enables
source maps before it imports the bundle, so a failed assertion names
its line in incremental-update.ts. The bundle directory uses the real
temp path, because macOS's /var symlink offset the map's relative
paths.
- Both freshness checks run before the bundle directory is created.
- The base step also takes the freshness helper from the PR head, so
both sides run the same runner code.
Co-authored-by: Isaac <no-reply@databricks.com>
…ade #2374)
Conflicts resolved as the standing rule says: main's tooling, with this
branch's oracle changes reapplied.
- package.json, packages/db/package.json: main's dependency versions,
plus the @tanstack/oracles and @tanstack/eslint-plugin-oracles
previews. fast-check stays at ^4.10.2 here and in
db-adapter-conformance.
- packages/db-ivm/vite.config.ts: main's publishableModuleOutput build
output, plus the stripOracleProbes() plugin.
- electric-sdk-delivery-oracle: main's resume-checkpoint check and
model, plus this branch's owns entry.
- solid-db row-materialization-oracle: main's keyed visibility
histories, plus this branch's case route.
- pnpm-lock.yaml: resolved from the public registry by a temporary CI
job (run 42653737079, which also checked a frozen install), because
the local registry proxy does not yet carry #2374's versions.
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
The incremental update benchmark now measures TanStack DB as a production app runs it. Before this change, the benchmark imported db's source through
tsx. It therefore measured development-mode code that every production build removes:process.env.NODE_ENVchecks, development-only branches, and private members without their mangled names.scripts/bench/production.mjs, bundles a benchmark with esbuild against the builtdistof db and db-ivm. The bundle is minified, andprocess.env.NODE_ENVis set toproduction. These are the settings that the size scenarios inscripts/size-compare.mjsuse.distthat is older than its package's sources. It also fails if the bundle includes any module from a package'ssrc, so a benchmark cannot measure a development copy of a package beside itsdist.incremental-update.ts.tscrun does not change.The maintainer asked for this change on 2026-11-09. The trigger was #2378: it flagged 5 writes in 100-row configurations. Those writes ran development-only state probes that a production bundle removes.
Verification
packages/db/distand 28 files frompackages/db-ivm/dist. It has no inputs fromsrc, and the output contains noNODE_ENVreads.scripts/bench/incremental-update.ts:1:7.node --test scripts/bench/inputs-changed.test.mjs scripts/bench/compare-incremental-update.test.mjs: 22 of 22 pass.Limits
node_moduleswithout a bundler run code that this benchmark does not measure.✅ Checklist
pnpm test. I ran the benchmark's own tests and the bundled benchmark.🚀 Release Impact
This pull request and its description were written by Isaac.
Summary by CodeRabbit