Skip to content

fix(compression): dedupe large blocks by full content - #621

Merged
VickyXAI merged 1 commit into
BlockRunAI:mainfrom
Tyagiquamar:fix/observation-dedup-full-content
Nov 19, 2026
Merged

VickyXAI merged 1 commit into
BlockRunAI:mainfrom
Tyagiquamar:fix/observation-dedup-full-content

Conversation

@Tyagiquamar

@Tyagiquamar Tyagiquamar commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Large-block deduplication in the observation compression layer now compares full message content, so two messages that merely share a leading 200 characters are no longer collapsed into a false "same content" pointer.

Root Cause & Problem

deduplicateLargeBlocks in src/compression/layers/observation.ts used msg.content.slice(0, 200) as the block identity key. Any two long messages with the same first 200 characters were treated as identical, and the later one was replaced with [See message #N - same content]. The text after the prefix was discarded, even though it can carry distinct and important information. The emitted pointer also asserted "same content" when the content was not actually the same.

Technical Solution

Use the complete msg.content string as the map key. This keeps the map-based exact-match behavior and removes the prefix-only false positives, with no new dependencies. Added src/compression/layers/observation.test.ts with two cases: one asserts that distinct messages sharing the first 200 characters both survive compression, and one asserts that genuinely identical blocks are still collapsed. The first case fails on the original code with expected '[See message #1 - same content]' to contain 'BETA_UNIQUE'.

Test Plan

Commands run on Windows with Node 24.22.1:

  • npx vitest run src/compression/layers/observation.test.ts on the unfixed code: FAILED as expected, 1 failed / 1 passed, with AssertionError: expected '[See message #1 - same content]' to contain 'BETA_UNIQUE'.
  • npx prettier --check src/compression/layers/observation.ts src/compression/layers/observation.test.ts: exit code 0, All matched files use Prettier code style! (after prettier --write on the same two files).
  • npx eslint src/compression/layers/observation.ts src/compression/layers/observation.test.ts: exit code 0, no diagnostics.
  • npm run typecheck (tsc --noEmit): exit code 0, no errors.
  • npx vitest run src/compression/layers/observation.test.ts after the fix: PASS, 2 passed / 2 passed.

Only the two changed files were touched. No new dependencies were added. The full repository test suite was not run locally.

Summary by CodeRabbit

  • Bug Fixes
    • Distinct long messages that share the same first 200 characters are no longer mistakenly treated as duplicates. Truly repeated messages continue to be marked as identical.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Large observation messages are now deduplicated by full content rather than by the first 200 characters. Tests cover distinct messages with a shared prefix and identical messages.

Changes: Observation deduplication

Layer / File(s) Summary
Full-content matching and tests
src/compression/layers/observation.ts, src/compression/layers/observation.test.ts
deduplicateLargeBlocks uses the full message content as its matching key. Tests verify that messages with a shared 200-character prefix but different tails remain distinct, while identical messages are marked "same content".

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · up to 97584

Large observation messages that share an opening but differ afterwards are no longer replaced by a "same content" pointer, so their text is preserved. Exact duplicates are still collapsed. The change is small, tested, and easy to revert, so merge risk is minimal.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the change: deduplicating large blocks by full content in the compression layer. This matches the summary, which changes deduplicateLargeBlocks from a 200-character prefix key to the f…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the change: deduplicating large blocks by full content in the compression layer. This matches the summary, which changes deduplicateLargeBlocks from a 200-character prefix key to the full message content, and the objectives, which describe the same fix. The title is specific and concise, follows the conventional commit style, and identifies the primary change.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between aa91d65 and 9758487.

Files selected for processing: 2

  • src/compression/layers/observation.test.ts
  • src/compression/layers/observation.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@VickyXAI VickyXAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: full-content key fixes the prefix false-positive; compression tests pass on Node 24. Thanks!

@VickyXAI
VickyXAI merged commit 443e7fa into BlockRunAI:main Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants