Skip to content

[Pipe] Support pooled TsFile parsing - #18482

Open
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:feat/pipe-tsfile-parser-pool
Open

[Pipe] Support pooled TsFile parsing#18482
Caideyipi wants to merge 4 commits into
apache:masterfrom
Caideyipi:feat/pipe-tsfile-parser-pool

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Pooled TsFile parsing

  • Adds the per-pipe processor attribute processor.tsfile-parser.parallelism.
  • Runs built-in DoNothingProcessor TsFile parsing in a shared parser pool while keeping custom processors single-threaded.
  • Uses the existing global and per-pipe-region parser admission limits, with non-positive per-pipe-region values following the global limit.

Ordering, lifecycle, and memory safety

  • Preserves ordered progress reporting across concurrently parsed TsFiles and keeps dependent events behind the parser barrier.
  • Allows heartbeat and progress-report control events to bypass the parser barrier.
  • Preserves parser state across retryable failures and handles STOP/START or task close without losing the current TsFile.
  • Bounds parsed-tablet output queue memory and reserves downstream tablet capacity to avoid parser/sink memory deadlocks.

Scope

This ports the TsFile parser pooling changes from the hotfix branch to master. The TsFile local sink implementation, writer, sink-specific tests, and the local-sink-based parser pool IT are intentionally excluded.

Verification

.\mvnw.cmd -pl iotdb-core/datanode -am -Dtest=UnboundedBlockingPendingQueueTest,PipeProcessorSubtaskExecutorTest,PipeRawTabletInsertionEventTest,PipeMemoryManagerTest,PipeEventCollectorTest -Dsurefire.failIfNoSpecifiedTests=false test

Result: 28 reactor modules succeeded. Node-commons tests: 3 passed. Datanode tests: 19 passed. Checkstyle and Spotless passed.


This PR has:

  • been self-reviewed.
  • considered concurrent read and write behavior.
  • added or updated configuration documentation.
  • added comments explaining non-obvious ordering, retry, and memory behavior.
  • added or updated unit tests for the new code paths.

Key changed/added classes
  • PipeProcessorSubtask
  • PipeEventCollector
  • PipeTsFileInsertionEvent
  • PipeRawTabletInsertionEvent
  • PipeMemoryManager
  • BlockingPendingQueue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant