Skip to content

[To dev/1.3] [Pipe] Support pooled TsFile parsing - #18483

Open
Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:feat/pipe-tsfile-parser-pool-dev-1.3
Open

[To dev/1.3] [Pipe] Support pooled TsFile parsing#18483
Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:feat/pipe-tsfile-parser-pool-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Pooled TsFile parsing

  • Adds opt-in processor.tsfile-parser.parallelism for the built-in do-nothing processor.
  • Runs admitted TsFile parser tasks in a shared executor while respecting the existing global and per-Pipe-region parser admission limits.
  • Keeps custom processors and the default configuration serial.

Ordering, retry, and lifecycle

  • Lets the source TsFile own the ordered commit id while parsed tablets retain its progress reference, so progress is committed only after all generated output is released.
  • Preserves parser progress across retryable failures, retries permanent failures through the existing subtask retry path, and releases in-flight or pending events on close.
  • Allows progress-report and heartbeat control events to pass the parser barrier without splitting a run of parseable TsFiles.

Parser output memory backpressure

  • Byte-accounts parsed tablets waiting in the sink queue and reserves downstream memory headroom before publishing them, preventing parallel parsers from exhausting the memory needed to drain the queue.
  • Accounts tablet sink batches as tablet memory and releases queue reservations on poll, clear, and discard paths.
  • Adds focused tests for parser concurrency, retry, progress ownership, memory admission, and pending-queue reservation release.

Scope

This PR intentionally does not include the TsFile local sink, its writer/tests, or the source-TsFile grouping metadata used only by that sink. Configurable TsFile parser selection is already present on dev/1.3 via #18449 and is not duplicated here.

Verification

  • mvn spotless:apply -DskipTests
  • mvn -pl iotdb-core/datanode -am '-Dtest=PipeProcessorSubtaskExecutorTest,PipeRawTabletInsertionEventTest,PipeMemoryManagerTest,PropertiesTest' '-Dsurefire.failIfNoSpecifiedTests=false' '-DskipITs' '-Dcheckstyle.skip=true' '-Dspotless.check.skip=true' test (26 tests, 0 failures)
  • mvn -pl iotdb-core/node-commons -am '-Dtest=BlockingPendingQueueTest' '-Dsurefire.failIfNoSpecifiedTests=false' '-DskipITs' '-Dcheckstyle.skip=true' '-Dspotless.check.skip=true' test (2 tests, 0 failures)
  • mvn -pl 'iotdb-core/node-commons,iotdb-core/datanode' -am '-DskipTests' '-Dspotless.check.skip=true' checkstyle:check (0 violations)

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for non-trivial methods.
  • added comments explaining intent where the concurrency and memory behavior is not obvious.
  • added or updated unit tests for new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeProcessorSubtask
  • PipeEventCollector
  • PipeRawTabletInsertionEvent
  • PipeTsFileInsertionEvent
  • PipeMemoryManager
  • BlockingPendingQueue / UnboundedBlockingPendingQueue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant