Skip to content

JITSU-294: Support multi-partition Bulker batch topics - #1641

Open
absorbb wants to merge 2 commits into
newjitsufrom
feat/jitsu-294-multipartition-batch
Open

absorbb wants to merge 2 commits into
newjitsufrom
feat/jitsu-294-multipartition-batch

Conversation

@absorbb

@absorbb absorbb commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Large workspace backups currently concentrate writes on a single Kafka partition. JITSU-294 lets selected batch topics use multiple partitions while retaining one scheduled destination writer and a one-partition default.

  • Require an ordinary batch member to own every partition of its topic before writing. Overlapping logical-shard instances with split assignments fail closed until one member has full ownership; retry groups still support shared assignments.
  • Track batch windows and checkpoints by topic/partition. Commit only processed partitions after destination completion; replay and retry/dead-letter transactions carry all affected offsets. Epoch fencing remains in place, and failed Kafka commits restart/replay instead of rewriting group offsets through the admin API.
  • Add BULKER_KAFKA_BATCH_TOPIC_PARTITIONS, an exact-topic JSON allowlist for partition counts. Existing-topic expansion additionally requires BULKER_KAFKA_BATCH_TOPIC_PARTITION_EXPANSION=true. Retry groups can handle more partitions than members, and stale-topic detection uses the newest partition timestamp.
  • Synchronize retry-topic checks with metadata refresh, fixing a pre-existing concurrent map/readiness race exposed by the integration race run.
  • Preserve aggregate queue metrics and add per-partition lag. Include a staged rollout/rollback guide and real Kafka coverage for destination failure, retry-publication failure, mixed due/future retries, and live partition expansion.

Validation (from bulker/bulkerapp):

  • go test ./app -count=1 — full Kafka/Postgres integration and unit suite.
  • Targeted batch, consumer runtime, partition provisioning and topic-map tests with -race.
  • go build ./....

The initial full -race integration run exposed the application race fixed here and an independent race in compose-go v2.12.1 (WithServicesTransform) used by the Docker test fixture. Do not treat the full race suite as green. Static-member replacement tests remain conditional on BULKER_TEST_KAFKA_BOOTSTRAP_SERVERS.

Rollout: deploy compatible consumers everywhere before enabling expansion. Kafka cannot shrink an expanded topic; do not roll back to single-partition-only consumers. Cross-partition ordering is not guaranteed and delivery remains at-least-once. Production multi-broker load comparison and a large-workspace canary remain rollout gates; this PR does not change production partition counts.

jitsu-code-review[bot]
jitsu-code-review Bot previously approved these changes Nov 17, 2026

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 58046da5cf5d1a36e219a49b92604b55ecf3ae82

Reviewed the full diff and found one data-loss issue. Targeted tests passed on rerun after a mock-cluster timeout; real-Kafka integration tests were not run.

Comment thread bulker/bulkerapp/app/topic_manager.go Resolved

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 9bed1a73f0983174d55439a2f24de70a709f3ba6

Reviewed the full diff and prior discussion. No high-confidence defects found. Focused batch, offset, partition configuration, and ownership tests passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant