Skip to content

Pass only the required Kafka consumer properties to the spout lag monitor - #8988

Merged
rzo1 merged 1 commit into
masterfrom
fix/spout-lag-consumer-property-filter
Aug 22, 2026
Merged

Pass only the required Kafka consumer properties to the spout lag monitor#8988
rzo1 merged 1 commit into
masterfrom
fix/spout-lag-consumer-property-filter

Conversation

@rzo1

@rzo1 rzo1 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

TopologySpoutLag copied every config.-prefixed key out of the submitted spout json_conf into the properties file handed to the storm-kafka-monitor process.

Only the properties the monitor actually reads are now forwarded, and anything dropped is logged by name so a lag display that stops working can be diagnosed. topics, groupid, bootstrap.servers and security.protocol are unaffected, since they travel via argv. Lag monitoring remains off by default. Adds TopologySpoutLagTest.

@rzo1 rzo1 added this to the 3.1.0 milestone Aug 21, 2026
@rzo1 rzo1 self-assigned this Aug 21, 2026

@reiabreu reiabreu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer: this comment was generated with the help of an LLM.

Approving — the allowlist approach is sound (forwarding only connection-relevant consumer properties is safe by construction). One compatibility note worth calling out in the release notes: because topology-supplied config.* properties are now filtered, any lag-monitor setup that relied on a topology-provided sasl.jaas.config (or another now-dropped property) to authenticate to the brokers will stop reporting lag after upgrade, until the monitor is given its own credentials on the UI host. It's the correct behavior, just a silent change for those setups.

@rzo1
rzo1 force-pushed the fix/spout-lag-consumer-property-filter branch from fe2b3f4 to a0ee4ff Compare August 22, 2026 16:54
@rzo1
rzo1 merged commit 27ae976 into master Aug 22, 2026
2 checks passed
@rzo1
rzo1 deleted the fix/spout-lag-consumer-property-filter branch August 22, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants