Add NATS JetStream as a message queue provider - #1053
Add NATS JetStream as a message queue provider#1053rohanshirsath-droadmap wants to merge 1 commit into
Conversation
Adds NATS JetStream as a 5th internal MQ option alongside RabbitMQ/SQS/GCP PubSub/Azure Service Bus, following the exact same MQConfigAdapter/MQInfra interface pattern as the existing providers. - internal/config: NATSConfig, wired into MQsConfig same as the others - internal/mqinfra: stream/consumer provisioning, explicit DLQ handling since JetStream has no broker-native dead-letter mechanism - internal/mqs: NATSQueue implementing the Queue interface - Unit tests for config DLQ-name derivation and queue defaults/connect failure, matching the existing RabbitMQ test coverage - .env.example and contributing/mq.md updated per the MQ contribution checklist Why NATS: JetStream is a single lightweight Go binary with built-in persistence, no separate broker cluster to operate — a good fit for teams already running lean, Kubernetes-native infrastructure who don't want to stand up RabbitMQ just for this. Verified running in production against a real multi-tenant webhook gateway.
|
@alexluong @alexbouchardd would appreciate a look when you have a moment — this closes #1052 and follows the existing MQ provider pattern closely, happy to adjust based on any feedback. |
|
Thanks! Agreed that NATS is a good natural fit. #910 already adds support, but multi-tenancy has caused some concerns. Is there a specific reason for a new PR? We'll review and merge whichever seems most appropriate! Just to set expectations, we have a big backlog of work; it might take some time. |
|
@alexbouchardd Thank you for your fast response and bringing up #910! Reviewed it carefully — I believe it solves two distinct things and not the That also explains why the multi-tenancy issue need not be applied here — the complexity of #910 is about setting up a separate NATS connection per customer tenant (account-specific credentials, watched accounts directory) to aggregate as a publish source. According to Alex's comment in #910, the design principle that needs to be followed is "Outpost consumes from one To add more, even putting aside the differences between #910 and publish-mq, the multi-tenancy approach of this PR is not new to — it is already what RabbitMQ uses currently (one shared exchange/queue for all Outpost tenants, and isolation handled by destination routing layer of Outpost, not by the message broker). No urgency on the backlog at all! |
|
Valid! We'll get back to this PR as soon as soon as we are done with some other big pieces of work |
Closes #1052
Summary
Adds NATS JetStream as a 5th internal MQ option, alongside RabbitMQ/SQS/GCP
PubSub/Azure Service Bus — following the exact same MQConfigAdapter/MQInfra
interface pattern the existing providers use. Purely additive: no existing
provider's code is touched.
Why NATS
JetStream is a single lightweight Go binary with built-in persistence — no
separate broker cluster to operate. Good fit for teams already running lean,
Kubernetes-native infrastructure who don't want to stand up RabbitMQ just
for this.
What's included (per the MQ contribution checklist)
internal/mqsinterface implementation (queue_nats.go)internal/mqinfrainterface implementation (nats.go)connect-failure handling
.env.exampleentriescontributing/mq.mdupdated (Supported MQs list + a NATS behaviorsection matching the AWS SQS/RabbitMQ ones)
Testing
Running in production against a real multi-tenant webhook gateway,
verified end to end (publish → durable delivery → ack). All existing
tests pass unmodified (
go test -short ./...).