Skip to content

Add NATS JetStream as a message queue provider - #1053

Open
rohanshirsath-droadmap wants to merge 1 commit into
hookdeck:mainfrom
rohanshirsath-droadmap:add-nats-jetstream-provider
Open

Add NATS JetStream as a message queue provider#1053
rohanshirsath-droadmap wants to merge 1 commit into
hookdeck:mainfrom
rohanshirsath-droadmap:add-nats-jetstream-provider

Conversation

@rohanshirsath-droadmap

Copy link
Copy Markdown

Closes #1052

Summary

Adds NATS JetStream as a 5th internal MQ option, alongside RabbitMQ/SQS/GCP
PubSub/Azure Service Bus — following the exact same MQConfigAdapter/MQInfra
interface pattern the existing providers use. Purely additive: no existing
provider's code is touched.

Why NATS

JetStream is a single lightweight Go binary with built-in persistence — no
separate broker cluster to operate. Good fit for teams already running lean,
Kubernetes-native infrastructure who don't want to stand up RabbitMQ just
for this.

What's included (per the MQ contribution checklist)

  • internal/mqs interface implementation (queue_nats.go)
  • internal/mqinfra interface implementation (nats.go)
  • Unit tests for config DLQ-name derivation, queue defaults, and
    connect-failure handling
  • .env.example entries
  • contributing/mq.md updated (Supported MQs list + a NATS behavior
    section matching the AWS SQS/RabbitMQ ones)

Testing

Running in production against a real multi-tenant webhook gateway,
verified end to end (publish → durable delivery → ack). All existing
tests pass unmodified (go test -short ./...).

Adds NATS JetStream as a 5th internal MQ option alongside RabbitMQ/SQS/GCP
PubSub/Azure Service Bus, following the exact same MQConfigAdapter/MQInfra
interface pattern as the existing providers.

- internal/config: NATSConfig, wired into MQsConfig same as the others
- internal/mqinfra: stream/consumer provisioning, explicit DLQ handling
  since JetStream has no broker-native dead-letter mechanism
- internal/mqs: NATSQueue implementing the Queue interface
- Unit tests for config DLQ-name derivation and queue defaults/connect
  failure, matching the existing RabbitMQ test coverage
- .env.example and contributing/mq.md updated per the MQ contribution
  checklist

Why NATS: JetStream is a single lightweight Go binary with built-in
persistence, no separate broker cluster to operate — a good fit for teams
already running lean, Kubernetes-native infrastructure who don't want to
stand up RabbitMQ just for this. Verified running in production against a
real multi-tenant webhook gateway.
@rohanshirsath-droadmap

Copy link
Copy Markdown
Author

@alexluong @alexbouchardd would appreciate a look when you have a moment — this closes #1052 and follows the existing MQ provider pattern closely, happy to adjust based on any feedback.

@alexbouchardd alexbouchardd added enhancement New feature or request internalmq Relates to operational MQ labels Aug 31, 2026
@alexbouchardd

alexbouchardd commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Thanks! Agreed that NATS is a good natural fit. #910 already adds support, but multi-tenancy has caused some concerns. Is there a specific reason for a new PR?

We'll review and merge whichever seems most appropriate!

Just to set expectations, we have a big backlog of work; it might take some time.

@rohanshirsath-droadmap

Copy link
Copy Markdown
Author

@alexbouchardd Thank you for your fast response and bringing up #910!

Reviewed it carefully — I believe it solves two distinct things and not the
same slot.
#910 is a PublishMQ implementation (reading from a user's NATS environment), and the problem statement there itself has listed "Internal MQ support (delivery/log)" as "Out of scope". And this PR does solve just that
— provides NATS as an internal MQ provider for Outpost's delivery/log queues, just like RabbitMQ/SQS/GCP/Azure already do.

That also explains why the multi-tenancy issue need not be applied here — the complexity of #910 is about setting up a separate NATS connection per customer tenant (account-specific credentials, watched accounts directory) to aggregate as a publish source.

According to Alex's comment in #910, the design principle that needs to be followed is "Outpost consumes from one
publish source over one connection" — and per-tenant/per-account connections will increase with the tenants * replicas count. This PR is exactly this "single connection" design pattern but for a different integration point: one NATS connection per Outpost deployment, similar to how RabbitMQ is consumed currently.

To add more, even putting aside the differences between #910 and publish-mq, the multi-tenancy approach of this PR is not new to — it is already what RabbitMQ uses currently (one shared exchange/queue for all Outpost tenants, and isolation handled by destination routing layer of Outpost, not by the message broker).

No urgency on the backlog at all!

@alexbouchardd

Copy link
Copy Markdown
Contributor

Valid! We'll get back to this PR as soon as soon as we are done with some other big pieces of work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request internalmq Relates to operational MQ

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add NATS JetStream as a message queue provider

2 participants