[SPIKE] Add plan for host audit ingestion in the primary instance - #5809
Draft
johnsimons wants to merge 8 commits into
Draft
[SPIKE] Add plan for host audit ingestion in the primary instance#5809johnsimons wants to merge 8 commits into
johnsimons wants to merge 8 commits into
Conversation
This design document outlines the architectural strategy and work plan for moving audit ingestion into the primary ServiceControl process for SQL Server and PostgreSQL persistence, including support for scaled-out ingestion-only workers.
First step of hosting audit ingestion in the primary instance. Contracts only, no runtime behavior change: nothing resolves or calls any of this yet, and no shipped persister advertises audit support. - IIngestionUnitOfWork gains a nullable Audit child alongside Monitoring and Recoverability, so a batch can record audit messages and saga snapshots in the same transaction as known endpoints. - New query contracts for the two entry points that are remote-only today: IAuditCountsDataStore and ISagaHistoryDataStore. - IFailedAuditImportDataStore mirrors IFailedErrorImportDataStore, and FailedAuditImport.DeriveKey gives failed audit imports the same deterministic key with native-id fallback, so competing consumers do not write a row per delivery attempt. - PersistenceManifest gains SupportsAuditIngestion. Absent means false, so RavenDB and every legacy manifest are unaffected. Both EF manifests declare it explicitly false until EF audit persistence lands. - IMessagesViewDataStore documents the precedence, paging and counting rules a persister must honour once one local result set unions failed and audited messages. - ServiceControl.Persistence.Tests.AuditCapable is a test-only persister that advertises audit support and delegates everything else to a real persister, so later pull requests can compose and start an audit-capable primary host before any shipped persister is one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adapts the standalone audit instance's receiving, enrichment, fault handling, forwarding and metrics into the primary project, wired to primary persistence and primary settings. The standalone audit executable is behaviorally unchanged. What the copy does differently: - Endpoint detection writes through IMonitoringIngestionUnitOfWork instead of sending RegisterNewEndpoint through IMessageSession, the same way ErrorProcessor does. IMessageSession and the ICommand overload on the enricher context are gone, so the runtime composes in a host with no NServiceBus endpoint. - Audit messages, saga snapshots and detected endpoints commit through one primary IIngestionUnitOfWork. - Transport output is dispatched through the receiving infrastructure's own dispatcher, matching the error ingestion scale-out work. Retry acknowledgements stay transport operations, because the acknowledgement queue is named by whichever instance issued the retry. - ITransportCustomization.CreateTransportInfrastructure takes a per receiver concurrency, so audit ingestion keeps its own concurrency in a host whose shared TransportSettings carries the primary endpoint's. - The meter is renamed to Particular.ServiceControl and the primary gains the three OpenTelemetry packages plus OtlpEndpointUrl. - The copied custom checks are renamed so they do not collide with the standalone audit instance reporting into this same primary. AuditComponent registers all of it only when the configured persister advertises SupportsAuditIngestion, so no shipped configuration changes behavior. The composition tests run against the audit capable test persister and assert what the host does and does not register, including that disabling ingestion stops only the receiver. Shared projects touched: ServiceControl.Transports gains an optional argument with the existing behavior as its default, so the audit executable's receiver is created exactly as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds --audit-ingestion-only to the command line and Help.txt, with the three checks the plan calls for. The host itself is not composed yet, so the command fails after its checks pass; no shipped persister advertises audit support, so in practice it fails on the storage check. - Storage that does not advertise SupportsAuditIngestion is rejected by name, driven by the manifest rather than by resolving optional services. RavenDB falls out of this without a special case. - The two ingestion only modes cannot be combined. Each queue gets its own worker pool so they can be scaled independently. - File system body storage is rejected unless ServiceControl/MessageBody/FileSystem/PathIsShared asserts the path is a shared mount. Nothing in the file system settings distinguishes a shared mount from a node local directory, so the operator has to say so. The same check now guards --error-ingestion-only, which PR #5801 documented as a known gap. No installer changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two audit entry points that were remote only become ordinary scatter
gather APIs over the new contracts, so a primary holding audit data
answers them from its own storage and still merges in any remotes.
Routes and authorization policies are unchanged: audit counts stay on
error:messages:view and sagas/{id} stays on error:sagas:view, which is
what a primary with an audit remote already serves them under.
- GetAuditCountsForEndpointApi drops the IMessagesViewDataStore it never
used, along with the comment saying it would never be implemented here.
- GetSagaByIdApi stops deriving from ScatterGatherRemoteOnly, which
nothing else used, so that base and its NoOpStore are gone.
- A persister without audit support falls back to an empty local source,
registered with TryAdd after the persister, so those hosts behave
exactly as before and the APIs still resolve.
LocalMessagesView.Merge puts the precedence, paging and counting rules in
one place for any persister that returns failed and audited messages from
one query: the failed row wins for a message that both failed and was
audited, the total counts it once, and the local result is already one
page. MessageViewComparer moves to the persistence project so a local
merge can order rows the same way the scatter gather does.
IBodyStorage documents the arbitration order for a store holding both
kinds of body, and AuditBodyKeyspace prefixes audit bodies so they do not
collide with failed message bodies in the same store.
The audit capable test persister implements all of it, so the rules have
a running implementation before the EF work starts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two primary owned capabilities depend on an audit remote existing today, and both break once a primary holds audit data itself. /api/connection is what ServicePulse and the Platform Connector plugin read to configure endpoints. With no audit remote it stops advertising MessageAudit.AuditQueue and SagaAudit.SagaAuditQueue, so endpoints cannot be told where to send audit or saga data at all. An audit capable primary now supplies the same two shapes the audit instance supplies, so nothing downstream sees a difference. SagaUpdatedHandler resolves the audit queue through the same IPlatformConnectionBuilder as before, reading either a remote's parsed JSON or the local provider's object, so a misdirected saga audit message is forwarded rather than failed. Audit throughput collection is likewise driven entirely by remotes. With local audit and no remotes, AuditQueues stays empty and the local audit and audit.log queues are counted as customer endpoints in the licensing throughput report, which is an accuracy defect rather than cosmetic, and AuditServicesData comes out blank. IAuditQuery gains an optional local source that contributes the local queue names, version, transport and retention alongside the remotes, so the existing platform endpoint exclusion, service metadata and connection diagnostics all work unchanged. Retention is reported exactly as configured. Where ServiceControl/AuditRetentionPeriod is unset the existing minimum retention gate warns, rather than this guessing a default: what null means is still an open item on the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the host --audit-ingestion-only runs: an audit receiver, the endpoint monitor it needs, and health endpoints, with no NServiceBus endpoint and none of the work a deployment may only do once. The component list is HeartbeatMonitoring and CustomChecks only. HeartbeatMonitoring is there because the audit enricher asks the endpoint monitor whether an endpoint is new before recording it, so without it every audited message writes a known endpoint upsert. CustomChecks is there so a stuck worker reports somewhere. EventLog and ExternalIntegrations are absent because audit ingestion raises no domain events and no integration events; the composition test states that, so a future registration forces a decision rather than slipping in. The ingestion only conditions scattered through the host become one Settings.IngestionOnly, so both modes agree on what they do not do: no endpoint, no retention sweep, no platform connection providers, no heartbeat checking. Ingestion health checks move to the components that host the ingestion they report on, so this host answers for audit-ingestion and not for error-ingestion, and the error ingestion only host is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/audit-ingestion-in-the-primary.md covers the three deployment modes, the settings and their key collisions with a standalone audit instance, queue ownership and what ingestion only workers never provision, the body storage rule, the health endpoints, and the query behavior including the precedence, paging and counting rules. The primary logs a warning at startup when it both ingests audit messages and has audit remotes configured, because that is the shape most likely to hit the setting collisions the doc describes. An architecture test pins the boundary the plan depends on: the primary must not reference ServiceControl.Audit, which stays a standalone composition root. The packaging tests confirm the copied runtime ships inside the existing primary artifact, that the three new OpenTelemetry references change no deployment unit, and that the persisters folder is still exactly the four shipped storages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is just a SPIKE to see what it would take to get audit ingestion hosted in the same process as error ingestion to simplify deployments and configuration.
This spike is only for EF implementation, the architecture for Raven is unchanged.
This spike, is composed of 7 commits that can be reviewed individually.
This PR does not actually implement EF storage for audits; it just adds the foundation for that work to be done next.
This code was written entirely by Claude Opus 5 and needs to be thoroughly reviewed, but the gist is correct.