diff --git a/docs/audit-ingestion-in-the-primary.md b/docs/audit-ingestion-in-the-primary.md new file mode 100644 index 0000000000..0d9b03dc88 --- /dev/null +++ b/docs/audit-ingestion-in-the-primary.md @@ -0,0 +1,114 @@ +# Audit ingestion in the primary instance + +## Overview + +Storage that advertises `SupportsAuditIngestion` in its `persistence.manifest` can hold audit data +alongside the primary's own data, which lets the primary ServiceControl process ingest the audit +queue itself instead of relying on a separate ServiceControl.Audit instance. + +The standalone RavenDB audit instance is unaffected. RavenDB does not advertise audit support, does +not gain combined hosting, and keeps its own executable, settings, API and installers. + +No shipped persister advertises audit support yet, so on every existing deployment the audit +component registers nothing and behavior is unchanged. + +## Deployment modes + +| Mode | How | What it runs | +| --- | --- | --- | +| Normal primary, audit ingestion on | Default where the persister advertises audit support | The audit receiver, the audit capabilities, the primary API and everything a normal primary runs | +| Normal primary, audit ingestion off | `ServiceControl/IngestAuditMessages=false` | Everything above except the audit receiver. Local audit queries, failed audit tooling and `/api/connection` stay active, because other processes may still be ingesting | +| Audit ingestion only | `ServiceControl.exe --audit-ingestion-only` | The audit receiver, the endpoint monitor it depends on, this node's custom checks, and the health endpoints. No NServiceBus endpoint, no API, no retention, no licensing | + +`--audit-ingestion-only` and `--error-ingestion-only` cannot be combined. Each queue gets its own +worker pool so the two can be scaled independently, so run one process per mode. + +## Settings + +The primary reads the audit settings under the same key names the audit instance uses, so an audit +capable primary is configured exactly the way an audit instance is configured today. + +| Setting | Default | Notes | +| --- | --- | --- | +| `ServiceControl/IngestAuditMessages` | `true` | Applies to the normal primary only. Always on under `--audit-ingestion-only`, and has no effect where the persister does not support audit | +| `ServiceBus/AuditQueue` | `audit` | The queue this instance drains | +| `ServiceBus/AuditLogQueue` | the subscoped audit queue name | Only used when forwarding is on | +| `ServiceControl/ForwardAuditMessages` | `false` | | +| `ServiceControl/AuditRetentionPeriod` | unset | Already existed. Validated between 1 hour and 365 days | +| `ServiceControl/MaximumAuditIngestionConcurrencyLevel` | `32` | Independent of the primary endpoint's concurrency, which is what `MaximumConcurrencyLevel` sets | +| `ServiceControl/TimeToRestartAuditIngestionAfterFailure` | 60 seconds | Mirrors the error equivalent | +| `ServiceControl/OtlpEndpointUrl` | unset | Enables the OpenTelemetry metrics exporter | +| `ServiceControl/MessageBody/FileSystem/PathIsShared` | `false` | Required by both ingestion only modes when body storage is the file system | + +### Setting collisions + +`ServiceControl` and `ServiceControl.Audit` settings can both be set by bare environment variable +name, and `ServiceBus/AuditQueue` is literally the same key for both processes. A combined primary +and a standalone audit instance sharing one environment file therefore collide on +`INGESTAUDITMESSAGES`, `AUDITRETENTIONPERIOD`, `FORWARDAUDITMESSAGES` and `SERVICEBUS_AUDITQUEUE`. + +That combination is unsupported. The primary logs a warning at startup when it has audit ingestion +enabled and audit remotes configured at the same time, because that is the shape most likely to hit +the collision. + +## Queue ownership + +The setup path creates the audit queue, and the audit forwarding queue when forwarding is enabled. +Ingestion only workers run no installers: they never create queues, never apply database migrations +and never provision body storage. Run setup from a normal instance before starting any worker. + +Transport operations remain in the audit ingestion path for two reasons only: + +- **Forwarding**, when `ForwardAuditMessages` is on. +- **Retry acknowledgements**. `ServiceControl.Retry.AcknowledgementQueue` is stamped by whichever + instance issued the retry, so the acknowledgement cannot be short-circuited into the local + database. In a combined host it is dispatched to the local error queue and comes straight back in + through local error ingestion, which is exactly what happens today. + +Endpoints detected from audit headers are written straight to the shared `KnownEndpoints` table +through the ingestion unit of work, rather than sent to the primary's input queue. + +## Body storage + +Audit and failed message bodies share one store, and each owns a prefixed keyspace, so an edited +message's failed body and its audited body do not collide. `IBodyStorage.TryFetch` resolves in a +fixed order: failed message by `UniqueMessageId`, then failed message by `MessageId`, then audit +message by `UniqueMessageId`. + +Every ingesting process must write bodies somewhere every host can read. Blob and S3 storage +qualify. File system storage qualifies only if the path is a shared mount, which nothing in the +settings can detect, so both ingestion only modes refuse to start unless +`ServiceControl/MessageBody/FileSystem/PathIsShared` asserts it. + +## Health endpoints + +Both ingestion only hosts map the same two routes, anonymously, returning JSON: + +- `/health` is liveness. It answers "is this process still serving" and is what a container health + check should restart on. +- `/health/ready` additionally reports whether the ingestion this host exists to do is happening. + An audit ingestion only host answers for `audit-ingestion` and not for `error-ingestion`. + +## Querying + +Local audit data is served through the existing primary routes under their existing policies: +`/api/messages` and its variants on `error:messages:view`, `/api/sagas/{id}` on +`error:sagas:view`, and `endpoints/{endpoint}/audit-count` on `error:messages:view`. A primary +configured with an audit remote already serves that remote's audit data under those policies today, +so nothing about the `my/routes` manifest or ServicePulse navigation changes. + +Additional audit remotes keep working. The scatter gather runs the local query first and merges the +remotes after, so a primary can hold audit data locally, query remotes, or both. + +Where one local result set contains both failed and audited messages, three rules apply, and +`LocalMessagesView.Merge` implements them for any persister: + +1. **Precedence.** A message that both failed and was audited shows as failed. +2. **Paging.** The local result is already at most one page, after deduplication. +3. **Counting.** A message that both failed and was audited is counted once. + +## Packaging + +The audit runtime ships inside the existing primary artifact. There is no new assembly and no new +deployment unit. The primary gains three OpenTelemetry package references, which the copied +ingestion metrics use, exported only when `OtlpEndpointUrl` is set. diff --git a/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs b/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs index 0c35b77398..149e4b1106 100644 --- a/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs +++ b/src/Particular.LicensingComponent.UnitTests/AuditQuery_Tests.cs @@ -74,6 +74,43 @@ public async Task Should_return_audit_remotes() } } + [Test] + public async Task Should_return_the_local_audit_source_alongside_the_remotes() + { + //Arrange + var auditQuery = new AuditQuery(NullLogger.Instance, new FakeEndpointApi(), new FakeAuditCountApi(), + new ConfigurationApi_ReturningOneValidAuditConfig(), new LocalAuditSource_ForThisInstance()); + + //Act + var remotes = await auditQuery.GetAuditRemotes(); + + //Assert + Assert.That(remotes, Has.Count.EqualTo(2), "The local audit source and the remote should both be reported"); + + var local = remotes.Single(remote => remote.ApiUri == "http://localhost:33333/api/"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(local.Queues, Does.Contain("audit"), "the local audit queue must be recognised as a platform endpoint"); + Assert.That(local.Queues, Does.Contain("audit.log")); + Assert.That(local.Transport, Is.EqualTo("LearningTransport"), "the report's audit service metadata is built from this"); + } + } + + [Test] + public async Task Should_not_report_a_local_audit_source_that_is_disabled() + { + //Arrange + var auditQuery = new AuditQuery(NullLogger.Instance, new FakeEndpointApi(), new FakeAuditCountApi(), + new ConfigurationApi_ReturningOneValidAuditConfig(), new LocalAuditSource_Disabled()); + + //Act + var remotes = await auditQuery.GetAuditRemotes(); + + //Assert + Assert.That(remotes, Has.Count.EqualTo(1)); + } + [Test] public async Task Should_return_successful_audit_connection_if_instances_exist_and_are_online() { @@ -204,6 +241,29 @@ public Task> GetEndpoints(CancellationToken cancellationToken = d } + class LocalAuditSource_ForThisInstance : ILocalAuditSource + { + public bool Enabled => true; + + public RemoteInstanceInformation Describe() => new() + { + ApiUri = "http://localhost:33333/api/", + VersionString = "6.0.0", + SemanticVersion = new NuGet.Versioning.SemanticVersion(6, 0, 0), + Status = "online", + Retention = TimeSpan.FromDays(10), + Queues = ["audit", "audit.log"], + Transport = "LearningTransport" + }; + } + + class LocalAuditSource_Disabled : ILocalAuditSource + { + public bool Enabled => false; + + public RemoteInstanceInformation Describe() => throw new InvalidOperationException("Describe must not be called when the source is disabled."); + } + class AuditCountApi_ReturningThreeAuditCounts : IAuditCountApi { public async Task> GetEndpointAuditCounts(string endpoint, CancellationToken cancellationToken = default) diff --git a/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs b/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs index 619a6ebae6..a4869ce996 100644 --- a/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs +++ b/src/Particular.LicensingComponent/AuditThroughput/AuditQuery.cs @@ -7,7 +7,7 @@ using ServiceControl.Api; using AuditCount = Contracts.AuditCount; - public class AuditQuery(ILogger logger, IEndpointsApi endpointsApi, IAuditCountApi auditCountApi, IConfigurationApi configurationApi) : IAuditQuery + public class AuditQuery(ILogger logger, IEndpointsApi endpointsApi, IAuditCountApi auditCountApi, IConfigurationApi configurationApi, ILocalAuditSource? localAuditSource = null) : IAuditQuery { // Customers are expected to run at least version 4.29 for their Audit instances public SemanticVersion MinAuditCountsVersion => new(4, 29, 0); @@ -45,6 +45,11 @@ public async Task> GetAuditRemotes(CancellationT var remotes = await configurationApi.GetRemoteConfigs(cancellationToken); var remotesInfo = new List(); + if (localAuditSource is { Enabled: true }) + { + remotesInfo.Add(localAuditSource.Describe()); + } + if (remotes.Any()) { List queues = []; diff --git a/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs b/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs new file mode 100644 index 0000000000..01afb4f805 --- /dev/null +++ b/src/Particular.LicensingComponent/AuditThroughput/ILocalAuditSource.cs @@ -0,0 +1,15 @@ +namespace Particular.LicensingComponent.AuditThroughput; + +using Particular.LicensingComponent.Contracts; + +/// +/// Audit throughput collection is driven entirely by audit remotes. A primary that holds audit data +/// itself has no remote to describe it, so without this its own audit queues are counted as customer +/// endpoints and the audit service metadata in the licensing report is blank. +/// +public interface ILocalAuditSource +{ + bool Enabled { get; } + + RemoteInstanceInformation Describe(); +} diff --git a/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs b/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs index 9e8db1f5e4..01637e50f7 100644 --- a/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs +++ b/src/Particular.LicensingComponent/LicensingComponentHostBuilderExtensions.cs @@ -1,4 +1,4 @@ -namespace Particular.LicensingComponent; +namespace Particular.LicensingComponent; using AuditThroughput; using BrokerThroughput; diff --git a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj index 028c471d49..4eafb776f3 100644 --- a/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj +++ b/src/ServiceControl.AcceptanceTests.PostgreSql/ServiceControl.AcceptanceTests.PostgreSql.csproj @@ -12,6 +12,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj index 7cc11bc96c..3413910f2e 100644 --- a/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj +++ b/src/ServiceControl.AcceptanceTests.RavenDB/ServiceControl.AcceptanceTests.RavenDB.csproj @@ -36,6 +36,8 @@ + + diff --git a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj index be951002d9..63e7d5f98f 100644 --- a/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj +++ b/src/ServiceControl.AcceptanceTests.SqlServer/ServiceControl.AcceptanceTests.SqlServer.csproj @@ -12,6 +12,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs new file mode 100644 index 0000000000..fa36935479 --- /dev/null +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_composing_audit_ingestion_in_the_primary.cs @@ -0,0 +1,173 @@ +namespace ServiceControl.AcceptanceTests.Auditing +{ + using System; + using System.IO; + using System.Linq; + using System.Runtime.Loader; + using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NUnit.Framework; + using Particular.LicensingComponent.AuditThroughput; + using Particular.ServiceControl; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.CompositeViews.MessageCounting; + using ServiceControl.Connection; + using ServiceControl.Infrastructure; + using ServiceControl.Infrastructure.WebApi; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Tests.AuditCapable; + using ServiceControl.SagaAudit; + + // The inner persistence type reaches the test persister through an environment variable, which is + // process wide, so these cannot run alongside anything else that sets it. + [NonParallelizable] + class When_composing_audit_ingestion_in_the_primary : AcceptanceTest + { + [Test] + public async Task Should_host_the_audit_runtime_when_the_persister_advertises_audit_support() + { + var (app, services) = await BuildHost(auditCapable: true); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.True); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + + Assert.That(app.Services.GetService(), Is.Not.Null, + "without it the local audit queues are counted as customer endpoints in the licensing report"); + Assert.That(services.Any(descriptor => + descriptor.ServiceType == typeof(IProvidePlatformConnectionDetails) + && descriptor.ImplementationType == typeof(AuditPlatformConnectionDetailsProvider)), + Is.True, + "/api/connection must still tell endpoints where to send audit and saga data"); + } + } + finally + { + await app.DisposeAsync(); + } + } + + [Test] + public async Task Should_keep_every_audit_capability_but_the_receiver_when_ingestion_is_disabled() + { + var (app, services) = await BuildHost(auditCapable: true, settings => settings.IngestAuditMessages = false); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.False, + "the receiver is the only thing the setting turns off, because other processes may still be ingesting"); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Not.Null); + } + } + finally + { + await app.DisposeAsync(); + } + } + + [Test] + public async Task Should_host_nothing_audit_related_on_a_persister_without_audit_support() + { + var (app, services) = await BuildHost(auditCapable: false); + + try + { + using (Assert.EnterMultipleScope()) + { + Assert.That(HostsAuditIngestion(services), Is.False); + Assert.That(app.Services.GetService(), Is.Null); + Assert.That(app.Services.GetService(), Is.Null); + + Assert.That(app.Services.GetService(), Is.Not.Null, + "the audit routes stay served from the configured remotes, so the APIs must still resolve"); + Assert.That(app.Services.GetService(), Is.Not.Null); + Assert.That(app.Services.GetService(), Is.Null); + } + } + finally + { + await app.DisposeAsync(); + } + } + + // The registrations are inspected rather than resolved. A normal primary hosts an NServiceBus + // endpoint, and constructing every hosted service without starting it fails inside the + // transport's receive component. + static bool HostsAuditIngestion(IServiceCollection services) => + services.Any(descriptor => + descriptor.ServiceType == typeof(IHostedService) && descriptor.ImplementationType == typeof(AuditIngestion)); + + async Task<(WebApplication App, IServiceCollection Services)> BuildHost(bool auditCapable, Action customize = null) + { + var settings = await CreateSettings(auditCapable); + + customize?.Invoke(settings); + + var endpointConfiguration = new EndpointConfiguration(settings.InstanceName); + endpointConfiguration.AssemblyScanner().Disable = true; + + var hostBuilder = WebApplication.CreateBuilder(); + hostBuilder.AddServiceControl(settings, endpointConfiguration); + hostBuilder.AddServiceControlApi(settings.CorsSettings); + + return (hostBuilder.Build(), hostBuilder.Services); + } + + async Task CreateSettings(bool auditCapable) + { + var persistenceType = StorageConfiguration.PersistenceType; + + if (auditCapable) + { + // The test persister delegates everything but the audit contracts to the real one, so the + // host under test is the real host apart from the capability its manifest advertises. + Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); + persistenceType = AuditCapablePersistenceName; + } + + var settings = new Settings(TransportIntegration.TypeName, persistenceType, + CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) + { + InstanceName = $"AuditComposition.{Guid.NewGuid():n}", + TransportConnectionString = TransportIntegration.ConnectionString, + MaximumConcurrencyLevel = 2, + DisableHealthChecks = true, + AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default + }; + + await StorageConfiguration.CustomizeSettings(settings); + + return settings; + } + + [TearDown] + public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); + + static LoggingSettings CreateLoggingSettings() + { + var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); + Directory.CreateDirectory(logPath); + return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); + } + + const string AuditCapablePersistenceName = "AuditCapableTest"; + + static readonly string InnerPersistenceTypeVariable = + AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); + } +} diff --git a/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs new file mode 100644 index 0000000000..86d16ebf29 --- /dev/null +++ b/src/ServiceControl.AcceptanceTests/Auditing/When_hosting_audit_ingestion_only.cs @@ -0,0 +1,154 @@ +namespace ServiceControl.AcceptanceTests.Auditing +{ + using System; + using System.IO; + using System.Linq; + using System.Runtime.Loader; + using System.Threading.Tasks; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NUnit.Framework; + using Particular.LicensingComponent.AuditThroughput; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Hosting.Commands; + using ServiceControl.Infrastructure; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Tests.AuditCapable; + + // The inner persistence type reaches the test persister through an environment variable, which is + // process wide, so these cannot run alongside anything else that sets it. + [NonParallelizable] + class When_hosting_audit_ingestion_only : AcceptanceTest + { + [Test] + public async Task Should_ingest_without_an_endpoint_and_without_the_single_owner_services() + { + var settings = await CreateSettings(auditCapable: true); + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + var hostedServices = host.Services.GetServices() + .Select(hostedService => hostedService.GetType().Name) + .ToArray(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(host.Services.GetService(), Is.Null, + "the host must not run an NServiceBus endpoint"); + Assert.That(host.Services.GetService(), Is.Not.Null); + + Assert.That(host.Services.GetService(), Is.Null, + "an ingestion only worker never changes the database schema"); + Assert.That(host.Services.GetService(), Is.Null, + "an ingestion only worker never provisions external storage"); + Assert.That(host.Services.GetService(), Is.Null, + "licensing throughput is owned by the normal primary, and would be counted once per node"); + Assert.That(host.Services.GetService(), Is.Not.Null); + + Assert.That(hostedServices, Is.EquivalentTo(ExpectedHostedServices), + "the set of hosted services in the audit ingestion only host changed. Every one of " + + "these runs on every ingestion node, so decide whether that is safe before updating " + + "this list. Audit ingestion raises no domain events and no integration events, which " + + "is why EventLog and ExternalIntegrations are not registered."); + } + } + finally + { + await host.DisposeAsync(); + } + } + + [Test] + public async Task Should_report_audit_ingestion_readiness() + { + var settings = await CreateSettings(auditCapable: true); + + var host = AuditIngestionOnlyCommand.BuildHost(settings); + + try + { + var readiness = host.Services.GetRequiredService(); + + var report = await readiness.CheckHealthAsync(registration => registration.Tags.Contains("ready")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(report.Entries.Keys, Does.Contain("audit-ingestion")); + Assert.That(report.Entries.Keys, Does.Not.Contain("error-ingestion"), + "this host does not ingest error messages, so it must not answer for them"); + } + } + finally + { + await host.DisposeAsync(); + } + } + + [Test] + public async Task Should_refuse_to_start_against_storage_without_audit_support() + { + var settings = await CreateSettings(auditCapable: false); + + var exception = Assert.ThrowsAsync(() => + new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); + + Assert.That(exception.Message, Does.Contain("supports audit ingestion")); + } + + static readonly string[] ExpectedHostedServices = + [ + "GenericWebHostService", // health endpoints only, no ServiceControl API + nameof(AuditIngestion), // the reason this host exists + "HeartbeatMonitoringHostedService", // warms the endpoint monitor, does not check heartbeats + "InternalCustomChecksHostedService", // reports this node's ingestion health to the database + "MetricsReporterHostedService", + "HealthCheckPublisherHostedService", // inert, no IHealthCheckPublisher is registered + "ExternalIntegrationRequestsDataStore" // registered by the persister; its drain is inert here, nothing calls Subscribe + ]; + + [TearDown] + public void ClearInnerPersistenceType() => Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, null); + + async Task CreateSettings(bool auditCapable) + { + var persistenceType = StorageConfiguration.PersistenceType; + + if (auditCapable) + { + Environment.SetEnvironmentVariable(InnerPersistenceTypeVariable, persistenceType); + persistenceType = AuditCapablePersistenceName; + } + + var settings = new Settings(TransportIntegration.TypeName, persistenceType, + CreateLoggingSettings(), forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)) + { + InstanceName = $"AuditIngestOnly.{Guid.NewGuid():n}", + TransportConnectionString = TransportIntegration.ConnectionString, + MaximumConcurrencyLevel = 2, + AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default + }; + + await StorageConfiguration.CustomizeSettings(settings); + + return settings; + } + + static LoggingSettings CreateLoggingSettings() + { + var logPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName()); + Directory.CreateDirectory(logPath); + return new LoggingSettings(Settings.SettingsRootNamespace, defaultLevel: LogLevel.Debug, logPath: logPath); + } + + const string AuditCapablePersistenceName = "AuditCapableTest"; + + static readonly string InnerPersistenceTypeVariable = + AuditCapableTestPersistenceConfiguration.InnerPersistenceTypeSetting.ToUpperInvariant(); + } +} diff --git a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs index 4c2cc1d874..4d72f33717 100644 --- a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs +++ b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs @@ -88,6 +88,7 @@ public Task ProvisionQueues(TransportSettings transportSettings, IEnumerable CreateTransportInfrastructure(string name, TransportSettings transportSettings, OnMessage onMessage = null, OnError onError = null, Func onCriticalError = null, TransportTransactionMode preferredTransactionMode = TransportTransactionMode.ReceiveOnly, + int? maxConcurrency = null, CancellationToken cancellationToken = default) => throw new NotImplementedException(); public string ToTransportQualifiedQueueName(string queueName) => queueName; diff --git a/src/ServiceControl.Audit/Auditing/AuditIngestion.cs b/src/ServiceControl.Audit/Auditing/AuditIngestion.cs index e500c3aee7..ef38365232 100644 --- a/src/ServiceControl.Audit/Auditing/AuditIngestion.cs +++ b/src/ServiceControl.Audit/Auditing/AuditIngestion.cs @@ -138,7 +138,7 @@ async Task SetUpAndStartInfrastructure(CancellationToken cancellationToken) errorHandlingPolicy.OnError, OnCriticalError, TransportTransactionMode.ReceiveOnly, - cancellationToken + cancellationToken: cancellationToken ); messageReceiver = transportInfrastructure.Receivers[inputEndpoint]; diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest index 433c31a5ae..49b6aa4414 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/persistence.manifest @@ -4,6 +4,7 @@ "Description": "PostgreSQL ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.PostgreSql", "TypeName": "ServiceControl.Persistence.EFCore.PostgreSql.PostgreSqlPersistenceConfiguration, ServiceControl.Persistence.EFCore.PostgreSql", + "SupportsAuditIngestion": false, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest index 9c1f04c024..598ba9168c 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/persistence.manifest @@ -4,6 +4,7 @@ "Description": "SQL Server ServiceControl persister", "AssemblyName": "ServiceControl.Persistence.EFCore.SqlServer", "TypeName": "ServiceControl.Persistence.EFCore.SqlServer.SqlServerPersistenceConfiguration, ServiceControl.Persistence.EFCore.SqlServer", + "SupportsAuditIngestion": false, "Settings": [ { "Name": "ServiceControl/Database/ConnectionString", diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs index acac97d01f..24f79b17ad 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/UnitOfWork/EFIngestionUnitOfWork.cs @@ -33,6 +33,9 @@ public EFIngestionUnitOfWork(IAsyncDisposable scope, ServiceControlDbContext dbC public IRecoverabilityIngestionUnitOfWork Recoverability { get; } + // Stays null until the EF audit persistence lands and the manifest advertises SupportsAuditIngestion. + public IAuditIngestionUnitOfWork? Audit => null; + internal void Record(RecordedFailedProcessingAttempt attempt) => failedProcessingAttempts.Enqueue(attempt); internal void RecordBodyWrite(Task bodyWrite) => bodyWrites.Enqueue(bodyWrite); diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig b/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig new file mode 100644 index 0000000000..ca5ad8bd2e --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/.editorconfig @@ -0,0 +1,5 @@ +[*.cs] + +# Justification: Test project +dotnet_diagnostic.CA2007.severity = none +dotnet_diagnostic.PS0018.severity = none diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs new file mode 100644 index 0000000000..fb79fc30cc --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableBodyStorage.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.IO; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations.BodyStorage; + + // The third step of the arbitration order IBodyStorage states: a failed message body wins, and the + // audit copy answers only when no failed message holds one. + class AuditCapableBodyStorage(IBodyStorage inner, InMemoryAuditStore auditStore) : IBodyStorage + { + public async Task TryFetch(string bodyId, CancellationToken cancellationToken = default) + { + var failedMessageBody = await inner.TryFetch(bodyId, cancellationToken); + + if (failedMessageBody.State != MessageBodyState.NotFound) + { + return failedMessageBody; + } + + var body = auditStore.BodyFor(bodyId); + + if (body == null) + { + return MessageBodyResult.NotFound(); + } + + return body.Length == 0 + ? MessageBodyResult.Empty() + : MessageBodyResult.Available(new MessageBodyStreamContent(new MemoryStream(body, writable: false), "application/json", body.Length, bodyId)); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs new file mode 100644 index 0000000000..f10e9f8032 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWork.cs @@ -0,0 +1,54 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Concurrent; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.MessageAuditing; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.SagaAudit; + + // Recording is buffered and only visible after Complete, so tests see the same all or nothing + // batch behaviour a real persister gives them. + class AuditCapableIngestionUnitOfWork(IIngestionUnitOfWork inner, InMemoryAuditStore auditStore) + : IIngestionUnitOfWork, IAuditIngestionUnitOfWork + { + readonly ConcurrentQueue<(ProcessedMessage Message, byte[] Body)> processedMessages = new(); + readonly ConcurrentQueue sagaSnapshots = new(); + + public IMonitoringIngestionUnitOfWork? Monitoring => inner.Monitoring; + + public IRecoverabilityIngestionUnitOfWork? Recoverability => inner.Recoverability; + + public IAuditIngestionUnitOfWork? Audit => this; + + public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) + { + processedMessages.Enqueue((processedMessage, body.ToArray())); + return Task.CompletedTask; + } + + public Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default) + { + sagaSnapshots.Enqueue(sagaSnapshot); + return Task.CompletedTask; + } + + public async Task Complete(CancellationToken cancellationToken = default) + { + await inner.Complete(cancellationToken); + + while (processedMessages.TryDequeue(out var processedMessage)) + { + auditStore.Record(processedMessage.Message, processedMessage.Body); + } + + while (sagaSnapshots.TryDequeue(out var sagaSnapshot)) + { + auditStore.Record(sagaSnapshot); + } + } + + public ValueTask DisposeAsync() => inner.DisposeAsync(); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs new file mode 100644 index 0000000000..cb5eb477fb --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableIngestionUnitOfWorkFactory.cs @@ -0,0 +1,14 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.UnitOfWork; + + class AuditCapableIngestionUnitOfWorkFactory(IIngestionUnitOfWorkFactory inner, InMemoryAuditStore auditStore) : IIngestionUnitOfWorkFactory + { + public async ValueTask StartNew(CancellationToken cancellationToken = default) => + new AuditCapableIngestionUnitOfWork(await inner.StartNew(cancellationToken), auditStore); + + public bool CanIngestMore() => inner.CanIngestMore(); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs new file mode 100644 index 0000000000..74d5dfe37d --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableMessagesViewDataStore.cs @@ -0,0 +1,59 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Persistence.Infrastructure; + + // One local result set holding both failed and audited messages, merged under the precedence, + // paging and counting rules IMessagesViewDataStore states. + class AuditCapableMessagesViewDataStore(IMessagesViewDataStore inner, InMemoryAuditStore auditStore) : IMessagesViewDataStore + { + public async Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessages(pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), + Audited(includeSystemMessages, timeSentRange), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesForEndpoint(string endpointName, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesForEndpoint(endpointName, pagingInfo, sortInfo, includeSystemMessages, timeSentRange, cancellationToken), + Audited(includeSystemMessages, timeSentRange).Where(message => message.ReceivingEndpoint?.Name == endpointName), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesByConversation(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesByConversation(conversationId, pagingInfo, sortInfo, includeSystemMessages, cancellationToken), + Audited(includeSystemMessages).Where(message => message.ConversationId == conversationId), pagingInfo, sortInfo); + + public async Task>> GetAllMessagesForSearch(string searchTerms, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.GetAllMessagesForSearch(searchTerms, pagingInfo, sortInfo, timeSentRange, cancellationToken), + Audited(includeSystemMessages: true, timeSentRange).Where(message => Matches(message, searchTerms)), pagingInfo, sortInfo); + + public async Task>> SearchEndpointMessages(string endpointName, string searchKeyword, PagingInfo pagingInfo, SortInfo sortInfo, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default) => + Merge(await inner.SearchEndpointMessages(endpointName, searchKeyword, pagingInfo, sortInfo, timeSentRange, cancellationToken), + Audited(includeSystemMessages: true, timeSentRange) + .Where(message => message.ReceivingEndpoint?.Name == endpointName && Matches(message, searchKeyword)), pagingInfo, sortInfo); + + IEnumerable Audited(bool includeSystemMessages, DateTimeRange? timeSentRange = null) => + auditStore.MessageViews + .Where(message => includeSystemMessages || !message.IsSystemMessage) + .Where(message => InRange(message, timeSentRange)); + + static bool InRange(MessagesView message, DateTimeRange? timeSentRange) => + timeSentRange == null + || (message.TimeSent >= timeSentRange.From && message.TimeSent <= timeSentRange.To); + + static bool Matches(MessagesView message, string searchTerms) => + searchTerms == null + || (message.MessageType?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false) + || (message.MessageId?.Contains(searchTerms, StringComparison.OrdinalIgnoreCase) ?? false); + + static QueryResult> Merge(QueryResult> failed, IEnumerable audited, PagingInfo pagingInfo, SortInfo sortInfo) => + LocalMessagesView.Merge( + [.. failed.Results ?? []], + [.. audited], + pagingInfo, + MessageViewComparer.FromSortInfo(sortInfo), + failed.QueryStats.ETag, + failed.QueryStats.IsStale); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs new file mode 100644 index 0000000000..238ca820b4 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistence.cs @@ -0,0 +1,59 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Linq; + using Microsoft.Extensions.DependencyInjection; + using ServiceControl.Operations.BodyStorage; + using ServiceControl.Persistence.UnitOfWork; + + class AuditCapableTestPersistence(IPersistence inner) : IPersistence + { + public void AddPersistence(IServiceCollection services) + { + inner.AddPersistence(services); + + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + Decorate(services, (inner, provider) => + new AuditCapableIngestionUnitOfWorkFactory(inner, provider.GetRequiredService())); + Decorate(services, (inner, provider) => + new AuditCapableMessagesViewDataStore(inner, provider.GetRequiredService())); + Decorate(services, (inner, provider) => + new AuditCapableBodyStorage(inner, provider.GetRequiredService())); + } + + public void AddInstaller(IServiceCollection services) => inner.AddInstaller(services); + + static void Decorate(IServiceCollection services, Func decorate) + where TService : class + { + var descriptor = services.LastOrDefault(d => d.ServiceType == typeof(TService)) + ?? throw new InvalidOperationException($"The delegated persister registered no {typeof(TService).Name}."); + + services.Remove(descriptor); + + services.Add(new ServiceDescriptor(typeof(TService), + provider => decorate(ResolveInner(provider, descriptor), provider), + descriptor.Lifetime)); + } + + static TService ResolveInner(IServiceProvider provider, ServiceDescriptor descriptor) + where TService : class + { + if (descriptor.ImplementationInstance is TService instance) + { + return instance; + } + + if (descriptor.ImplementationFactory is not null) + { + return (TService)descriptor.ImplementationFactory(provider); + } + + return (TService)ActivatorUtilities.CreateInstance(provider, descriptor.ImplementationType!); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs new file mode 100644 index 0000000000..58cd720adf --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/AuditCapableTestPersistenceConfiguration.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using ServiceControl.Configuration; + + /// + /// A persister that exists only so tests can compose a primary host whose manifest advertises audit + /// support, before any shipped persister does. Everything except the audit contracts is delegated to + /// the persister named by the setting, so the error side of + /// the host is the real thing. Delete it once a shipped manifest sets SupportsAuditIngestion. + /// + public class AuditCapableTestPersistenceConfiguration : IPersistenceConfiguration + { + public const string InnerPersistenceTypeSetting = "AuditCapableTestInnerPersistenceType"; + + public PersistenceSettings CreateSettings(SettingsRootNamespace settingsRootNamespace) => + CreateInnerConfiguration(settingsRootNamespace).CreateSettings(settingsRootNamespace); + + public IPersistence Create(PersistenceSettings settings) => + new AuditCapableTestPersistence(CreateInnerConfiguration(PrimaryRootNamespace).Create(settings)); + + static IPersistenceConfiguration CreateInnerConfiguration(SettingsRootNamespace settingsRootNamespace) + { + var persistenceType = SettingsReader.Read(settingsRootNamespace, InnerPersistenceTypeSetting) + ?? throw new InvalidOperationException( + $"The audit capable test persister needs the {settingsRootNamespace}/{InnerPersistenceTypeSetting} setting to name the persister it delegates to."); + + var manifest = PersistenceManifestLibrary.Find(persistenceType) + ?? throw new InvalidOperationException($"No persistence manifest matches '{persistenceType}'."); + + var configurationType = Type.GetType(manifest.TypeName, throwOnError: true)!; + + return (IPersistenceConfiguration)Activator.CreateInstance(configurationType)!; + } + + static readonly SettingsRootNamespace PrimaryRootNamespace = new("ServiceControl"); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs new file mode 100644 index 0000000000..8c25475784 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditCountsDataStore.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + + class InMemoryAuditCountsDataStore(InMemoryAuditStore auditStore) : IAuditCountsDataStore + { + public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) + { + IList counts = [.. auditStore.CountsFor(endpointName).Select(count => new AuditCount { UtcDate = count.UtcDate, Count = count.Count })]; + + return Task.FromResult(new QueryResult>(counts, new QueryStatsInfo(string.Empty, counts.Count, false))); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs new file mode 100644 index 0000000000..0179f4ca61 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryAuditStore.cs @@ -0,0 +1,102 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Collections.Concurrent; + using System.Collections.Generic; + using System.Linq; + using NServiceBus; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.MessageAuditing; + using ServiceControl.Operations; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + public class InMemoryAuditStore + { + readonly ConcurrentQueue processedMessages = new(); + readonly ConcurrentQueue sagaSnapshots = new(); + readonly ConcurrentDictionary failedImports = new(); + + public void Record(ProcessedMessage processedMessage, byte[] body) => + processedMessages.Enqueue(new AuditRecord(processedMessage, body)); + + public void Record(SagaSnapshot sagaSnapshot) => sagaSnapshots.Enqueue(sagaSnapshot); + + public void Record(FailedAuditImport failedImport) => failedImports[failedImport.Id] = failedImport; + + public IReadOnlyList FailedImports => [.. failedImports.Values]; + + public bool RemoveFailedImport(string id) => failedImports.TryRemove(id, out _); + + public IReadOnlyList MessageViews => [.. processedMessages.Select(record => ToMessagesView(record.Message))]; + + public byte[]? BodyFor(string uniqueMessageId) => + processedMessages.FirstOrDefault(record => record.Message.UniqueMessageId == uniqueMessageId)?.Body; + + public IReadOnlyList<(DateTime UtcDate, long Count)> CountsFor(string endpointName) => + [ + .. processedMessages + .Where(record => EndpointOf(record.Message) == endpointName) + .GroupBy(record => record.Message.ProcessedAt.Date) + .Select(group => (UtcDate: group.Key, Count: (long)group.Count())) + .OrderBy(count => count.UtcDate) + ]; + + public SagaHistory? HistoryFor(Guid sagaId) + { + var snapshots = sagaSnapshots.Where(snapshot => snapshot.SagaId == sagaId).ToList(); + + if (snapshots.Count == 0) + { + return null; + } + + return new SagaHistory + { + Id = sagaId, + SagaId = sagaId, + SagaType = snapshots[0].SagaType, + Changes = [.. snapshots.OrderByDescending(snapshot => snapshot.FinishTime).Select(ToStateChange)] + }; + } + + static MessagesView ToMessagesView(ProcessedMessage message) => new() + { + Id = message.Id, + MessageId = Metadata(message, "MessageId"), + MessageType = Metadata(message, "MessageType"), + SendingEndpoint = Metadata(message, "SendingEndpoint"), + ReceivingEndpoint = Metadata(message, "ReceivingEndpoint"), + TimeSent = Metadata(message, "TimeSent"), + ProcessedAt = message.ProcessedAt, + CriticalTime = Metadata(message, "CriticalTime"), + ProcessingTime = Metadata(message, "ProcessingTime"), + DeliveryTime = Metadata(message, "DeliveryTime"), + IsSystemMessage = Metadata(message, "IsSystemMessage"), + ConversationId = Metadata(message, "ConversationId"), + Headers = [.. message.Headers.Select(header => new KeyValuePair(header.Key, header.Value))], + Status = MessageStatus.Successful, + MessageIntent = Metadata(message, "MessageIntent"), + BodyUrl = $"/messages/{message.UniqueMessageId}/body" + }; + + static T? Metadata(ProcessedMessage message, string key) => + message.MessageMetadata.TryGetValue(key, out var value) && value is T typed ? typed : default; + + static SagaStateChange ToStateChange(SagaSnapshot snapshot) => new() + { + StartTime = snapshot.StartTime, + FinishTime = snapshot.FinishTime, + Status = snapshot.Status, + StateAfterChange = snapshot.StateAfterChange, + InitiatingMessage = snapshot.InitiatingMessage, + OutgoingMessages = snapshot.OutgoingMessages, + Endpoint = snapshot.Endpoint + }; + + static string? EndpointOf(ProcessedMessage message) => + message.Headers.GetValueOrDefault(Headers.ProcessingEndpoint); + + sealed record AuditRecord(ProcessedMessage Message, byte[] Body); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs new file mode 100644 index 0000000000..f7436437c4 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemoryFailedAuditImportDataStore.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations; + + class InMemoryFailedAuditImportDataStore(InMemoryAuditStore auditStore) : IFailedAuditImportDataStore + { + public Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default) + { + auditStore.Record(failure); + return Task.CompletedTask; + } + + public async Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default) + { + foreach (var failedImport in auditStore.FailedImports) + { + if (failedImport.Message is null) + { + continue; + } + + await processMessage(failedImport.Message, cancellationToken); + auditStore.RemoveFailedImport(failedImport.Id); + } + } + + public Task QueryContainsFailedImports(CancellationToken cancellationToken = default) => + Task.FromResult(auditStore.FailedImports.Count > 0); + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs new file mode 100644 index 0000000000..3bae1d4383 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/InMemorySagaHistoryDataStore.cs @@ -0,0 +1,20 @@ +namespace ServiceControl.Persistence.Tests.AuditCapable +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + class InMemorySagaHistoryDataStore(InMemoryAuditStore auditStore) : ISagaHistoryDataStore + { + public Task> QuerySagaHistoryById(Guid sagaId, CancellationToken cancellationToken = default) + { + var history = auditStore.HistoryFor(sagaId); + + return Task.FromResult(history is null + ? QueryResult.Empty() + : new QueryResult(history, new QueryStatsInfo(string.Empty, history.Changes.Count, false))); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj b/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj new file mode 100644 index 0000000000..4de524ff84 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/ServiceControl.Persistence.Tests.AuditCapable.csproj @@ -0,0 +1,16 @@ + + + + net10.0 + enable + + + + + + + + + + + diff --git a/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest b/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest new file mode 100644 index 0000000000..c214727198 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.AuditCapable/persistence.manifest @@ -0,0 +1,9 @@ +{ + "Name": "AuditCapableTest", + "DisplayName": "Audit capable test persister", + "Description": "Test only persister that advertises audit support and delegates everything else to a real persister", + "AssemblyName": "ServiceControl.Persistence.Tests.AuditCapable", + "TypeName": "ServiceControl.Persistence.Tests.AuditCapable.AuditCapableTestPersistenceConfiguration, ServiceControl.Persistence.Tests.AuditCapable", + "IsSupported": false, + "SupportsAuditIngestion": true +} diff --git a/src/ServiceControl.Persistence.Tests/RetryStateTests.cs b/src/ServiceControl.Persistence.Tests/RetryStateTests.cs index 56df79608c..3646e229e0 100644 --- a/src/ServiceControl.Persistence.Tests/RetryStateTests.cs +++ b/src/ServiceControl.Persistence.Tests/RetryStateTests.cs @@ -503,6 +503,7 @@ public Task CreateTransportInfrastructure(string name, Func onCriticalError = null, NServiceBus.TransportTransactionMode preferredTransactionMode = NServiceBus.TransportTransactionMode.ReceiveOnly, + int? maxConcurrency = null, CancellationToken cancellationToken = default) => Task.FromResult(TransportInfrastructure); public void CustomizeAuditEndpoint(NServiceBus.EndpointConfiguration endpointConfiguration, TransportSettings transportSettings) => throw new NotImplementedException(); public void CustomizeMonitoringEndpoint(NServiceBus.EndpointConfiguration endpointConfiguration, TransportSettings transportSettings) => throw new NotImplementedException(); diff --git a/src/ServiceControl.Persistence/FailedAuditImport.cs b/src/ServiceControl.Persistence/FailedAuditImport.cs new file mode 100644 index 0000000000..3a8f46ad8c --- /dev/null +++ b/src/ServiceControl.Persistence/FailedAuditImport.cs @@ -0,0 +1,32 @@ +namespace ServiceControl.Operations +{ + using System; + using System.Collections.Generic; + using ServiceControl.Persistence.Infrastructure; + + public class FailedAuditImport + { + public required string Id { get; set; } + public FailedTransportMessage? Message { get; set; } + public string? ExceptionInfo { get; set; } + + public static Guid DeriveKey(IReadOnlyDictionary headers, string nativeMessageId) + { + try + { + if (Guid.TryParse(headers.UniqueId(), out var uniqueMessageId)) + { + return uniqueMessageId; + } + } + catch (Exception) + { + // UniqueId() derives the processing endpoint, which throws when the audited message + // carries no endpoint header. Malformed messages are a leading cause of import + // failure, so fall back to a key derived from the id the transport always supplies. + } + + return DeterministicGuid.MakeId(nativeMessageId); + } + } +} diff --git a/src/ServiceControl.Persistence/IAuditCountsDataStore.cs b/src/ServiceControl.Persistence/IAuditCountsDataStore.cs new file mode 100644 index 0000000000..c5cb0090df --- /dev/null +++ b/src/ServiceControl.Persistence/IAuditCountsDataStore.cs @@ -0,0 +1,13 @@ +namespace ServiceControl.Persistence +{ + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + + public interface IAuditCountsDataStore + { + Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/IBodyStorage.cs b/src/ServiceControl.Persistence/IBodyStorage.cs index 7d7715437f..34d280206f 100644 --- a/src/ServiceControl.Persistence/IBodyStorage.cs +++ b/src/ServiceControl.Persistence/IBodyStorage.cs @@ -7,9 +7,28 @@ public interface IBodyStorage { + /// + /// Resolves a message body from wherever it was stored. A persister that holds audit data as + /// well as failed messages resolves in one fixed order, because a message that both failed and + /// was audited has two bodies and an edited message's two bodies differ: + /// + /// failed message by UniqueMessageId, + /// failed message by MessageId, + /// audit message by UniqueMessageId, including a body held inline for full text search. + /// + /// Task TryFetch(string bodyId, CancellationToken cancellationToken = default); } + /// + /// Audit and failed message bodies share one external store, and the two sides key bodies + /// differently, so each owns a prefixed keyspace. Retention must sweep both. + /// + public static class AuditBodyKeyspace + { + public static string ExternalBodyId(Guid uniqueMessageId) => $"audit-{uniqueMessageId}"; + } + public enum MessageBodyState { NotFound, diff --git a/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs b/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs new file mode 100644 index 0000000000..5ac5a9c60a --- /dev/null +++ b/src/ServiceControl.Persistence/IFailedAuditImportDataStore.cs @@ -0,0 +1,14 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Operations; + + public interface IFailedAuditImportDataStore + { + Task StoreFailedAuditImport(FailedAuditImport failure, CancellationToken cancellationToken = default); + Task ProcessFailedAuditImports(Func processMessage, CancellationToken cancellationToken = default); + Task QueryContainsFailedImports(CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/IMessagesViewDataStore.cs b/src/ServiceControl.Persistence/IMessagesViewDataStore.cs index 66b45c9302..7ae6549af7 100644 --- a/src/ServiceControl.Persistence/IMessagesViewDataStore.cs +++ b/src/ServiceControl.Persistence/IMessagesViewDataStore.cs @@ -7,6 +7,19 @@ namespace ServiceControl.Persistence using CompositeViews.Messages; using Infrastructure; + /// + /// The single local source of message views. A persister that also holds audit data returns failed + /// and audited messages from one query, which puts three rules on the result it returns. + /// + /// Precedence. For a given {ReceivingEndpoint.Name}-{MessageId} the failed row must come + /// before the audit row, because ScatterGatherApiMessageView + /// deduplicates with TryAdd and would otherwise show a failed message as successfully processed. + /// Paging. At most PagingInfo.PageSize rows after deduplication, because the scatter gather + /// truncates and would silently drop rows if each source contributed a full page. + /// Counting. A message that both failed and was audited counts once in + /// , not once per source. + /// + /// public interface IMessagesViewDataStore { Task>> GetAllMessages(PagingInfo pagingInfo, SortInfo sortInfo, bool includeSystemMessages, DateTimeRange? timeSentRange = null, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs b/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs new file mode 100644 index 0000000000..928b62061b --- /dev/null +++ b/src/ServiceControl.Persistence/ISagaHistoryDataStore.cs @@ -0,0 +1,13 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + public interface ISagaHistoryDataStore + { + Task> QuerySagaHistoryById(Guid sagaId, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/LocalMessagesView.cs b/src/ServiceControl.Persistence/LocalMessagesView.cs new file mode 100644 index 0000000000..379d65c5c0 --- /dev/null +++ b/src/ServiceControl.Persistence/LocalMessagesView.cs @@ -0,0 +1,61 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Collections.Generic; + using System.Linq; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Persistence.Infrastructure; + + /// + /// Merges the failed and audited halves of one local result set under the three rules + /// states. A persister that holds both kinds of message + /// returns them through this, so precedence, paging and counting are defined in one place rather + /// than re-derived per provider and per query. + /// + public static class LocalMessagesView + { + public static QueryResult> Merge( + IReadOnlyCollection failedMessages, + IReadOnlyCollection auditedMessages, + PagingInfo pagingInfo, + IComparer? order = null, + string eTag = "", + bool isStale = false) + { + ArgumentNullException.ThrowIfNull(failedMessages); + ArgumentNullException.ThrowIfNull(auditedMessages); + ArgumentNullException.ThrowIfNull(pagingInfo); + + var deduplicated = new Dictionary(failedMessages.Count + auditedMessages.Count); + + // Failed first. A message that both failed and was audited must show as failed, and the + // scatter gather deduplicates with TryAdd, so whichever row is seen first wins for good. + foreach (var message in failedMessages.Concat(auditedMessages)) + { + deduplicated.TryAdd(DeduplicationKey(message), message); + } + + var merged = deduplicated.Values.ToList(); + + if (order != null) + { + merged.Sort(order); + } + + // The total is counted after deduplication, so a message that both failed and was audited + // counts once rather than once per source. + var totalCount = merged.Count; + + IList page = merged.Take(pagingInfo.PageSize).ToList(); + + return new QueryResult>(page, new QueryStatsInfo(eTag, totalCount, isStale)); + } + + /// + /// The key ScatterGatherApiMessageView deduplicates on, so a local merge and a cross + /// instance merge agree on what counts as the same message. + /// + public static string DeduplicationKey(MessagesView message) => + $"{message.ReceivingEndpoint?.Name}-{message.MessageId}"; + } +} diff --git a/src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs b/src/ServiceControl.Persistence/MessageViewComparer.cs similarity index 85% rename from src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs rename to src/ServiceControl.Persistence/MessageViewComparer.cs index 0da0816e8c..602b4977af 100644 --- a/src/ServiceControl/CompositeViews/Messages/MessageViewComparer.cs +++ b/src/ServiceControl.Persistence/MessageViewComparer.cs @@ -4,7 +4,7 @@ namespace ServiceControl.CompositeViews.Messages using System.Collections.Generic; using Persistence.Infrastructure; - static class MessageViewComparer + public static class MessageViewComparer { public static IComparer FromSortInfo(SortInfo sortInfo) { @@ -49,17 +49,11 @@ public Comparer(Func comparerFunc) this.comparerFunc = comparerFunc; } - public int Compare(MessagesView x, MessagesView y) - { - return comparerFunc(x, y); - } + public int Compare(MessagesView? x, MessagesView? y) => comparerFunc(x!, y!); - public IComparer Reverse() - { - return new Reverse(this); - } + public IComparer Reverse() => new Reverse(this); - Func comparerFunc; + readonly Func comparerFunc; } class Reverse : IComparer @@ -69,7 +63,7 @@ public Reverse(IComparer inner) this.inner = inner; } - public int Compare(MessagesView x, MessagesView y) => inner.Compare(y, x); + public int Compare(MessagesView? x, MessagesView? y) => inner.Compare(y, x); readonly IComparer inner; } } diff --git a/src/ServiceControl.Persistence/PersistenceManifest.cs b/src/ServiceControl.Persistence/PersistenceManifest.cs index 1c1c6dff73..d0b2d1319c 100644 --- a/src/ServiceControl.Persistence/PersistenceManifest.cs +++ b/src/ServiceControl.Persistence/PersistenceManifest.cs @@ -24,6 +24,13 @@ public class PersistenceManifest public bool IsSupported { get; set; } = true; + /// + /// Whether this persister can store and query audit data alongside the primary data, which is + /// what lets the primary instance ingest the audit queue itself. Absent means false, so RavenDB + /// and every legacy manifest stay audit free. + /// + public bool SupportsAuditIngestion { get; set; } + public string[] Aliases { get; set; } = []; internal bool IsMatch(string persistenceType) => diff --git a/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj b/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj index f4f4299197..3b1d6b8c88 100644 --- a/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj +++ b/src/ServiceControl.Persistence/ServiceControl.Persistence.csproj @@ -6,6 +6,7 @@ + diff --git a/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs index 7ba823d5d6..69bf7d37f1 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/FallbackIngestionUnitOfWork.cs @@ -22,6 +22,7 @@ public FallbackIngestionUnitOfWork(IIngestionUnitOfWork primary, IIngestionUnitO Recoverability = primary.Recoverability ?? fallback.Recoverability ?? throw new InvalidOperationException("Fallback unit of work must implement Recoverability"); + Audit = primary.Audit ?? fallback.Audit; } public override Task Complete(CancellationToken cancellationToken = default) diff --git a/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs new file mode 100644 index 0000000000..844315c080 --- /dev/null +++ b/src/ServiceControl.Persistence/UnitOfWork/IAuditIngestionUnitOfWork.cs @@ -0,0 +1,15 @@ +namespace ServiceControl.Persistence.UnitOfWork +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.MessageAuditing; + using ServiceControl.SagaAudit; + + public interface IAuditIngestionUnitOfWork + { + Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default); + + Task RecordSagaSnapshot(SagaSnapshot sagaSnapshot, CancellationToken cancellationToken = default); + } +} diff --git a/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs b/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs index 0971aaaaab..2d6df24848 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/IIngestionUnitOfWork.cs @@ -1,4 +1,4 @@ -namespace ServiceControl.Persistence.UnitOfWork +namespace ServiceControl.Persistence.UnitOfWork { using System; using System.Threading; @@ -8,6 +8,12 @@ public interface IIngestionUnitOfWork : IAsyncDisposable { IMonitoringIngestionUnitOfWork? Monitoring { get; } IRecoverabilityIngestionUnitOfWork? Recoverability { get; } + + /// + /// Null unless the persister advertises SupportsAuditIngestion in its manifest. + /// + IAuditIngestionUnitOfWork? Audit { get; } + Task Complete(CancellationToken cancellationToken = default); } -} \ No newline at end of file +} diff --git a/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs b/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs index 32431e061c..3e1b02459b 100644 --- a/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs +++ b/src/ServiceControl.Persistence/UnitOfWork/IngestionUnitOfWorkBase.cs @@ -19,6 +19,7 @@ public async ValueTask DisposeAsync() public IMonitoringIngestionUnitOfWork? Monitoring { get; protected set; } public IRecoverabilityIngestionUnitOfWork? Recoverability { get; protected set; } + public IAuditIngestionUnitOfWork? Audit { get; protected set; } public virtual Task Complete(CancellationToken cancellationToken = default) => Task.CompletedTask; } } diff --git a/src/ServiceControl.Transports/TransportCustomization.cs b/src/ServiceControl.Transports/TransportCustomization.cs index a03fcb2ee1..8de0534169 100644 --- a/src/ServiceControl.Transports/TransportCustomization.cs +++ b/src/ServiceControl.Transports/TransportCustomization.cs @@ -25,7 +25,12 @@ public interface ITransportCustomization Task ProvisionQueues(TransportSettings transportSettings, IEnumerable additionalQueues, CancellationToken cancellationToken = default); string ToTransportQualifiedQueueName(string queueName); - Task CreateTransportInfrastructure(string name, TransportSettings transportSettings, OnMessage onMessage = null, OnError onError = null, Func onCriticalError = null, TransportTransactionMode preferredTransactionMode = TransportTransactionMode.ReceiveOnly, CancellationToken cancellationToken = default); + /// + /// Creates transport infrastructure for one receiver. overrides + /// the shared , which a combined host cannot use for + /// every receiver because error and audit ingestion are scaled independently. + /// + Task CreateTransportInfrastructure(string name, TransportSettings transportSettings, OnMessage onMessage = null, OnError onError = null, Func onCriticalError = null, TransportTransactionMode preferredTransactionMode = TransportTransactionMode.ReceiveOnly, int? maxConcurrency = null, CancellationToken cancellationToken = default); } public abstract class TransportCustomization : ITransportCustomization where TTransport : TransportDefinition @@ -159,7 +164,7 @@ public virtual async Task ProvisionQueues(TransportSettings transportSettings, I await transportInfrastructure.Shutdown(cancellationToken); } - public async Task CreateTransportInfrastructure(string name, TransportSettings transportSettings, OnMessage onMessage = null, OnError onError = null, Func onCriticalError = null, TransportTransactionMode preferredTransactionMode = TransportTransactionMode.ReceiveOnly, CancellationToken cancellationToken = default) + public async Task CreateTransportInfrastructure(string name, TransportSettings transportSettings, OnMessage onMessage = null, OnError onError = null, Func onCriticalError = null, TransportTransactionMode preferredTransactionMode = TransportTransactionMode.ReceiveOnly, int? maxConcurrency = null, CancellationToken cancellationToken = default) { var transport = CreateTransport(transportSettings, preferredTransactionMode); @@ -190,13 +195,11 @@ public async Task CreateTransportInfrastructure(string if (createReceiver) { - if (!transportSettings.MaxConcurrency.HasValue) - { - throw new ArgumentException("MaxConcurrency is not set in TransportSettings"); - } + var receiverConcurrency = maxConcurrency ?? transportSettings.MaxConcurrency + ?? throw new ArgumentException("MaxConcurrency is not set in TransportSettings and no per receiver concurrency was supplied"); var transportInfrastructureReceiver = transportInfrastructure.Receivers[name]; - await transportInfrastructureReceiver.Initialize(new PushRuntimeSettings(transportSettings.MaxConcurrency.Value), onMessage, onError, cancellationToken); + await transportInfrastructureReceiver.Initialize(new PushRuntimeSettings(receiverConcurrency), onMessage, onError, cancellationToken); } return transportInfrastructure; diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt index be3c421a56..53537d2012 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.CustomCheckDetails.approved.txt @@ -1,5 +1,7 @@ Configuration: Saga Audit Configuration Health: ServiceControl Primary Instance Health: ServiceControl Remotes +ServiceControl Health: Audit Message Ingestion (local) +ServiceControl Health: Audit Message Ingestion Process ServiceControl Health: Error Message Ingestion ServiceControl Health: Error Message Ingestion Process \ No newline at end of file diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt index 203d0facb5..50073c8573 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/APIApprovals.PlatformSampleSettings.approved.txt @@ -68,7 +68,15 @@ "ForwardErrorMessages": false, "IngestErrorMessages": true, "RunRetryProcessor": true, + "AuditQueue": "audit", + "AuditLogQueue": "audit.log", + "ForwardAuditMessages": false, + "IngestAuditMessages": true, + "MaximumAuditIngestionConcurrencyLevel": 32, + "TimeToRestartAuditIngestionAfterFailure": "00:01:00", + "OtlpEndpointUrl": null, "ErrorIngestionOnly": false, + "AuditIngestionOnly": false, "AuditRetentionPeriod": null, "ErrorRetentionPeriod": "10.00:00:00", "EventsRetentionPeriod": "14.00:00:00", diff --git a/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs new file mode 100644 index 0000000000..ade60f8ec0 --- /dev/null +++ b/src/ServiceControl.UnitTests/Hosting/AuditIngestionOnlyCommandTests.cs @@ -0,0 +1,83 @@ +namespace ServiceControl.UnitTests.Hosting +{ + using System; + using System.Threading.Tasks; + using NUnit.Framework; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Hosting.Commands; + + // Environment variables are process wide, so these cannot run alongside anything else that reads them. + [TestFixture] + [NonParallelizable] + public class AuditIngestionOnlyCommandTests + { + [TestCase("RavenDB")] + [TestCase("SQLServer")] + [TestCase("PostgreSQL")] + public void Should_refuse_to_start_against_storage_without_audit_support(string persistenceType) + { + var settings = CreateSettings(persistenceType); + + var exception = Assert.ThrowsAsync(() => + new AuditIngestionOnlyCommand().Execute(new HostArguments([]), settings)); + + Assert.That(exception.Message, Does.Contain("supports audit ingestion")); + } + + [Test] + public void Should_refuse_to_combine_the_two_ingestion_only_modes() + { + var exception = Assert.Throws(() => + IngestionOnlyGuards.EnsureModesAreNotCombined(errorIngestionOnly: true, auditIngestionOnly: true)); + + Assert.That(exception.Message, Does.Contain("cannot be combined")); + } + + [Test] + public void Should_refuse_file_system_body_storage_that_is_not_asserted_as_shared() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem"); + + var exception = Assert.Throws(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + + Assert.That(exception.Message, Does.Contain(IngestionOnlyGuards.SharedBodyStoragePathKey)); + } + + [Test] + public void Should_accept_file_system_body_storage_asserted_as_shared() + { + using var storageType = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem"); + using var pathIsShared = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_PATHISSHARED", "true"); + + Assert.DoesNotThrow(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + } + + [Test] + public void Should_ignore_body_storage_that_every_host_can_already_read() + { + using var _ = new EnvironmentVariableScope("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "AzureBlob"); + + Assert.DoesNotThrow(() => + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only")); + } + + static Settings CreateSettings(string persistenceType) => + new("LearningTransport", persistenceType, forwardErrorMessages: false, errorRetentionPeriod: TimeSpan.FromDays(10)); + + sealed class EnvironmentVariableScope : IDisposable + { + readonly string name; + + public EnvironmentVariableScope(string name, string value) + { + this.name = name; + Environment.SetEnvironmentVariable(name, value); + } + + public void Dispose() => Environment.SetEnvironmentVariable(name, null); + } + } +} diff --git a/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs new file mode 100644 index 0000000000..976797662c --- /dev/null +++ b/src/ServiceControl.UnitTests/Infrastructure/PersistenceManifestAuditCapabilityTests.cs @@ -0,0 +1,57 @@ +namespace ServiceControl.UnitTests.Infrastructure +{ + using System; + using System.IO; + using System.Text.Json; + using NUnit.Framework; + using ServiceControl.Persistence; + + [TestFixture] + public class PersistenceManifestAuditCapabilityTests + { + [Test] + public void Absent_property_means_no_audit_support() + { + var manifest = Deserialize(""" + { + "Name": "Whatever", + "DisplayName": "Whatever", + "Description": "Whatever", + "AssemblyName": "Whatever", + "TypeName": "Whatever, Whatever" + } + """); + + Assert.That(manifest.SupportsAuditIngestion, Is.False); + } + + [TestCase("ServiceControl.Persistence.RavenDB")] + [TestCase("ServiceControl.Persistence.EFCore.SqlServer")] + [TestCase("ServiceControl.Persistence.EFCore.PostgreSql")] + public void Shipped_primary_persisters_do_not_advertise_audit_support(string projectName) + { + var manifest = ReadManifest(projectName); + + Assert.That(manifest.SupportsAuditIngestion, Is.False, + $"{projectName} advertises audit ingestion, which makes the primary host ingest the audit queue. " + + "Only flip this once that persister can store and query audit data."); + } + + [Test] + public void The_test_persister_advertises_audit_support() + { + var manifest = ReadManifest("ServiceControl.Persistence.Tests.AuditCapable"); + + Assert.That(manifest.SupportsAuditIngestion, Is.True); + } + + static PersistenceManifest ReadManifest(string projectName) => + Deserialize(File.ReadAllText(Path.Combine(SourceDirectory, projectName, "persistence.manifest"))); + + static PersistenceManifest Deserialize(string json) => + JsonSerializer.Deserialize(json) ?? throw new InvalidOperationException("The manifest is empty or invalid."); + + static string SourceDirectory => + Path.GetFullPath(Path.Combine(TestContext.CurrentContext.TestDirectory, "..", "..", "..", "..")); + } +} diff --git a/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs b/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs new file mode 100644 index 0000000000..99900e76d5 --- /dev/null +++ b/src/ServiceControl.UnitTests/Infrastructure/PrimaryAssemblyBoundaryTests.cs @@ -0,0 +1,21 @@ +namespace ServiceControl.UnitTests.Infrastructure +{ + using System.Linq; + using NUnit.Framework; + using Particular.ServiceControl; + + [TestFixture] + public class PrimaryAssemblyBoundaryTests + { + [Test] + public void The_primary_does_not_reference_the_standalone_audit_executable() + { + var referenced = typeof(HostingComponent).Assembly.GetReferencedAssemblies().Select(name => name.Name); + + Assert.That(referenced, Does.Not.Contain("ServiceControl.Audit"), + "ServiceControl.Audit is a standalone composition root holding RavenDB persistence selection, standalone " + + "settings, API hosting, installer commands and its own NServiceBus endpoint. The primary owns a copy of " + + "the audit runtime instead, so that project stays off its reference graph."); + } + } +} diff --git a/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs b/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs new file mode 100644 index 0000000000..8e1dbb1256 --- /dev/null +++ b/src/ServiceControl.UnitTests/ScatterGather/LocalMessagesViewTests.cs @@ -0,0 +1,95 @@ +namespace ServiceControl.UnitTests.ScatterGather +{ + using System; + using System.Collections.Generic; + using System.Linq; + using NUnit.Framework; + using ServiceControl.CompositeViews.Messages; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.Infrastructure; + + [TestFixture] + public class LocalMessagesViewTests + { + [Test] + public void A_message_that_both_failed_and_was_audited_shows_as_failed() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("Receiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + Assert.That(result.Results.Single().Status, Is.EqualTo(MessageStatus.Failed)); + } + + [Test] + public void A_message_that_both_failed_and_was_audited_is_counted_once() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("Receiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results, Has.Count.EqualTo(1)); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(1)); + } + } + + [Test] + public void The_same_message_id_on_a_different_endpoint_is_a_different_message() + { + var failed = Message("Receiver", "1", MessageStatus.Failed); + var audited = Message("OtherReceiver", "1", MessageStatus.Successful); + + var result = LocalMessagesView.Merge([failed], [audited], new PagingInfo()); + + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(2)); + } + + [Test] + public void A_full_page_from_each_source_is_truncated_to_one_page() + { + var pagingInfo = new PagingInfo(pageSize: 5); + var failed = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"failed-{i}", MessageStatus.Failed)).ToArray(); + var audited = Enumerable.Range(0, 5).Select(i => Message("Receiver", $"audited-{i}", MessageStatus.Successful)).ToArray(); + + var result = LocalMessagesView.Merge(failed, audited, pagingInfo); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Results, Has.Count.EqualTo(5), "the scatter gather truncates to a page, so the page must already be the local answer"); + Assert.That(result.QueryStats.TotalCount, Is.EqualTo(10), "the total counts every distinct message, not just the page"); + } + } + + [Test] + public void The_page_is_taken_after_the_requested_order_is_applied() + { + var pagingInfo = new PagingInfo(pageSize: 2); + var failed = new[] { Message("Receiver", "c", MessageStatus.Failed), Message("Receiver", "a", MessageStatus.Failed) }; + var audited = new[] { Message("Receiver", "b", MessageStatus.Successful) }; + + var result = LocalMessagesView.Merge(failed, audited, pagingInfo, MessageViewComparer.FromSortInfo(new SortInfo("message_id", "asc"))); + + Assert.That(result.Results.Select(message => message.MessageId), Is.EqualTo(new[] { "a", "b" }).AsCollection); + } + + [Test] + public void The_local_key_matches_the_key_the_scatter_gather_deduplicates_on() + { + var message = Message("Receiver", "1", MessageStatus.Failed); + + Assert.That(LocalMessagesView.DeduplicationKey(message), Is.EqualTo("Receiver-1")); + } + + static MessagesView Message(string receivingEndpoint, string messageId, MessageStatus status) => new() + { + MessageId = messageId, + Status = status, + ReceivingEndpoint = new EndpointDetails { Name = receivingEndpoint, Host = "host", HostId = Guid.NewGuid() } + }; + } +} diff --git a/src/ServiceControl.slnx b/src/ServiceControl.slnx index 622050f094..39fb0c9392 100644 --- a/src/ServiceControl.slnx +++ b/src/ServiceControl.slnx @@ -51,6 +51,7 @@ + diff --git a/src/ServiceControl/Auditing/AuditComponent.cs b/src/ServiceControl/Auditing/AuditComponent.cs new file mode 100644 index 0000000000..5d28f29868 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditComponent.cs @@ -0,0 +1,93 @@ +namespace ServiceControl.Auditing +{ + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using Particular.LicensingComponent.AuditThroughput; + using Particular.ServiceControl; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Connection; + using ServiceControl.CustomChecks; + using ServiceControl.Infrastructure; + using ServiceControl.Infrastructure.Health; + using ServiceControl.Persistence; + using ServiceControl.Transports; + + // Registers nothing unless the configured persister advertises audit support in its manifest, so + // hosts on a persister that cannot store audit data behave exactly as they did before. + class AuditComponent : ServiceControlComponent + { + public override void Setup(Settings settings, IComponentInstallationContext context, IHostApplicationBuilder hostBuilder) + { + if (!SupportsAuditIngestion(settings)) + { + return; + } + + context.CreateQueue(settings.AuditQueue); + + if (settings.ForwardAuditMessages && settings.AuditLogQueue != null) + { + context.CreateQueue(settings.AuditLogQueue); + } + } + + public override void Configure(Settings settings, ITransportCustomization transportCustomization, IHostApplicationBuilder hostBuilder) + { + if (!SupportsAuditIngestion(settings)) + { + return; + } + + WarnAboutSettingCollisions(settings); + + var services = hostBuilder.Services; + + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + services.AddCustomCheck(); + services.AddCustomCheck(); + + services.AddHealthChecks() + .AddCheck("audit-ingestion", tags: [HealthCheckExtensions.ReadyTag]); + + if (settings.IngestAuditMessages) + { + services.AddHostedService(); + } + + if (!settings.IngestionOnly) + { + // Registered before the licensing component's own fallback, which uses TryAdd. + services.AddSingleton(); + services.AddPlatformConnectionProvider(); + } + + hostBuilder.AddAuditIngestionOpenTelemetry(settings); + } + + // ServiceControl and ServiceControl.Audit settings can both be set by bare environment variable + // name, and ServiceBus/AuditQueue is literally the same key for both processes, so a combined + // primary and a standalone audit instance sharing one environment file collide. That + // combination is unsupported, and this is the shape most likely to hit it. + static void WarnAboutSettingCollisions(Settings settings) + { + if (settings.RemoteInstances.Length == 0) + { + return; + } + + LoggerUtil.CreateStaticLogger(typeof(AuditComponent), settings.LoggingSettings.LogLevel) + .LogWarning("This instance ingests audit messages itself and also has {RemoteInstanceCount} audit remote(s) configured. " + + "Running both is not supported: the two processes read the same setting names, so a shared environment file makes them " + + "collide on the audit queue, retention, forwarding and ingestion settings.", settings.RemoteInstances.Length); + } + + internal static bool SupportsAuditIngestion(Settings settings) => + PersistenceManifestLibrary.Find(settings.PersistenceType)?.SupportsAuditIngestion ?? false; + } +} diff --git a/src/ServiceControl/Auditing/AuditEnricherContext.cs b/src/ServiceControl/Auditing/AuditEnricherContext.cs new file mode 100644 index 0000000000..846e0de77d --- /dev/null +++ b/src/ServiceControl/Auditing/AuditEnricherContext.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Auditing +{ + using System.Collections.Generic; + using System.Linq; + using NServiceBus.Transport; + using ServiceControl.Operations; + + // Unlike the standalone audit instance there is no ICommand overload. Endpoints detected from audit + // headers are collected here and written through the monitoring unit of work, the same way the error + // path does it, rather than sent to the primary's input queue. + class AuditEnricherContext(IReadOnlyDictionary headers, IList outgoingSends, IDictionary metadata) + { + List newEndpoints; + + public IReadOnlyDictionary Headers { get; } = headers; + + public IDictionary Metadata { get; } = metadata; + + public IEnumerable NewEndpoints => newEndpoints ?? Enumerable.Empty(); + + public void Add(EndpointDetails endpointDetails) + { + newEndpoints ??= []; + + newEndpoints.Add(endpointDetails); + } + + public void AddForSend(TransportOperation transportOperation) => outgoingSends.Add(transportOperation); + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestion.cs b/src/ServiceControl/Auditing/AuditIngestion.cs new file mode 100644 index 0000000000..a6742330e2 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestion.cs @@ -0,0 +1,372 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Channels; + using System.Threading.Tasks; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Infrastructure; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.Transports; + + class AuditIngestion : BackgroundService + { + public AuditIngestion( + Settings settings, + ITransportCustomization transportCustomization, + TransportSettings transportSettings, + IFailedAuditImportDataStore failedImportsStore, + AuditIngestionCustomCheck.State ingestionState, + AuditIngestor auditIngestor, + IIngestionUnitOfWorkFactory unitOfWorkFactory, + IHostApplicationLifetime applicationLifetime, + AuditIngestionMetrics metrics, + ILogger logger) + { + inputEndpoint = settings.AuditQueue; + this.transportCustomization = transportCustomization; + this.transportSettings = transportSettings; + this.auditIngestor = auditIngestor; + this.unitOfWorkFactory = unitOfWorkFactory; + this.settings = settings; + this.applicationLifetime = applicationLifetime; + this.metrics = metrics; + this.logger = logger; + + // Audit ingestion is scaled independently of the primary endpoint, whose concurrency the + // shared TransportSettings carries, so the receiver gets its own value. + maxConcurrency = settings.MaximumAuditIngestionConcurrencyLevel; + MaxBatchSize = maxConcurrency; + + channel = Channel.CreateBounded(new BoundedChannelOptions(MaxBatchSize) + { + SingleReader = true, + SingleWriter = false, + AllowSynchronousContinuations = false, + FullMode = BoundedChannelFullMode.Wait + }); + + errorHandlingPolicy = new AuditIngestionFaultPolicy(failedImportsStore, settings.LoggingSettings, OnCriticalError, metrics, logger); + + watchdog = new Watchdog( + "audit message ingestion", + EnsureStarted, + EnsureStopped, + ingestionState.ReportError, + ingestionState.Clear, + settings.TimeToRestartAuditIngestionAfterFailure, + logger); + } + + public override async Task StartAsync(CancellationToken cancellationToken = default) + { + await watchdog.Start(() => applicationLifetime.StopApplication(), cancellationToken); + await base.StartAsync(cancellationToken); + } + + protected override async Task ExecuteAsync(CancellationToken cancellationToken = default) + { + try + { + var contexts = new List(MaxBatchSize); + + while (await channel.Reader.WaitToReadAsync(cancellationToken)) + { + // will only enter here if there is something to read. + try + { + using var batchMetrics = metrics.BeginBatch(MaxBatchSize); + + // as long as there is something to read this will fetch up to MaximumConcurrency items + while (channel.Reader.TryRead(out var context)) + { + contexts.Add(context); + } + + await auditIngestor.Ingest(contexts, messageDispatcher, cancellationToken); + + batchMetrics.Complete(contexts.Count); + } + catch (OperationCanceledException e) when (cancellationToken.IsCancellationRequested) + { + // signal all message handling tasks to terminate + foreach (var context in contexts) + { + _ = context.GetTaskCompletionSource().TrySetException(e); + } + + logger.LogInformation(e, "Batch cancelled"); + break; + } + catch (Exception e) + { + // signal all message handling tasks to terminate + foreach (var context in contexts) + { + _ = context.GetTaskCompletionSource().TrySetException(e); + } + + logger.LogInformation(e, "Ingesting messages failed"); + } + finally + { + contexts.Clear(); + } + } + // will fall out here when writer is completed + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + // ExecuteAsync cancelled + } + } + + public override async Task StopAsync(CancellationToken cancellationToken = default) + { + try + { + // Order matters. Receiving stops under the shutdown token rather than a cancelled + // one, so messages already being processed finish and their receives commit instead + // of being abandoned and redelivered after having been forwarded. Nothing new enters + // the channel after this, and the infrastructure stays up until the channel drains. + await EnsureReceivingStopped(cancellationToken); + channel.Writer.Complete(); + await base.StopAsync(cancellationToken); + } + finally + { + // Tears the infrastructure down, now that nothing is left to dispatch. + await watchdog.Stop(cancellationToken); + } + } + + Task OnCriticalError(string failure, Exception exception, CancellationToken cancellationToken) + { + logger.LogCritical(exception, "OnCriticalError. '{Failure}'", failure); + return watchdog.OnFailure(failure, cancellationToken); + } + + async Task EnsureStarted(CancellationToken cancellationToken) + { + try + { + await startStopSemaphore.WaitAsync(cancellationToken); + + var canIngest = unitOfWorkFactory.CanIngestMore(); + + logger.LogDebug("Ensure started {CanIngest}", canIngest); + + if (canIngest) + { + await SetUpAndStartInfrastructure(cancellationToken); + } + else + { + await StopAndTeardownInfrastructure(cancellationToken); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + try + { + await StopAndTeardownInfrastructure(cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception teardownException) + { + throw new AggregateException(e, teardownException); + } + + throw; + } + finally + { + startStopSemaphore.Release(); + } + } + + async Task SetUpAndStartInfrastructure(CancellationToken cancellationToken) + { + if (messageReceiver != null) + { + logger.LogDebug("Infrastructure already Started"); + return; + } + + try + { + logger.LogInformation("Starting infrastructure"); + transportInfrastructure = await transportCustomization.CreateTransportInfrastructure( + inputEndpoint, + transportSettings, + OnMessage, + errorHandlingPolicy.OnError, + OnCriticalError, + TransportTransactionMode.ReceiveOnly, + maxConcurrency, + cancellationToken); + + messageReceiver = transportInfrastructure.Receivers[inputEndpoint]; + messageDispatcher = transportInfrastructure.Dispatcher; + + await auditIngestor.VerifyCanReachForwardingAddress(messageDispatcher, cancellationToken); + await messageReceiver.StartReceive(cancellationToken); + + logger.LogInformation(LogMessages.StartedInfrastructure); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Failed to start infrastructure"); + throw; + } + } + + async Task StopAndTeardownInfrastructure(CancellationToken cancellationToken) + { + if (transportInfrastructure == null) + { + logger.LogDebug("Infrastructure already Stopped"); + return; + } + + try + { + logger.LogInformation("Stopping infrastructure"); + try + { + await StopReceiving(cancellationToken); + } + finally + { + await transportInfrastructure.Shutdown(cancellationToken); + } + + messageReceiver = null; + transportInfrastructure = null; + receiveStopped = false; + + logger.LogInformation(LogMessages.StoppedInfrastructure); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Failed to stop infrastructure"); + throw; + } + } + + async Task EnsureStopped(CancellationToken cancellationToken) + { + try + { + await startStopSemaphore.WaitAsync(cancellationToken); + + // By passing a CancellationToken in the cancelled state we stop receivers ASAP and + // still correctly stop/shutdown + await StopAndTeardownInfrastructure(new CancellationToken(canceled: true)); + } + finally + { + startStopSemaphore.Release(); + } + } + + async Task EnsureReceivingStopped(CancellationToken cancellationToken) + { + await startStopSemaphore.WaitAsync(cancellationToken); + + try + { + await StopReceiving(cancellationToken); + } + finally + { + startStopSemaphore.Release(); + } + } + + // Stops the receiver on its own, leaving the infrastructure up. Idempotent because a + // shutdown stops receiving before draining and then tears down, so this runs twice. + async Task StopReceiving(CancellationToken cancellationToken) + { + if (messageReceiver == null || receiveStopped) + { + return; + } + + await messageReceiver.StopReceive(cancellationToken); + receiveStopped = true; + } + + async Task OnMessage(MessageContext messageContext, CancellationToken cancellationToken) + { + using var messageIngestionMetrics = metrics.BeginIngestion(messageContext); + + if (settings.MessageFilter != null && settings.MessageFilter(messageContext)) + { + messageIngestionMetrics.Skipped(); + return; + } + + var taskCompletionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + messageContext.SetTaskCompletionSource(taskCompletionSource); + + await channel.Writer.WriteAsync(messageContext, cancellationToken); + _ = await taskCompletionSource.Task; + + messageIngestionMetrics.Success(); + } + + TransportInfrastructure transportInfrastructure; + IMessageReceiver messageReceiver; + bool receiveStopped; + + // Left in place when the infrastructure is torn down. A shutdown drains before tearing down, + // so this is still usable there. + IMessageDispatcher messageDispatcher; + + readonly int MaxBatchSize; + readonly int maxConcurrency; + readonly SemaphoreSlim startStopSemaphore = new(1); + readonly string inputEndpoint; + readonly ITransportCustomization transportCustomization; + readonly TransportSettings transportSettings; + readonly AuditIngestor auditIngestor; + readonly AuditIngestionFaultPolicy errorHandlingPolicy; + readonly IIngestionUnitOfWorkFactory unitOfWorkFactory; + readonly Settings settings; + readonly Channel channel; + readonly Watchdog watchdog; + readonly IHostApplicationLifetime applicationLifetime; + readonly AuditIngestionMetrics metrics; + readonly ILogger logger; + + internal static class LogMessages + { + internal const string StartedInfrastructure = "Started infrastructure"; + internal const string StoppedInfrastructure = "Stopped infrastructure"; + } + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs b/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs new file mode 100644 index 0000000000..e1ad653b01 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestionCustomCheck.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using NServiceBus.CustomChecks; + + class AuditIngestionCustomCheck(AuditIngestionCustomCheck.State criticalErrorHolder) + : CustomCheck("Audit Message Ingestion Process", "ServiceControl Health", TimeSpan.FromSeconds(5)) + { + public override Task PerformCheck(CancellationToken cancellationToken = default) + { + var failure = criticalErrorHolder.GetLastFailure(); + return failure == null + ? successResult + : Task.FromResult(CheckResult.Failed(failure)); + } + + static readonly Task successResult = Task.FromResult(CheckResult.Pass); + + public class State + { + volatile string lastFailure; + + public void Clear() => lastFailure = null; + public void ReportError(string failure) => lastFailure = failure; + public string GetLastFailure() => lastFailure; + } + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs new file mode 100644 index 0000000000..7b3e6c0cbe --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestionFaultPolicy.cs @@ -0,0 +1,112 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Diagnostics; + using System.IO; + using System.Runtime.InteropServices; + using System.Runtime.Versioning; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Installers; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Configuration; + using ServiceControl.Infrastructure; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class AuditIngestionFaultPolicy + { + public AuditIngestionFaultPolicy( + IFailedAuditImportDataStore store, + LoggingSettings loggingSettings, + Func onCriticalError, + AuditIngestionMetrics metrics, + ILogger logger) + { + this.store = store; + this.metrics = metrics; + this.logger = logger; + failureCircuitBreaker = new ImportFailureCircuitBreaker(onCriticalError); + + if (!AppEnvironment.RunningInContainer) + { + logPath = Path.Combine(loggingSettings.LogPath, "FailedImports", "Audit"); + Directory.CreateDirectory(logPath); + } + } + + public async Task OnError(ErrorContext errorContext, CancellationToken cancellationToken = default) + { + using var errorMetrics = metrics.BeginErrorHandling(errorContext); + + //Same as recoverability policy in NServiceBusFactory + if (errorContext.ImmediateProcessingFailures < 3) + { + errorMetrics.Retry(); + return ErrorHandleResult.RetryRequired; + } + + await Handle(errorContext, cancellationToken); + return ErrorHandleResult.Handled; + } + + async Task Handle(ErrorContext errorContext, CancellationToken cancellationToken) + { + var failure = new FailedAuditImport + { + Message = new FailedTransportMessage + { + Id = errorContext.MessageId, + Headers = errorContext.Headers, + Body = errorContext.Body.ToArray() + }, + ExceptionInfo = errorContext.Exception.ToFriendlyString(), + Id = FailedAuditImport.DeriveKey(errorContext.Headers, errorContext.MessageId).ToString() + }; + + try + { + await DoLogging(errorContext.Exception, failure, cancellationToken); + } + finally + { + failureCircuitBreaker.Increment(errorContext.Exception); + } + } + + async Task DoLogging(Exception exception, FailedAuditImport failure, CancellationToken cancellationToken) + { + logger.LogError(exception, "Failed importing audit message"); + + await store.StoreFailedAuditImport(failure, cancellationToken); + + if (!AppEnvironment.RunningInContainer) + { + var filePath = Path.Combine(logPath, $"FailedAuditImports_{failure.Id.Replace("/", "_")}.txt"); + await File.WriteAllTextAsync(filePath, failure.ExceptionInfo, cancellationToken); + + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + WriteToEventLog($"An audit message import has failed. A log file has been written to {filePath}"); + } + } + } + + [SupportedOSPlatform("windows")] + static void WriteToEventLog(string message) + { +#if DEBUG + EventSourceCreator.Create(); +#endif + EventLog.WriteEntry(EventSourceCreator.SourceName, message, EventLogEntryType.Error); + } + + readonly IFailedAuditImportDataStore store; + readonly AuditIngestionMetrics metrics; + readonly ImportFailureCircuitBreaker failureCircuitBreaker; + readonly string logPath; + readonly ILogger logger; + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestionOpenTelemetry.cs b/src/ServiceControl/Auditing/AuditIngestionOpenTelemetry.cs new file mode 100644 index 0000000000..8415b3c710 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestionOpenTelemetry.cs @@ -0,0 +1,49 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Diagnostics; + using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.Hosting; + using Microsoft.Extensions.Logging; + using OpenTelemetry.Metrics; + using OpenTelemetry.Resources; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing.Metrics; + using ServiceControl.Infrastructure; + + static class AuditIngestionOpenTelemetry + { + public static void AddAuditIngestionOpenTelemetry(this IHostApplicationBuilder builder, Settings settings) + { + if (string.IsNullOrEmpty(settings.OtlpEndpointUrl)) + { + return; + } + + if (!Uri.TryCreate(settings.OtlpEndpointUrl, UriKind.Absolute, out var otelMetricsUri)) + { + throw new UriFormatException($"Invalid OtlpEndpointUrl: {settings.OtlpEndpointUrl}"); + } + + var version = FileVersionInfo.GetVersionInfo(typeof(AuditIngestionOpenTelemetry).Assembly.Location).ProductVersion; + + builder.Services.AddOpenTelemetry() + .ConfigureResource(b => b.AddService( + serviceName: settings.InstanceName, + serviceVersion: version, + autoGenerateServiceInstanceId: true)) + .WithMetrics(b => + { + b.AddAuditIngestionMetrics(); + b.AddOtlpExporter(e => e.Endpoint = otelMetricsUri); + if (Debugger.IsAttached) + { + b.AddConsoleExporter(); + } + }); + + var logger = LoggerUtil.CreateStaticLogger(typeof(AuditIngestionOpenTelemetry), settings.LoggingSettings.LogLevel); + logger.LogInformation("OpenTelemetry metrics exporter enabled: {OtlpEndpointUrl}", settings.OtlpEndpointUrl); + } + } +} diff --git a/src/ServiceControl/Auditing/AuditIngestor.cs b/src/ServiceControl/Auditing/AuditIngestor.cs new file mode 100644 index 0000000000..520ec57f09 --- /dev/null +++ b/src/ServiceControl/Auditing/AuditIngestor.cs @@ -0,0 +1,180 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Routing; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.Transports; + + class AuditIngestor + { + public AuditIngestor( + Settings settings, + IIngestionUnitOfWorkFactory unitOfWorkFactory, + IEndpointInstanceMonitoring endpointInstanceMonitoring, + ITransportCustomization transportCustomization, + ILogger logger) + { + this.settings = settings; + this.unitOfWorkFactory = unitOfWorkFactory; + this.logger = logger; + + logQueueAddress = transportCustomization.ToTransportQualifiedQueueName(settings.AuditLogQueue); + + IEnrichImportedAuditMessages[] enrichers = + [ + new AuditMessageTypeEnricher(), + new AuditEnrichWithTrackingIds(), + new AuditProcessingStatisticsEnricher(), + new DetectNewEndpointsFromAuditImportsEnricher(endpointInstanceMonitoring), + new DetectSuccessfulRetriesEnricher(), + new SagaRelationshipsEnricher() + ]; + + processor = new AuditProcessor(enrichers, logger); + } + + public async Task Ingest(List contexts, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + var stored = await Store(contexts, dispatcher, cancellationToken); + + try + { + if (settings.ForwardAuditMessages) + { + await Forward(stored, logQueueAddress, dispatcher, cancellationToken); + } + + foreach (var context in contexts) + { + context.GetTaskCompletionSource().TrySetResult(true); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Forwarding messages failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + } + + async Task> Store(IReadOnlyList contexts, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + // deliberately not using the using statement because we dispose async explicitly + IIngestionUnitOfWork unitOfWork = null; + try + { + unitOfWork = await unitOfWorkFactory.StartNew(cancellationToken); + + var storedContexts = await processor.Process(contexts, unitOfWork, dispatcher, cancellationToken); + + await unitOfWork.Complete(cancellationToken); + + return storedContexts; + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Bulk insertion failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + finally + { + if (unitOfWork != null) + { + try + { + // this can throw even though dispose is never supposed to throw + await unitOfWork.DisposeAsync(); + } + catch (Exception e) + { + logger.LogWarning(e, "Bulk insertion dispose failed"); + + // making sure to rethrow so that all messages get marked as failed + throw; + } + } + } + } + + static Task Forward(IReadOnlyCollection messageContexts, string forwardingAddress, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + var transportOperations = new List(messageContexts.Count); + MessageContext anyContext = null; + foreach (var messageContext in messageContexts) + { + if (messageContext.Extensions.TryGet("AuditType", out string auditType) + && auditType != "ProcessedMessage") + { + continue; + } + + anyContext = messageContext; + var outgoingMessage = new OutgoingMessage( + messageContext.NativeMessageId, + messageContext.Headers, + messageContext.Body); + + // Forwarded messages should last as long as possible + outgoingMessage.Headers.Remove(Headers.TimeToBeReceived); + + transportOperations.Add(new TransportOperation(outgoingMessage, new UnicastAddressTag(forwardingAddress))); + } + + return anyContext != null + ? dispatcher.Dispatch(new TransportOperations([.. transportOperations]), anyContext.TransportTransaction, cancellationToken) + : Task.CompletedTask; + } + + public async Task VerifyCanReachForwardingAddress(IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + if (!settings.ForwardAuditMessages) + { + return; + } + + try + { + var transportOperations = new TransportOperations( + new TransportOperation( + new OutgoingMessage(Guid.Empty.ToString("N"), [], Array.Empty()), + new UnicastAddressTag(logQueueAddress))); + + await dispatcher.Dispatch(transportOperations, new TransportTransaction(), cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + throw new Exception($"Unable to write to forwarding queue {settings.AuditLogQueue}", e); + } + } + + readonly AuditProcessor processor; + readonly IIngestionUnitOfWorkFactory unitOfWorkFactory; + readonly Settings settings; + readonly string logQueueAddress; + readonly ILogger logger; + } +} diff --git a/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs b/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs new file mode 100644 index 0000000000..7181afd9ab --- /dev/null +++ b/src/ServiceControl/Auditing/AuditPlatformConnectionDetailsProvider.cs @@ -0,0 +1,43 @@ +namespace ServiceControl.Auditing +{ + using System.Threading; + using System.Threading.Tasks; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Connection; + + // /api/connection is what ServicePulse and the Platform Connector plugin read to configure + // endpoints. Without an audit remote it stops advertising where audit and saga data should go, so a + // primary holding audit data locally supplies the same shapes the audit instance supplies. + class AuditPlatformConnectionDetailsProvider(Settings settings) : IProvidePlatformConnectionDetails + { + public Task ProvideConnectionDetails(PlatformConnectionDetails connection, CancellationToken cancellationToken = default) + { + connection.Add("MessageAudit", new MessageAuditConnectionDetails + { + Enabled = true, + AuditQueue = settings.AuditQueue + }); + + connection.Add("SagaAudit", new SagaAuditConnectionDetails + { + Enabled = true, + SagaAuditQueue = settings.AuditQueue + }); + + return Task.CompletedTask; + } + + // HINT: These should match the types in the PlatformConnector package + public class MessageAuditConnectionDetails + { + public bool Enabled { get; set; } + public string AuditQueue { get; set; } + } + + public class SagaAuditConnectionDetails + { + public bool Enabled { get; set; } + public string SagaAuditQueue { get; set; } + } + } +} diff --git a/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs b/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs new file mode 100644 index 0000000000..4d4dcb3f2a --- /dev/null +++ b/src/ServiceControl/Auditing/AuditProcessingStatisticsEnricher.cs @@ -0,0 +1,65 @@ +namespace ServiceControl.Auditing +{ + using System; + using NServiceBus; + + class AuditProcessingStatisticsEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var headers = context.Headers; + var metadata = context.Metadata; + var processingEnded = DateTime.MinValue; + var startTime = DateTime.MinValue; + var processingStarted = DateTime.MinValue; + + if (headers.TryGetValue(Headers.TimeSent, out var timeSentValue)) + { + startTime = DateTimeOffsetHelper.ToDateTimeOffset(timeSentValue).UtcDateTime; + metadata.Add("TimeSent", startTime); + } + + if (headers.TryGetValue(Headers.DeliverAt, out var deliverAtValue)) + { + startTime = DateTimeOffsetHelper.ToDateTimeOffset(deliverAtValue).UtcDateTime; + } + + if (headers.TryGetValue(Headers.ProcessingStarted, out var processingStartedValue)) + { + processingStarted = DateTimeOffsetHelper.ToDateTimeOffset(processingStartedValue).UtcDateTime; + } + + if (headers.TryGetValue(Headers.ProcessingEnded, out var processingEndedValue)) + { + processingEnded = DateTimeOffsetHelper.ToDateTimeOffset(processingEndedValue).UtcDateTime; + } + + var criticalTime = TimeSpan.Zero; + + if (processingEnded != DateTime.MinValue && startTime != DateTime.MinValue) + { + criticalTime = processingEnded - startTime; + } + + metadata.Add("CriticalTime", criticalTime); + + var processingTime = TimeSpan.Zero; + + if (processingEnded != DateTime.MinValue && processingStarted != DateTime.MinValue) + { + processingTime = processingEnded - processingStarted; + } + + metadata.Add("ProcessingTime", processingTime); + + var deliveryTime = TimeSpan.Zero; + + if (processingStarted != DateTime.MinValue && startTime != DateTime.MinValue) + { + deliveryTime = processingStarted - startTime; + } + + metadata.Add("DeliveryTime", deliveryTime); + } + } +} diff --git a/src/ServiceControl/Auditing/AuditProcessor.cs b/src/ServiceControl/Auditing/AuditProcessor.cs new file mode 100644 index 0000000000..2a5dc7018c --- /dev/null +++ b/src/ServiceControl/Auditing/AuditProcessor.cs @@ -0,0 +1,169 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using System.Text.Json; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus; + using NServiceBus.Transport; + using ServiceControl.EndpointPlugin.Messages.SagaState; + using ServiceControl.Infrastructure; + using ServiceControl.MessageAuditing; + using ServiceControl.Operations; + using ServiceControl.Persistence; + using ServiceControl.Persistence.UnitOfWork; + using ServiceControl.SagaAudit; + + class AuditProcessor(IEnrichImportedAuditMessages[] enrichers, ILogger logger) + { + public async Task> Process(IReadOnlyList contexts, IIngestionUnitOfWork unitOfWork, IMessageDispatcher dispatcher, CancellationToken cancellationToken = default) + { + var audit = unitOfWork.Audit + ?? throw new InvalidOperationException("The configured persistence does not support audit ingestion."); + var monitoring = unitOfWork.Monitoring + ?? throw new InvalidOperationException("The configured persistence does not support monitoring."); + + var storedContexts = new List(contexts.Count); + + var tasks = new List(contexts.Count); + foreach (var context in contexts) + { + tasks.Add(ProcessMessage(context, dispatcher, cancellationToken)); + } + + await Task.WhenAll(tasks); + + var knownEndpoints = new Dictionary(); + + foreach (var context in contexts) + { + // Any message context that failed during processing will have a faulted task and should be skipped + if (context.GetTaskCompletionSource().Task.IsFaulted) + { + continue; + } + + if (context.Extensions.TryGet(out ProcessedMessage processedMessage)) + { + await audit.RecordProcessedMessage(processedMessage, context.Body, cancellationToken); + } + else if (context.Extensions.TryGet(out SagaSnapshot sagaSnapshot)) + { + await audit.RecordSagaSnapshot(sagaSnapshot, cancellationToken); + } + + if (context.Extensions.TryGet>(out var newEndpoints)) + { + foreach (var endpointDetails in newEndpoints) + { + RecordKnownEndpoint(endpointDetails, knownEndpoints); + } + } + + storedContexts.Add(context); + } + + foreach (var endpoint in knownEndpoints.Values) + { + await monitoring.RecordKnownEndpoint(endpoint, cancellationToken); + } + + return storedContexts; + } + + async Task ProcessMessage(MessageContext context, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + if (context.Headers.TryGetValue(Headers.EnclosedMessageTypes, out var messageType) + && messageType == typeof(SagaUpdatedMessage).FullName) + { + ProcessSagaAuditMessage(context); + } + else + { + await ProcessAuditMessage(context, dispatcher, cancellationToken); + } + } + + void ProcessSagaAuditMessage(MessageContext context) + { + try + { + using var stream = new ReadOnlyStream(context.Body); + var message = JsonSerializer.Deserialize(stream, SagaAuditMessagesSerializationContext.Default.SagaUpdatedMessage); + + var sagaSnapshot = SagaSnapshotFactory.Create(message); + + context.Extensions.Set("AuditType", "SagaSnapshot"); + context.Extensions.Set(sagaSnapshot); + } + catch (Exception e) + { + logger.LogWarning(e, "Processing of saga audit message '{NativeMessageId}' failed", context.NativeMessageId); + + // releasing the failed message context early so that they can be retried outside the current batch + context.GetTaskCompletionSource().TrySetException(e); + } + } + + async Task ProcessAuditMessage(MessageContext context, IMessageDispatcher dispatcher, CancellationToken cancellationToken) + { + if (!context.Headers.TryGetValue(Headers.MessageId, out var messageId)) + { + messageId = DeterministicGuid.MakeId(context.NativeMessageId).ToString(); + } + + try + { + var metadata = new Dictionary + { + ["MessageId"] = messageId, + ["MessageIntent"] = context.Headers.MessageIntent() + }; + + var messagesToEmit = new List(); + var enricherContext = new AuditEnricherContext(context.Headers, messagesToEmit, metadata); + + foreach (var enricher in enrichers) + { + enricher.Enrich(enricherContext); + } + + var auditMessage = new ProcessedMessage(context.Headers, new Dictionary(metadata)); + + //Do not hook into the incoming transaction + await dispatcher.Dispatch(new TransportOperations([.. messagesToEmit]), new TransportTransaction(), cancellationToken); + + context.Extensions.Set("AuditType", "ProcessedMessage"); + context.Extensions.Set(auditMessage); + context.Extensions.Set(enricherContext.NewEndpoints); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogWarning(e, "Processing of message '{MessageId}' failed", messageId); + + // releasing the failed message context early so that they can be retried outside the current batch + context.GetTaskCompletionSource().TrySetException(e); + } + } + + static void RecordKnownEndpoint(EndpointDetails observedEndpoint, Dictionary observedEndpoints) + { + var uniqueEndpointId = $"{observedEndpoint.Name}{observedEndpoint.HostId}"; + if (!observedEndpoints.ContainsKey(uniqueEndpointId)) + { + observedEndpoints.Add(uniqueEndpointId, new KnownEndpoint + { + EndpointDetails = observedEndpoint, + HostDisplayName = observedEndpoint.Host, + Monitored = false + }); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DefaultEnrichers.cs b/src/ServiceControl/Auditing/DefaultEnrichers.cs new file mode 100644 index 0000000000..f14f7e4e8f --- /dev/null +++ b/src/ServiceControl/Auditing/DefaultEnrichers.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Auditing +{ + using System.Linq; + using NServiceBus; + + class AuditMessageTypeEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var isSystemMessage = false; + string messageType = null; + + if (context.Headers.ContainsKey(Headers.ControlMessageHeader)) + { + isSystemMessage = true; + } + + if (context.Headers.TryGetValue(Headers.EnclosedMessageTypes, out var enclosedMessageTypes)) + { + messageType = GetMessageType(enclosedMessageTypes); + isSystemMessage = DetectSystemMessage(messageType); + context.Metadata.Add("SearchableMessageType", messageType.Replace(".", " ").Replace("+", " ")); + } + + context.Metadata.Add("IsSystemMessage", isSystemMessage); + context.Metadata.Add("MessageType", messageType); + } + + static bool DetectSystemMessage(string messageTypeString) => + messageTypeString.Contains("NServiceBus.Scheduling.Messages.ScheduledTask"); + + static string GetMessageType(string messageTypeString) => + messageTypeString.Contains(',') ? messageTypeString.Split(',').First() : messageTypeString; + } + + class AuditEnrichWithTrackingIds : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + if (context.Headers.TryGetValue(Headers.ConversationId, out var conversationId)) + { + context.Metadata.Add("ConversationId", conversationId); + } + + if (context.Headers.TryGetValue(Headers.RelatedTo, out var relatedToId)) + { + context.Metadata.Add("RelatedToId", relatedToId); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs new file mode 100644 index 0000000000..cfdb2d603a --- /dev/null +++ b/src/ServiceControl/Auditing/DetectNewEndpointsFromAuditImportsEnricher.cs @@ -0,0 +1,46 @@ +namespace ServiceControl.Auditing +{ + using System; + using ServiceControl.Contracts.Operations; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class DetectNewEndpointsFromAuditImportsEnricher(IEndpointInstanceMonitoring monitoring) : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var sendingEndpoint = EndpointDetailsParser.SendingEndpoint(context.Headers); + + // SendingEndpoint will be null for messages that are from v3.3.x endpoints because we don't + // have the relevant information via the headers, which were added in v4. + if (sendingEndpoint != null) + { + TryAddEndpoint(sendingEndpoint, context); + context.Metadata.Add("SendingEndpoint", sendingEndpoint); + } + + var receivingEndpoint = EndpointDetailsParser.ReceivingEndpoint(context.Headers); + // The ReceivingEndpoint will be null for messages from v3.3.x endpoints that were successfully + // processed because we dont have the information from the relevant headers. + if (receivingEndpoint != null) + { + TryAddEndpoint(receivingEndpoint, context); + context.Metadata.Add("ReceivingEndpoint", receivingEndpoint); + } + } + + void TryAddEndpoint(EndpointDetails endpointDetails, AuditEnricherContext context) + { + // for backwards compat with version before 4_5 we might not have a hostid + if (endpointDetails.HostId == Guid.Empty) + { + return; + } + + if (monitoring.IsNewInstance(endpointDetails)) + { + context.Add(endpointDetails); + } + } + } +} diff --git a/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs b/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs new file mode 100644 index 0000000000..9d953cf0b9 --- /dev/null +++ b/src/ServiceControl/Auditing/DetectSuccessfulRetriesEnricher.cs @@ -0,0 +1,45 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Collections.Generic; + using NServiceBus; + using NServiceBus.Routing; + using NServiceBus.Transport; + + class DetectSuccessfulRetriesEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) + { + var headers = context.Headers; + var isOldRetry = headers.TryGetValue("ServiceControl.RetryId", out _); + var isNewRetry = headers.TryGetValue("ServiceControl.Retry.UniqueMessageId", out var newRetryMessageId); + var isAckHandled = headers.ContainsKey("ServiceControl.Retry.AcknowledgementSent"); + var hasAckQueue = headers.TryGetValue("ServiceControl.Retry.AcknowledgementQueue", out var ackQueue); + + var hasBeenRetried = isOldRetry || isNewRetry; + + context.Metadata.Add("IsRetried", hasBeenRetried); + + if (!hasBeenRetried || isAckHandled) + { + //The message has not been sent for retry from ServiceControl or the endpoint indicated that is already has sent a retry acknowledgement to the + //ServiceControl main instance. Nothing to do. + return; + } + + if (hasAckQueue && isNewRetry) + { + // The acknowledgement queue is named by whichever instance issued the retry, so this stays a + // transport operation rather than a direct write to the local recoverability unit of work. + // In a combined host it simply comes back in through local error ingestion. + var ackMessage = new OutgoingMessage(Guid.NewGuid().ToString(), new Dictionary + { + ["ServiceControl.Retry.Successful"] = DateTimeOffsetHelper.ToWireFormattedString(DateTimeOffset.UtcNow), + ["ServiceControl.Retry.UniqueMessageId"] = newRetryMessageId + }, Array.Empty()); + var ackOperation = new TransportOperation(ackMessage, new UnicastAddressTag(ackQueue)); + context.AddForSend(ackOperation); + } + } + } +} diff --git a/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs b/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs new file mode 100644 index 0000000000..ffc5c9600f --- /dev/null +++ b/src/ServiceControl/Auditing/FailedAuditImportCustomCheck.cs @@ -0,0 +1,29 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.CustomChecks; + using ServiceControl.Persistence; + + // Deliberately named and categorised differently from the standalone audit instance's check, which + // reports into this same primary through ReportCustomChecksTo and would otherwise collide. + class FailedAuditImportCustomCheck(IFailedAuditImportDataStore store, ILogger logger) + : CustomCheck("Audit Message Ingestion (local)", "ServiceControl Health", TimeSpan.FromHours(1)) + { + public override async Task PerformCheck(CancellationToken cancellationToken = default) + { + if (await store.QueryContainsFailedImports(cancellationToken)) + { + logger.LogWarning(message); + return CheckResult.Failed(message); + } + + return CheckResult.Pass; + } + + const string message = @"One or more audit messages have failed to import properly into ServiceControl and have been stored in the ServiceControl database. +The import of these messages could have failed for a number of reasons and ServiceControl is not able to automatically reimport them. For guidance on how to resolve this see https://docs.particular.net/servicecontrol/import-failed-messages"; + } +} diff --git a/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs b/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs new file mode 100644 index 0000000000..1468902d89 --- /dev/null +++ b/src/ServiceControl/Auditing/IEnrichImportedAuditMessages.cs @@ -0,0 +1,7 @@ +namespace ServiceControl.Auditing +{ + interface IEnrichImportedAuditMessages + { + void Enrich(AuditEnricherContext context); + } +} diff --git a/src/ServiceControl/Auditing/ImportFailedAudits.cs b/src/ServiceControl/Auditing/ImportFailedAudits.cs new file mode 100644 index 0000000000..248f4975a6 --- /dev/null +++ b/src/ServiceControl/Auditing/ImportFailedAudits.cs @@ -0,0 +1,70 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging; + using NServiceBus.Extensibility; + using NServiceBus.Transport; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Operations; + using ServiceControl.Persistence; + + class ImportFailedAudits( + IFailedAuditImportDataStore failedAuditStore, + AuditIngestor auditIngestor, + Lazy messageDispatcher, + Settings settings, + ILogger logger) + { + public async Task Run(CancellationToken cancellationToken = default) + { + await auditIngestor.VerifyCanReachForwardingAddress(messageDispatcher.Value, cancellationToken); + + var succeeded = 0; + var failed = 0; + + await failedAuditStore.ProcessFailedAuditImports(async (transportMessage, token) => + { + try + { + var messageContext = new MessageContext( + transportMessage.Id, + transportMessage.Headers, + transportMessage.Body, + EmptyTransaction, + settings.AuditQueue, + EmptyContextBag); + var taskCompletionSource = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + messageContext.SetTaskCompletionSource(taskCompletionSource); + + await auditIngestor.Ingest([messageContext], messageDispatcher.Value, token); + + await taskCompletionSource.Task; + + succeeded++; + logger.LogDebug("Successfully re-imported failed audit message {MessageId}", transportMessage.Id); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + throw; + } + catch (Exception e) + { + logger.LogError(e, "Error while attempting to re-import failed audit message {MessageId}", transportMessage.Id); + failed++; + } + }, cancellationToken); + + logger.LogInformation("Done re-importing failed audits. Successfully re-imported {SuccessCount} messages. Failed re-importing {FailureCount} messages", succeeded, failed); + + if (failed > 0) + { + logger.LogWarning("{FailureCount} messages could not be re-imported. This could indicate a problem with the data. Contact Particular support if you need help with recovering the messages", failed); + } + } + + static readonly TransportTransaction EmptyTransaction = new(); + static readonly ContextBag EmptyContextBag = new(); + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditBatchMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditBatchMetrics.cs new file mode 100644 index 0000000000..edc993f218 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditBatchMetrics.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Auditing.Metrics +{ + using System; + using System.Diagnostics; + using System.Diagnostics.Metrics; + + public record AuditBatchMetrics(int MaxBatchSize, Histogram BatchDuration, Action IsSuccess) : IDisposable + { + public void Dispose() + { + var isSuccess = actualBatchSize > 0; + + IsSuccess(isSuccess); + + string result; + + if (isSuccess) + { + result = actualBatchSize == MaxBatchSize ? "full" : "partial"; + } + else + { + result = "failed"; + } + + BatchDuration.Record(sw.Elapsed.TotalSeconds, new TagList { { "result", result } }); + } + + public void Complete(int size) => actualBatchSize = size; + + int actualBatchSize = -1; + readonly Stopwatch sw = Stopwatch.StartNew(); + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditErrorMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditErrorMetrics.cs new file mode 100644 index 0000000000..f52f286b0c --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditErrorMetrics.cs @@ -0,0 +1,22 @@ +namespace ServiceControl.Auditing.Metrics +{ + using System; + using System.Diagnostics.Metrics; + using NServiceBus.Transport; + + public record AuditErrorMetrics(ErrorContext Context, Counter Failures) : IDisposable + { + public void Dispose() + { + var tags = AuditIngestionMetrics.GetMessageTags(Context.Headers); + + tags.Add("result", retry ? "retry" : "stored-poison"); + + Failures.Add(1, tags); + } + + public void Retry() => retry = true; + + bool retry; + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs new file mode 100644 index 0000000000..aaa5b0f9b8 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetrics.cs @@ -0,0 +1,76 @@ +namespace ServiceControl.Auditing.Metrics +{ + using System.Collections.Generic; + using System.Diagnostics; + using System.Diagnostics.Metrics; + using NServiceBus; + using NServiceBus.Transport; + using ServiceControl.EndpointPlugin.Messages.SagaState; + + public class AuditIngestionMetrics + { + public const string MeterName = "Particular.ServiceControl"; + + public static readonly string BatchDurationInstrumentName = $"{InstrumentPrefix}.batch_duration_seconds"; + public static readonly string MessageDurationInstrumentName = $"{InstrumentPrefix}.message_duration_seconds"; + + public AuditIngestionMetrics(IMeterFactory meterFactory) + { + var meter = meterFactory.Create(MeterName, MeterVersion); + + batchDuration = meter.CreateHistogram(BatchDurationInstrumentName, unit: "seconds", "Message batch processing duration in seconds"); + ingestionDuration = meter.CreateHistogram(MessageDurationInstrumentName, unit: "seconds", description: "Audit message processing duration in seconds"); + consecutiveBatchFailureGauge = meter.CreateObservableGauge($"{InstrumentPrefix}.consecutive_batch_failures_total", () => consecutiveBatchFailures, description: "Consecutive audit ingestion batch failures"); + failureCounter = meter.CreateCounter($"{InstrumentPrefix}.failures_total", description: "Audit ingestion failure count"); + } + + public AuditMessageMetrics BeginIngestion(MessageContext messageContext) => new(messageContext, ingestionDuration); + + public AuditErrorMetrics BeginErrorHandling(ErrorContext errorContext) => new(errorContext, failureCounter); + + public AuditBatchMetrics BeginBatch(int maxBatchSize) => new(maxBatchSize, batchDuration, RecordBatchOutcome); + + public static TagList GetMessageTags(Dictionary headers) + { + var tags = new TagList(); + + if (headers.TryGetValue(Headers.EnclosedMessageTypes, out var messageType)) + { + tags.Add("message.category", messageType == SagaUpdateMessageType ? "saga-update" : "audit-message"); + } + else + { + tags.Add("message.category", "control-message"); + } + + return tags; + } + + void RecordBatchOutcome(bool success) + { + if (success) + { + consecutiveBatchFailures = 0; + } + else + { + consecutiveBatchFailures++; + } + } + + long consecutiveBatchFailures; + + readonly Histogram batchDuration; +#pragma warning disable IDE0052 + // this can be changed to Gauge once we can use the latest version of System.Diagnostics.DiagnosticSource + readonly ObservableGauge consecutiveBatchFailureGauge; +#pragma warning restore IDE0052 + readonly Histogram ingestionDuration; + readonly Counter failureCounter; + + const string MeterVersion = "0.1.0"; + const string InstrumentPrefix = "sc.audit.ingestion"; + + static readonly string SagaUpdateMessageType = typeof(SagaUpdatedMessage).FullName; + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs new file mode 100644 index 0000000000..3a524b63f3 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditIngestionMetricsConfiguration.cs @@ -0,0 +1,20 @@ +namespace ServiceControl.Auditing.Metrics +{ + using OpenTelemetry.Metrics; + + public static class AuditIngestionMetricsConfiguration + { + public static void AddAuditIngestionMetrics(this MeterProviderBuilder builder) + { + builder.AddMeter(AuditIngestionMetrics.MeterName); + + // Note: Views can be replaced by new InstrumentAdvice { HistogramBucketBoundaries = [...] }; once we can update to the latest OpenTelemetry packages + builder.AddView( + instrumentName: AuditIngestionMetrics.MessageDurationInstrumentName, + new ExplicitBucketHistogramConfiguration { Boundaries = [0.01, 0.05, 0.1, 0.5, 1, 5] }); + builder.AddView( + instrumentName: AuditIngestionMetrics.BatchDurationInstrumentName, + new ExplicitBucketHistogramConfiguration { Boundaries = [0.01, 0.05, 0.1, 0.5, 1, 5] }); + } + } +} diff --git a/src/ServiceControl/Auditing/Metrics/AuditMessageMetrics.cs b/src/ServiceControl/Auditing/Metrics/AuditMessageMetrics.cs new file mode 100644 index 0000000000..52dfb04cc4 --- /dev/null +++ b/src/ServiceControl/Auditing/Metrics/AuditMessageMetrics.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Auditing.Metrics +{ + using System; + using System.Diagnostics; + using System.Diagnostics.Metrics; + using NServiceBus.Transport; + + public record AuditMessageMetrics(MessageContext Context, Histogram Duration) : IDisposable + { + public void Skipped() => result = "skipped"; + + public void Success() => result = "success"; + + public void Dispose() + { + var tags = AuditIngestionMetrics.GetMessageTags(Context.Headers); + + tags.Add("result", result); + Duration.Record(sw.Elapsed.TotalSeconds, tags); + } + + string result = "failed"; + + readonly Stopwatch sw = Stopwatch.StartNew(); + } +} diff --git a/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs b/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs new file mode 100644 index 0000000000..b145529241 --- /dev/null +++ b/src/ServiceControl/Auditing/PrimaryLocalAuditSource.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Auditing +{ + using System; + using System.Diagnostics; + using NuGet.Versioning; + using Particular.LicensingComponent.AuditThroughput; + using Particular.LicensingComponent.Contracts; + using ServiceBus.Management.Infrastructure.Settings; + + // Without this the local audit and audit log queues are counted as customer endpoints in the + // licensing throughput report, and the report's audit service metadata is blank. + class PrimaryLocalAuditSource(Settings settings) : ILocalAuditSource + { + public bool Enabled => true; + + public RemoteInstanceInformation Describe() + { + var version = FileVersionInfo.GetVersionInfo(typeof(PrimaryLocalAuditSource).Assembly.Location).ProductVersion; + + return new RemoteInstanceInformation + { + ApiUri = settings.ApiUrl, + VersionString = version, + SemanticVersion = SemanticVersion.TryParse(version ?? string.Empty, out var semanticVersion) ? semanticVersion : null, + Status = "online", + // Retention is reported as configured. When it is not configured the existing minimum + // retention gate warns, rather than this guessing a default on the operator's behalf. + Retention = settings.AuditRetentionPeriod ?? TimeSpan.Zero, + Queues = [settings.AuditQueue, settings.AuditLogQueue], + Transport = settings.TransportType + }; + } + } +} diff --git a/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs b/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs new file mode 100644 index 0000000000..a029bac8f1 --- /dev/null +++ b/src/ServiceControl/Auditing/SagaRelationshipsEnricher.cs @@ -0,0 +1,9 @@ +namespace ServiceControl.Auditing +{ + using ServiceControl.SagaAudit; + + class SagaRelationshipsEnricher : IEnrichImportedAuditMessages + { + public void Enrich(AuditEnricherContext context) => InvokedSagasParser.Parse(context.Headers, context.Metadata); + } +} diff --git a/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs b/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs index dbf265b0b6..e63379da01 100644 --- a/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs +++ b/src/ServiceControl/CompositeViews/AuditCounts/GetAuditCountsForEndpointApi.cs @@ -13,25 +13,18 @@ using Persistence.Infrastructure; using ServiceBus.Management.Infrastructure.Settings; - // The endpoint is included for consistency reasons but is actually not required here because the query - // is forwarded to the remote instance. But this at least enforces us to declare the controller action - // with the necessary parameter and not accessing the endpoint becomes an implementation details of the scatter - // gather approach here. public record AuditCountsForEndpointContext(PagingInfo PagingInfo, string Endpoint) : ScatterGatherContext(PagingInfo); public class GetAuditCountsForEndpointApi( - IMessagesViewDataStore dataStore, + IAuditCountsDataStore dataStore, Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) - : ScatterGatherApi>(dataStore, settings, httpClientFactory, httpContextAccessor, logger) + : ScatterGatherApi>(dataStore, settings, httpClientFactory, httpContextAccessor, logger) { - static readonly IList Empty = new List(0).AsReadOnly(); - protected override Task>> LocalQuery(AuditCountsForEndpointContext input, CancellationToken cancellationToken = default) => - // Will never be implemented on the primary instance - Task.FromResult(new QueryResult>(Empty, QueryStatsInfo.Zero)); + DataStore.QueryAuditCounts(input.Endpoint, cancellationToken); protected override IList ProcessResults(AuditCountsForEndpointContext input, QueryResult>[] results) => results.SelectMany(r => r.Results) diff --git a/src/ServiceControl/CompositeViews/Messages/ScatterGatherRemoteOnly.cs b/src/ServiceControl/CompositeViews/Messages/ScatterGatherRemoteOnly.cs deleted file mode 100644 index 379070101b..0000000000 --- a/src/ServiceControl/CompositeViews/Messages/ScatterGatherRemoteOnly.cs +++ /dev/null @@ -1,23 +0,0 @@ -namespace ServiceControl.CompositeViews.Messages -{ - using System.Net.Http; - using System.Threading; - using System.Threading.Tasks; - using Microsoft.AspNetCore.Http; - using Microsoft.Extensions.Logging; - using Persistence.Infrastructure; - using ServiceBus.Management.Infrastructure.Settings; - - public abstract class ScatterGatherRemoteOnly(Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) - : ScatterGatherApi(NoOpStore.Instance, settings, httpClientFactory, httpContextAccessor, logger) - where TIn : ScatterGatherContext - where TOut : class - { - protected sealed override Task> LocalQuery(TIn input, CancellationToken cancellationToken = default) => QueryResult.Empty(); - } - - public sealed class NoOpStore - { - public static NoOpStore Instance => field ??= new NoOpStore(); - } -} \ No newline at end of file diff --git a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs index 5e4b20e731..7b486e3d15 100644 --- a/src/ServiceControl/CustomChecks/CustomChecksComponent.cs +++ b/src/ServiceControl/CustomChecks/CustomChecksComponent.cs @@ -29,7 +29,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddEventLogMapping(); hostBuilder.Services.AddEventLogMapping(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs index 6d86407288..9e1782a333 100644 --- a/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs +++ b/src/ServiceControl/ExternalIntegrations/ExternalIntegrationsComponent.cs @@ -18,7 +18,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { services.AddDomainEventHandler(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { services.AddHostedService(); } diff --git a/src/ServiceControl/HostApplicationBuilderExtensions.cs b/src/ServiceControl/HostApplicationBuilderExtensions.cs index d8790c9151..09c7f76314 100644 --- a/src/ServiceControl/HostApplicationBuilderExtensions.cs +++ b/src/ServiceControl/HostApplicationBuilderExtensions.cs @@ -35,7 +35,7 @@ static class HostApplicationBuilderExtensions { public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, Settings settings, EndpointConfiguration configuration, params ReadOnlySpan components) { - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { ArgumentNullException.ThrowIfNull(configuration); } @@ -95,7 +95,7 @@ public static void AddServiceControl(this IHostApplicationBuilder hostBuilder, S services.AddMetrics(settings.PrintMetrics); services.AddServiceControlHealthChecks(); - if (settings.ErrorIngestionOnly) + if (settings.IngestionOnly) { // Ingestion receives through its own transport infrastructure and forwards through // that same infrastructure's dispatcher, so the endpoint is not hosted at all. @@ -153,6 +153,7 @@ Audit Retention Period (optional): {settings.AuditRetentionPeriod} Error Retention Period: {settings.ErrorRetentionPeriod} Ingest Error Messages: {settings.IngestErrorMessages} Error Ingestion Only: {settings.ErrorIngestionOnly} +Audit Ingestion Only: {settings.AuditIngestionOnly} Forwarding Error Messages: {settings.ForwardErrorMessages} ServiceControl Logging Level: {settings.LoggingSettings.LogLevel} Selected Transport Customization: {settings.TransportType} diff --git a/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs new file mode 100644 index 0000000000..eb86a3e7c5 --- /dev/null +++ b/src/ServiceControl/Hosting/Commands/AuditIngestionOnlyCommand.cs @@ -0,0 +1,62 @@ +namespace ServiceControl.Hosting.Commands +{ + using System; + using System.Threading; + using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; + using Particular.ServiceControl; + using Particular.ServiceControl.Hosting; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + using ServiceControl.Infrastructure.Health; + using ServiceControl.Monitoring; + + /// + /// Runs a host that does nothing but drain the audit queue into the shared database, so several + /// processes can ingest against one database. Everything a deployment may only run once, the + /// failed audit reimport command, API hosting, retention and licensing, stays with the primary + /// instance, and this host never provisions queues, schema or body storage. + /// + class AuditIngestionOnlyCommand : AbstractCommand + { + public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) + { + IngestionOnlyGuards.EnsureStorageSupportsAuditIngestion(settings); + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--audit-ingestion-only"); + + var app = BuildHost(settings); + + await app.RunAsync(settings.RootUrl); + } + + internal static WebApplication BuildHost(Settings settings, Action customize = null) + { + settings.AuditIngestionOnly = true; + settings.IngestAuditMessages = true; + settings.IngestErrorMessages = false; + settings.RunRetryProcessor = false; + + var hostBuilder = WebApplication.CreateBuilder(); + + hostBuilder.AddServiceControl(settings, configuration: null, Components); + + customize?.Invoke(hostBuilder); + + var app = hostBuilder.Build(); + + app.MapServiceControlHealthChecks(); + + return app; + } + + // EventLog and ExternalIntegrations are deliberately absent: audit ingestion raises no domain + // events and no integration events. Hosting would claim the instance queue, and Licensing would + // count throughput once per node. + static ServiceControlComponent[] Components => + [ + new HeartbeatMonitoringComponent(), + new CustomChecks.CustomChecksComponent(), + new AuditComponent() + ]; + } +} diff --git a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs index 1fc1c68e4e..02f3ad3824 100644 --- a/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs +++ b/src/ServiceControl/Hosting/Commands/ErrorIngestionOnlyCommand.cs @@ -29,6 +29,7 @@ class ErrorIngestionOnlyCommand : AbstractCommand public override async Task Execute(HostArguments args, Settings settings, CancellationToken cancellationToken = default) { EnsureStorageCanScaleOut(settings); + IngestionOnlyGuards.EnsureBodyStorageIsReadableByEveryHost("--error-ingestion-only"); var app = BuildHost(settings); diff --git a/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs b/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs new file mode 100644 index 0000000000..f235f5a083 --- /dev/null +++ b/src/ServiceControl/Hosting/Commands/IngestionOnlyGuards.cs @@ -0,0 +1,65 @@ +namespace ServiceControl.Hosting.Commands +{ + using System; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Configuration; + using ServiceControl.Persistence; + + /// + /// The startup checks both ingestion only modes share. They are deliberately driven by the + /// persistence manifest and by settings, never by resolving optional services or by catching a + /// startup failure, so an unsupported deployment fails with a message that names what to change. + /// + static class IngestionOnlyGuards + { + public const string SharedBodyStoragePathKey = "MessageBody/FileSystem/PathIsShared"; + + const string BodyStorageTypeKey = "MessageBody/StorageType"; + const string FileSystemBodyStorage = "FileSystem"; + + public static void EnsureStorageSupportsAuditIngestion(Settings settings) + { + var manifest = PersistenceManifestLibrary.Find(settings.PersistenceType); + + if (manifest?.SupportsAuditIngestion != true) + { + throw new Exception( + $"--audit-ingestion-only requires storage that supports audit ingestion, but this instance is configured to use '{settings.PersistenceType}'. " + + "Hosting audit ingestion in the primary instance is not supported for this storage type."); + } + } + + public static void EnsureModesAreNotCombined(bool errorIngestionOnly, bool auditIngestionOnly) + { + if (errorIngestionOnly && auditIngestionOnly) + { + throw new Exception( + "--error-ingestion-only and --audit-ingestion-only cannot be combined. Each queue gets its own worker pool so the two can be scaled independently, " + + "so run one process per mode."); + } + } + + /// + /// Nothing in the file system body storage settings distinguishes a shared mount from a node + /// local directory, so an ingestion only worker requires the operator to assert it explicitly. + /// Without it, bodies written by a worker are unreadable by every other host. + /// + public static void EnsureBodyStorageIsReadableByEveryHost(string mode) + { + var storageType = SettingsReader.Read(Settings.SettingsRootNamespace, BodyStorageTypeKey); + + if (!string.Equals(storageType, FileSystemBodyStorage, StringComparison.OrdinalIgnoreCase)) + { + return; + } + + if (!SettingsReader.Read(Settings.SettingsRootNamespace, SharedBodyStoragePathKey, false)) + { + throw new Exception( + $"{mode} is configured for file system body storage, which every host must be able to read. " + + $"Set {Settings.SettingsRootNamespace}/{SharedBodyStoragePathKey} to true to assert that the configured path is a shared mount, " + + "or use blob or S3 body storage."); + } + } + } +} diff --git a/src/ServiceControl/Hosting/Help.txt b/src/ServiceControl/Hosting/Help.txt index 4925b8c494..4c9530d956 100644 --- a/src/ServiceControl/Hosting/Help.txt +++ b/src/ServiceControl/Hosting/Help.txt @@ -19,8 +19,20 @@ share the ingestion load. Requires SQL Server or PostgreSQL storage, and require has already been provisioned by a normal instance. Exactly one normal instance must still be running: it owns the retry pipeline, the retention sweep, integration event dispatch and heartbeat monitoring. -Message bodies must be stored somewhere every host can read, so this mode should not be combined with -file system body storage unless the path is a shared mount. +Message bodies must be stored somewhere every host can read. With file system body storage the mode +refuses to start unless ServiceControl/MessageBody/FileSystem/PathIsShared is set to true, which +asserts that the configured path is a shared mount. + +AUDIT INGESTION ONLY + + ServiceControl.exe --audit-ingestion-only + +Runs a host that only drains the audit queue into the configured database, so several processes can +share the ingestion load. Requires storage that supports audit ingestion, and requires that the +database has already been provisioned by a normal instance. It cannot be combined with +--error-ingestion-only: each queue gets its own worker pool so the two can be scaled independently. + +The same body storage rule applies as for error ingestion only. SERVICE INSTALL AND UNINSTALL AND CONFIGURATION OPTIONS diff --git a/src/ServiceControl/Hosting/HostArguments.cs b/src/ServiceControl/Hosting/HostArguments.cs index b260543662..6c4a0a9955 100644 --- a/src/ServiceControl/Hosting/HostArguments.cs +++ b/src/ServiceControl/Hosting/HostArguments.cs @@ -1,4 +1,4 @@ -namespace Particular.ServiceControl.Hosting +namespace Particular.ServiceControl.Hosting { using System; using System.IO; @@ -11,6 +11,9 @@ class HostArguments { public HostArguments(string[] args) { + var errorIngestionOnly = false; + var auditIngestionOnly = false; + if (SettingsReader.Read(Settings.SettingsRootNamespace, "MaintenanceMode")) { args = [.. args, "-m"]; @@ -53,12 +56,17 @@ public HostArguments(string[] args) } }; - var errorIngestionOnlyOptions = new OptionSet + var ingestionOnlyOptions = new OptionSet { { "error-ingestion-only", "Run only error ingestion, for scaling out ingestion across several processes", - s => Command = typeof(ErrorIngestionOnlyCommand) + s => errorIngestionOnly = true + }, + { + "audit-ingestion-only", + "Run only audit ingestion, for scaling out ingestion across several processes", + s => auditIngestionOnly = true } }; @@ -85,10 +93,19 @@ public HostArguments(string[] args) return; } - errorIngestionOnlyOptions.Parse(args); + ingestionOnlyOptions.Parse(args); + + IngestionOnlyGuards.EnsureModesAreNotCombined(errorIngestionOnly, auditIngestionOnly); + + if (errorIngestionOnly) + { + Command = typeof(ErrorIngestionOnlyCommand); + return; + } - if (Command == typeof(ErrorIngestionOnlyCommand)) + if (auditIngestionOnly) { + Command = typeof(AuditIngestionOnlyCommand); return; } diff --git a/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs b/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs new file mode 100644 index 0000000000..aeef4345ce --- /dev/null +++ b/src/ServiceControl/Infrastructure/Health/AuditIngestionHealthCheck.cs @@ -0,0 +1,29 @@ +namespace ServiceControl.Infrastructure.Health +{ + using System.Threading; + using System.Threading.Tasks; + using Microsoft.Extensions.Diagnostics.HealthChecks; + using ServiceBus.Management.Infrastructure.Settings; + using ServiceControl.Auditing; + + /// + /// Reports the state the audit ingestion watchdog publishes, which covers a batch that keeps + /// failing as well as a receiver that fails to start. + /// + class AuditIngestionHealthCheck(AuditIngestionCustomCheck.State ingestionState, Settings settings) : IHealthCheck + { + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) + { + if (!settings.IngestAuditMessages) + { + return Task.FromResult(HealthCheckResult.Healthy("Audit ingestion is disabled")); + } + + var failure = ingestionState.GetLastFailure(); + + return Task.FromResult(failure == null + ? HealthCheckResult.Healthy("Ingesting audit messages") + : HealthCheckResult.Unhealthy(failure)); + } + } +} diff --git a/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs b/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs index 64fd04f30e..d709a85968 100644 --- a/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs +++ b/src/ServiceControl/Infrastructure/Health/HealthCheckExtensions.cs @@ -14,11 +14,12 @@ static class HealthCheckExtensions public const string LivenessPath = "/health"; public const string ReadinessPath = "/health/ready"; - const string ReadyTag = "ready"; + internal const string ReadyTag = "ready"; + // The individual checks are added by the components that host the work they report on, so a + // host that does not ingest error messages does not answer for error ingestion. public static void AddServiceControlHealthChecks(this IServiceCollection services) => - services.AddHealthChecks() - .AddCheck("error-ingestion", tags: [ReadyTag]); + services.AddHealthChecks(); /// /// Liveness answers "is this process still serving", and is what a container health check diff --git a/src/ServiceControl/Infrastructure/Settings/Settings.cs b/src/ServiceControl/Infrastructure/Settings/Settings.cs index 3850d65b52..32b1834a7d 100644 --- a/src/ServiceControl/Infrastructure/Settings/Settings.cs +++ b/src/ServiceControl/Infrastructure/Settings/Settings.cs @@ -1,4 +1,4 @@ -namespace ServiceBus.Management.Infrastructure.Settings +namespace ServiceBus.Management.Infrastructure.Settings { using System; using System.Collections.Generic; @@ -43,6 +43,7 @@ public Settings( InstanceName = SettingsReader.Read(SettingsRootNamespace, "InstanceName", InstanceName); LoadErrorIngestionSettings(); + LoadAuditIngestionSettings(); TransportConnectionString = GetConnectionString(); TransportType = transportType ?? SettingsReader.Read(SettingsRootNamespace, "TransportType"); @@ -79,6 +80,7 @@ public Settings( NotificationsFilter = SettingsReader.Read(SettingsRootNamespace, "NotificationsFilter"); RemoteInstances = GetRemoteInstances().ToArray(); TimeToRestartErrorIngestionAfterFailure = GetTimeToRestartErrorIngestionAfterFailure(); + TimeToRestartAuditIngestionAfterFailure = GetTimeToRestartIngestionAfterFailure("TimeToRestartAuditIngestionAfterFailure"); DisableExternalIntegrationsPublishing = SettingsReader.Read(SettingsRootNamespace, "DisableExternalIntegrationsPublishing", false); TrackInstancesInitialValue = SettingsReader.Read(SettingsRootNamespace, "TrackInstancesInitialValue", true); ShutdownTimeout = SettingsReader.Read(SettingsRootNamespace, "ShutdownTimeout", ShutdownTimeout); @@ -201,9 +203,36 @@ public TimeSpan HeartbeatGracePeriod public bool IngestErrorMessages { get; set; } = true; public bool RunRetryProcessor { get; set; } = true; + public string AuditQueue { get; set; } + public string AuditLogQueue { get; set; } + + public bool ForwardAuditMessages { get; set; } + + /// + /// Whether the normal primary host runs the audit receiver. Only has an effect where the + /// persister advertises audit support; always on under audit ingestion only. + /// + public bool IngestAuditMessages { get; set; } = true; + + public int MaximumAuditIngestionConcurrencyLevel { get; set; } + + public TimeSpan TimeToRestartAuditIngestionAfterFailure { get; set; } + + public string OtlpEndpointUrl { get; set; } = SettingsReader.Read(SettingsRootNamespace, nameof(OtlpEndpointUrl)); + // Set by the --error-ingestion-only command, never read from configuration. public bool ErrorIngestionOnly { get; set; } + // Set by the --audit-ingestion-only command, never read from configuration. + public bool AuditIngestionOnly { get; set; } + + /// + /// True in either ingestion only mode. These hosts run no NServiceBus endpoint, own none of the + /// work a deployment may only do once, and never provision anything. + /// + [JsonIgnore] + public bool IngestionOnly => ErrorIngestionOnly || AuditIngestionOnly; + public TimeSpan? AuditRetentionPeriod { get; set; } public TimeSpan ErrorRetentionPeriod { get; } @@ -369,10 +398,12 @@ TimeSpan GetErrorRetentionPeriod() return result; } - TimeSpan GetTimeToRestartErrorIngestionAfterFailure() + TimeSpan GetTimeToRestartErrorIngestionAfterFailure() => GetTimeToRestartIngestionAfterFailure("TimeToRestartErrorIngestionAfterFailure"); + + TimeSpan GetTimeToRestartIngestionAfterFailure(string settingName) { string message; - var valueRead = SettingsReader.Read(SettingsRootNamespace, "TimeToRestartErrorIngestionAfterFailure"); + var valueRead = SettingsReader.Read(SettingsRootNamespace, settingName); if (valueRead == null) { return TimeSpan.FromSeconds(60); @@ -382,21 +413,21 @@ TimeSpan GetTimeToRestartErrorIngestionAfterFailure() { if (ValidateConfiguration && result < TimeSpan.FromSeconds(5)) { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, value should be minimum 5 seconds."; + message = $"{settingName} setting is invalid, value should be minimum 5 seconds."; InternalLogger.Fatal(message); throw new Exception(message); } if (ValidateConfiguration && result > TimeSpan.FromHours(1)) { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, value should be maximum 1 hour."; + message = $"{settingName} setting is invalid, value should be maximum 1 hour."; InternalLogger.Fatal(message); throw new Exception(message); } } else { - message = "TimeToRestartErrorIngestionAfterFailure setting is invalid, please make sure it is a TimeSpan."; + message = $"{settingName} setting is invalid, please make sure it is a TimeSpan."; InternalLogger.Fatal(message); throw new Exception(message); } @@ -432,6 +463,32 @@ static string Subscope(string address) return $"{queue}.log@{machine}"; } + void LoadAuditIngestionSettings() + { + // Key names are deliberately the ones the standalone audit instance reads, so an operator + // configures a combined primary exactly as they configure an audit instance today. + var serviceBusRootNamespace = new SettingsRootNamespace("ServiceBus"); + AuditQueue = SettingsReader.Read(serviceBusRootNamespace, "AuditQueue", "audit"); + + if (string.IsNullOrEmpty(AuditQueue)) + { + throw new Exception("ServiceBus/AuditQueue value is required to start the instance"); + } + + IngestAuditMessages = SettingsReader.Read(SettingsRootNamespace, "IngestAuditMessages", true); + + AuditLogQueue = SettingsReader.Read(serviceBusRootNamespace, "AuditLogQueue", null); + + if (AuditLogQueue == null) + { + logger.LogInformation("No settings found for audit log queue to import, default name will be used"); + AuditLogQueue = Subscope(AuditQueue); + } + + ForwardAuditMessages = SettingsReader.Read(SettingsRootNamespace, "ForwardAuditMessages", false); + MaximumAuditIngestionConcurrencyLevel = SettingsReader.Read(SettingsRootNamespace, "MaximumAuditIngestionConcurrencyLevel", 32); + } + void LoadErrorIngestionSettings() { var serviceBusRootNamespace = new SettingsRootNamespace("ServiceBus"); diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs index 542901eed0..7f15781d4e 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringComponent.cs @@ -31,7 +31,7 @@ public override void Configure(Settings settings, ITransportCustomization transp { hostBuilder.Services.AddHostedService(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddHostedService(); } @@ -52,7 +52,7 @@ public override void Configure(Settings settings, ITransportCustomization transp hostBuilder.Services.AddErrorMessageEnricher(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { hostBuilder.Services.AddPlatformConnectionProvider(); } diff --git a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs index 779a2fb46f..35efea4e21 100644 --- a/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs +++ b/src/ServiceControl/Monitoring/HeartbeatMonitoringHostedService.cs @@ -27,7 +27,7 @@ public async Task StartAsync(CancellationToken cancellationToken = default) // An ingestion only host receives no heartbeats, so it has nothing to check and would // only report every endpoint as dead. It still warms the monitor, because the error // enricher asks it whether an endpoint is new before recording it. - if (settings.ErrorIngestionOnly) + if (settings.IngestionOnly) { return; } diff --git a/src/ServiceControl/Operations/ErrorIngestion.cs b/src/ServiceControl/Operations/ErrorIngestion.cs index cc89696c80..ad92086020 100644 --- a/src/ServiceControl/Operations/ErrorIngestion.cs +++ b/src/ServiceControl/Operations/ErrorIngestion.cs @@ -216,7 +216,7 @@ async Task SetUpAndStartInfrastructure(CancellationToken cancellationToken) errorHandlingPolicy.OnError, OnCriticalError, TransportTransactionMode.ReceiveOnly, - cancellationToken + cancellationToken: cancellationToken ); messageReceiver = transportInfrastructure.Receivers[errorQueue]; diff --git a/src/ServiceControl/Persistence/EmptyAuditDataStores.cs b/src/ServiceControl/Persistence/EmptyAuditDataStores.cs new file mode 100644 index 0000000000..c0fcb92a00 --- /dev/null +++ b/src/ServiceControl/Persistence/EmptyAuditDataStores.cs @@ -0,0 +1,27 @@ +namespace ServiceControl.Persistence +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Api.Contracts; + using ServiceControl.Persistence.Infrastructure; + using ServiceControl.SagaAudit; + + // A primary whose persister holds no audit data still serves the audit routes, answering from its + // remotes alone. These stand in for the local source so the scatter gather is uniform and the APIs + // do not need to know which persister they are running on. + class EmptyAuditCountsDataStore : IAuditCountsDataStore + { + public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) => + Task.FromResult(new QueryResult>(Empty, QueryStatsInfo.Zero)); + + static readonly IList Empty = new List(0).AsReadOnly(); + } + + class EmptySagaHistoryDataStore : ISagaHistoryDataStore + { + public Task> QuerySagaHistoryById(Guid sagaId, CancellationToken cancellationToken = default) => + Task.FromResult(QueryResult.Empty()); + } +} diff --git a/src/ServiceControl/Persistence/PersistenceFactory.cs b/src/ServiceControl/Persistence/PersistenceFactory.cs index bc1c655cd8..274679128f 100644 --- a/src/ServiceControl/Persistence/PersistenceFactory.cs +++ b/src/ServiceControl/Persistence/PersistenceFactory.cs @@ -13,7 +13,7 @@ public static IPersistence Create(Settings settings, bool maintenanceMode = fals //HINT: This is false when executed from acceptance tests settings.PersisterSpecificSettings ??= persistenceConfiguration.CreateSettings(Settings.SettingsRootNamespace); settings.PersisterSpecificSettings.MaintenanceMode = maintenanceMode; - settings.PersisterSpecificSettings.RunRetentionSweep = !settings.ErrorIngestionOnly; + settings.PersisterSpecificSettings.RunRetentionSweep = !settings.IngestionOnly; var persistence = persistenceConfiguration.Create(settings.PersisterSpecificSettings); return persistence; diff --git a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs index 2447bfe400..7918f73cb3 100644 --- a/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs +++ b/src/ServiceControl/Persistence/PersistenceServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence { using Microsoft.Extensions.DependencyInjection; + using Microsoft.Extensions.DependencyInjection.Extensions; using ServiceBus.Management.Infrastructure.Settings; static class PersistenceServiceCollectionExtensions @@ -10,6 +11,11 @@ public static void AddPersistence(this IServiceCollection services, Settings set { var persistence = PersistenceFactory.Create(settings, maintenanceMode); persistence.AddPersistence(services); + + // Only an audit capable persister registers these, so the rest fall back to a local source + // that holds nothing and the audit routes answer from the configured remotes alone. + services.TryAddSingleton(); + services.TryAddSingleton(); } } } diff --git a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs index ab17793427..d3ab20dfa4 100644 --- a/src/ServiceControl/Recoverability/RecoverabilityComponent.cs +++ b/src/ServiceControl/Recoverability/RecoverabilityComponent.cs @@ -11,6 +11,7 @@ using ExternalIntegrations; using Infrastructure.BackgroundTasks; using Infrastructure.DomainEvents; + using Infrastructure.Health; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; @@ -80,7 +81,7 @@ public override void Configure(Settings settings, ITransportCustomization transp services.AddSingleton(); services.AddSingleton(); - if (!settings.ErrorIngestionOnly) + if (!settings.IngestionOnly) { services.AddHostedService(provider => provider.GetRequiredService()); } @@ -109,6 +110,8 @@ public override void Configure(Settings settings, ITransportCustomization transp //Health checks services.AddCustomCheck(); services.AddCustomCheck(); + services.AddHealthChecks() + .AddCheck("error-ingestion", tags: [HealthCheckExtensions.ReadyTag]); //External integration services.AddIntegrationEventPublisher(); diff --git a/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs b/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs index 2c4d975e6a..02ec07895b 100644 --- a/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs +++ b/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs @@ -40,7 +40,7 @@ ILogger logger public async Task StartAsync(CancellationToken cancellationToken = default) { - transportInfrastructure = await transportCustomization.CreateTransportInfrastructure(InputAddress, transportSettings, Handle, faultManager.OnError, (_, __, ___) => Task.CompletedTask, TransportTransactionMode.SendsAtomicWithReceive, cancellationToken); + transportInfrastructure = await transportCustomization.CreateTransportInfrastructure(InputAddress, transportSettings, Handle, faultManager.OnError, (_, __, ___) => Task.CompletedTask, TransportTransactionMode.SendsAtomicWithReceive, cancellationToken: cancellationToken); messageReceiver = transportInfrastructure.Receivers[InputAddress]; messageDispatcher = transportInfrastructure.Dispatcher; diff --git a/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs b/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs index 559fa1f143..a94ac10ed2 100644 --- a/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs +++ b/src/ServiceControl/SagaAudit/GetSagaByIdApi.cs @@ -3,17 +3,23 @@ namespace ServiceControl.SagaAudit using System; using System.Linq; using System.Net.Http; + using System.Threading; + using System.Threading.Tasks; using CompositeViews.Messages; using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Logging; + using Persistence; using Persistence.Infrastructure; using ServiceBus.Management.Infrastructure.Settings; public record SagaByIdContext(PagingInfo PagingInfo, Guid SagaId) : ScatterGatherContext(PagingInfo); - public class GetSagaByIdApi(Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) - : ScatterGatherRemoteOnly(settings, httpClientFactory, httpContextAccessor, logger) + public class GetSagaByIdApi(ISagaHistoryDataStore dataStore, Settings settings, IHttpClientFactory httpClientFactory, IHttpContextAccessor httpContextAccessor, ILogger logger) + : ScatterGatherApi(dataStore, settings, httpClientFactory, httpContextAccessor, logger) { + protected override Task> LocalQuery(SagaByIdContext input, CancellationToken cancellationToken = default) => + DataStore.QuerySagaHistoryById(input.SagaId, cancellationToken); + protected override SagaHistory ProcessResults(SagaByIdContext input, QueryResult[] results) { var nonEmptyCount = results.Count(x => x.Results != null); @@ -37,4 +43,4 @@ protected override SagaHistory ProcessResults(SagaByIdContext input, QueryResult return firstResult; } } -} \ No newline at end of file +} diff --git a/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs b/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs index 25f1f73096..613e7db277 100644 --- a/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs +++ b/src/ServiceControl/SagaAudit/SagaUpdatedHandler.cs @@ -53,12 +53,12 @@ async Task RefreshAuditQueue(CancellationToken cancellationToken) var connectionDetails = await connectionBuilder.BuildPlatformConnection(cancellationToken); // First instance is named `SagaAudit`, following instance `SagaAudit1`..`SagaAuditN` - if (connectionDetails.ToDictionary().TryGetValue("SagaAudit", out var sagaAuditObj) && sagaAuditObj is JsonElement sagaAudit) + if (connectionDetails.ToDictionary().TryGetValue("SagaAudit", out var sagaAudit)) { // Pick any audit queue, assume all instance are based on competing consumer - auditQueueName = sagaAudit.GetProperty("SagaAuditQueue").GetString(); + auditQueueName = ReadSagaAuditQueue(sagaAudit); nextAuditQueueNameRefresh = DateTime.UtcNow.AddMinutes(5); - logger.LogInformation("Refreshed audit queue name '{AuditQueueName}' from ServiceControl Audit instance. Will continue to use this value for forwarding saga update messages for the next 5 minutes", auditQueueName); + logger.LogInformation("Refreshed audit queue name '{AuditQueueName}'. Will continue to use this value for forwarding saga update messages for the next 5 minutes", auditQueueName); } } catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) @@ -76,6 +76,15 @@ async Task RefreshAuditQueue(CancellationToken cancellationToken) } } + // A remote instance's details arrive as parsed JSON, a local audit capable primary's as the + // object its provider added. + static string ReadSagaAuditQueue(object sagaAudit) => sagaAudit switch + { + JsonElement json => json.GetProperty("SagaAuditQueue").GetString(), + Auditing.AuditPlatformConnectionDetailsProvider.SagaAuditConnectionDetails local => local.SagaAuditQueue, + _ => null + }; + static string auditQueueName; static DateTime nextAuditQueueNameRefresh; static readonly SemaphoreSlim semaphore = new(1); diff --git a/src/ServiceControl/ServiceControl.csproj b/src/ServiceControl/ServiceControl.csproj index 201bf2478b..4401575e13 100644 --- a/src/ServiceControl/ServiceControl.csproj +++ b/src/ServiceControl/ServiceControl.csproj @@ -35,6 +35,9 @@ + + + diff --git a/src/ServiceControl/ServiceControlMainInstance.cs b/src/ServiceControl/ServiceControlMainInstance.cs index 10a7ba1a0b..bded219889 100644 --- a/src/ServiceControl/ServiceControlMainInstance.cs +++ b/src/ServiceControl/ServiceControlMainInstance.cs @@ -1,5 +1,6 @@ namespace Particular.ServiceControl { + using global::ServiceControl.Auditing; using global::ServiceControl.CustomChecks; using global::ServiceControl.EventLog; using global::ServiceControl.ExternalIntegrations; @@ -14,6 +15,7 @@ static class ServiceControlMainInstance new ExternalIntegrationsComponent(), new RecoverabilityComponent(), new HeartbeatMonitoringComponent(), + new AuditComponent(), new CustomChecksComponent(), new LicensingComponent() }; diff --git a/src/audit-ingestion-in-primary-plan.md b/src/audit-ingestion-in-primary-plan.md new file mode 100644 index 0000000000..446c31ad11 --- /dev/null +++ b/src/audit-ingestion-in-primary-plan.md @@ -0,0 +1,615 @@ +# Host Audit Ingestion in the Primary Instance + +## Summary + +For SQL Server and PostgreSQL persistence, move audit ingestion and the supporting audit capabilities into the primary ServiceControl process. A normal primary instance ingests audit messages by default once its persister advertises audit support. A setting disables its receiver. Additional primary processes can run with `--audit-ingestion-only` to scale ingestion through competing consumers. + +The existing standalone RavenDB audit instance remains supported and retains its current behavior. RavenDB does not gain combined hosting or audit-ingestion-only support. + +This plan covers the contracts, project boundaries, host composition, settings, and fail-fast command-line surface needed before EF audit persistence is implemented. It makes no installer changes, so RavenDB instances are unaffected by construction. It does not implement EF entities, migrations, SQL queries, retention algorithms, or provider registrations. + +## Goals + +- Host audit ingestion in the normal SQL Server or PostgreSQL primary instance. +- Allow audit ingestion to be disabled in the normal primary instance. +- Add `--audit-ingestion-only` so additional processes can scale audit ingestion. +- Include the audit capabilities that must exist when audit data is local: SagaAudit ingestion, failed audit imports, forwarding, ingestion metrics, health, and querying. +- Reuse primary-owned capabilities rather than duplicating them: endpoint detection, retry acknowledgement handling, body storage, retention sweeping, and known endpoints. +- Use the same database and existing primary EF persistence for primary and audit data. +- Keep existing primary API routes and their existing authorization policies. +- Continue supporting additional audit remotes through the existing scatter-gather. +- Restore the platform capabilities that today depend on an audit remote existing: platform connection details, saga audit forwarding, and licensing throughput collection. +- Put contracts and composition boundaries in place without starting the EF audit implementation. + +## Non-goals + +- Adding audit persistence to SQL Server or PostgreSQL in this work. +- Adding combined hosting or ingestion-only support to RavenDB. +- Replacing or removing the existing standalone RavenDB audit instance. +- Supporting the standalone `ServiceControl.Audit` executable with SQL Server or PostgreSQL. +- Finalizing provider-specific retention, partitioning, full-text search, or body storage implementations. +- Migrating existing RavenDB audit data into EF persistence. +- Running error ingestion and audit ingestion in the same ingestion-only worker. + +## Decisions + +### Data and persistence + +- Audit and primary data use the same database and connection configuration. +- Shared data, including known endpoints, uses the existing primary tables. +- Audit-owned data uses explicit table names in the existing default schema, for example `AuditMessages`, `FailedAuditImports`, and `SagaSnapshots`. +- SQL Server and PostgreSQL extend the existing `ServiceControl.Persistence.EFCore`, `ServiceControl.Persistence.EFCore.SqlServer`, and `ServiceControl.Persistence.EFCore.PostgreSql` projects. The three audit-specific EF projects from the earlier spike are not recreated. +- The `SagaSnapshots` table maps the existing `ServiceControl.SagaAudit.SagaSnapshot` type. That type lives in `ServiceControl.Audit.Persistence.SagaAudit` and is already on the primary's reference graph through `ServiceControl.SagaAudit`. It does not move. + +### Hosting + +- The normal primary retains the existing HTTP routes and serves local audit data through them. There is no separate SQL Server or PostgreSQL audit HTTP service. +- `--audit-ingestion-only` always ingests and does not host an NServiceBus endpoint. +- `--audit-ingestion-only` and `--error-ingestion-only` are mutually exclusive. Passing both fails at startup with a clear message. Each queue gets its own worker pool so the two can be scaled independently, and each keeps a single, auditable component list. Combining them is a possible follow-up. +- Disabling ingestion in the normal primary stops only its receiver. Local queries, SagaAudit, failed-import tooling, and other audit capabilities remain active because workers may still ingest. + +### API + +- Local audit data is served through the existing primary routes under their existing policies. `/api/messages` and its variants stay on `error:messages:view`, `/api/sagas/{id}` stays on `error:sagas:view`, and `endpoints/{endpoint}/audit-count` stays on `error:messages:view`. + A primary configured with an audit remote already serves that remote's audit data under `error:messages:view` today, so local audit data inherits an established gate and nothing about the `my/routes` manifest or ServicePulse navigation changes. The standalone audit instance keeps its own `audit:*` policies and its anonymous audit-count route. + +### Settings and installer + +- The primary reads the audit settings under the same key names the audit instance uses. Key names are reused rather than invented. +- No installer changes in this work. `ServiceControlInstaller.*`, `ServiceControl.Config` and `ServiceControl.Management.PowerShell` are untouched, so RavenDB instances behave exactly as they do today by construction. SCMU and PowerShell support for EF storage types is a separate, undecided workstream, and the audit settings belong to it. See "Settings and commands" for the handoff note. + +### Observability + +- The copied ingestion metrics keep their OpenTelemetry implementation. The primary gains the three OpenTelemetry package references and an `OtlpEndpointUrl` setting, and the meter is renamed from `Particular.ServiceControl.Audit` to `Particular.ServiceControl`. + Copying faithfully keeps the primary and audit implementations comparable for the later reuse assessment, and it opens the door to moving error ingestion onto the same instrumentation. The cost is three new packages in the shipped primary artifact. + +## What the primary already owns + +Several capabilities the earlier draft treated as "moving from audit" already exist on the primary. The plan reuses them rather than copying an audit equivalent. + +| Capability | Where it already lives | Consequence for this work | +| --- | --- | --- | +| Local-first scatter-gather | `ScatterGatherApi.Execute` runs the local query first, then remotes | No new "query coordinator" is needed. With zero remotes it is already a local-only query. | +| Message view queries | `IMessagesViewDataStore` | The five audit message queries extend this contract. Its EF implementation performs the union. | +| Saga history DTOs | `ServiceControl.Audit.Persistence.SagaAudit`, referenced through `ServiceControl.SagaAudit` | No new DTO project. `GetSagaByIdApi` stops being remote-only. | +| Audit retention period setting | `ServiceControl/AuditRetentionPeriod`, already validated and published in `/api/configuration` | Reuse it. Define what `null` means now that it drives behavior. | +| Full-text search toggle | `PersistenceSettings.EnableFullTextSearchOnBodies` | One value governs error and audit bodies. | +| Max body size | `BodyStorageSettings.MaxBodySizeToStore` | One value governs error and audit bodies. | +| Body storage and installers | `IBodyStorage`, `IBodyStoragePersistence`, FileSystem, AzureBlob and S3 implementations plus installers | Audit bodies use the same store. There is no database body store. | +| Retention sweeping | `RetentionSweeper`, a `BackgroundService` registered inside `BasePersistence` | Audit retention extends the sweeper. There is no host-level retention component. | +| Endpoint detection pattern | `DetectNewEndpointsFromErrorImportsEnricher` plus `unitOfWork.Monitoring.RecordKnownEndpoint` in `ErrorProcessor` | The audit enricher adopts the same shape. | +| Retry acknowledgement handling | `RetryConfirmationProcessor`, driven by acknowledgements arriving on the error queue | Unchanged. See "Retry acknowledgements" below. | +| Internal custom checks | `services.AddCustomCheck()` plus `InternalCustomChecksHostedService` | The copied audit checks register through DI, not through `configuration.AddCustomCheck`. | +| Saga audit misconfiguration handling | `SagaUpdatedHandler` and `SagaAuditMisconfigurationCustomCheck` | Both need work. See "Platform connection details". | + +## Project and boundary assessment + +Do not add a new runtime project for the first implementation. Copy the audit runtime behavior needed by the SQL Server and PostgreSQL primary host into the `ServiceControl` project, then adapt that copy to primary persistence, primary settings, and the endpoint-free ingestion-only profile. + +The copied primary implementation includes: + +- Audit receiving, batching, and shutdown coordination. +- Audit message parsing and enrichment. +- Saga snapshot and relationship processing. +- Failed-ingestion handling and failed-audit orchestration. +- Forwarding orchestration. +- Ingestion metrics and readiness state. +- Registrations for the normal and ingestion-only primary hosts. + +Do not make the primary executable reference `ServiceControl.Audit.csproj`. That project remains a standalone composition root containing RavenDB persistence selection, standalone settings, API hosting, installer commands, and its own NServiceBus endpoint. + +Keep these concerns in the existing standalone audit executable: + +- RavenDB persistence loading and lifecycle. +- Standalone audit settings and maintenance mode. +- Standalone audit HTTP API composition. +- Standalone installers and queue setup behavior. +- The existing audit NServiceBus endpoint and its `ReportCustomChecksTo` reporting. + +### Shared surface + +The two executables are not isolated. Four projects sit underneath both, and this work must change at least one of them. + +| Project | Shared how | Risk | +| --- | --- | --- | +| `ServiceControl.SagaAudit` | Compiled into `ServiceControl.Audit` by source, referenced as a project by `ServiceControl.Persistence` | A change to `SagaSnapshotFactory` or `InvokedSagasParser` silently changes the shipped audit executable. | +| `ServiceControl.Audit.Persistence.SagaAudit` | Referenced by the audit persisters, the Raven primary persister, and transitively by `ServiceControl.Persistence` | The saga DTOs are shared. Changing their shape affects Raven audit documents. | +| `ServiceControl.Infrastructure` | `Watchdog`, `DeterministicGuid`, `ReadOnlyStream`, `LoggerUtil`, `Permissions` | A shutdown or watchdog change for the primary changes audit shutdown too. | +| `ServiceControl.Transports` | `ITransportCustomization.CreateTransportInfrastructure` must change for per-receiver concurrency | Both hosts create their receivers through this method. | + +Any pull request touching these four projects runs the full audit acceptance suite and states in its description why the change is safe for the audit executable. + +Note that the baseline already breaks the "audit runtime untouched" guarantee: PR #5800 modifies `AuditIngestion`, `AuditIngestor` and `AuditPersister` in the standalone audit project. The guarantee this plan makes is narrower and honest: no *behavioral* change to the standalone audit executable, verified by its acceptance suite. + +### Divergence and later reuse + +The copied implementations are expected to diverge initially. The primary copy removes endpoint assumptions, uses the primary persistence unit of work, and participates in local queries. The RavenDB implementation remains optimized for its existing standalone process. Once both paths are stable, compare them and extract shared code only where doing so removes meaningful duplication without coupling their composition roots. + +Do not add a separate contracts project. Define the new primary audit contracts in `ServiceControl.Persistence`. Leave the current RavenDB audit persistence contracts and implementation untouched unless a later reuse refactor demonstrates a clear benefit. + +## Target host profiles + +| Capability | Normal SQL/Postgres primary | `--audit-ingestion-only` | Standalone RavenDB audit | +| --- | --- | --- | --- | +| Audit receiver | Enabled by setting, default on when persistence supports audit | Always enabled | Unchanged | +| Primary NServiceBus endpoint | Yes | No | Not applicable | +| Existing audit NServiceBus endpoint | No | No | Unchanged | +| Primary API | Yes | Health endpoints only, mapped as minimal API routes, no controllers | Existing audit API unchanged | +| Local audit queries | Yes | No | Unchanged | +| Optional remote audit queries | Yes | No | Unchanged | +| Endpoint discovery | Shared persistence unit of work | Shared persistence unit of work | Unchanged | +| Retry acknowledgement dispatch | Yes | Yes | Unchanged | +| Retry acknowledgement recording | Yes, via error ingestion | No | Not applicable | +| Forwarding | Yes | Yes | Unchanged | +| Failed-audit storage | Yes | Yes | Unchanged | +| Failed-audit reimport command | Yes | No | Unchanged | +| Retention | Inside the persister, gated by `RunRetentionSweep` | Off, `RunRetentionSweep` false | Unchanged, RavenDB document expiry | +| Platform connection details for audit | Yes, local provider | No | Unchanged | +| Licensing, throughput, email, event dispatch | Yes | No | Unchanged | +| Internal custom checks | Yes | Yes | Unchanged | +| Liveness and readiness | Yes | Yes | Unchanged unless adopted separately | + +## Persistence contract direction + +The current primary persistence design already exposes capability-specific children from `IIngestionUnitOfWork`. Add audit as a sibling to monitoring and recoverability: + +```csharp +public interface IIngestionUnitOfWork : IAsyncDisposable +{ + IMonitoringIngestionUnitOfWork Monitoring { get; } + IRecoverabilityIngestionUnitOfWork Recoverability { get; } + IAuditIngestionUnitOfWork Audit { get; } + Task Complete(CancellationToken cancellationToken = default); +} +``` + +The audit child initially expresses only the operations the runtime requires, without defining EF storage details: + +- Record a processed audit message and its body reference. +- Record a Saga snapshot. + +During a batch, the audit runtime uses the existing capability children as well: + +- `Monitoring.RecordKnownEndpoint(...)` records endpoints detected from audit headers. +- `Audit.RecordProcessedMessage(...)` records an audit message. +- `Audit.RecordSagaSnapshot(...)` records a Saga snapshot. +- `Complete(...)` commits all derived state atomically where the persistence supports it. + +This replaces the earlier spike's duplicated `KnownEndpoints` table, insert-only staging table, and reconciliation process. + +### Query contracts + +Extend `IMessagesViewDataStore` rather than adding a parallel audit query contract. Its five existing queries are exactly what the scatter-gather APIs call. The EF implementation unions failed messages and audit messages, subject to the precedence rule below. + +Only three genuinely new query contracts are required: + +- Audit counts per endpoint. +- Saga history by saga id. +- Audit body resolution, folded into `IBodyStorage` rather than added alongside it. + +Plus two ingestion-side contracts: + +- Failed audit import storage and reimport selection, mirroring `IFailedErrorImportDataStore`. +- Persistence capability discovery. + +Do not move the existing monolithic `IAuditDataStore` into primary persistence. + +### Capability discovery + +Audit support is declared in `persistence.manifest`, read by `ServiceControl.Persistence.PersistenceManifest`. Add: + +```json +"SupportsAuditIngestion": true +``` + +to the SQL Server and PostgreSQL manifests. The property is absent from the RavenDB manifest and from every file under `LegacyArtifacts`, and absent means false. + +The installer has its own `PersistenceManifest` class over the same file. It does not need the property yet, and this work does not add it there. Whenever SCMU gains EF storage support it can pick the property up from the same file, so there is one source of truth waiting for it. + +Host composition must not infer support by resolving optional services or catching startup failures. + +## Removing the ingestion endpoint dependency + +The ingestion-only process must not host an NServiceBus endpoint. Follow the approach established by the error-ingestion scale-out work: + +- The receiver owns its low-level transport infrastructure and dispatcher. +- Forwarding uses the dispatcher belonging to the receiving infrastructure. +- Shutdown stops receiving under the real shutdown token, completes the writer, drains the channel, and only then tears down transport infrastructure. +- `HostInformation` and critical-error handling are supplied directly by the host. +- `IMessageSession` is absent and tested as absent. + +### Endpoint detection + +`DetectNewEndpointsFromAuditImportsEnricher` currently sends a `RegisterNewEndpoint` command through `IMessageSession`, routed to the primary's queue, where `RegisterNewEndpointHandler` calls `EndpointInstanceMonitoring.EndpointDetected`. + +In the primary copy it instead writes through `IMonitoringIngestionUnitOfWork.RecordKnownEndpoint`, matching `ErrorProcessor`. Both paths write the same `KnownEndpoints` table. + +This is the only use of `AuditEnricherContext.AddForSend(ICommand)` in the tree. Once it is gone, `IMessageSession` drops out of the copied `AuditPersister` entirely, and the `ICommand` overload is deleted from the copied `AuditEnricherContext`. Saga relationship enrichment and saga snapshot processing never needed the endpoint. + +One behavioral difference must be characterized before the switch. The command path raises the `EndpointDetected` domain event, which `MonitoringDataPersister` handles and which anything downstream of the domain event observes. The unit-of-work path does not. Write a test that pins the current observable outcome, then decide whether the audit path must raise it. + +### Retry acknowledgements + +Do not short-circuit retry acknowledgements into `IRecoverabilityIngestionUnitOfWork`. + +`DetectSuccessfulRetriesEnricher` does not perform a round trip to the primary endpoint. It emits a raw transport operation to whatever queue the `ServiceControl.Retry.AcknowledgementQueue` header names. That header is stamped by the instance that issued the retry, using its own error queue address. The receiving instance turns it into `RecordSuccessfulRetry` through `RetryConfirmationProcessor` on the normal error ingestion path. + +Writing directly to the local recoverability unit of work is only correct when the acknowledgement queue resolves to this instance's error queue. Where the retry was issued by a different primary, a direct write records the confirmation in the wrong database and the real owner never resolves the failed message. The endpoint-side acknowledgement, signalled by `ServiceControl.Retry.AcknowledgementSent`, arrives on the error queue regardless, so the transport path cannot be removed anyway. + +In a combined host the acknowledgement is dispatched to the local error queue and comes straight back into local error ingestion. That is one broker round trip, it is exactly what happens today, and it is provably correct. Keep it. Revisit the optimization only with a rule that compares the acknowledgement queue against the local error queue. + +Transport operations therefore remain in the audit ingestion path for two reasons only: forwarding, and the retry acknowledgement. + +## API and query behavior + +Keep the existing primary API routes and policies used by ServicePulse, including messages, searches, conversations, audit counts, bodies, and Saga history. + +No new query coordinator is required. `ScatterGatherApi.Execute` already runs the local query first and the remotes after, so a primary with no remotes already performs a local-only query. Two existing entry points must stop being remote-only: + +- `GetAuditCountsForEndpointApi.LocalQuery` returns `Empty` with the comment "Will never be implemented on the primary instance". It takes an `IMessagesViewDataStore` it never uses. Both the comment and the unused dependency go. +- `GetSagaByIdApi` derives from `ScatterGatherRemoteOnly`. It becomes a normal `ScatterGatherApi` over the new saga history contract. + +### Precedence, paging, and counting + +`ScatterGatherApiMessageView.ProcessResults` deduplicates on `{ReceivingEndpoint.Name}-{MessageId}` using `TryAdd`, and relies on a documented invariant: the first result set comes from the main instance, so failed-message data wins over audit data. + +Once one local result set contains both failed and audited rows through a union, that cross-source invariant becomes an intra-list ordering requirement on the SQL. If an audit row precedes the failed row for the same key, ServicePulse shows a message as successfully processed when it actually failed. + +The local query contract must therefore state three rules, and each needs a test: + +1. **Precedence.** Within a single local result set, the failed-message row for a given `{ReceivingEndpoint.Name}-{MessageId}` must precede the audit row. Either the union orders by source, or the local query deduplicates before returning. +2. **Paging.** `ProcessResults` truncates with `Take(PageSize)`. A local union that returns `PageSize` rows per source is silently truncated. The local query returns at most `PageSize` rows after its own deduplication. +3. **Counting.** `AggregateStats` sums `TotalCount` across sources. A message that both failed and was audited must be counted once by the local query, not once per source. + +RavenDB primary instances continue using their existing local-error-plus-remote-audit behavior, which these rules do not change. + +## Body storage + +Every ingestion process must write bodies to storage readable by the normal primary and every relevant worker. Blob, S3, and explicitly shared filesystem storage are acceptable. There is no database body store in the primary EF persistence. + +### Body id keyspace + +The two sides key bodies differently today, and merging them into one store without a rule produces wrong answers. + +- Audit body id is `Headers.MessageId`. +- Primary external body id is `UniqueMessageId`. +- `BodyStorage.TryFetch` resolves only against `FailedMessages`, first by `UniqueMessageId` and then falling back to `MessageId`. + +Without a rule, `GET /api/messages/{messageId}/body` for a message that both failed and was audited returns the failed copy, which for an edited message is a different body, and for an audited-only message returns 404 because nothing consults the audit tables. + +The plan adopts the existing precedent. `FailedErrorImportEntity.ExternalBodyId(...)` already prefixes a distinct keyspace inside the same store. Audit bodies use their own prefix. + +`IBodyStorage.TryFetch` gains an explicit arbitration order, stated once and tested: + +1. Failed message by `UniqueMessageId`. +2. Failed message by `MessageId`. +3. Audit message by `UniqueMessageId`, including bodies embedded in the row for full-text search. + +Retention must sweep both keyspaces. + +### Filesystem body storage in ingestion-only mode + +The earlier draft required rejecting a "node-local filesystem path" at startup. That check is not implementable. `FileSystemBodyStorageSettings` carries only a path, a compression threshold and a size cap, and nothing distinguishes a shared mount from a local directory. + +Instead, filesystem body storage requires an explicit opt-in in ingestion-only mode. Add a setting that asserts the path is shared. Without it, an ingestion-only worker configured for filesystem body storage fails at startup with a message naming the setting. + +Apply the same rule to `--error-ingestion-only`, which PR #5801 documents as a known gap. The two ingestion-only modes must not disagree about this. + +## Platform connection details + +Two primary-owned capabilities currently depend on an audit remote existing, and both break in combined mode. + +### Saga audit forwarding + +`SagaUpdatedHandler` throws `UnrecoverableException` when it cannot resolve `SagaAudit.SagaAuditQueue`. That key is produced only by the standalone audit instance's `ConnectionController` and reaches the primary only through `RemotePlatformConnectionDetailsProvider`. A combined primary with no remotes fails every misdirected `SagaUpdatedMessage`. + +### Endpoint configuration + +`/api/connection` is what ServicePulse and the Platform Connector plugin read to configure endpoints. Without an audit remote it stops advertising `MessageAudit.AuditQueue` and `SagaAudit.SagaAuditQueue`, so endpoints cannot be told where to send audit or saga data at all. + +### Correction + +Add an audit platform connection details provider to the primary, registered when the persister advertises audit support and the primary owns an audit queue. It supplies the same `MessageAudit` and `SagaAudit` shapes the audit instance supplies today, so ServicePulse and the plugin see no difference. + +`SagaUpdatedHandler` then resolves the local audit queue through the existing `IPlatformConnectionBuilder` with no change to its logic. Whether it should instead hand the snapshot straight to the audit unit of work is an open item, not a requirement of this plan. + +## Licensing and throughput + +Audit throughput collection is driven entirely by remote instances. `AuditQuery.GetAuditRemotes` derives audit queues, retention and version from `configurationApi.GetRemoteConfigs()`. `AuditThroughputCollectorHostedService.SaveAuditInstanceData` sets the static `AuditQueues` list from those remotes, and `PlatformEndpointHelper.IsPlatformEndpoint` uses that list to exclude platform queues from throughput. + +With local audit and no remotes: + +- `AuditQueues` stays empty, so the local `audit` and `audit.log` queues are counted as customer endpoints in the licensing throughput report. This is a licensing accuracy defect, not cosmetic. +- `SaveAuditServiceMetadata` records no audit versions or transports, so `AuditServicesData` in the report is blank. +- `TestAuditConnection` reports "No Audit Instances configured" in the ServicePulse diagnostics. + +`IAuditQuery` gains a local audit source alongside the remote one. It contributes the local audit queue names, the local audit retention period, and the local instance version, and it satisfies the existing "minimum 2 days retention" gate from local settings rather than from a remote's configuration payload. `Particular.LicensingComponent` is an affected project and is listed in the work plan. + +## Settings and commands + +### Runtime settings + +Settings reach the primary through `SettingsReader`, which reads environment variables, the registry and `ServiceControl.exe.config` independently of the installer. Nothing below requires an installer change to work. + +| Setting | Status | Notes | +| --- | --- | --- | +| `ServiceControl/IngestAuditMessages` | New on the primary | Effective default is `true` where the persister advertises audit support, `false` otherwise. Applies to the normal primary host only. Always on under `--audit-ingestion-only`. | +| `ServiceBus/AuditQueue` | Reused key name | Same key the audit instance reads. Default `audit`. Not written by SCMU on a primary, see the installer note. | +| `ServiceBus/AuditLogQueue` | Reused key name | Defaults to the subscoped audit queue name, matching the audit instance. Not written by SCMU on a primary. | +| `ServiceControl/ForwardAuditMessages` | Reused key name | Default `false`, matching the audit instance. Not written by SCMU on a primary. | +| `ServiceControl/AuditRetentionPeriod` | Already exists | `TimeSpan?`, already validated at min 1 hour and max 365 days, already published in `/api/configuration`. Define `null` as "use the persister default", and state that default. | +| `ServiceControl/EnableFullTextSearchOnBodies` | Already exists | One value governs error and audit bodies. | +| `ServiceControl/MessageBody/...` | Already exists | One body storage configuration governs error and audit bodies. | +| Maximum audit ingestion concurrency | New | See the transport change below. | +| `ServiceControl/TimeToRestartAuditIngestionAfterFailure` | New on the primary | Mirrors the existing error equivalent. | +| `ServiceControl/OtlpEndpointUrl` | New on the primary | Required by the copied OpenTelemetry metrics. | +| Shared filesystem body storage assertion | New | Required by ingestion-only mode. See "Body storage". | + +### Key collisions + +`ServiceControl` and `ServiceControl.Audit` settings can both be set by bare environment variable name. `ServiceControl.Audit` also falls back to `ServiceControl/IngestAuditMessages` for backwards compatibility, and `ServiceBus/AuditQueue` is literally the same key for both processes. + +The consequence is that a primary in combined mode and a standalone audit instance sharing one environment file will collide on `INGESTAUDITMESSAGES`, `AUDITRETENTIONPERIOD`, `FORWARDAUDITMESSAGES` and `SERVICEBUS_AUDITQUEUE`. + +That combination is documented as unsupported. The primary logs a warning at startup when it has audit ingestion enabled and audit remotes configured at the same time, because that is the shape most likely to hit the collision. + +### Installer: out of scope, with one handoff + +No files under `ServiceControlInstaller.Engine`, `ServiceControl.Config` or `ServiceControl.Management.PowerShell` change in this work. RavenDB instances therefore behave exactly as they do today, by construction rather than by test. That is the requirement. + +Windows deployments of an audit-capable primary configure these settings the same way EF instances are configured today, through environment variables or the config file directly, because SCMU does not yet support EF storage types at all. + +The handoff, for whoever picks up SCMU and PowerShell support for EF storage types: + +`ServiceBus/AuditQueue`, `ServiceBus/AuditLogQueue` and `ServiceControl/ForwardAuditMessages` are declared `RemovedFrom = 4.0.0` in `ServiceControlSettings`, and `ServiceControlAppConfig.UpdateSettings` calls `RemoveIfRetired` on each one. Once SCMU manages an audit-capable primary, applying settings would strip that instance's audit queue configuration out of `ServiceControl.exe.config`. + +Version gating cannot express "supported on EF, retired on RavenDB", so the gate has to move to the persister. `IServiceControlInstance` already exposes `PersistenceManifest` through `IPersistenceConfig`, so `ServiceControlAppConfig` has what it needs: drop `RemovedFrom` from the three `SettingInfo` declarations, branch on the manifest, and call `settings.Remove(name)` on the non-audit branch to preserve today's RavenDB behavior. The three `RemoveIfRetired` calls become no-ops once `RemovedFrom` is gone, so they have to be deleted rather than left in place. + +This is recorded so the trap is visible, not so it is fixed here. It is only reachable once SCMU can create an EF instance. + +### Transport concurrency + +`TransportSettings` is a singleton registered once by `AddTransportForPrimary`, and `CreateTransportInfrastructure` reads `transportSettings.MaxConcurrency.Value` for its `PushRuntimeSettings`. `CustomizePrimaryEndpoint` sets the default to 10; `CustomizeAuditEndpoint` sets it to 32. In a combined host only the primary path runs, so audit ingestion would run at 10 rather than 32, a threefold regression against the standalone instance, and the proposed per-receiver setting could not be honored at all. + +`ITransportCustomization.CreateTransportInfrastructure` gains an explicit concurrency argument, and `AuditIngestion` derives its channel bound and batch size from that value rather than from the shared singleton. This changes a project shared with the audit executable, so it runs the audit suite. Consider folding it into PR #5800 while that is still open. + +### Commands + +Add `--audit-ingestion-only` to the primary command-line parser and to `Help.txt`. + +Before EF audit persistence exists, the command is present but fails with a clear message that the selected persistence does not support audit ingestion. It also fails for RavenDB, and it fails when combined with `--error-ingestion-only`. + +The normal primary must not activate audit ingestion until a persister advertises audit support, so the groundwork merges without changing existing behavior. + +The setup command provisions the audit queue and optional audit forwarding queue for audit-capable primaries, through the existing `IComponentInstallationContext.CreateQueue` mechanism. The deployment or update setup path owns all infrastructure changes, including database schema migrations, queue creation, and body storage provisioning. Ingestion-only workers do not run installers or perform any setup or upgrade work. + +## Scale-out rules + +Per-message operations run on every normal or ingestion-only receiver and must be safe with concurrent writers: + +- Audit message and Saga snapshot inserts. +- Known endpoint upserts. +- Failed audit import storage. +- Body storage writes. +- Audit forwarding. +- Retry acknowledgement dispatch. + +Only the normal primary runs singleton work: + +- Failed-audit reimport commands. +- API hosting and remote aggregation. +- Licensing and throughput ownership. +- Email notifications. +- Event and integration dispatch polling. +- Retention, which lives inside the persister and is gated by `RunRetentionSweep`. + +### Idempotency + +The earlier draft asserted idempotency as an acceptance criterion. The current code does not have it, so it is a requirement with named keys. + +- `AuditIngestionFaultPolicy` sets `FailedAuditImport.Id = Guid.NewGuid()`. With competing consumers plus immediate retries, one poison message writes a new row per attempt per worker, the custom check fires permanently, and `--import-failed-audits` reprocesses duplicates. Replace it with a deterministic key plus a native-id fallback, modelled on `FailedErrorImport.DeriveKey`. +- `ProcessedMessage.Id` is `ProcessedMessages-{processingStartedTicks}-{ProcessingId()}`, and `ProcessingId()` returns a fresh `Guid` whenever message id, processing endpoint or processing-started headers are missing. State the deduplication key for audit rows, including that degenerate case. +- `ProcessingEndpointName()` throws for headers it cannot resolve. The per-message failure path already handles that, and the failed-import key must not depend on it. + +### Ingestion-only component list + +PR #5801 registers `EventLog`, `ExternalIntegrations`, `Recoverability`, `HeartbeatMonitoring` and `CustomChecks` in `--error-ingestion-only`, with the reasoning that which node ingests a given message is arbitrary, so nodes behaving differently makes derived data a coin flip per message. + +The audit ingestion-only host registers: + +| Component | Reason | +| --- | --- | +| `HeartbeatMonitoring` | `DetectNewEndpointsFromAuditImportsEnricher` asks `EndpointInstanceMonitoring.IsNewInstance`, which must be warmed from persistence. Without it every audited message writes a known-endpoint upsert. | +| `CustomChecks` | A stuck worker must report somewhere. Without it, an ingestion failure on a worker is invisible. | + +It does not register `Hosting`, which claims the instance queue, or `Licensing`, which would count throughput once per node. `EventLog` and `ExternalIntegrations` are not required because audit ingestion raises no domain events and no integration events. State that explicitly in the composition test so a future registration forces a decision. + +## Observability, health, and packaging + +- The copied `IngestionMetrics` keeps its OpenTelemetry implementation. `ServiceControl.csproj` gains `OpenTelemetry.Exporter.Console`, `OpenTelemetry.Exporter.OpenTelemetryProtocol` and `OpenTelemetry.Extensions.Hosting`, and the primary gains an `OtlpEndpointUrl` setting wired the same way the audit host wires it. The meter is renamed to `Particular.ServiceControl`. +- The copied custom checks are renamed so they do not collide with the standalone audit instance reporting the same names to the same primary through `ReportCustomChecksTo`. `FailedAuditImportCustomCheck` also drops its `ServiceControl.Audit Health` category, which would otherwise appear on a process that is not the audit instance. They register through `services.AddCustomCheck()`, not `configuration.AddCustomCheck`. +- `/health` and `/health/ready` follow the error-ingestion-only conventions from PR #5803, mapped as minimal API routes. +- `AuditIngestor.VerifyCanReachForwardingAddress` dispatches an empty probe message to the log queue on every infrastructure start. With N workers restarting under the watchdog, N probes accumulate. Decide whether ingestion-only workers verify forwarding at all, and what happens when the log queue does not exist because setup has not run. +- Confirm the copied runtime ships inside the existing primary artifact without adding another assembly, and that the new package references do not break `ServiceControlInstaller.Packaging.UnitTests`. + +## Work plan + +### 1. Establish the baseline + +- Land or rebase on PRs #5800, #5801 and #5803. None of them is merged, and `RunRetentionSweep`, `--error-ingestion-only`, `/health` and dispatcher-as-argument do not exist without them. +- Record the exact service composition of the existing standalone audit host. +- Write the characterization tests listed below. The existing audit acceptance suite is dominated by CORS, HTTPS, forwarded headers and OIDC, and has no coverage at all for forwarding, retention, queue setup or shutdown ordering. + +Characterization tests to write, not to assume: + +| Behavior | Why it matters | +| --- | --- | +| Audit forwarding to the log queue, including the startup probe | Nothing covers forwarding today. | +| Failed audit import round trip through `--import-failed-audits` | Establishes the duplicate-row behavior before the key changes. | +| `EndpointDetected` domain event on audit-discovered endpoints | Pins the observable difference the enricher change introduces. | +| Retry acknowledgement dispatch and recording end to end | Pins the behavior the plan deliberately leaves alone. | +| Audit shutdown with a non-empty channel and forwarding on | Pins PR #5800's fix. | +| Audit queue provisioning through setup | Nothing covers it. | +| `/api/connection` payload with and without an audit remote | Pins what the Platform Connector plugin receives. | + +### 2. Persistence contracts, capability model, and a test persister + +- Add the audit child to the primary ingestion unit of work. +- Add the failed-import, audit count, saga history and capability contracts. Extend `IMessagesViewDataStore` rather than duplicating it. +- Add `SupportsAuditIngestion` to `ServiceControl.Persistence.PersistenceManifest` and to the two EF `persistence.manifest` files. +- Add a test persister for the primary that advertises audit support, since none exists. There is no in-memory primary persister today: the only manifests are RavenDB, EFCore.SqlServer and EFCore.PostgreSql, and `ServiceControl.Persistence.Tests.InMemory` is a test context, not a persister. Either a fake registered in the acceptance-test host or a new in-memory persister is acceptable, but the plan needs one of them by name. +- Do not move the standalone RavenDB audit implementation onto the new contracts. +- Do not add EF entities, mappings, migrations, or SQL. + +No runtime behavior change. + +### 3. Copy and adapt the audit runtime + +Merged into one pull request, because a copy nothing constructs is reviewable but not verifiable. + +- Copy the receiving, parsing, enrichment, fault handling, forwarding orchestration, metrics and readiness behavior into `ServiceControl`, changing namespaces and dependencies so the copy is owned by the primary project. +- Do not copy standalone API composition, RavenDB settings, installers, maintenance mode, or persistence loading. +- Replace endpoint-registration commands with direct monitoring unit-of-work calls, and remove `IMessageSession` and the `ICommand` overload from the copy. +- Pass the low-level dispatcher into operations that perform transport output. +- Add the per-receiver concurrency argument to `CreateTransportInfrastructure`. +- Add the OpenTelemetry references and `OtlpEndpointUrl`, and rename the meter. +- Register the copy against the test persister from step 2, and add a composition test that actually starts it. This is what makes the pull request verifiable. +- Leave the source implementation in `ServiceControl.Audit` behaviorally unchanged, and run its acceptance suite. + +### 4. Settings and the fail-fast command + +- Add the runtime settings from the table above. No installer changes. +- Add `--audit-ingestion-only` parsing, `Help.txt`, and the three fail-fast paths: unsupported persistence, RavenDB, and combination with `--error-ingestion-only`. +- Add the audit queue and audit log queue to the setup component installation context. + +### 5. Normal primary audit composition + +- Add an audit component to the primary component model. +- Register the full audit capability in the normal primary profile when the persister advertises audit support. +- Keep all audit capabilities except the receiver when normal-primary ingestion is disabled. +- Add exact hosted-service composition tests against the test persister. + +### 6. Local query composition + +- Convert `GetAuditCountsForEndpointApi` and `GetSagaByIdApi` to local-capable APIs. +- Implement the precedence, paging and counting rules against the test persister. +- Fold audit body resolution into `IBodyStorage` with the stated arbitration order and the prefixed keyspace. +- Keep existing routes and authorization policies. + +### 7. Platform connection details, saga forwarding, and licensing + +- Add the local audit platform connection details provider. +- Verify `SagaUpdatedHandler` resolves the local audit queue. +- Add the local audit source to `IAuditQuery` and `AuditThroughputCollectorHostedService`, so the local audit queues are recognized as platform endpoints and audit service metadata is populated. + +### 8. Ingestion-only composition + +- Add the dedicated host builder path. +- Register the component list from "Ingestion-only component list", and assert the exact hosted-service set. +- Add `/health` and `/health/ready`. +- Reject unsupported persistence, RavenDB, mode combination, and filesystem body storage without the shared-path assertion. + +### 9. Packaging and documentation + +- Confirm the copied runtime ships in the existing primary artifact and that the new package references pass the packaging tests. +- Keep the standalone RavenDB audit artifact and manifests unchanged. +- Document the normal, disabled-ingestion, and ingestion-only deployment modes. +- Document queue ownership, body storage requirements, health endpoints, the setting collisions, and unsupported combinations. + +### 10. Reassess reuse after delivery + +- Compare the stable primary and RavenDB implementations. +- Identify code that remains behaviorally identical and has compatible dependencies. +- Extract shared code only when the resulting boundary is simpler than maintaining the copies. +- Treat a shared hosting project as an optional follow-up, not a prerequisite for EF audit support. + +## Pull request sequence + +1. Persistence contracts, capability model, and the primary test persister. No runtime behavior change. +2. Copy and adapt the audit runtime, registered against the test persister and exercised by a composition test. +3. Settings and the fail-fast command-line mode. No installer changes. +4. Local query composition, including audit counts, saga history, and body arbitration. +5. Platform connection details, saga forwarding, and the local licensing throughput source. +6. Ingestion-only host composition and health checks. +7. Packaging, documentation, and architecture tests. + +Each pull request leaves the full RavenDB audit suite passing, states why any change to the four shared projects is safe, and avoids activating unsupported EF audit behavior. + +## Validation and acceptance criteria + +### Existing behavior + +- The standalone RavenDB audit executable has no observable behavior or configuration changes, verified by its acceptance suite on every pull request. +- RavenDB primary instances continue querying configured audit remotes. +- Existing SQL Server and PostgreSQL primary instances behave as before until their persistence advertises audit support. +- No file under `ServiceControlInstaller.Engine`, `ServiceControl.Config` or `ServiceControl.Management.PowerShell` is modified. RavenDB install, upgrade and settings-apply behavior is therefore unchanged by construction, and needs no new test to prove it. + +### Contracts + +- Audit ingestion can persist an audit message and Saga snapshot without depending on RavenDB types. +- Endpoint discovery uses the existing monitoring persistence contract and writes the same `KnownEndpoints` rows as the error path. +- Persistence capability checks produce deterministic startup validation, driven by the manifest rather than by resolving optional services. + +### Normal primary composition + +- Audit ingestion can be enabled or disabled independently of the remaining audit capabilities. +- Existing API routes resolve using local audit query contracts, under their existing policies. +- Local results can be combined with remotes. +- Precedence, paging and counting rules hold for a merged local result. A message that both failed and was audited appears once, shows as failed, and is counted once. +- `/api/connection` advertises the local audit queue, and a misdirected saga audit message is forwarded rather than failed. +- The local audit queue and audit log queue are recognized as platform endpoints by throughput collection. + +### Ingestion-only composition + +- Every registered hosted service resolves without an NServiceBus endpoint. +- `IMessageSession` is absent. +- Installer services such as `IDatabaseMigrator` and body storage provisioners are absent. +- Starting an ingestion-only worker never changes the database schema, creates queues, or provisions external storage. +- The exact hosted-service set is asserted so future registrations force an explicit scale-out decision. +- RavenDB, persistence without audit support, combination with `--error-ingestion-only`, and unasserted filesystem body storage each fail clearly at startup. +- Liveness and readiness endpoints return JSON responses. + +### Scale-out semantics + +- Concurrent workers can process the same audit queue using competing consumers. +- A redelivered audit message produces one row, not one per delivery, including when the processing-started header is absent. +- A poison audit message produces one failed-import row, not one per attempt per worker. +- Shutdown drains accepted messages before transport infrastructure is torn down. +- Forwarded messages are not duplicated during graceful shutdown. +- Audit ingestion concurrency is independent of error ingestion concurrency. + +## Constraints for the later EF implementation + +The earlier audit EF spike remains useful evidence, especially for retention, full-text search, and body storage. The later implementation should revisit it using the current primary EF architecture as the authority. + +Important retained findings are: + +- PostgreSQL can use range partitioning and partition removal for retention. +- SQL Server needs a provider-specific strategy because full-text indexes prevent equivalent partition truncation. +- Retention requires distributed locking. This belongs to the EF implementation, not to host composition. Audit retention extends `RetentionSweeper`, which already deletes bodies through `IBodyStoragePersistence` and is gated by `RunRetentionSweep`. +- Cleanup capacity must be proportional to ingestion rate. A fixed delete batch can fall behind. +- Full-text search remains provider-specific, and is governed by the existing `EnableFullTextSearchOnBodies` setting shared with error bodies. +- Body storage lifecycle must align with audit retention, and must sweep the prefixed audit keyspace as well as the error keyspace. +- Stable lock ordering and provider-specific upsert behavior are requirements, following the `INSERT ... ON CONFLICT` and `MERGE WITH (HOLDLOCK)` patterns the error batch writer already uses. +- `--audit-ingestion-only` must never apply EF migrations, modify the database schema, create queues, or provision body storage. It assumes the deployment or update setup path has already prepared all required infrastructure. + +Unlike the spike, the implementation uses one primary EF model and migration stream, one shared known-endpoint table, and no audit-to-primary endpoint reconciliation process. + +## Open items + +These do not block the groundwork, but they need answers before or during the EF implementation. + +1. Should `SagaUpdatedHandler` forward a misdirected saga audit message to the local audit queue, or hand the snapshot straight to the audit unit of work? Forwarding preserves today's behavior and its warning. Direct handling removes a broker round trip. +2. Should the audit path raise the `EndpointDetected` domain event that the command path raises today? The characterization test in step 1 answers what is currently observable. +3. Do ingestion-only workers verify the forwarding address at startup? N workers restarting under the watchdog put N probe messages in the log queue. +4. What is the retention lock scope? One lock for the whole sweeper, or separate error and audit locks so a slow audit sweep does not block error retention. +5. What is the default audit retention period when `ServiceControl/AuditRetentionPeriod` is null? The audit instance defaults to 30 days while SCMU and the Dockerfile default to 7. +6. Should a later release combine the two ingestion-only modes, or replace both flags with a single `--ingestion-only` governed by `IngestErrorMessages` and `IngestAuditMessages`? +7. Handed to the SCMU and PowerShell workstream for EF storage types, not answered here: how are the audit settings surfaced for a Windows primary, and what fixes the `RemoveIfRetired` trap described in "Installer: out of scope, with one handoff"? Nothing in this plan is blocked on it, because SCMU cannot create an EF instance today. + +## Reference pull requests + +- [Audit EF spike: #5318](https://github.com/Particular/ServiceControl/pull/5318) +- [Scale out error ingestion 1/3, dispatcher ownership: #5800](https://github.com/Particular/ServiceControl/pull/5800) +- [Scale out error ingestion 2/3, ingestion-only host: #5801](https://github.com/Particular/ServiceControl/pull/5801) +- [Scale out error ingestion 3/3, health endpoints: #5803](https://github.com/Particular/ServiceControl/pull/5803)