Observability Stack — OpenSearch, OpenTelemetry and RKE2

A version-pinned, progressively validated observability platform for RKE2 and Java microservices. The reference layout keeps stateful observability services outside the application cluster. Hostnames, credentials, storage paths, and environment-specific deployment settings belong in private configuration.

This implementation replaces the repository's former SigNoz/Grafana experiment. SigNoz and Grafana are not components of this implementation.

Public reference boundary

This repository is a sanitized engineering reference, not an export of bank servers. Installation paths in templates are illustrative Linux defaults; directory layouts, host inventories, actual environment overrides, organization identifiers, credential paths and rollback artifacts belong in private, access-controlled infrastructure records. Some historical UAT qualification results are described here without releasing raw reports. Prior Git commits may still contain older deployment references until a separate historical audit is completed.

Current stack

Layer	Component	Validated version	Responsibility
Application	OpenTelemetry Java agent	2.30.0	Automatic traces, correlated logs, and runtime/application metrics
Kubernetes	OpenTelemetry Operator	0.154.0	Opt-in agent injection for agentless images
Kubernetes	OpenTelemetry Collector Contrib	0.156.0	OTLP gateway, Kubernetes enrichment, and signal routing
Kubernetes	Fluent Bit	4.1.0	Container stdout collection for workloads not exporting OTLP logs
Processing	Data Prepper	2.16.0	Trace/log ingestion and v2 service-map generation
Metrics	Prometheus	3.14.0	Metrics storage, remote-write ingestion, alert evaluation, and optional derived/reporting rules
Alerting	Alertmanager	0.31.1	Alert routing and notification delivery
Host metrics	node_exporter	1.12.1	Linux host CPU, memory, disk, filesystem, network, load, and uptime metrics
Host processes	process-exporter	0.8.7	Selected process-level metrics on the observability host
Storage and UI	OpenSearch and OpenSearch Dashboards	3.6.0	Logs, traces, application maps, dashboards, and investigation
Platform	RKE2 / Kubernetes	v1.35.7+rke2r1 / v1.35.7	Application orchestration

The versions above document a pinned reference configuration, not a live inventory for any particular bank. Use the compatibility and validation procedures before deploying to another environment.

See Version and Lifecycle Policy for the selection rules and production promotion process.

Architecture

flowchart TB
    subgraph RKE2["RKE2 application cluster"]
        Apps["Java microservices"]
        FB["Fluent Bit DaemonSet"]
        Operator["OpenTelemetry Operator"]
        Collector["OpenTelemetry Collector gateway"]

        Operator -. "opt-in injection" .-> Apps
        Apps -->|"OTLP traces, logs, metrics"| Collector
        FB -->|"stdout logs for non-OTLP workloads"| Collector
    end

    subgraph OBS["Dedicated observability platform"]
        DP["Data Prepper"]
        Prom["Prometheus"]
        OS["OpenSearch"]
        UI["OpenSearch Dashboards"]

        DP --> OS
        OS --> UI
        Prom --> UI
    end

    Collector -->|"traces and logs"| DP
    Collector -->|"remote-write metrics"| Prom

Loading

OpenSearch, OpenSearch Dashboards, Data Prepper, and Prometheus run on dedicated observability infrastructure. The Operator and Collector run once per Kubernetes cluster—not once per application or node.

For the production topology, failure domains, scaling boundaries, and network flows, read Scalable Observability Architecture.

Instrumentation modes

Every Java container must run exactly one OpenTelemetry Java agent.

Application image	Mode	Rule
Already starts with -javaagent	Embedded agent	Configure OTLP environment; never add Operator injection
Contains no agent	Operator injection	Add the explicit inject-java annotation

Applications exporting logs directly through OTLP use fluentbit.io/exclude: "true" to prevent Fluent Bit from sending an uncorrelated duplicate.

See Kubernetes Telemetry Onboarding for the in-cluster OpenTelemetry Collector/Operator manifests, the double-agent guardrail, validation, and rollback. Linux host metrics are onboarded separately with node_exporter.

Repository layout

.
├── compose.yaml                         # Single-node OpenSearch lab only
├── dashboards/                         # OpenSearch Dashboards export policy
├── docs/
│   ├── 01-version-policy.md
│   ├── 02-architecture.md
│   ├── 03-infrastructure-prerequisites.md
│   ├── 04-lab-preflight-validation.md
│   ├── 05-kubernetes-telemetry-onboarding.md
│   ├── 06-observability-host-bootstrap.md
│   ├── 07-validation-and-evidence.md
│   ├── 08-reference-uat-baseline.md
│   ├── 09-reference-runtime-capture.md
│   ├── 10-sanitized-reference-runtime-contract.md
│   ├── 11-live-closeout.md
│   ├── 12-developer-search-standardization.md
│   ├── 13-explore-log-datasets.md
│   ├── 14-alerting-standard.md
│   ├── 15-ubuntu-2404-cleanroom-qualification.md
│   └── 16-linux-host-metrics-onboarding.md
├── evidence/                           # Evidence policy; generated output stays local
├── kubernetes/
│   ├── collector/                       # Collector, Service, RBAC, and configuration
│   ├── examples/                        # Mutually exclusive Java onboarding modes
│   ├── instrumentation/                 # Shared opt-in Instrumentation resource
│   └── operator/                        # Pinned Operator Helm values
├── prometheus/
│   └── rules/                           # Standardized alerts plus optional recording/reporting rules
├── scripts/
│   └── validate-platform.sh             # Read-only end-to-end acceptance checks
├── prometheus.yml                       # Remote-write receiver and local scrape baseline
└── versions.env                         # Frozen component baseline

Validated results

The functional lab has demonstrated:

• stable RKE2 operation through a five-minute validation window;
• a healthy, pinned Operator and Collector;
• least-privilege Collector Kubernetes RBAC;
• Kubernetes metadata enrichment across traces, logs, and metrics;
• a gateway-to-payment distributed trace and v2 service-map relationship;
• trace- and span-correlated application logs;
• duplicate-log prevention between OTLP and Fluent Bit;
• automatic service-derived RED metric discovery for newly instrumented services;
• request, 4xx, 5xx, latency, availability, and error-budget calculations;
• a fresh Ubuntu 24.04.5 LTS dedicated-host clean-room bootstrap, secured activation, and five-minute runtime soak with all seven observability services healthy, all configured Prometheus targets UP, and zero unhealthy Prometheus rules.

This is a validated functional baseline, not a claim that the single-node lab topology is production ready.

Documentation path

Read the repository in this order:

1. Version and Lifecycle Policy
2. Scalable Observability Architecture
3. Infrastructure Prerequisites
4. Lab Preflight Validation
5. Kubernetes Telemetry Onboarding
6. Dedicated Observability Host Bootstrap
7. Validation and Evidence
8. Reference UAT Observability Baseline
9. Reference Runtime Capture
10. Sanitized Reference Runtime Contract
11. Live Observability Closeout
12. Developer Search Standardization
13. Explore Logs Dataset Separation
14. Alerting Standard
15. Ubuntu 24.04 Clean-Room Qualification
16. Linux Host Metrics Onboarding

Lab-only OpenSearch start

The root compose.yaml starts only OpenSearch and OpenSearch Dashboards for single-node functional testing. It is not the production deployment method.

export OPENSEARCH_INITIAL_ADMIN_PASSWORD='replace-with-a-strong-random-secret'
docker compose config
docker compose up -d

Do not store the password in Git or shell history in a real environment. Data Prepper and Prometheus are managed separately in the validated lab and will receive their own sanitized installation stages.

Production gates still open

Before production promotion, the project still requires:

• Collector gateway high availability and capacity tests;
• authenticated TLS or mTLS for every cross-host telemetry path;
• Kubernetes NetworkPolicies and host firewall rules;
• external secret management and credential rotation;
• OpenSearch multi-node sizing, retention, snapshots, and recovery testing;
• Prometheus retention, backup, and HA decision;
• alert routing, notification ownership, and runbooks;
• UAT soak, failure, and rollback tests at representative application scale.

Security

This public repository must never contain production addresses, hostnames, usernames, passwords, tokens, certificates, kubeconfigs, trace IDs, transaction IDs, or customer data. Environment-specific values belong in private overlays or an approved secrets manager.