Skip to content

Add OpenMetrics-based Datadog Agent integration - #25795

Open
jszwedko wants to merge 10 commits into
masterfrom
jszwedko/datadog-agent-openmetrics-integration
Open

jszwedko wants to merge 10 commits into
masterfrom
jszwedko/datadog-agent-openmetrics-integration

Conversation

@jszwedko

@jszwedko jszwedko commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a datadog_agent integration that monitors the Datadog Agent through its OpenMetrics telemetry endpoint (/telemetry), and marks the expvar-based Agent Metrics integration as deprecated in favor of it.

By default it collects the telemetry analogous to what go_expvar.d/agent_stats.yaml.example collected (forwarder, DogStatsD, aggregator, scheduler, Logs Agent), using the names from the Agent's openmetrics.d/agent_stats.yaml.example under the datadog.agent namespace. Counters are submitted with native OpenMetrics V2 types (<name>.count). tag_by_endpoint defaults to false because the forwarder telemetry has its own endpoint label (the intake endpoint), which would be ambiguous next to the scraped URL.

The README includes a "Migrating from Agent Metrics" section that maps each old metric to its replacement and covers the behavior changes (counters as .count, combined metrics distinguished by tags, Go runtime metrics opt-in).

The new integration uses the datadog-agent-openmetrics app_id. app_ids can't be reused, so agent_metrics keeps datadog-agent.

Motivation

DataDog/datadog-agent#55721 deprecated collecting the Agent's internal telemetry via expvars in favor of the OpenMetrics endpoint, which has typed metrics, richer tags, and more metrics. That PR only added an example openmetrics.d config. This gives users a first-class integration to migrate to.

Some of the default metrics (aggregator flush stats, scheduler.queues.size, etc.) were added to the telemetry endpoint by that PR, so they require Agent 7.86+. Shipping a datadog_agent.d/conf.yaml.example with the Agent is a follow-up in datadog-agent.

Tests: a unit test runs the check against a fixture captured from a real Agent's /telemetry (plus samples for families that only appear under load) and asserts every documented metric is collected and nothing else is. The E2E test scrapes the E2E Agent container's own telemetry on a free port, so it doesn't reach an Agent installed on the host.

Checklist before requesting review

  • I self-reviewed the full diff, and this description matches the final change
  • Tests cover the change, or this description explains why none are needed
  • Automated review comments, including Codex review, were addressed or answered
  • I, the human author, understand this change, can explain it to reviewers, and believe it meets the repository standards

🤖 Generated with Claude Code

Add a `datadog_agent` integration that scrapes the Agent's `/telemetry`
OpenMetrics endpoint, collecting the telemetry analogous to what the
expvar-based Agent Metrics integration collected. Mark Agent Metrics as
deprecated in favor of it, and move the `datadog-agent` app_id to the new
integration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

  2/2 jobs

✅ 2 passed · nothing failed

Batches · ✅ batch-01 2/2

Dispatcher finished on 3d27d19 — GitHub Run · Dispatcher Logs.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

evalya-impact-summary

Evalya CI impact

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   4 (always emitted)
Changed files:   29

Full impact report
Learn more about CI impact filtering

@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Pipelines  Tests  Code Coverage

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 1 Pipeline job failed

Validate repository | Run Validations / Validate

View more details · View in GitHub Actions

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 93.18%
• Overall Coverage: 90.69% (+1.08%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9be1bdc | Docs | View more details | Give us feedback!

Comment thread agent_metrics/manifest.json Resolved
@dd-octo-sts

dd-octo-sts Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Disk usage change

Commit 9be1bdc compared against cd6ae55.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_agent (NEW) 0.0.1 Integration 🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB 🔺 +39.87 KiB

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
datadog_agent (NEW) 0.0.1 Integration 🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB 🔺 +11.63 KiB

The Agent always serves /telemetry; telemetry.enabled only enables
additional telemetry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jszwedko
jszwedko marked this pull request as ready for review October 26, 2026 18:04
@jszwedko
jszwedko requested a review from a team October 26, 2026 18:04
@jszwedko
jszwedko requested review from a team as code owners October 26, 2026 18:04
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-26T18:04:23.132440Z 986bb1f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jszwedko

Copy link
Copy Markdown
Contributor Author

Left some feedback from Docs and approved the PR.

@drichards-87 thanks! I applied all of the suggestions in a6c12a7.

drichards-87
drichards-87 previously approved these changes Oct 30, 2026
…integration

app_ids can't be reused, so agent_metrics keeps datadog-agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed drichards-87’s stale review November 2, 2026 19:16

Review from drichards-87 is dismissed. Related teams and files:

  • documentation
    • datadog_agent/manifest.json
@steveny91

Copy link
Copy Markdown
Collaborator

Looks good. I have one question, do we expect users to migrate existing to this integration? The change from the counter that the old integration used to collect might behave differently here with a different name with the .count affix. Would that be of concern?

steveny91
steveny91 previously approved these changes Nov 2, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed steveny91’s stale review November 2, 2026 21:02

Review from steveny91 is dismissed. Related teams and files:

  • agent-integrations
    • datadog_agent/README.md
@jszwedko

jszwedko commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Looks good. I have one question, do we expect users to migrate existing to this integration? The change from the counter that the old integration used to collect might behave differently here with a different name with the .count affix. Would that be of concern?

@steveny91 We are primarily targeting new users that are starting to collect telemetry for the Datadog Agent but hopefully some existing users migrate too (there are no plans to remove the old integration though). Your question prompted me to consider where we should have a small blurb for customers migrating so I pushed that up as f5dcf50.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread datadog_agent/README.md Resolved
drichards-87
drichards-87 previously approved these changes Nov 2, 2026
Comment thread datadog_agent/README.md Resolved
Comment thread datadog_agent/README.md Resolved
Comment thread datadog_agent/README.md Resolved
Comment thread datadog_agent/README.md Resolved
Comment thread datadog_agent/README.md Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed drichards-87’s stale review November 2, 2026 23:14

Review from drichards-87 is dismissed. Related teams and files:

  • documentation
    • datadog_agent/README.md
@jszwedko

jszwedko commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @drichards-87 ! I applied the suggested updates in 9be1bdc

@dd-octo-sts

dd-octo-sts Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Validation Report

Validation Description Status
pr-description Validate the pull request checklist and visible description length

To fix: Tick every item in .github/PULL_REQUEST_TEMPLATE.md and keep the description under 3,000 visible characters.
❌

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (21)
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file ✅
ci Validate CI configuration and code coverage settings ✅
codeowners Validate every integration has a CODEOWNERS entry ✅
config Validate default configuration files against spec.yaml ✅
dep Verify dependency pins are consistent and Agent-compatible ✅
http Validate integrations use the HTTP wrapper correctly ✅
imports Validate check imports do not use deprecated modules ✅
integration-style Validate check code style conventions ✅
jmx-metrics Validate JMX metrics definition files and config ✅
labeler Validate PR labeler config matches integration directories ✅
legacy-signature Validate no integration uses the legacy Agent check signature ✅
license-headers Validate Python files have proper license headers ✅
licenses Validate third-party license attribution list ✅
metadata Validate metadata.csv metric definitions ✅
models Validate configuration data models match spec.yaml ✅
openmetrics Validate OpenMetrics integrations disable the metric limit ✅
package Validate Python package metadata, naming, and wheel contents ✅
qa-label Validate the pull request declares whether it needs QA for the next Agent release ✅
readmes Validate README files have required sections ✅
saved-views Validate saved view JSON file structure and fields ✅
version Validate version consistency between package and changelog ✅

View full run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants