Skip to content

Add DBM support to IBM DB2 - #25329

Merged
jasonmp85 merged 44 commits into
masterfrom
dbm/db2
Oct 26, 2026
Merged

jasonmp85 merged 44 commits into
masterfrom
dbm/db2

Conversation

@sethsamuel

@sethsamuel sethsamuel commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Add opt-in Database Monitoring (dbm: true) for IBM Db2 LUW:

  • Collect per-query execution counts, elapsed and CPU time, rows read/returned, and lock waits from the package cache. Send obfuscated full query text with table, command, and SQL-comment metadata.
  • Add instance metadata and configurable database identifiers shared by DBM payloads and integration metrics. Attribute monitoring queries to the Agent, and use the shared DBM framework for background collection, custom queries, and collector health.
  • Document fixed-server deployment and HADR behavior. Add unit, integration, packaged-Agent E2E, and HADR takeover coverage, with amd64 GitLab CI environments for Db2 11.1, 11.5, and 12.1.

Validation: 49 unit tests passed, along with ddev lint and configuration/model validation. Live Db2 integration and E2E tests were not run locally.

Motivation

This is the first merge of the Db2 DBM development branch into master, bringing query metrics and full query text into the existing IBM Db2 integration. Schema collection remains a separate follow-up in #25330.

Earlier component reviews:

Checklist before requesting review

  • I self-reviewed the full diff, and this description matches the final change
  • Tests cover the change, or this description explains why none are needed
  • Automated review comments, including Codex review, were addressed or answered
  • I, the human author, understand this change, can explain it to reviewers, and believe it meets the repository standards

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines  Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 94.65%
• Overall Coverage: 94.99% (+5.42%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1092181 | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

evalya-impact-summary

Evalya CI impact

Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   4 (always emitted)
Changed files:   0

Full impact report
Learn more about CI impact filtering

@sethsamuel sethsamuel added the qa/skip-qa Automatically skip this PR for the next QA label Sep 23, 2026
@dd-octo-sts

dd-octo-sts Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Disk usage change

Commit 1092181 compared against 1e3691a.

Uncompressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
ibm_db2 4.4.0 Integration 🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB 🔺 +26.85 KiB

Compressed

linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB
Details
Name Version Type linux-aarch64 linux-x86_64 macos-aarch64 macos-x86_64 windows-x86_64
ibm_db2 4.4.0 Integration 🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB 🔺 +9.17 KiB

@chouetz chouetz added the internal Identify a non-fork PR label Oct 2, 2026
@jasonmp85
jasonmp85 added this pull request to stack #25731 October 21, 2026 17:33
@jasonmp85
jasonmp85 force-pushed the dbm/db2 branch 2 times, most recently from 14e5734 to 4017aa7 Compare October 22, 2026 01:17
Comment thread ibm_db2/datadog_checks/ibm_db2/ibm_db2.py
Comment thread ibm_db2/datadog_checks/ibm_db2/query_metrics.py
Comment thread ibm_db2/datadog_checks/ibm_db2/ibm_db2.py Outdated
Comment thread ibm_db2/datadog_checks/ibm_db2/data/conf.yaml.example
Comment thread ibm_db2/datadog_checks/ibm_db2/query_metrics.py Outdated
jasonmp85

This comment was marked as duplicate.

sethsamuel and others added 8 commits October 22, 2026 13:51
Build the generated InstanceConfig model once in __init__, as postgres
does, instead of reading loose values from the instance dict. Global
custom query merging moves into build_config.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Base metric queries and table space state tracking move to
MetricsCollector in metrics.py, and custom query collection moves to
CustomMetricsCollector in custom_metrics.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Db2Connection owns the ibm_db connection, reconnect-on-failure, and row
iteration, and get_connection_data becomes a module-level function.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Submission methods are looked up on the check rather than the
collector, which has no gauge/monotonic_count, so every custom metric
was being rejected as an invalid submission method.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Switch the test environment to icr.io/db2_community/db2 for 11.5.5.1
and 12.1.5.0. The official image needs privileged mode and an explicit
instance password, and creates the instance at startup, so the fixture
waits for setup to finish before creating the test database.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Custom queries now run on their own thread and connection, registered
with the DatabaseCheck async job lifecycle. The check and the job close
their connections on shutdown.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@AAraKKe AAraKKe left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi team! Some comments below, I understand why you are trying to run tests in GitLab but that is a hard blocker to merge this PR, take a look at the comment about that as the strongest one that I have. I would propose to drop 11.1 al together since it is fully EOL with not even extended support and supporting it in GitLab means not knowing anything about the tests on the integration at all in the standard quality validation checks we currently have.

Comment thread .ddev/config.toml Outdated
Comment thread ibm_db2/datadog_checks/ibm_db2/custom_metrics.py Outdated
Comment thread ibm_db2/tests/test_integration_e2e.py
Comment thread ibm_db2/hatch.toml Outdated
Comment thread ibm_db2/tests/test_hadr.py
Comment thread ibm_db2/tests/test_query_metrics.py
Comment thread ibm_db2/tests/test_hadr.py
Comment thread ibm_db2/datadog_checks/ibm_db2/custom_metrics.py
Comment thread ibm_db2/datadog_checks/ibm_db2/query_metrics.py
@dd-octo-sts

dd-octo-sts Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dispatcher tests: passed

Dispatcher beta: informational only
Existing CI remains the merge signal.

  10/10 jobs

✅ 10 passed · nothing failed

Batches · ✅ batch-01 10/10

Dispatcher finished on cd6ae55 — GitHub Run · Dispatcher Logs.

@jasonmp85

Copy link
Copy Markdown
Contributor

I'm checking out the ACR question both otherwise have addressed the feedback

@jasonmp85

Copy link
Copy Markdown
Contributor

@AAraKKe I think I've addressed your feedback. There might be a CI flake right now, but if my last commit fixes that it's good to merge.

Any chance of a ✅ 😄 ?

@AAraKKe AAraKKe left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets go! 🚀🚀🚀

Thanks for taking a look, the failed job right now does seem more an issue with the testing harness than anything else because I saw it passing before. I will take a look tomorrow since I am in rotation this week.

@dd-octo-sts

dd-octo-sts Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 22 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file ✅
ci Validate CI configuration and code coverage settings ✅
codeowners Validate every integration has a CODEOWNERS entry ✅
config Validate default configuration files against spec.yaml ✅
dep Verify dependency pins are consistent and Agent-compatible ✅
http Validate integrations use the HTTP wrapper correctly ✅
imports Validate check imports do not use deprecated modules ✅
integration-style Validate check code style conventions ✅
jmx-metrics Validate JMX metrics definition files and config ✅
labeler Validate PR labeler config matches integration directories ✅
legacy-signature Validate no integration uses the legacy Agent check signature ✅
license-headers Validate Python files have proper license headers ✅
licenses Validate third-party license attribution list ✅
metadata Validate metadata.csv metric definitions ✅
models Validate configuration data models match spec.yaml ✅
openmetrics Validate OpenMetrics integrations disable the metric limit ✅
package Validate Python package metadata, naming, and wheel contents ✅
pr-description Validate the pull request checklist and visible description length ✅
qa-label Validate the pull request declares whether it needs QA for the next Agent release ✅
readmes Validate README files have required sections ✅
saved-views Validate saved view JSON file structure and fields ✅
version Validate version consistency between package and changelog ✅

View full run

@jasonmp85
jasonmp85 added this pull request to the merge queue Oct 26, 2026
Merged via the queue into master with commit 3891dd4 Oct 26, 2026
81 of 84 checks passed
@jasonmp85
jasonmp85 deleted the dbm/db2 branch October 26, 2026 18:56
@dd-octo-sts dd-octo-sts Bot added this to the 7.86.0 milestone Oct 26, 2026
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/datadog-agent that referenced this pull request Oct 27, 2026
### What does this PR do?

Update `go-sqllexer` from v0.2.5 to [v0.2.6](https://github.com/DataDog/go-sqllexer/tree/v0.2.6), which includes the merged [Db2 literal-handling fix](DataDog/go-sqllexer#115). Keep the version and checksums consistent across Agent modules.

### Motivation

Db2 treats backslashes as ordinary characters inside string literals. The current lexer treats them as escapes, so `VALUES 'path\', 'secret'` can normalize to `VALUES ? secret ?`, exposing a literal value. With this fix, it becomes `VALUES ?, ?`.

This is needed for the [Db2 DBM integration](DataDog/integrations-core#25329) just merged into integrations-core.

### Describe how you validated your changes

- Obfuscator suite passed with race detection; `dda inv tidy` passed without additional changes.
- A rebuilt Linux/amd64 Agent using the same lexer code passed all 13 native Db2 obfuscation corpus pairs. The v0.2.5 build failed two backslash cases.
- Verified the signed v0.2.6 tag resolves through the public Go proxy; it differs from the native-tested commit only by the release changelog.


Co-authored-by: jason.petersen <jason.petersen@datadoghq.com>
@jasonmp85 jasonmp85 mentioned this pull request Oct 27, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants