Skip to content

feat(webhooks): Return an error code when a webhook test can't connect - #9019

Open
khvn26 wants to merge 3 commits into
mainfrom
fix/webhook-test-error-reason
Open

khvn26 wants to merge 3 commits into
mainfrom
fix/webhook-test-error-reason

Conversation

@khvn26

@khvn26 khvn26 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

In this PR, we surface non-HTTP errors during webhook testing as POSIX error names (e.g. ECONNREFUSED, EAI_NONAME), or OpenSSL certificate verification error names (e.g. X509_V_ERR_CERT_HAS_EXPIRED):

The code is derived only from the exception type, errno and OpenSSL verify code, and does not touch sensitive data. The codes do reveal coarse reachability, e.g. refused vs timed out. Response timing already gives that away, and SSRF validation still rejects internal addresses before any request is made.

How did you test this code?

  • Unit tests
  • Ran this branch locally (API + dashboard) with SSRF_ALLOWED_HOSTS=localhost and a private-CA HTTPS server on localhost:7000. curl --cacert to the endpoint succeeds, and the dashboard's "Test your webhook" shows the code above.
  • Ran get_request_error_code inside the released flagsmith/flagsmith:2.292.0 image (Python 3.13, requests 2.33.1, urllib3 2.8.0) against real failures.

Since #7550, a webhook test that fails before getting an HTTP response
returns the same message whatever went wrong, and the exception isn't
logged. Self-hosted users can't tell a TLS trust problem from DNS, a
proxy, a firewall or a timeout.

Add a `code` to that 400 response: the POSIX error name
(`ECONNREFUSED`, `ETIMEDOUT`, `ECONNRESET`, ...) or, for certificate
failures, the Node.js-style OpenSSL verification code
(`UNABLE_TO_GET_ISSUER_CERT_LOCALLY`, `CERT_HAS_EXPIRED`, ...). The code
is derived from the exception type and errno only, so no hostnames,
addresses or ports reach the client.
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
docs Ignored Ignored Preview Nov 19, 2026 3:42pm UTC
flagsmith-frontend-preview Ignored Ignored Preview Nov 19, 2026 3:42pm UTC
flagsmith-frontend-staging Ignored Ignored Preview Nov 19, 2026 3:42pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds a helper that maps request exceptions to error codes. The webhook request error response includes the derived code. The serializer and OpenAPI schema document optional status and code fields. Unit tests cover error-code mappings and webhook responses.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Low

Merge Risk: Minimal · up to 1417a

Webhook test failures now report an error code, and the certificate and proxy-timeout cases raised earlier are handled and covered by unit tests. No actionable merge risk remains.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 6280127a-ce24-4081-a03d-29c2a86532f8
Commits

Reviewing files that changed from the base of the PR and between f835498 and 1417a19.

Files selected for processing: 6

  • api/core/network.py
  • api/tests/unit/core/test_network.py
  • api/tests/unit/webhooks/test_unit_webhooks.py
  • api/webhooks/serializers.py
  • api/webhooks/views.py
  • openapi.yaml
Files with no reviewable changes: 1

  • api/tests/unit/webhooks/test_unit_webhooks.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

Finishing Touches
Fix failing CI checks

  • Commit to this branch
  • Create a new PR

@github-actions github-actions Bot added api Issue related to the REST API feature New feature or request labels Nov 19, 2026
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.93%. Comparing base (7ec6a0e) to head (1417a19).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9019   +/-   ##
=======================================
  Coverage   98.92%   98.93%           
=======================================
  Files        1713     1714    +1     
  Lines       72566    72657   +91     
=======================================
+ Hits        71789    71880   +91     
  Misses        777      777           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@khvn26 khvn26 changed the title feat(webhooks): return an error code when a webhook test can't connect feat(webhooks): Return an error code when a webhook test can't connect Nov 19, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Nov 19, 2026
@khvn26
khvn26 marked this pull request as ready for review November 19, 2026 15:25
@khvn26
khvn26 requested review from a team as code owners November 19, 2026 15:25
@khvn26
khvn26 requested review from emyller and removed request for a team November 19, 2026 15:25
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-e2e:pr-9019 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-api-test:pr-9019 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-9019 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-api:pr-9019 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-9019 Finished ✅ Results ✅

@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Nov 19, 2026
Comment thread api/core/network.py Outdated
# RFC 6598 carrier-grade NAT; not caught by ipaddress.is_private.
_SHARED_ADDRESS_SPACE = ipaddress.ip_network("100.64.0.0/10")

# OpenSSL X509_V_ERR_* verification results, named as Node.js reports them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why "as Node.js reports them"? I thought we'd updated the webhook testing process to be run through the python backend to make it as close as possible to the real thing.

@khvn26 khvn26 Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not about verification process, it's about presentation. Basically, the agent had "POSIX/Node.js-style errors" in the brief because they are the most universally comprehensive to my taste. Happy to remove any Node.js references if you'd like.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think it's confusing.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor
✅ private-cloud · depot-ubuntu-latest-arm-16 — run #23470 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  35.8 seconds
commit  1417a19
info  🔄 Run: #23470 (attempt 1)

🗂️ Previous results
✅ private-cloud · depot-ubuntu-latest-16 — run #23470 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  5 passed

Details

stats  5 tests across 5 suites
duration  2.9 seconds
commit  1417a19
info  🔄 Run: #23470 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #23470 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  36.5 seconds
commit  1417a19
info  🔄 Run: #23470 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #23470 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  31.8 seconds
commit  1417a19
info  🔄 Run: #23470 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #23467 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  36.1 seconds
commit  f835498
info  🔄 Run: #23467 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #23467 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  35.6 seconds
commit  f835498
info  🔄 Run: #23467 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-16 — run #23467 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  5 passed

Details

stats  5 tests across 4 suites
duration  3.1 seconds
commit  f835498
info  🔄 Run: #23467 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #23467 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  30.4 seconds
commit  f835498
info  🔄 Run: #23467 (attempt 1)

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 6 files

Confidence score: 5/5

  • In api/webhooks/serializers.py, the code help text says “POSIX error name,” but the values also include Node.js-style names, which could mislead API users. Update the help text to describe the values accurately.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/webhooks/serializers.py">

<violation number="1" location="api/webhooks/serializers.py:40">
P3: The `code` help_text says "POSIX error name", but the values come from `core/network.py` and include Node.js-style names such as `ENOTFOUND`, `ESOCKETTIMEDOUT`, and `ERR_TLS_CERT_ALTNAME_INVALID`. It can also be `UNKNOWN`. Update the help_text to say POSIX or Node.js-style, and list the fallback.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread api/webhooks/serializers.py Outdated
code = serializers.CharField(
required=False,
help_text=(
"A POSIX error name, or a TLS error code, when "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The code help_text says "POSIX error name", but the values come from core/network.py and include Node.js-style names such as ENOTFOUND, ESOCKETTIMEDOUT, and ERR_TLS_CERT_ALTNAME_INVALID. It can also be UNKNOWN. Update the help_text to say POSIX or Node.js-style, and list the fallback.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/webhooks/serializers.py, line 40:

<comment>The `code` help_text says "POSIX error name", but the values come from `core/network.py` and include Node.js-style names such as `ENOTFOUND`, `ESOCKETTIMEDOUT`, and `ERR_TLS_CERT_ALTNAME_INVALID`. It can also be `UNKNOWN`. Update the help_text to say POSIX or Node.js-style, and list the fallback.</comment>

<file context>
@@ -29,5 +29,15 @@ class TestWebhookSuccessResponseSerializer(serializers.Serializer[None]):
+    code = serializers.CharField(
+        required=False,
+        help_text=(
+            "A POSIX error name, or a TLS error code, when "
+            "the webhook could not be reached."
+        ),
</file context>

Comment thread api/tests/unit/core/test_network.py Resolved
Comment thread api/core/network.py Resolved
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Visual Regression

26 screenshots compared. See report for details.
View full report

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 63730013-119c-461c-a801-31c8bcbd2130
Commits

Reviewing files that changed from the base of the PR and between fe9b4cd and f835498.

Files selected for processing: 6

  • api/core/network.py
  • api/tests/unit/core/test_network.py
  • api/tests/unit/webhooks/test_unit_webhooks.py
  • api/webhooks/serializers.py
  • api/webhooks/views.py
  • openapi.yaml

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment thread api/core/network.py Outdated
21: "UNABLE_TO_VERIFY_LEAF_SIGNATURE",
22: "CERT_CHAIN_TOO_LONG",
23: "CERT_REVOKED",
24: "INVALID_CA",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Account for OpenSSL 3 verification codes.

On OpenSSL 3.0, verification code 24 means NO_ISSUER_PUBLIC_KEY; INVALID_CA is 79. This entry reports the wrong cause for code 24, while an invalid CA returns the generic EPROTO. Account for the runtime version and add tests for these mappings. Keep the existing expiry and issuer-trust mappings. (raw.githubusercontent.com)

Comment thread api/core/network.py Outdated
Comment on lines +48 to +51
if isinstance(error, requests.exceptions.ConnectTimeout):
return "ETIMEDOUT"
if isinstance(error, requests.exceptions.ReadTimeout):
return "ESOCKETTIMEDOUT"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Recognise timeouts inside proxy exception chains.

If an HTTPS proxy connection times out, urllib3 can wrap ConnectTimeoutError in ProxyError. Requests then raises requests.exceptions.ProxyError, so these top-level checks do not match. The chain classifier has no nested timeout check and returns UNKNOWN when the socket timeout has no errno. The dependency sources establish this wrapping path. (raw.githubusercontent.com)

Classify nested timeout types and add a proxy-timeout test. Exclude NewConnectionError from a ConnectTimeoutError check so refused connections still return ECONNREFUSED. (raw.githubusercontent.com)

- Name certificate failures with OpenSSL's X509_V_ERR_* constants,
  numbered as in OpenSSL 3+, where 24 is NO_ISSUER_PUBLIC_KEY and
  INVALID_CA is 79.
- Name DNS failures with their getaddrinfo EAI_* code, instead of
  reporting every resolver error as ENOTFOUND.
- Report read timeouts, and timeouts connecting to a proxy, as
  ETIMEDOUT.
- Omit `code` when the cause isn't recognised, rather than returning
  UNKNOWN.
- Build real urllib3 objects in tests, removing the type ignores.
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Nov 19, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 6 files (changes from recent commits).

Confidence score: 4/5

  • In api/core/network.py, unmapped certificate verification codes are reported as EPROTO, which misleads users about the trust failure. Return None for unmapped codes so they’re omitted.
  • In api/core/network.py, verify code 24 is mislabeled on Python builds linked to OpenSSL 1.1.1, obscuring invalid-CA failures. Select the name based on ssl.OPENSSL_VERSION_INFO.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="api/core/network.py">

<violation number="1" location="api/core/network.py:37">
P2: This labels `verify_code == 24` incorrectly on Python builds linked to OpenSSL 1.1.1, where code 24 is `X509_V_ERR_INVALID_CA`. Select the name based on `ssl.OPENSSL_VERSION_INFO` so invalid-CA failures are not misreported.</violation>

<violation number="2" location="api/core/network.py:70">
P2: An unmapped certificate verification code is reported as `EPROTO`, which tells the user this is a protocol error rather than a trust failure. Return `None` for unmapped verify codes so the code is omitted, as the commit message describes.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread api/core/network.py
def _get_exception_error_code(exc: BaseException) -> str | None:
# SSL errors subclass OSError, with an OpenSSL-specific errno.
if isinstance(exc, ssl.SSLCertVerificationError):
return _CERT_VERIFY_ERROR_NAMES.get(exc.verify_code, "EPROTO")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: An unmapped certificate verification code is reported as EPROTO, which tells the user this is a protocol error rather than a trust failure. Return None for unmapped verify codes so the code is omitted, as the commit message describes.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/core/network.py, line 70:

<comment>An unmapped certificate verification code is reported as `EPROTO`, which tells the user this is a protocol error rather than a trust failure. Return `None` for unmapped verify codes so the code is omitted, as the commit message describes.</comment>

<file context>
@@ -10,63 +10,74 @@
     # SSL errors subclass OSError, with an OpenSSL-specific errno.
     if isinstance(exc, ssl.SSLCertVerificationError):
-        return _CERT_VERIFY_ERROR_CODES.get(exc.verify_code, "EPROTO")
+        return _CERT_VERIFY_ERROR_NAMES.get(exc.verify_code, "EPROTO")
     if isinstance(exc, ssl.SSLError):
         return "EPROTO"
</file context>
Suggested change
return _CERT_VERIFY_ERROR_NAMES.get(exc.verify_code, "EPROTO")
return _CERT_VERIFY_ERROR_NAMES.get(exc.verify_code)

Comment thread api/core/network.py
21: "X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE",
22: "X509_V_ERR_CERT_CHAIN_TOO_LONG",
23: "X509_V_ERR_CERT_REVOKED",
24: "X509_V_ERR_NO_ISSUER_PUBLIC_KEY",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This labels verify_code == 24 incorrectly on Python builds linked to OpenSSL 1.1.1, where code 24 is X509_V_ERR_INVALID_CA. Select the name based on ssl.OPENSSL_VERSION_INFO so invalid-CA failures are not misreported.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At api/core/network.py, line 37:

<comment>This labels `verify_code == 24` incorrectly on Python builds linked to OpenSSL 1.1.1, where code 24 is `X509_V_ERR_INVALID_CA`. Select the name based on `ssl.OPENSSL_VERSION_INFO` so invalid-CA failures are not misreported.</comment>

<file context>
@@ -10,63 +10,74 @@
+    21: "X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE",
+    22: "X509_V_ERR_CERT_CHAIN_TOO_LONG",
+    23: "X509_V_ERR_CERT_REVOKED",
+    24: "X509_V_ERR_NO_ISSUER_PUBLIC_KEY",
+    25: "X509_V_ERR_PATH_LENGTH_EXCEEDED",
+    26: "X509_V_ERR_INVALID_PURPOSE",
</file context>
Suggested change
24: "X509_V_ERR_NO_ISSUER_PUBLIC_KEY",
24: "X509_V_ERR_NO_ISSUER_PUBLIC_KEY" if ssl.OPENSSL_VERSION_INFO >= (3, 0, 0) else "X509_V_ERR_INVALID_CA",

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants