Skip to content

feat(cdk): section-aware CHECK for the stepped setup form (check-section) - #1255

Open
Serhii Lazebnyi (lazebnyi) wants to merge 6 commits into
mainfrom
feat/check-section
Open

Serhii Lazebnyi (lazebnyi) wants to merge 6 commits into
mainfrom
feat/check-section

Conversation

@lazebnyi

@lazebnyi Serhii Lazebnyi (lazebnyi) commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What

CDK side of the stepped setup form: CHECK becomes section-aware.

A connector whose spec declares sections next to groups (each section listing the groups of properties it shows, in form order) can be checked one section at a time. The configuration JSON then carries a top-level check-section key naming the section, and only holds the fields of that section and of the sections before it. Without the key, CHECK is exactly today's full check.

"groups":   [{"id": "db"}, {"id": "auth"}],
"sections": [
  {"id": "connection",  "title": "Connection",      "order": 0, "groups": ["db"],   "check": true},
  {"id": "credentials", "title": "Credentials",     "order": 1, "groups": ["auth"], "check": true},
  {"id": "review",      "title": "Review & set up", "order": 2, "groups": []}
]

A section declares "check": true when the platform should run a CHECK with check-section once the user completes it. A section without the flag exists for layout only and is never checked on its own: the platform never sends its id, and the connector rejects it with a config error if it does. The typical case is a final review step, which the full check before creating the source covers anyway.

check-section Entrypoint
absent source.check(config), unchanged
the last declared section source.check(config) without the key: the configuration is complete by then
any other declared section source.check_section(section, config) without the key
unknown section, non-string value, or a spec without sections failed CONNECTION_STATUS with a config error naming the declared sections
a declared section without "check": true failed CONNECTION_STATUS with a config error saying the section has no check

Before any of that, the partial config is validated against the spec narrowed to the section: required only keeps the properties whose group belongs to that section or to an earlier one, and every property keeps its own schema, so the present fields are still validated in full.

How

  • airbyte_cdk/sources/utils/check_section.py: CHECK_SECTION_KEY, declared_sections, checkable_sections, requested_section, is_last_section, strip_check_section, section_spec.
  • AirbyteEntrypoint.check: resolves the section, narrows the validation, dispatches (_check). discover and read are untouched.
  • BaseConnector.check_section(logger, section, config): new hook, fails with a system_error unless the connector overrides it, so a spec declaring sections without an implementation is loud.
  • AbstractSource.check_section wraps a new check_section_connection(logger, section, config) -> (bool, error), the counterpart of check_connection, so sources keep the tuple-returning style.

The contract for implementations: validate the fields of the section, the fields of earlier sections it depends on, and the server-side prerequisites those fields describe, and nothing beyond; report a missing or invalid field through a config error naming it. The spec orders sections so that no section relies on a later one.

Tests

  • unit_tests/sources/utils/test_check_section.py: section resolution (absent, declared, unknown, non-string, no sections, section without check), last section, key stripping, required narrowing (first, middle and last section; other spec fields kept), root cross-property constraints dropped.
  • unit_tests/test_entrypoint.py: check end to end with a sectioned spec: delegation with the partial config, full check for the last section, failure of check_section reported as a failed status, system error when the source does not implement it, narrowed validation still rejecting a missing section field, full validation without the key, invalid section values, spec without sections.
  • unit_tests/sources/test_abstract_source.py: check_section wraps check_section_connection; default is a system error.

First consumer: source-s3 (airbytehq/airbyte#90132), which declares bucket / files / review sections and checks the bucket access for the first one.

🤖 Generated with Claude Code

A connector whose spec declares `sections` (each listing the `groups` of
properties it shows) can be checked one section at a time. The config
JSON then carries a top-level `check-section` key naming the section and
only holds the fields collected up to that section.

The entrypoint resolves the key against the spec: no key keeps today's
full check; an unknown section, a non-string value or a spec without
sections fail with a config error; the last section runs the full check
because the configuration is complete by then; any other section is
delegated to the connector's new `check_section`. The partial config is
validated against the spec narrowed to the section (`required` keeps the
properties whose group belongs to that section or an earlier one).

`BaseConnector.check_section` fails with a system error unless the
connector overrides it; `AbstractSource.check_section` wraps a new
`check_section_connection`, the counterpart of `check_connection`.
@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@feat/check-section#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch feat/check-section

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

PyTest Results (Fast)

5 939 tests  +37   5 908 ✅ +37   6m 14s ⏱️ -7s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 27a68d0. ± Comparison against base commit 3e85723.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

PyTest Results (Full)

5 942 tests  +37   5 911 ✅ +37   7m 57s ⏱️ - 3m 6s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 27a68d0. ± Comparison against base commit 3e85723.

♻️ This comment has been updated with latest results.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread airbyte_cdk/sources/utils/check_section.py
Comment thread airbyte_cdk/sources/utils/check_section.py Outdated
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Comment thread airbyte_cdk/connector.py Outdated
A section check validates the partial config against the spec narrowed
to the section, but root-level `oneOf`, `anyOf`, `allOf`, `not`,
`if`/`then`/`else`, `dependencies`, `dependentRequired` and
`dependentSchemas` can still require fields of later sections and
blocked the check before the connector ran. The narrowed spec now drops
those keywords; every present field keeps its own schema and the full
check applies the constraints again.

Also documents and tests that a null `check-section` is rejected like
any other non-string value.
Comment thread airbyte_cdk/connector.py
Comment thread airbyte_cdk/sources/utils/check_section.py Outdated
The default `check_section` returned nothing but a raise while
declaring an `AirbyteConnectionStatus`; it now returns a failed status
naming the missing implementation, and `check_section_connection`
reuses that message as its (False, error) tuple. This also removes the
local import of AirbyteTracedException in connector.py. The config
error helper in check_section.py is inlined.
@lazebnyi

Copy link
Copy Markdown
Contributor Author

Patrick Nilan (@pnilan) I marked it as ready for review to get Devin’s AI review while I’m still fixing things, but thanks for taking an early look!

@pnilan Patrick Nilan (pnilan) left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple changes. Main one is the check_section question.

EDIT: Just saw your comment above! Makes sense, sorry for jumping on it quickly. Was excited to see what you had going!

Both defaults now build their message from one constant,
CHECK_SECTION_NOT_IMPLEMENTED, instead of AbstractSource reaching into
BaseConnector for it, and the docstrings describe the two methods as
the section counterparts of check and check_connection.
A section declares `"check": true` when the platform should run a CHECK
with `check-section` once the user completes it. `requested_section`
rejects a section without the flag with a config error, so a section
which exists for layout only (such as a final review step covered by
the full check before creation) is never checked on its own.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants