[Anatolii Yatsuk (tolik0)]

Stricter standard tests, no runtime change. On the next CDK release, a Python or manifest-only connector whose acceptance-test-config.yml yields no runnable test scenario, or that has no such file, fails collection instead of passing with every scenario test skipped, unless the file declares a bypass_reason. Connector CI installs airbyte-cdk[dev] unpinned, so this reaches every connector at once. On master fd347841abe (2026-10-27) that is 35 connectors (10 without a bypass, 25 without the file). Monorepo changes fix all of them and must merge first: airbytehq/airbyte#88931 (bypass reasons, 25 connectors), airbytehq/airbyte#88932 (legacy list sections), and airbytehq/airbyte#90137 (bypass reasons for 10 Python destinations whose CI is already red on master, split out of #88931, plus the run module 9 of them were missing). With all three merged, the classifier finds 0 connectors in either group; with only #88931 and #88932, those 10 destinations lack a file.

What

Implements P0.3 of the CAT → FAST coverage epic (airbytehq/airbyte-internal-issues#17131, under airbytehq/airbyte-internal-issues#17129): an empty test-scenario set must fail, not silently skip.

Until now, pytest_generate_tests parametrized scenario with whatever get_scenarios() returned. An empty list made pytest skip every scenario test as got empty parameter set and the run passed. A connector whose configs were never collected, whose file was missing, or whose layout the reader did not understand looked the same as a tested one.

How

• Zero scenarios is an error. When no scenario can run and no bypass_reason explains why, collection fails with one message that names the file and says why each section and entry produced nothing (e.g. `acceptance_tests.spec.tests[0]`: no `config_path` and no `bypass_reason` (keys: `spec_path`)), followed by the fix. The error is raised with pytrace=False outside the except block, so it prints once, without a traceback.

• Explicit, machine-readable bypass. A bypass_reason counts in three places:

  • at the top level of the file: the whole connector. A file holding only this key is the opt-out for a connector without test configs. Combined with runnable configs it is an error.
  • on a section (connection: {bypass_reason: ...}), the layout CAT used and that ~285 connectors have today. It is ignored when the section lists a runnable config, as before.
  • on an entry without config_path. An entry with a config_path always runs.

  Only the collected sections count (spec, connection, discovery, basic_read, full_refresh). An incremental bypass or an IAM-role config does not make a connector "bypassed".

• Bypasses are visible. Each scenario test gets one extra skipped param per distinct reason ([bypassed-connection-discovery-basic_read-full_refresh], [bypassed-connector]) whose skip message is acceptance-test-config.yml bypasses <scope>: <reason>, and collection prints one line per connector, e.g. source-hibob: acceptance-test-config.yml declares 1 bypass(es), reported as skipped tests: the connector: No test config in GSM for this connector. The reasons land in the JUnit XML.

• Legacy list-format sections (connection: [...]) are read as the section's tests list, through section_entries(), which the spec backward-compatibility waiver reader from feat(standard-tests): check spec backward compatibility against the published version #1220 now shares.

• Legacy top-level tests: fails with one message naming the file and the migration (rename to acceptance_tests:, nest entries under tests). Converting those 34 connectors is airbytehq/airbyte-internal-issues#16264.

• A missing file is an error for Python and manifest-only connectors. Java connectors (language:java in metadata.yaml) are reported as bypassed, since their Gradle suites test them.

• Suites with their own scenarios. A suite that overrides get_scenarios() supplies its scenarios in code, so its acceptance-test-config.yml is not read for bypasses: a missing file, a legacy layout or a top-level bypass_reason changes nothing. If the override returns no scenario, collection fails with <Suite>.get_scenarios() returned no test scenarios.

• No values in failure messages. An entry the scenario model rejects (e.g. empty_streams as plain strings) is reported by location, field and rule; pydantic's input values are not printed.

Changes

• airbyte_cdk/test/standard_tests/_acceptance_test_config.py (new, private): loading, layout errors, scenario and bypass collection, and the empty-set message.
• docker_base.py: acceptance_test_config and get_scenarios() use it. New get_scenario_bypasses() and describe_empty_scenarios(), which skip the file when a subclass overrides get_scenarios(). get_scenarios() raises MissingAcceptanceTestConfigError (a FileNotFoundError) or AcceptanceTestConfigError (a ValueError) instead of warning, so existing except FileNotFoundError / except ValueError callers keep working.
• pytest_hooks.py: the failure, the bypass params and the collection line (pytest_report_collectionfinish).
• scenario.py: one field, ConnectorTestScenario.bypass_reason, holding the reason on the placeholder scenario of a skipped bypass test. Kept minimal on purpose (see Rollout).
• _spec_compatibility.py: reads the spec section through the shared section_entries().
• cdk-migrations.md: new ## Upgrading to the next minor release (unreleased) section with a before/now table, the three bypass forms and the migration steps.

Fleet impact

Classifier over the 594 non-archived Python and manifest-only connectors on airbyte master (fd347841abe, re-run 2026-10-27; first run on 1e22e3c), and over master plus both fleet PRs:

Group	Master count	Today	After this release, fleet PRs not merged	After this release, fleet PRs merged
Runnable scenarios	244	Scenarios run	Same scenarios (identical count per connector)	Same, 245
Only declared bypasses	286	Every scenario test skipped as an empty parameter set	Skipped with the declared reason	Same, 321
No runnable config, no bypass	10	Silently skipped	Collection fails	0, bypass declared (PR A)
No acceptance-test-config.yml	25	UserWarning, silently skipped	Collection fails	0, bypass-only file (PR A)
Legacy list-format sections	3	Sections ignored	Configs run: 2 connectors fail (dead or missing GSM configs), 1 config error	Converted, runnable configs pass (PR B)
Legacy top-level tests:	28	Collection fails	Collection fails, clearer message	Unchanged, tracked in airbytehq/airbyte-internal-issues#16264

The first run also found one invalid entry (expect_records without path); the P1.5 cleanup removed it from master.

Certified connectors newly declaring a bypass: destination-pinecone (GSM holds a config no entry lists) and source-granola (no GSM config).

The connectors kept in the private enterprise repository, whose CI runs the same standard tests, are not affected: the same classifier finds every Python and manifest-only connector there runnable.

Rollout

Release gate. This PR stays in draft until all of these are done:

• test(connectors): declare why 25 connectors have no standard-test scenario airbyte#88931 merged
• test(connectors): move legacy list-format acceptance test sections to the current layout airbyte#88932 merged
• fix(connectors): add the missing run module to 9 Python destinations and declare their standard-test bypass airbyte#90137 merged
• Classifier re-run on airbyte master right before the release is tagged: 0 connectors without a bypass, 0 without the file (connectors added after the fleet PRs can land uncovered, as source-github-g2 did)
• Release draft notes that connector test suites (not syncs) of connectors without a runnable config or a bypass_reason now fail collection, linking the cdk-migrations.md section

If the fleet PRs cannot land before the next release, split this PR: ship the bypass reading and reporting first, and make the empty set a failure one release later.

• Merge order: test(connectors): declare why 25 connectors have no standard-test scenario airbyte#88931, test(connectors): move legacy list-format acceptance test sections to the current layout airbyte#88932 and fix(connectors): add the missing run module to 9 Python destinations and declare their standard-test bypass airbyte#90137 first, then release this. The follow-up's destinations fail connector CI today (image spec failures, a revoked GSM credential, a broken build), so it needs those fixed first or a merge with the failures noted. All are safe on the released CDK: the 37 connectors touched at the time collected with exit 0 under airbyte-cdk[dev] 7.36.0, and on 2026-10-27 the 35 connectors of #88931 and the follow-up collect with exit 0 under 7.39.1, the release CI installs today. PR A's new files carry acceptance_tests: {} next to the bypass_reason because 7.36.0 rejects a file without acceptance_tests. Neither PR bumps a version (CheckVersionIncrement skips: only acceptance-test-config.yml changed).
• Ship as a minor (feat, no !), with the migration note under the shared "next minor release (unreleased)" heading.
• P1.5 (feat(standard-tests): remove the ignored scenario fields and reject acceptance-test-config keys that are not read #1226) is merged and its conflicts are resolved here. feat(standard-tests): remove the ignored scenario fields and reject acceptance-test-config keys that are not read #1226 merged on 2026-10-20 and shipped in 7.39.0. This branch merged main the same day: scenario.py, docker_base.py and cdk-migrations.md are resolved (feat(standard-tests): remove the ignored scenario fields and reject acceptance-test-config keys that are not read #1226's notes now sit under ## Upgrading to 7.39.0), and bypass_reason is dropped from AWAITING_READER in test_scenario_keys.py, since this PR reads it. The branch still merges cleanly with main.
• feat(standard-tests): remove the ignored scenario fields and reject acceptance-test-config keys that are not read #1226's monorepo cleanup landed as test(connectors): remove acceptance-test-config keys the standard tests never read (1/8: certified manifest-only) airbyte#88948–#88955 instead of #88927 (closed). PR A and PR B both merge cleanly with current master.
• With feat(standard-tests): check primary keys, state messages and error traces in read tests, and report stream statuses #1230 (P1.2): if feat(standard-tests): check primary keys, state messages and error traces in read tests, and report stream statuses #1230 merges first, this PR must add a minimal acceptance-test-config.yml to feat(standard-tests): check primary keys, state messages and error traces in read tests, and report stream statuses #1230's tmp fixture, because a missing file is now an error.
• Open draft fix(standard-tests): honor acceptance-test bypass_reason for basic_read #1096 (honor acceptance-test bypass_reason for basic_read) overlaps the bypass handling and can be closed in favor of this one.

Follow-ups

Left for later, from review:

• The YAML parse error includes PyYAML's source snippet of the offending line. Fix: build the message from the problem and the line and column numbers only.

Test plan

• unit_tests/test/test_empty_scenarios.py (45 tests, pytester runs of the real hooks): every bypass form and its scope; the empty-set message; section bypass ignored next to runnable entries; blank and non-string reasons; top-level bypass with and without runnable entries; legacy list sections; legacy top-level tests:; unusable layouts; missing file for Python, manifest-only and Java; collection failures print once, without a traceback or chained exception; skipped bypass params, ids and grouping; validation errors without values; a suite overriding get_scenarios() runs its own scenarios with no file, a legacy file or a top-level bypass, and fails when it returns none; the suite create_connector_test_suite(connector_directory=...) builds for airbyte-cdk connector test skips a bypass-only file and fails collection on a zero-scenario file. The no-traceback checks assert that no pytest_hooks.py: frame is printed; switching either pytrace=False to True, or dropping the override check, fails tests.

• poetry run pytest unit_tests/test: 870 passed, 9 skipped (2026-10-28, head 2e8cc3bf). Earlier, with unit_tests/resources: 834 passed, 20 skipped. The skip count depends on the registry: pokeapi's spec backward-compatibility test skips while the registry publishes a newer version than its fixture.

• poe lint and ruff format --check clean; poe type-check reports no error in the changed files (the 51 existing errors are unchanged against main); Prettier clean on the new migration section.

• CLI proofs with this branch's airbyte-cdk, on copies of monorepo connectors with GSM secrets fetched by airbyte-cdk secrets fetch:

  Connector	Shape	Ref	Result
  source-pokeapi	runnable	master	connector test exit 0, 9 passed, 3 skipped
  source-100ms	section bypasses only	master	exit 0, 3 passed, 9 skipped with the declared reasons
  source-hibob	no runnable config, no bypass	master	exit 2, "No test scenarios for connector 'source-hibob'"
  source-clazar	no file	master	exit 2, "Acceptance test config file not found"
  source-activecampaign	legacy top-level tests:	master	exit 2, migration message
  source-hibob, source-clazar	top-level bypass	PR A	exit 0, 3 passed, 9 skipped "bypasses the connector: No test config in GSM for this connector"
  destination-pinecone, source-granola	top-level bypass	PR A	collect: bypass line printed
  source-pardot	legacy list, converted	PR B	connector test exit 0 (5 passed, 16 skipped); image test exit 0
  source-google-directory	legacy list, converted	PR B	image test exit 0, 4 passed (discover kept)
  source-netsuite	legacy list, converted	PR B	image test exit 0, 3 passed

• Classifier: master has 10 + 25 connectors to fix; master plus #88931, #88932 and the follow-up has 0 and 0; without the follow-up, 0 and 10 (re-run 2026-10-27 on fd347841abe; the 10th no-bypass connector is source-github-g2, added to master on 2026-10-20 and now covered by PR A).

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/empty-scenarios-fail#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/empty-scenarios-fail

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 948 tests  +110   5 917 ✅ +110   6m 25s ⏱️ - 1m 44s
    1 suites ±  0      31 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 2e8cc3b. ± Comparison against base commit 8b721cd.

This pull request removes 2 and adds 112 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_api_budget_is_set_before_dynamic_streams_evaluated
unit_tests.test.test_scenario_keys ‑ test_fields_awaiting_a_reader_are_still_unread[bypass_reason]

unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_building_a_source_does_not_check_the_component_schema[False]
unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_building_a_source_does_not_check_the_component_schema[True]
unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_legacy_source_delegates_to_public_manifest_validator
unit_tests.source_declarative_manifest.test_source_declarative_manifest_imports ‑ test_spec_check_discover_do_not_import_pandas_or_numpy
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_builds_only_the_first_stream_count_in_order[one-stream]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_builds_only_the_first_stream_count_in_order[three-streams]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_legacy_manifest_source
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_non_positive_count_or_disabled_availability[0-True]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_non_positive_count_or_disabled_availability[1-False]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_override_for_concurrent_source
…

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 951 tests  +110   5 920 ✅ +110   11m 27s ⏱️ - 5m 14s
    1 suites ±  0      31 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 2e8cc3b. ± Comparison against base commit 8b721cd.

This pull request removes 2 and adds 112 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.test_concurrent_declarative_source ‑ test_api_budget_is_set_before_dynamic_streams_evaluated
unit_tests.test.test_scenario_keys ‑ test_fields_awaiting_a_reader_are_still_unread[bypass_reason]

unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_building_a_source_does_not_check_the_component_schema[False]
unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_building_a_source_does_not_check_the_component_schema[True]
unit_tests.legacy.sources.declarative.test_manifest_declarative_source ‑ test_legacy_source_delegates_to_public_manifest_validator
unit_tests.source_declarative_manifest.test_source_declarative_manifest_imports ‑ test_spec_check_discover_do_not_import_pandas_or_numpy
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_builds_only_the_first_stream_count_in_order[one-stream]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_builds_only_the_first_stream_count_in_order[three-streams]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_legacy_manifest_source
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_non_positive_count_or_disabled_availability[0-True]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_for_non_positive_count_or_disabled_availability[1-False]
unit_tests.sources.declarative.checks.test_check_dynamic_stream ‑ test_check_dynamic_stream_uses_streams_override_for_concurrent_source
…

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Requesting changes for the release ordering only. The code itself looks sound to me.

Done well:

• the bypass model (top-level, per section, per entry) and the "top-level bypass next to runnable configs" guard;
• failure messages that name keys and sections, never values;
• the cdk-migrations.md section;
• strong tests: 18 of 24 deliberate mutations are caught.

Rollout: this is breaking for connector test suites (not syncs) and ships as a minor, which is fine only if airbytehq/airbyte#88931 and #88932 merge first (inline comment on pytest_hooks.py). I independently verified that the files in both fleet PRs load and collect correctly under this branch and under the released CLI.

Not inline (low, for later): YAML parse errors echo the offending source line; a bypass_reason on a runnable entry is silently dropped; AcceptanceTestConfigError and ScenarioBypass are importable only from a private module; the PR body's fleet counts are stale.