[Anatolii Yatsuk (tolik0)]

Part of airbytehq/airbyte-internal-issues#17135 (P1.2 "Restore read assertions", epic airbytehq/airbyte-internal-issues#17129). Follows #1218.

Fleet impact: of the 88 manifest-only connectors whose basic read passes today, 2 go red, both community connectors and both on the primary key check. The state check and the two trace tests turn no connector red. The stream status check is advisory (a warning), so it turns no connector red either. Details under "Fleet impact".

Merge order: merge and release this PR first, then merge the connector opt-outs in airbytehq/airbyte#88937 right after the release. The released CDK (7.41.0) rejects the opt-out key that #88937 adds, so merging #88937 first would turn both connectors red until this PR is released.

What

#1218 restored per-stream record presence and schema validation in test_basic_read. Four more checks of CAT's basic read test were still missing from the Standard Tests. This PR enforces three of them, with CAT's config flags as the opt-outs, and reports the fourth without enforcing it:

1. Primary keys (CAT _validate_primary_keys_data_type). For every config under basic_read, every record of a stream with a source_defined_primary_key must have a usable key: no key field holds an object or an array, and not every key field is missing or null. Opt out with validate_primary_keys_data_type: false.
2. State messages (CAT _validate_state_messages). Every stream the read covers must be named by a STREAM state message (or a GLOBAL state's stream_states); no state uses the deprecated LEGACY type; every state carries sourceStats.recordCount. Opt out with validate_state_messages: false.
3. A failing read reports a well-formed error trace (CAT test_airbyte_trace_message_on_failure). test_fail_read_with_bad_catalog, which is CAT's test, now checks that its error traces are well-formed, and a new per-connector test test_fail_read_with_invalid_config does the same for a config the spec rejects. Opt out with expect_trace_message_on_failure: false in a basic_read entry: as in CAT, it skips the bad-catalog test for that config, and it skips the new test for the connector.
4. Stream statuses, advisory (CAT _validate_stream_statuses). For every config under basic_read, every stream the read covers, and no other stream, should report STARTED, then only RUNNING, then COMPLETE. Problems are reported as a StreamStatusWarning (exported from airbyte_cdk.test.standard_tests, like DiscoveredCatalogWarning) and do not fail the test. There is no config key; -W error::airbyte_cdk.test.standard_tests.StreamStatusWarning makes it fail.

Failure messages name the stream, the record number and the key path, never a value:

Records have invalid primary keys. Destinations deduplicate on the primary key, ...
- Stream 'issues' (primary key `$.id`): 2 of 40 records have an invalid primary key.
    - record #3: key field `$.id` is an object; a primary key field must hold a scalar (a string, a number or a boolean)

The read's state messages are incomplete. To skip this check for a config, set `validate_state_messages: false` ...
- 1 of 3 configured streams emitted no `STREAM` or `GLOBAL` state message: 'events'. ...

How

• _read_assertions.py: _StreamPrimaryKeyChecker (runs in the same single pass over the records as the schema check), _find_state_message_failures, assert_error_trace_messages, make_invalid_config. assert_read_records gains validate_primary_keys and state_messages, both default off, so the Docker read test, which P1.4 (airbytehq/airbyte-internal-issues#17137) is changing in parallel, is not touched. It takes record messages plus the configured catalog, so P1.4 can pass the new arguments as is.
• ConnectorTestScenario: validate_primary_keys_data_type, validate_state_messages, expect_trace_message_on_failure, with CAT's names and defaults (true). _dedup_scenarios keeps an opt-out declared by any entry for the config, as it already did for validate_schema. These three keys and validate_schema are accepted only in basic_read entries (BASIC_READ_ONLY_FIELDS), as in CAT, so an entry of another section cannot turn the config's basic_read checks off through the merge. For the three new keys that is how main behaves; validate_schema was accepted in any section until now, and no monorepo config sets it outside basic_read.
• _read_assertions.py: find_stream_status_problems and StreamStatusWarning; test_basic_read warns with the problems before the enforced checks run, so a config that fails them still reports its stream statuses.
• test_basic_read passes the new arguments for basic_read configs. A config read only because of full_refresh keeps the "some records" floor, as in feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218.
• cdk-migrations.md: a section under "Upgrading to the next minor release (unreleased)".

Decisions

Stream statuses are advisory, with no config key. CAT ran _validate_stream_statuses by default for certified connectors only, and refused to turn it off for them. The Standard Tests have no notion of a certified connector (only test_strictness_level, which P3.3, airbytehq/airbyte-internal-issues#17146, is replacing), and the check has never been measured across the fleet. Reporting it as a warning gets that measurement from every CI run without turning anything red. Enforcing it, and for which connectors, is a later change.

Primary keys follow CAT exactly. A composite key may have null parts as long as one part is present, and booleans count as scalars (the message calls them "boolean", not "integer"). The stricter "every key part present" rule was measured only (see below), not enforced. As in CAT, validate_primary_keys_data_type: false is allowed at test_strictness_level: high; only validate_schema: false is refused there.

State: the per-stream version, not CAT's count. CAT asserted len(state_messages) >= len(streams). That is not vacuous, but one stream that checkpoints many times hides another stream that emitted no state at all. This PR checks coverage per stream (matched on namespace and name). The LEGACY check is kept; CAT's version crashed on GLOBAL states, which this one handles. The sourceStats check is kept, but it is vacuous for CDK connectors, because the CDK entrypoint sets sourceStats.recordCount on every state it emits; it guards non-CDK sources and the Docker path. The CDK emits a final {"__ab_no_cursor_state_message": true} state for every full-refresh stream it completes, so a CDK connector fails only when a stream ends abnormally or custom code drops the state.

The failure input for the trace test is a config the connector's spec rejects. CAT's trace test used the valid config plus a broken catalog, and FAST's test_fail_read_with_bad_catalog already is that test. It now uses the same well-formedness check instead of only requiring some trace message, so there is no duplicate. The new test needs a different failure, one that reaches the connector:

• The config holds only a marker key (__airbyte_standard_tests_invalid_config) and none of the spec's required top-level properties. The CDK entrypoint rejects it against the spec with a config_error trace, before any request is sent.
• It needs no secret and runs once per connector, so connectors whose basic_read configs are all secret-backed are covered too.
• Invalid real-looking credentials were rejected: they send failed logins to vendor test accounts, which some vendors answer with lockouts or CAPTCHAs.
• A missing stream was rejected: AbstractSource reports it as INCOMPLETE without an error.
• The test skips when the spec requires no top-level property (no config is certainly invalid) and when any basic_read entry sets expect_trace_message_on_failure: false: the connector does not promise an error trace for a failed read. CAT had no such test; its flag skipped the bad-catalog test, and test_fail_read_with_bad_catalog now honours it per config too.
• In-process, entrypoint_wrapper turns any uncaught exception into a well-formed trace message, so this check mostly guards the CDK and the harness. The same helper is meant for the Docker path (P1.4), where a crashing image can emit nothing.

source-s3's hand-ported CAT harness (integration_tests/test_acceptance.py), named as prior art in the issue, has none of these three checks, so they are ported from CAT's semantics directly.

Fleet impact

I ran this branch's checks in-process (as airbyte-cdk connector test does) against airbytehq/airbyte master at 1e22e3c689c. The population was every manifest-only source with a basic_read config: 185 connectors. 16 were skipped, because their test secret holds a single-use rotating refresh token, and a local run would invalidate the copy CI uses. Each basic_read scenario was read once (20-minute timeout, and 45 minutes for a second try of the 6 that timed out), and the verdict on main (presence and schema, from #1218) and the new checks were taken from the same read. test_fail_read_with_invalid_config ran once per connector, and test_fail_read_with_bad_catalog on the first read scenario.

Connectors	Count
Run	169
No verdict for the read: the read did not finish within 45 minutes locally (both trace tests passed)	4
Already red before this change	77
Green before and after	86
Green before, red after	2 (none certified)

Already red before this change: 41 have expired, revoked or under-scoped test credentials, or hit plan limits; 8 have no test secret in GSM; 3 fail to load with the current CDK or have an acceptance-test-config.yml the harness cannot parse; 25 fail on other API or connector errors, or on #1218's checks. None of the reads that reached the checks failed the new ones, so they hide nothing that is known.

Connector	Check	Stream	Problem
source-google-calendar	primary key	colors	The key fields calendar and event hold objects.
source-statuspage	primary key	subscribers_histogram_by_state	The records have no id, the declared key.

Both are real key problems, not false positives, and CAT would have failed them too. Changing a primary key is a breaking change, so airbytehq/airbyte#88937 opts the two configs out with validate_primary_keys_data_type: false instead, and the key fixes are left for separate PRs.

Per check:

• Primary keys: 81 of 97 scenarios that reached the checks read at least one stream with a primary key; 2 failed. The stricter "every key field present" rule, measured as an advisory only, flags no other stream.
• State: 96 scenarios passed and 1 is opted out (source-chargebee, which already declares validate_state_messages: false). No connector fails. source-monday also declares the opt-out; it was among the 16 connectors skipped for their rotating refresh token.
• test_fail_read_with_invalid_config: 161 passed and 6 skipped because the spec requires no top-level property. 2 failed, both on connectors whose every test already fails (one cannot load its custom components with the current CDK, one has an acceptance-test-config.yml the harness cannot parse).
• test_fail_read_with_bad_catalog: 159 passed. The 9 failures are connectors that are red before: 8 have no test secret, and 1 cannot load its custom components.

The CDK's Python test connectors, run in-process with this branch: source-hardcoded-records passes (the new trace test skips, as its spec requires nothing). source-shopify's test_fail_read_with_invalid_config passes; its two test_basic_read scenarios fail before any new check, because the test store's Europe/Kyiv time zone is missing from the pendulum 2 time zone data, which this PR does not touch. source-google-drive cannot be created by the harness (its constructor needs catalog, config and state), and the new trace test skips on its existing expect_trace_message_on_failure: false.

Python sources. Since airbytehq/airbyte#88945, Python sources run this suite in-process with the CDK they pin. I ran the 26 that pin a CDK with the in-process suite (6.46.0 or later) against airbytehq/airbyte master at eaf94ec6fea, as a CDK bump would: each connector's own lock on Python 3.11, with this branch's wheel (at cb7d7d6) installed over the locked CDK. None uses a rotating refresh token. 17 ship their own integration_tests/standard_tests.py.

• Read checks: of 63 basic_read scenarios, 47 reached the checks, including all 14 of source-azure-blob-storage, all 21 of source-s3, all 3 of source-gcs and both of source-google-drive. All 47 pass the primary key and state checks, and none reports a StreamStatusWarning. 13 failed before any check, the same way on main: expired or rejected credentials (appsflyer, asana, freshcaller, jina-ai-reader, microsoft-sharepoint), API errors (cart), no config.json in the test secret (box-data-extract), and two failures that only happen locally (faker's processes under macOS spawn, and google-ads, whose gRPC calls cannot pass the local TLS-inspecting proxy). source-google-drive was rerun at f547853 with the proxy's CA bundle set for httplib2; its first run had failed to list files for that reason. The other 3 belong to source-facebook-marketing and source-shopify, whose own suites override test_basic_read: they get the new read checks only once that override is removed.
• test_fail_read_with_invalid_config: 16 passed and 7 skipped. 3 failed at collection on the legacy top-level tests: layout, as every test of those connectors already does (amazon-sqs, commercetools, firebase-realtime-database).
• test_fail_read_with_bad_catalog: 20 passed and 1 skipped (source-google-drive, on its expect_trace_message_on_failure: false, which this test honours since f547853); 1 is red before (box-data-extract, no config.json).
• The four sources that pin a 6.x CDK (amazon-sqs, commercetools, faker, microsoft-sharepoint) installed and ran this 7.x wheel without import errors.

Rollout

The connector monorepo installs the airbyte-cdk CLI unpinned, so manifest-only connectors, which run the in-process suite through airbyte-cdk connector test, pick the new checks up with the release, without a per-connector CDK bump. Python sources run the same suite in-process with the CDK they pin (airbytehq/airbyte#88945), so they get the checks when they bump to this release: 22 of the 58 live Python sources allow it, including all 9 certified ones. 17 Python sources ship their own integration_tests/standard_tests.py; source-facebook-marketing and source-shopify override test_basic_read, so the new read checks reach them only once the override goes. The Python-source run under "Fleet impact" covers the 26 that run the suite today. This ships as a minor (feat, no !), as standard-test changes do.

Merge order:

1. This PR, then the CDK release.
2. test(connectors): opt google-calendar and statuspage out of the primary key check airbyte#88937, right after the release: primary key opt-outs for source-google-calendar and source-statuspage (test-only, no connector release). Re-run its connector tests against the released CLI first.

#88937 cannot go first. It adds validate_primary_keys_data_type: false to two basic_read entries, and releases 7.39.0 through 7.41.0 reject that key as one the standard tests do not read. With the unpinned CLI, both connectors would then fail scenario collection, and so every test, until this PR is released. Its green checks predate 7.39.0: they ran on 2026-10-14, and loading its two configs with 7.41.0 fails with that error today.

In this order, between the release and the merge of #88937, the two connectors fail only the new primary key check, and only on a PR that touches them.

Landing order for airbytehq/airbyte-internal-issues#17135 (updated)

1. feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218 (merged, 7.36.0): per-stream record presence, record-vs-schema validation, failure messages that name the stream and the record path.
2. This PR: primary-key data types, per-stream state messages, and a well-formed error trace on a failing read, enforced; stream status completeness, reported as a warning.
3. Later, after a measurement pass: enforce stream status completeness (decide whether for every connector or only certified ones, as CAT did); strict extra columns (fail_on_extra_columns) and data points (validate_data_points) behind flags, since CAT never applied fail_on_extra_columns and defaulted validate_data_points to false.

The Docker read test (test_docker_image_build_and_read) is not changed here; P1.4 (airbytehq/airbyte-internal-issues#17137) owns it.

Test plan

• poetry run pytest unit_tests/test/ -p no:cacheprovider: 978 passed, 9 skipped (at f547853). With -W error::airbyte_cdk.test.standard_tests.StreamStatusWarning, only test_basic_read_fails_without_a_declared_status[no_records] fails, because its fake source emits nothing.
• source_pokeapi_w_components_py test_basic_read with -W error::airbyte_cdk.test.standard_tests.StreamStatusWarning: passes, so a real declarative source reports complete stream statuses.
• New tests in unit_tests/test/test_read_keys_state_and_traces.py cover each check, including the stream status warning, directly and through SourceTestSuiteBase (fake sources that emit a null key, no state, a malformed trace, or a read that succeeds on the invalid config), the opt-outs (including expect_trace_message_on_failure skipping test_fail_read_with_bad_catalog), high strictness, and opt-outs surviving _dedup_scenarios in both orders. unit_tests/test/test_scenario_keys.py checks that each basic_read opt-out is rejected in every other section, from the file and through get_scenarios(), and still accepted when a scenario is merged or copied. Each of these changes now fails a test: reverting the bad-catalog check to assert result.trace_messages, dropping the record path guard, counting invalid records per problem, dropping the recordCount is None check, the boolean guard on emitted_at, the STREAM type check or the STREAM / GLOBAL without object case, the invalid-config assert result.errors, or validate_schema from BASIC_READ_ONLY_FIELDS. A test also pins that stream statuses are reported when an enforced check fails.
• ruff format, ruff check, mypy airbyte_cdk/test: clean.
• source_pokeapi_w_components_py standard tests (the CDK fixture): 11 passed, 11 skipped; test_fail_read_with_invalid_config, test_fail_read_with_bad_catalog and test_basic_read pass. The extra skip is the spec backward-compatibility test: the fixture's version is older than the published one.
• CDK Test Connectors matrix, in-process with this branch: source-hardcoded-records green; source-shopify, source-google-drive and source-intercom as described under "Fleet impact" (source-intercom is manifest-only and is green in the fleet run).
• Fleet runs described above (manifest-only and Python sources).

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/read-assertions-slice-2#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/read-assertions-slice-2

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

6 184 tests  +230   6 153 ✅ +230   6m 23s ⏱️ +53s
    1 suites ±  0      31 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit f547853. ± Comparison against base commit d5dd299.

This pull request removes 2 and adds 232 tests. Note that renamed tests count towards both.

unit_tests.test.test_scenario_keys ‑ test_legacy_key_is_accepted_and_dropped_in_its_section[basic_read-expect_trace_message_on_failure]
unit_tests.test.test_scenario_keys ‑ test_legacy_key_is_accepted_and_dropped_in_its_section[basic_read-validate_state_messages]

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_fail_read_with_invalid_config
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_bootstrap_and_replay_observability_are_deduplicated_and_aged
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_commit_observability_counts_completed_pages_per_partition
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_empty_partitioned_state_still_bootstraps
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_empty_state_still_bootstraps[empty-state]
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_empty_state_still_bootstraps[no-state]
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_empty_state_still_bootstraps[nothing-committed-yet]
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_malformed_partitioned_sync_token_state_fails[both-shapes]
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_malformed_partitioned_sync_token_state_fails[empty-token-entry]
unit_tests.sources.declarative.incremental.test_sync_token_cursor ‑ test_malformed_partitioned_sync_token_state_fails[integer-token-entry]
…

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

6 187 tests   6 156 ✅  12m 5s ⏱️
    1 suites     31 💤
    1 files        0 ❌

Results for commit f547853.

♻️ This comment has been updated with latest results.

[Baz (bazarnov)]

Thanks for the thorough work. The checks match CAT where the body says they do, and test_fail_read_with_invalid_config fails no live manifest-only source on its own check: of 507, 479 pass, 16 skip on a spec without top-level required, and 12 already fail (11 at collection on the legacy tests: layout, source-braze on importing its components). Three comments should land before merge: the test for the bad-catalog check, the Python-source rollout, and the 7.41.0 range. The rest are optional.

• airbytehq/airbyte#88937: verified locally with this branch and the connectors' test secrets. Without the opt-outs only the primary key check fails, with them both connectors pass, and on 7.41.0 both fail collection, so the merge order in the body holds.
• #1231 conflicts with this PR in cdk-migrations.md and calls assert_read_records without the new arguments; after its rebase it should pass them.
• Behind failures that already happen (manifest and code read only, unverified): source-coingecko-coins market_chart (the primary key prices is an array in its schema), and source-firebase-realtime-database (a Python source that emits no state; its legacy tests: layout already fails collection).

[Baz (bazarnov)]

Thanks for the quick turnaround. All eight points are addressed and pinned: I re-ran the mutations from my review on f547853, and each now fails a test (11 of 11, including the new STREAM/GLOBAL-without-object branch, the validate_schema rule and the invalid-config message). The new validate_schema rule turns nothing red: all 8 monorepo uses sit in basic_read, and no airbyte-enterprise config sets it. CI is green except Check: destination-motherduck, which fails on other PRs too (dead MotherDuck token).

Merge order as in the body: this PR >> 7.42.0 >> airbytehq/airbyte#88937. Its description still says to merge it first; the order here is the right one.