Skip to content

Glue DataLakeCatalog uses stale schema after ALTER ADD COLUMN #2085

Description

@alsugiliazova

Problem

For an Iceberg table in a Glue-backed DataLakeCatalog, consecutive ALTER TABLE ... ADD COLUMN commands succeed and the new columns are readable, but SHOW CREATE TABLE still returns the original schema. A subsequent attempt to drop one of the added columns also fails because ClickHouse cannot find it.

The same scenario works correctly with the REST catalog.

Tested with https://altinity-build-artifacts.s3.amazonaws.com/PRs/1841/78d272e18d7641866ac4f4ded573708cf005a3f9/build_arm_release/clickhouse-common-static_26.3.13.20001.altinityantalya_arm64.deb

Reproduction

Create an unpartitioned Iceberg table containing one nullable String column named name, then run:

SET allow_insert_into_iceberg = 1;

ALTER TABLE datalake_db.`namespace.table`
ADD COLUMN column_a Nullable(String);

ALTER TABLE datalake_db.`namespace.table`
ADD COLUMN column_b Nullable(Int64);

The columns are immediately available:

SELECT * FROM datalake_db.`namespace.table`;
Alice	\N	\N

Now run:

SHOW CREATE TABLE datalake_db.`namespace.table`;

Actual behavior

SHOW CREATE TABLE returns only the original column:

CREATE TABLE datalake_db.`namespace.table`
(
    `name` Nullable(String)
)
ENGINE = Iceberg('http://minio:9000/warehouse/data/')

column_a and column_b are missing even though both ALTER commands succeeded and SELECT * exposes both columns.

Dropping one of the added columns then fails:

SET allow_insert_into_iceberg = 1;
ALTER TABLE datalake_db.`namespace.table` DROP COLUMN column_a;
Code: 10. DB::Exception: Wrong column name.
Cannot find column `column_a` to drop.
(NOT_FOUND_COLUMN_IN_BLOCK)

Expected behavior

SHOW CREATE TABLE should return the current Iceberg schema:

(
    `name` Nullable(String),
    `column_a` Nullable(String),
    `column_b` Nullable(Int64)
)

All operations that resolve the table schema should use the same current metadata version.

DROP COLUMN column_a should succeed and leave name and column_b in the current schema.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions