-
Notifications
You must be signed in to change notification settings - Fork 63
Open
Description
Hey,
When using the connector, I noticed that all rows with the 'Upsert' operation are getting ignored, and their destinations remain the same.
This is my connector config -
{
"connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector",
"transforms.TimestampConverter.target.type": "string",
"iceberg.tables.cdc-field": "_cdc.op",
"iceberg.tables.evolve-schema-enabled": "true",
"auto.register.schemas": "false",
"transforms.TimestampConverter.field": "kafka_date",
"iceberg.tables.upsert-mode-enabled": "true",
"tasks.max": "2",
"transforms": "dbztransform, kafka_date, TimestampConverter",
"iceberg.tables.auto-create-enabled": "true",
"transforms.kafka_date.type": "org.apache.kafka.connect.transforms.InsertField$Value",
"iceberg.tables": "main.bronze_streams_cdc",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"iceberg.catalog.token": ==================",
"iceberg.tables.default-partition-by": "kafka_date",
"topics": "bronze_streams_cdc",
"transforms.TimestampConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"iceberg.control.commit.interval-ms": "60000",
"iceberg.catalog.uri": "===================",
"transforms.TimestampConverter.format": "yyyy-MM-dd",
"value.converter.schema.registry.url": "===================",
"transforms.kafka_date.timestamp.field": "kafka_date",
"consumer.override.auto.offset.reset": "earliest",
"name": "data-streams-iceberg-test2",
"iceberg.catalog.warehouse": "poc_unified_data_lake",
"iceberg.catalog.type": "rest",
"transforms.dbztransform.type" : "io.tabular.iceberg.connect.transforms.DebeziumTransform",
"transforms.dbztransform.cdc.target.pattern":"poc_unified_data_lake.main.bronze_streams_cdc",
"iceberg.tables.default-id-columns":"id"
}
did someone else get into trouble when trying to enable the upsert mode?
henbh and asanisimov
Metadata
Metadata
Assignees
Labels
No labels