Error failed to get table info from metastore When Using Kafka Connect Iceberg with Hive SSL #328
Description
I am using kafka-connect-iceberg to connect to an Iceberg catalog backed by Hive Metastore (HMS) with SSL enabled. Below is my connector configuration:
class: io.tabular.iceberg.connect.IcebergSinkConnector
config:
topics: test123
iceberg.catalog.type: hive
iceberg.catalog.uri: 'thrift://...:9083'
iceberg.catalog.warehouse: 's3a://...'
iceberg.catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
iceberg.catalog.s3.endpoint: '...'
iceberg.catalog.s3.access-key-id: admin
iceberg.catalog.s3.secret-access-key: admin
iceberg.control.commit.timeout-ms: '10000'
iceberg.tables.auto-create-enabled: 'true'
iceberg.catalog: icebergtest
iceberg.tables: "test.test123"
iceberg.tables.evolve-schema-enabled: 'true'
iceberg.control.commit.interval-ms: '50000'
iceberg.catalog.client.region: us-east-1
iceberg.tables.upsert-mode-enabled: 'false'
iceberg.catalog.s3.path-style-access: 'true'
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: 'false'
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: 'false'
iceberg.catalog.hive.metastore.use.SSL: true
iceberg.catalog.hive.metastore.truststore.type: 'PKCS12'
iceberg.catalog.hive.metastore.truststore.path: '/opt/truststore.p12'
iceberg.catalog.hive.metastore.truststore.password: 'test'
iceberg.catalog.hive.metastore.client.auth.mode: 'PLAIN'
iceberg.catalog.hive.metastore.client.plain.username: 'abc'
iceberg.catalog.hive.metastore.client.plain.password: 'abc'
When running the connector, I encounter the following error:
2025-03-11 12:55:07,177 ERROR [test|task-0] WorkerSinkTask{id=test-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-test-0]
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:857)
Caused by: java.lang.RuntimeException: Failed to get table info from metastore test.test123
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:159)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)
... 11 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500)
... 27 more
Has anyone encountered this issue before? If so, how did you resolve it? I would really appreciate any guidance or suggestions to help me fix this problem. Thank you in advance for your support!