Self hosted iceberg kafka connector throwing the warning

[panthamb]

We are in process of deploying the self hosted iceberg kafka connector into AKS and using confluent cloud kafka to connect. We are getting following logs from AKS saying that confluent kafka disconnected.

[2025-04-14 21:15:08,162] INFO [Consumer clientId=connector-consumer-iceberg-sink-connector-0, groupId=connect-iceberg-sink-connector] Cancelled in-flight API_VERSIONS request with correlation id 5 due to node -1 being disconnected (elapsed time since creation: 5057ms, elapsed time since send: 5057ms, throttle time: 0ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient)

[2025-04-14 21:15:08,162] WARN [Consumer clientId=connector-consumer-iceberg-sink-connector-0, groupId=connect-iceberg-sink-connector] Bootstrap broker (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

We are using following configuration:
spec:
containers:
- name: kafka-connect
image:
imagePullPolicy: IfNotPresent
env:
- name: CONNECT_BOOTSTRAP_SERVERS
value: ""
password="";
- name: CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR
value: "1"
- name: CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR
value: "1"
- name: CONNECT_STATUS_STORAGE_REPLICATION_FACTOR
value: "1"
- name: CONNECT_GROUP_ID
value: "kafka-connect-cluster"
- name: CONNECT_CONFIG_STORAGE_TOPIC
value: "connect-configs"
- name: CONNECT_OFFSET_STORAGE_TOPIC
value: "connect-offsets"
- name: CONNECT_STATUS_STORAGE_TOPIC
value: "connect-status"
- name: CONNECT_PLUGIN_PATH
value: "/usr/share/java,/usr/share/confluent-hub-components"
- name: CONNECT_REST_ADVERTISED_HOST_NAME
value: "kafka-connect-service.kafka-connect.svc.cluster.local"
- name: CONNECT_REST_PORT
value: "8083"
- name: CONNECT_KEY_CONVERTER
value: "org.apache.kafka.connect.storage.StringConverter"
- name: CONNECT_VALUE_CONVERTER
value: "org.apache.kafka.connect.json.JsonConverter"
- name: CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE
value: "false"
# Azure credentials
- name: AZURE_TENANT_ID
valueFrom:
secretKeyRef:
name: azure-creds
key: tenant-id
- name: AZURE_CLIENT_ID
valueFrom:
secretKeyRef:
name: azure-creds
key: client-id
- name: AZURE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: azure-creds
key: client-secret

Anybody has as an idea why it is disconnecting with confluent kafka bootstrap URL?

[rahulgulati89]

The "broker disconnected" errors indicate that your Kafka connector is not providing the correct authentication credentials required to communicate with the Kafka broker. Depending on the authentication mechanism used at the broker level—such as Kerberos or OAuth—you must configure the appropriate client-side settings (e.g., JAAS configuration) in the Kafka connector to establish a successful connection.