Internal error when using topicctl with MSK IAM auth

Error:

time="2022-12-02 17:27:35" level=error msg="[58] SASL Authentication Failed: SASL Authentication failed: [redacted]: Internal error"

cluster.yaml

meta:
  name: redacted
  environment: redacted
  region: redacted
  description: |
    redacted
spec:
  bootstrapAddrs:
  - "redacted:9098"

  tls:
    enabled: true
  sasl:
    enabled: true
    mechanism: AWS-MSK-IAM

From Cloudwatch:

Failed authentication with INTERNAL_IP/INTERNAL_IP ([redacted]: Internal error) (org.apache.kafka.common.network.Selector)

Not sure what that would be, but I got https://github.com/provectus/kafka-ui working with the same cluster.

[oliveiragabriel07]

Same issue here. I'm using kafka 2.8.1 on AWS MSK

[raphael-caylent]

Same issue here. I tried it on Kafka 2.8.1, 2.7.2, and 2.6.3. Got the error on all of them. Also tried other versions of topicctl such as 1.8, 1.4, and 1.3 but no luck there.
Have someone ever got this working? And if so, which Kafka and topicctl versions?

[ghotiphud]

I found that setting the AWS_REGION=us-east-2 environment variable fixed the issue for me.

[raphael-caylent]

That's crazy.... I set the environment variable and it indeed worked. Thanks for the help @ghotiphud

Btw @ghotiphud did something led you to try this solution?

[ghotiphud]

After hunting through the other issues a bit, I ended up on this page: https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

Started trying the Environment Variables listed there to see if maybe there was a config that didn't get passed to the underlying library. Apparently that was AWS_REGION

[ghotiphud]

Seems like the fix might be passing the region from the cluster config yaml into the AWS session setup like:

// Create a Session with a custom region
sess, err := session.NewSession(&aws.Config{
	Region: aws.String("us-west-2"),
})

[hongbo-miao]

Thanks @ghotiphud ! ☺️
Just a friendly reminder for people in different regions: make sure to set the correct region. In my case, it is

export AWS_REGION=us-west-2