Roles with paths do not work when the path is included in their ARN in the aws-auth configmap

[jceresini]

I have a role with an ARN that looks like this: arn:aws:iam::XXXXXXXXXXXX:role/gitlab-ci/gitlab-runner. My aws-auth configmap was as follow:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/EKSWorkerNode
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/EKSServiceWorker
      username: kubernetes-admin
      groups:
        - system:masters
    - rolearn: arn:aws:iam::XXXXXXXXXXXX:role/gitlab-ci/gitlab-runner
      username: gitlab-admin
      groups:
        - system:masters

I repeated got unauthorized errors from the cluster until I updated the rolearn to arn:aws:iam::XXXXXXXXXXXX:role/gitlab-runner. After that change my access worked as expected.

If it makes a difference, I'm using assume-role on our gitlab-runner, and using aws eks update-kubeconfig --region=us-east-1 --name=my-cluster to get kubectl configured.

[beetahnator]

Running into the same issue here on EKS 1.14.6.

[casey-robertson]

Ahh....this explains our issue when testing with AWS SSO-created roles too. See the issue referenced in this document. This has been a problem for a quite a while (at least 14 months).

https://aws.amazon.com/blogs/opensource/integrating-ldap-ad-users-kubernetes-rbac-aws-iam-authenticator-project/

Pertinent passage:
For the rolearn be sure to remove the /aws-reserved/sso.amazonaws.com/ from the rolearn url, otherwise the arn will not be able to authorize as a valid user.

When we stumbled across this I assumed it was something about the SSO role but based on this issue it's probably the path.

[rlangfordBV]

We don't use EKS, but have had this issue with 1.12 and 1.14.6 with aws-iam-authenticator. If you edit the configmap to remove the /gitlab-ci portion, and restart the pods, you will likely find that access works.

My co-worker and I suspect that is because of the way that sts returns output for assumed role session arns.

We have a role arn:aws:iam::000000000000:/role/bosun/bosun_deploy that we use for cluster administration of our kops created clusters.

If you assume the role, and run aws sts get-caller-identity, we get the following:

{
    "UserId": "<redacted-AKID>:<redacted-userid>",
    "Account": "000000000000",
    "Arn": "arn:aws:sts::000000000000:assumed-role/bosun_deploy/<redacted-userid>"
}

I wish this was fixed, as of now, I'm not sure what to do other than creating a role with a shortened path and switch to it.

I suppose one can also just edit the role that gets input to the configmap itself.

[jceresini]

Yeah, removing the path is how I identified it as the cause of the issue.

The field name is rolearn and the path is part of the ARN for a given role.

I opened this so others running into the issue might find it, and also because I think something needs to address it, whether its documentation (though I don't think docs are sufficient without changing the name of the field in the configmap) or a bugfix

[jangrewe]

We just discovered the same, by using

$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
$ TOKEN=$(aws-iam-authenticator token -i fooCluster --token-only)
$ aws-iam-authenticator verify -i fooCluster -t ${TOKEN}

and comparing the roles that the Pod uses (containing a path) vs. the one that are set in the token (path missing).

For now our workaround is also adding a role mapping to an IAM Role that "doesn't actually exist".

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jceresini]

/remove-lifecycle stale

[arhea]

I was able to reproduce this issue. I created two roles: K8s-Admin and K8s-Admin-WithPath, I created the roles using the following commands:

  aws iam create-role \
  --role-name K8s-Admin \
  --description "Kubernetes administrator role (for AWS IAM Authenticator for Kubernetes)." \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::<account id>:root"},"Action":"sts:AssumeRole","Condition":{}}]}' \
  --output text \
  --query 'Role.Arn'

  aws iam create-role \
  --role-name K8s-Admin-WithPath \
  --path "/kubernetes/" \
  --description "Kubernetes administrator role (for AWS IAM Authenticator for Kubernetes)." \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::<account id>:root"},"Action":"sts:AssumeRole","Condition":{}}]}' \
  --output text \
  --query 'Role.Arn'

Mapped them to the cluster with:

eksctl create iamidentitymapping --cluster basic-demo --arn arn:aws:iam::<account id>:role/K8s-Admin--group system:masters --username iam-admin

eksctl create iamidentitymapping --cluster basic-demo --arn arn:aws:iam::<accound id>:role/kubernetes/K8s-Admin-WithPath --group system:masters --username iam-admin-withpath

Then attached the AWS ReadOnly policy to both roles. Next, I created two AWS CLI profiles sandbox-k8s-admin and sandbox-k8s-admin-withpath specifying the rolearn options to trigger an assume role. After creating the roles, I updated my local kubeconfig:

eksctl utils write-kubeconfig --cluster=basic-demo --profile=sandbox-k8s-admin --set-kubeconfig-context --region=us-east-2

kubectl get nodes
# returned list of nodes, expected

Then switched over to the role with the path

eksctl utils write-kubeconfig --cluster=basic-demo --profile=sandbox-k8s-admin-withpath --set-kubeconfig-context --region=us-east-2

kubectl get nodes
# error: You must be logged in to the server (Unauthorized)