TargetNotConnected exception trying to run ECS exec command.

[ccovey]

Describe the bug

I am trying to run an exec command on a task in an ECS cluster and I continue to get TargetNotConnectedException. I have run the exec checker and it looks like everything is set up correctly. I updated my ssm (I hope?) with the host management in systems manager. I'm not sure if this is a bug or if there is some bit of configuration I am missing. I am using the latest ecs optimized images. I'm haven't ssh'd to the ec2 instances directly (they have no internet access) but assume the ecs images should have everything. I also refreshed the images and added dnf -y install https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm in the user data just in case its not in the ecs image but still get the same error.

I have searched and found others with the issue but its usually something like having aws keys in env variables which I do not have. I'm pasting below my output from the exec checker in case I'm not seeing something.

AWS_REGION=us-east-2 bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) cdai-ecs-staging-cluster arn:aws:ecs:us-east-2:***:task/cdai-ecs-staging-cluster/*** --region=us-east-2
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.17.18 Python/3.9.20 Linux/6.1.129-138.220.amzn2023.x86_64 source/x86_64.amzn.2023)
  Session Manager Plugin | OK (1.2.707.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-east-2
Cluster: cdai-ecs-staging-cluster
Task   : arn:aws:ecs:us-east-2:***:task/cdai-ecs-staging-cluster/***
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::***:role/bastion
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | EC2
  ECS Agent Version      | 1.91.1
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "portal_nextjs"
    ----------
      Init Process Enabled (cdai-staging-task:102)
    ----------
         1. Enabled - "portal_nextjs"
    ----------
      Read-Only Root Filesystem (cdai-staging-task:102)
    ----------
         1. Disabled - "portal_nextjs"
  Task Role Permissions  | arn:aws:iam::***:role/cdai-staging-task-role
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          |
    Found existing endpoints for vpc-**:
      - com.amazonaws.us-east-2.s3
      - com.amazonaws.vpce.us-east-2.vpce-svc-**
      - com.amazonaws.us-east-2.secretsmanager
      - com.amazonaws.us-east-2.ssmmessages
  Environment Variables  | (cdai-staging-task:102)
       1. container "portal_nextjs"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

[ccovey@ip-172-31-6-61 ~]$ aws ecs execute-command     --cluster cdai-ecs-staging-cluster     --task arn:aws:ecs:us-east-2:**:task/cdai-ecs-staging-cluster/**--container portal_nextjs      --command "/bin/sh"     --interactive

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

I'm not sure if this is a bug or a config issue but I feel I have followed the steps to configure it properly and the checker reports the same. If you need more info let me know.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Able to execute commands on containers running on ecs

Current Behavior

TargetNotConnectedException

Reproduction Steps

Run something like the following command.

aws ecs execute-command --cluster cdai-ecs-staging-cluster --task arn:aws:ecs:us-east-2:***:task/cdai-ecs-staging-cluster/****--container portal_nextjs --command "/bin/sh" --interactive

This should execute properly but instead I receive the above error.

Possible Solution

No response

Additional Information/Context

No response

CLI version used

aws-cli/2.17.18 Python/3.9.20 Linux/6.1.129-138.220.amzn2023.x86_64 source/x86_64.amzn.2023

Environment details (OS name and version, etc.)

Amazon Linux release 2023.6.20250303 (Amazon Linux)

[adev-code]

Hi @ccovey, I tried transferring this issue to https://github.com/aws/amazon-ecs-agent but looks like I don't have permissions. Please feel free to open this issue to the ECS Repo as they have their own repo and model for this issue. For helpful guide also that may be relevant to the issue and ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html

Thank you.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.