Add support for an ~/.aws/config.d/ directory

[quinnypig]

Is your feature request related to a problem? Please describe.

My ~/.aws/config file is massive. I have a proliferation of personal accounts, work accounts, and each client grants me a profile within dozens of accounts as well. I'd like to bring some order to that chaos, as well as not have a single file that lists every AWS account to which I currently have configured access.

Describe the solution you'd like
Being able to break out profiles on a per-client, or along a personal / work divide via different configuration files enables a number of useful workflows. I'd like a subdirectory within ~/.aws/ named config.d; files within that directory can then be included when ~/.aws/config is read by various ecosystem tools. They can be independently managed, and not result in the obnoxious and manual / error prone approach of either editing the file by hand to reflect new accounts / the removal of old ones, or the somehow-worse problem of building a variety of "aws config file generator" scripts bespoke to every customer who writes one of the blasted things.

Describe alternatives you've considered
I have tried and discarded:

• Generating the config file from scripts
• Ignoring the problem in the forlorn hope it would annoy me less with time
• manually editing the file
• hiring an assistant to log me into AWS accounts manually so I don't have to worry about it

Additional context
The conf.d/ structure is well understood within Linux, and is a well adopted pattern ecosystem-wide.

[kellertk]

Hi there! Thanks for the feature request - I can absolutely see the benefits here and I'll make sure that we discuss it with the larger team.

There is one other option, though it's about as much work as hiring an assistant to log into accounts for you, and that's sourcing credentials with an external process. The big scary security warnings on that page are there to remind you that it's very easy to leak credentials into logs and the like if you're not careful, but it's possible for someone to write a conf.d parser as a script and emit those credentials to the SDK via credentials_process. Process-sourced credentials are actually used quite frequently internally at AWS, because we develop with ephemeral accounts, short credential lifetimes, and other internal-specific restrictions that are embodied in a process that vends credentials for us. But absolutely not everyone is going to want to write a custom client just to manage their SDK credentials, so this feature request still makes sense.

One thing that we're trying to do (which we haven't always done in the past) is not have the SDKs behave differently when it comes to reading config files, handling credentials, and that sort of thing. Which means we wouldn't want to implement this in just CLI, but all of the other SDKs as well. Because of that I'm going to move this issue into the repo that we're using to track cross-SDK feature requests. Expect some more information from us once we've had a chance to review this across the SDKs and prioritized this.

[ranman]

Another option is to use something like git's [includeIf "dir:~/work/"] functionality: https://git-scm.com/docs/git-config#_conditional_includes

[kellertk]

We discussed this feature internally across the other SDK teams and we're interested so far. We'll be having one of our engineers do a test implementation to look at things like ergonomics, performance, and effort required in all of the SDKs. Also, there is some complexity we'll have to work out, because we have both credentials and config files to consider, but we're looking at it :)

As always, your thumbs up reactions let us know how urgent this is for the community. Thanks!

[quinnypig]

Recently dealt with a customer with a few thousand AWS accounts. My kingdom for this feature!

[ronaldbradford]

+1 for me. I could definitely benefit from managing individual files not sed'ingy way thru a single configt file for existence before modified or removal of an unknown number of lines

[nitrocode]

The way I've worked around this limitation is by creating different configuration files per client and setting the AWS_CONFIG_FILE and using direnv

Example

clients/client1/.envrc
clients/client1/.aws-config
clients/client2/.envrc
clients/client2/.aws-config
# ...

Then each .envrc contains

export AWS_CONFIG_FILE=$(pwd)/.aws-config

Example (note that direnv allow is only needed the first time)

$ cd ~/git/work/client1
$ direnv allow
$ echo $AWS_CONFIG_FILE
AWS_CONFIG_FILE=/home/user/git/work/client1/.aws-config