AWS Command Line Interface (CLI) with separate credentials & config files
Many of our customers’ cloud engineers and architects–but also consultants at Nordcloud supporting them–routinely work with fairly complex cloud environments made of dozens of Amazon Web services (AWS) accounts hosting many solutions.
Some of our customers–and ourselves–joined the 100+ accounts club long ago and, with more workloads migrated to AWS or new solutions built on the platform, challenges posed by the ever-increasing number of projects and potentially growing number of accounts are already a reality and are not going away anytime soon.
One common challenge for cloud engineers is configuration and credentials management for programmatic access to these workloads and accounts with the AWS Command Line Interface (CLI). Without a structured, comprehensible and scalable approach, programmatic access can potentially have dire consequences.
This blog entry is an attempt (my attempt) at making this process easier and safer. It might not be the best and is definitely not the only solution but it is based on my own experience and is what I use for my own sanity.
Jump to Multiple credentials & config files directly to read about how I work with multiple credentials and config files.
Single credentials & config files
The AWS CLI uses two files to separate the sensitive credential information (
~/.aws/credentials) from the less sensitive configuration options (
~/.aws/config). The AWS CLI also supports named profiles which can help you deal with multiple access keys but also to assume IAM roles using these credentials.
A simple configuration with a
default profile and an additional named profile might look like this.
$ cat ~/.aws/credentials [default] aws_access_key_id = AOPAJ7NKIGAC4XXHOQIQ aws_secret_access_key = maYUE41voIojht7Bjryq21KngMex+blIwZ
$ cat ~/.aws/config [default] output = json region = eu-north-1 [profile fourteenislands.io] region = us-east-1 role_arn = arn:aws:iam::964957387984:role/fourteenislands.io source_profile = default
To protect myself from unintentional actions (we have all at least once experienced immediate regret after hitting
enter in the terminal) access keys associated with my
default profile do not allow any IAM action but
sts:AssumeRole to specific IAM roles.
In order to list the S3 buckets in account
964957387984 I would need to run:
$ aws s3 ls --profile fourteenislands.io
Sometimes one cannot even be bothered with configuring credentials and config files with profiles–especially within CI/CD pipelines–but just wants to run a command.
If you are one of those girls or guys, Nordcloud released assume-role-arn, an all-in-one library that does just that.
To list the S3 buckets in account
assume-role-arn installed I just need to:
- Have my access keys available as environment variables
$ export AWS_ACCESS_KEY_ID=AOPAJ7NKIGAC4XXHOQIQ $ export AWS_SECRET_ACCESS_KEY=maYUE41voIojht7Bjryq21KngMex+blIwZ
$ eval $(assume-role-arn -r arn:aws:iam::964957387984:role/fourteenislands.io) $ aws s3 ls
All this is definitely a good start and offers some flexibility but it is often not enough: what if I want to separate credential information and configuration options of customer A from those of customer B and keep the two files short and clean?
Multiple credentials & config files
I have always wanted to have separate credentials and config files for the different customers I work with but how would the files look like? Where would they be located on the filesystem? How would I use them when I need them and only when I need them?
First I came up with the following structure to organize AWS credentials under
~/Worskpace/aws on my filesystem (but whatever works best for you will do). Be aware that I have no files under
aws └── credentials ├── aws-cli-lroguet │ ├── config │ └── credentials └── aws-cli-nordcloud ├── config └── credentials
Load & unload credentials dynamically
With that structure in place I needed a mechanism to automatically load and unload credentials (the process must be safe and unloading credentials when they are not needed is a must) on demand. To do so, I leverage on two things:
AWS_SHARED_CREDENTIALS_FILEenvironment variables supported by the AWS CLI
- direnv, an environment switcher for the shell
$ brew install direnv
direnv to work properly it needs to be hooked into the shell. Each shell has its own extension mechanism and here is how I set it up on macOS:
$ echo 'eval "$(direnv hook bash)"' >> ~/.bash_profile $ source ~/.bash_profile
Load & unload credentials
The .envrc file
From that point on switching between credentials is as easy as navigating directories and all you need is a
.envrc file to tell
direnv which files (credentials and config) to load or unload depending on the current directory.
scripts ├── .envrc ├── .gitignore ├── build-and-push.sh ├── functions.sh ├── hugo.properties └── run-local.sh
$ cat .envrc export AWS_CONFIG_FILE=$HOME/Workspace/aws/credentials/aws-cli-lroguet/config export AWS_SHARED_CREDENTIALS_FILE=$HOME/Workspace/aws/credentials/aws-cli-lroguet/credentials
Loading & unloading
Loading and unloading the credentials (setting and unsetting the corresponding environment variables) happens during navigation.
$ cd scripts/ direnv: loading .envrc direnv: export +AWS_CONFIG_FILE +AWS_SHARED_CREDENTIALS_FILE $ aws s3 ls --profile fourteenislands.io 2019-03-20 19:25:26 cf-templates-db7lklce4srh-eu-north-1 2018-03-25 21:06:15 cf-templates-db7lklce4srh-eu-west-1 2018-03-27 10:16:13 cf-templates-db7lklce4srh-us-east-1 ... $ cd .. direnv: unloading $ aws s3 ls --profile fourteenislands.io The config profile (fourteenislands.io) could not be found
That’s it. Working with multiple credentials and config files for the AWS Command Line Interface (CLI) and switching forth and back between different customers and environments has never been easier and safer. For me at least. Please leave a comment below if you have any other suggestions or recommendations.