by generating a temporary IAM STS session with MFA then assuming cross-account IAM roles
I recently had the need to run some AWS commands across all AWS accounts in my AWS Organization. This was a bit more difficult to accomplish cleanly than I had assumed it might be, so I present the steps here for me to find when I search the Internet for it in the future.
You are also welcome to try out this approach, though if your account structure doesn’t match mine, it might require some tweaking.
Assumptions And Background
(Almost) all of my AWS accounts are in a single AWS Organization. This allows me to ask the Organization for the list of account ids.
I have a role named “admin” in each of my AWS accounts. It has a lot of power to do things. The default cross-account admin role name for accounts created in AWS Organizations is “OrganizationAccountAccessRole”.
I start with an IAM principal (IAM user or IAM role) that the aws-cli can access through a “source profile”. This principal has the power to assume the “admin” role in other AWS accounts. In fact, that principal has almost no other permissions.
I require MFA whenever a cross-account IAM role is assumed.
You can read about how I set up AWS accounts here, including the above configuration:
Creating AWS Accounts From The Command Line With AWS Organizations
I use and love the aws-cli and bash. You should, too, especially if you want to use the instructions in this guide.
I jump through some hoops in this article to make sure that AWS
credentials never appear in command lines, in the shell history, or in
files, and are not passed as environment variables to processes that
don’t need them (no export).
Setup
For convenience, we can define some bash functions that will improve clarity when we want to run commands in AWS accounts. These freely use bash variables to pass information between functions.
The aws-session-init function obtains temporary session credentials
using MFA (optional). These are used to generate temporary assume-role
credentials for each account without having to re-enter an MFA token
for each account. This function will accept optional MFA serial number
and source profile name. This is run once.
aws-session-init() {
# Sets: source_access_key_id source_secret_access_key source_session_token
local source_profile=${1:-${AWS_SESSION_SOURCE_PROFILE:?source profile must be specified}}
local mfa_serial=${2:-$AWS_SESSION_MFA_SERIAL}
local token_code=
local mfa_options=
if [ -n "$mfa_serial" ]; then
read -s -p "Enter MFA code for $mfa_serial: " token_code
echo
mfa_options="--serial-number $mfa_serial --token-code $token_code"
fi
read -r source_access_key_id \
source_secret_access_key \
source_session_token \
<<<$(aws sts get-session-token \
--profile $source_profile \
$mfa_options \
--output text \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]')
test -n "$source_access_key_id" && return 0 || return 1
}
The aws-session-set function obtains temporary assume-role
credentials for the specified AWS account and IAM role. This is run
once for each account before commands are run in that account.
aws-session-set() {
# Sets: aws_access_key_id aws_secret_access_key aws_session_token
local account=$1
local role=${2:-$AWS_SESSION_ROLE}
local name=${3:-aws-session-access}
read -r aws_access_key_id \
aws_secret_access_key \
aws_session_token \
<<<$(AWS_ACCESS_KEY_ID=$source_access_key_id \
AWS_SECRET_ACCESS_KEY=$source_secret_access_key \
AWS_SESSION_TOKEN=$source_session_token \
aws sts assume-role \
--role-arn arn:aws:iam::$account:role/$role \
--role-session-name "$name" \
--output text \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]')
test -n "$aws_access_key_id" && return 0 || return 1
}
The aws-session-run function runs a provided command, passing in AWS
credentials in environment variables for that process to use. Use this
function to prefix each command that needs to run in the currently set
AWS account/role.
aws-session-run() {
AWS_ACCESS_KEY_ID=$aws_access_key_id \
AWS_SECRET_ACCESS_KEY=$aws_secret_access_key \
AWS_SESSION_TOKEN=$aws_session_token \
"$@"
}
The aws-session-cleanup function should be run once at the end, to
make sure that no AWS credentials are left lying around in bash
variables.
aws-session-cleanup() {
unset source_access_key_id source_secret_access_key source_session_token
unset aws_access_key_id aws_secret_access_key aws_session_token
}
Running aws-cli Commands In Multiple AWS Accounts
After you have defined the above bash functions in your current shell, here’s an example for how to use them to run aws-cli commands across AWS accounts.
As mentioned in the assumptions, I have a role named “admin” in each account. If your role names are less consistent, you’ll need to do extra work to automate commands.
role="admin" # Yours might be called "OrganizationAccountAccessRole"
This command gets all of the account ids in the AWS Organization. You can use whatever accounts and roles you wish, as long as you are allowed to assume-role into them from the source profile.
accounts=$(aws organizations list-accounts \
--output text \
--query 'Accounts[].[JoinedTimestamp,Status,Id,Email,Name]' |
grep ACTIVE |
sort |
cut -f3) # just the ids
echo "$accounts"
Run the initialization function, specifying the aws-cli source profile
for assuming roles, and the MFA device serial number or ARN. These are
the same values as you would use for source_profile and mfa_serial
in the aws-cli config file for a profile that assumes an IAM
role. Your “source_profile” is probably “default”. If you
don’t use MFA for assuming a cross-account IAM role, then you may
leave MFA serial empty.
source_profile=default # The "source_profile" in your aws-cli config
mfa_serial=arn:aws:iam::YOUR_ACCOUNTID:mfa/YOUR_USER # Your "mfa_serial"
aws-session-init $source_profile $mfa_serial
Now, let’s iterate through the AWS accounts, running simple AWS CLI commands in each account. This example will output each AWS account id followed by the list of S3 buckets in that account.
for account in $accounts; do
# Set up temporary assume-role credentials for an account/role
# Skip to next account if there was an error.
aws-session-set $account $role || continue
# Sample command 1: Get the current account id (should match)
this_account=$(aws-session-run \
aws sts get-caller-identity \
--output text \
--query 'Account')
echo "Account: $account ($this_account)"
# Sample command 2: List the S3 buckets in the account
aws-session-run aws s3 ls
done
Wrap up by clearing out the bash variables holding temporary credentials.
aws-session-cleanup
Note: The credentials used by this approach are all temporary and use the default expiration. If any expire before you complete your tasks, you may need to adjust some of the commands and limits in your accounts.
Credits
Thanks to my role model, Jennine Townsend, the above code
uses a special bash syntax to set the AWS environment variables for
the aws-cli commands without an export, which would have made the
sensitive environment variables available to other commands we might
need to run. I guess nothing makes you as (justifiably) paranoid as
deep sysadmin experience.
Jennine also wrote code that demonstrates the same approach of STS get-session-token with MFA followed by STS assume-role for multiple roles, but I never quite understood what she was trying to explain to me until I tried to accomplish the same result. Now I see the light.
GitHub Repo
For my convenience, I’ve added the above functions into a GitHub repo,
so I can easily add them to my $HOME/.bashrc and use them in my
regular work.
Perhaps you may find it convenient as well. The README provides instructions for how I set it up, but again, your environment may need tailoring.