Creating Public AMIs Securely for EC2

| 2 Comments

Amazon published a tutorial about best practices in creating public AMIs for use on EC2 last week:

How To Share and Use Public AMIs in A Secure Manner

Though the general principles put forth in the tutorial are good, some of the specifics are flawed in how to accomplish those principles. (Comments here relate to the article update from June 7, 2011 3:45 AM GMT.)

The primary message of the article is that you should not publish private information on a public AMI. Excellent advice!

Unfortunately, the article seems to recommend or at least to assume that you are building the public AMI by taking a snapshot of a running instance. Though this method seems an easy way to build an AMI and is fine for private AMIs, it is is a dangerous approach for public AMIs because of how difficult it is to identify private information and to clear that private information from a running system in such a way that it does not leak into the public AMI.

The article recommends the use of the rm command to remove files containing confidential files. This is inadequate if you are building an EBS boot AMI for public use by taking a snapshot of the volume, as the deleted files are likely to remain in the EBS block device and are copied into the snapshot and public AMI.

Deleted files

In September, 2009, I published an article that describes this danger with an example of an EBS snapshot containing deleted files. (An EBS boot AMI is simply an EBS snapshot that has been registered).

Hidden Dangers in Creating Public EBS Snapshots on EC2

Almost immediately after I published that article with a sample public EBS snapshot, a reader found and restored the deleted file and redeemed the $100 Amazon gift certificate I had hidden in it. Now, imagine that the deleted file contained your AWS credentials with power over your entire infrastructure on EC2, power to charge tens of thousands of dollars to your credit card with a single API call, and that person finding them does not have the best of intentions. Ouch.

If you are generating a public AMI by creating an EBS snapshot of a running instance, then the use of rm is not sufficient to clear and protect secret information. As much as I love wipe, shred, and srm, even attempting to overwrite blocks may not be sufficient with modern journaling file systems.

History files

Amazon recommends “Always delete the shell history before creating your AMI” with an example of an rm of dot history files. Unfortunately, depending on exactly when this is run and what you do between that command and the creation of the AMI, a history file might be silently re-created and included on your AMI including commands that you typed before removing the history file. This is true even disregarding the fact that a file removed with rm isn’t really removed if you are creating an EBS boot AMI with a snapshot.

For example, simply exiting a shell will cause bash to re-create .bash_history with all the commands entered into that shell, perhaps including the ones where you had passed AWS credentials to commands.

You can test this with a sequence like:

  1. ssh to your server
  2. echo "my secret"
  3. rm .bash_history
  4. exit
  5. ssh to your server again
  6. cat .bash_history

It looks like you removed your secret, but there it is!

There are ways of configuring bash to not create history files, but bash is not the only program that creates history files, and you may have multiple users to worry about, and some information gets stored in system log files, and… It just gets messy. With security, you don’t want to say “I probably got everything”. You want to know that you are not at risk.

authorized_keys

Amazon recommends removing all authorized_keys files from your disk before creating the AMI. This is a great recommendation because you don’t want to release public AMIs with back doors letting you access your user’s servers. It’s bad form and makes you look suspicious and untrustworthy once discovered.

Unfortunately, this makes it difficult to reconnect to your running instance, so you need to make sure you keep some ssh connections alive. You would need to restore the authorized_keys to copy files or create new ssh connections to the instance, and then hope you remember to remove them again before taking the next snapshot.

This becomes error prone if you are in the test/development loop trying to perfect an AMI.

The article recommends you remove any unrecognized authorized_keys files and user login accounts when you run a public AMI. I would go further. If you run a public AMI that includes back doors like this, you should question what other security problems might exist with that AMI and get off of it as soon as possible. In fact, Amazon sends out alerts and turns off access when they find AMIs with pre-existing authorized_keys so it’s a bit odd this article seems to take such a security hole casually.

So now you’re sufficiently worried and are wondering how to create public AMIs without including your secret information or accidentally releasing AMIs with embarrassing back doors.

Recommendation 1

The cleanest and safest approach is to build the file system for the new image separate from the file system you are using with the running system. This means that you are creating a chroot environment with a complete operating system in a directory structure that is not the same root directory where you are logging in and running commands.

The great thing about this approach is that you know exactly what files you are putting on the new image; you only run the exact commands you want to in the chroot environment; no history or log files show up there that you don’t control; your authorized_keys files are never seen in that file system; and any files you delete are not leaked when you copy that file system onto a new EBS volume to snapshot and register the public AMI.

This is a reasonably advanced Linux concept and sometimes requires some special tools and knowledge, but the tools and knowledge are publicly available and folks in the community are standing by to help guide you when you run into problems.

I think this approach is made easiest with Ubuntu, as Canonical has provided downloadable file system images that are configured correctly for the EC2 environment and that can be easily customized to add software and configuration for making AMIs to your own specification.

I published a tutorial on how to build AMIs with Canonical’s downloadable images back in 2009 with Ubuntu Karmic (now past end of life). It is similar with modern versions of Ubuntu, and if there is interest I can publish a new article with the updated steps.

The code I use to build the Alestic Git Server AMIs uses this approach with Ubuntu 10.04 Natty and is available as a resource to study:

alestic-git-build-ami

Recommendation 2

If you aren’t building an Ubuntu AMI and your Linux distro does not provide clean, downloadable file systems for EC2 images, and you really believe you can identify which files on your running system contain private information, and you really want to create a public AMI from a running system, then here is how you can avoid releasing an AMI with recoverable deleted files.

  1. Start with a fresh instance of a clean public AMI to install and configure your software. If your instance has been running for a while, you really don’t know what has leaked into the file system in visible and deleted files

  2. Put as little private information on the running system as possible. If you need to have AWS credentials on the running system, drop them in a mounted ephemeral store disk (/mnt on some distros). Don’t type private information like keys or passwords into command lines where they might get dropped into history files.

  3. Delete all sensitive files and all authorized_keys. (This is still risky as you might miss files, and history files can sometimes reappear as described above.)

    Do not snapshot the live EBS volume as it still contains the deleted files and you don’t want to make them public in the new AMI. Instead,

  4. Create a new EBS volume, attach, and mount it on the running instance, say under /image

  5. Copy the root file system over to the new EBS volume. This only copies the current view of the undeleted files and does not copy the blocks containing the deleted files or any other modified file information. The command might look something like:

    rsync -axvSHAX / /image/
    
  6. umount and detach the new EBS volume.

  7. Create an EBS snapshot of the new EBS volume.

  8. Register the EBS snapshot as a new AMI.

Test by running an instance of the new AMI and verifying it works and contains no private information before you change its attributes so the public can run it. You can then terminate instances and delete the extra EBS volume. Only the EBS snapshot is required to be kept for the new AMI.

Warning: Even though these instructions are under a section titled “Recommendation 2” I would not build public AMIs this way myself. It just seems too risky that something sensitive might slip into the public AMI. It’s better than snapshotting a running system directly because it removes the possibility deleted files leak out, but “Recommendation 1” is a safer path.

Wrapping up

I’m a bit surprised that the above article’s rm/snapshot and auhorized_keys instructions came from Amazon as I’ve seen their security folks and documentation warn against these very practices in the past. Amazon knows better as an organization and I hope that this misinformation is corrected soon so that people building AMIs don’t follow the guidance and leave secret information or back doors on public AMIs.

This is a complicated topic, as security related issues tend to be. I welcome comments, clarifications, and questions on the original post. Articles from Alestic.com are republished on a couple specific sites with explicit permission, but I only read and respond to comments on my primary blog.

[Update 2011-06-17: Corrected comments about authorized_keys. Amazon does recommend removing all of these files before publishing a new public AMI and recommends removing “unrecognized” authorized_keys files when running public AMIs.]

2 Comments

Good post - although you might mention one simple thing.

For all the credentials you've used when creating your AMI, there's a very simple remedy to make sure they are not exposed: change them.

If it's AWS keys, simply make them inactive and start using new ones - it's not like they are a resource that will run out. You probably have the same key in a couple places where you need to remember to update them, but I'm guessing this is less of a hassle than trying to create a clean AMI. Same thing with AWS X.509 certificates. SSH keys might be simple or might be hard, depending on usage. Usually you won't end up copying your "shared everywhere" ssh private key to the VM, so changing should be simple. The common thing here is that the secret is totally random generated and worthless after all access has been revoked for it.

For passwords, things are a bit trickier. Not many people have completely separate passwords for every service, so revealing an old password is a problem. For this, the easiest way is to change the password before using it in an AMI, and change it back afterwards. The throwaway password can be completely random as it is used only for a short while and you don't have to even learn new passwords.

Obviously, if changing credentials in many places is too much of a hassle, you can do the same thing with the other AWS credentials: create temporary credentials for whatever you need and discard them afterwards. That way you can even be sure they aren't accepted at some place you've forgotten to revoke them from.

So, in summary: it's good to be careful when building an AMI and it's great if you can manage to do it without using any secrets on the machine - but if you use secrets, just change them and sleep your nights soundly.

nuutti.kotivuori: Rotating AWS credentials on a regular basis is a good policy. Handing out credentials to the public with the plan to change them quickly seems risky and a lot of work, especially if you release AMIs on a frequent basis or are not the only one building AMIs following your procedure and have to depend on everybody cleaning up after themselves. I'm going to stick with my recommendation to never let secret information touch the block device used to build the AMI.

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


More Entries

EBS-SSD Boot AMIs For Ubuntu On Amazon EC2
With Amazon’s announcement that SSD is now available for EBS volumes, they have also declared this the recommended EBS volume type. The good folks at Canonical are now building Ubuntu…
EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
When you need an AWS command line toolset not provided by Ubuntu packages, you can download the tools directly from Amazon and install them locally. In a previous article I…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…
You Should Use EBS Boot Instances on Amazon EC2
EBS boot vs. instance-store If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store…
Retrieve Public ssh Key From EC2
A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as…
Running EC2 Instances on a Recurring Schedule with Auto Scaling
Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time? Would you like to…