Hidden Dangers in Creating Public EBS Snapshots on EC2

| 13 Comments

Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.

You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.

For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.

The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.

Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.

This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.

To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.

$100 Reward

To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:

snap-d53484bc

This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.

The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.

The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.

Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).

Good luck!

13 Comments

Hi, Eric, I recovered the gift certificate code and redeemed it. Thank you!

Here's what I did:

1. create a volume from your snapshot using ElasticFox
2. mount the volume to confirm it was the correct volume
3. unmount the volume
4. "strings /dev/sdb >/mnt/alestic"
5. check the file size of /mnt/alestic to see how manageable it is to examine; it turned out to be 1.4Kb
6. examine the file with emacs, recover the following deleted information:

Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:
http://alestic.com/2009/09/ec2-public-ebs-danger
This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).
As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:
XXXX-XXXXXX-XXXX
If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.
If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!
For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc

Tweeting that I found it and writing up this reply took longer than recovering the data.

So, the moral of this story is to overwrite the empty space on an EBS volume before sharing it, right?

Pete

Amazon gives me a different message than the normal one that it lists when a gift certificate has already been claimed: "There's something wrong with the gift certificate/card claim code you entered. Please check for transposed digits, omitted digits, and similar errors." I tested it with another already used code I have and it provided a different result.

In any case, here's how I got the code:

root@ip-10-250-131-99:/# echo dump_unused | debugfs /dev/sdg
debugfs 1.41.4 (27-Jan-2009)
debugfs:
Unused block 30721 contains non-zero data:

Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:

http://alestic.com/2009/09/ec2-public-ebs-danger

This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).

As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:

XXXX-XXXXXX-XXXX

If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.

If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!

For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc

Pete: Congratulations! That was fast.

EBS snapshots only store modified blocks, so one problem with overwriting all unused blocks on the source volume is the increased size and resulting cost of the snapshot.

There are various tools like "wipe" ("apt-get install wipe" on Ubuntu) which can be used to delete files overwriting the contents, then again this might still leave data in the file system journal, depending on the file system being used.

My recommendation would be to always create a fresh EBS volume, copy the desired files onto it in a single, clean operation, then create the shared EBS snapshot.

Mark: I don't know why Amazon is giving a different error than the usual "already been redeemed". Perhaps it has to do with different accounts trying to use the same code.

Damn. I was too late. Although I read this 30-45mins, and there were NO comments :(

# convert snapshot to volume, and attach to instance at /dev/sdf

$ strings /dev/sdf

lost+found
README-1.txt
README-2.txt
This is a public EBS snapshot created as described in:
http://alestic.com/2009/09/ec2-public-ebs-danger
There is a deleted file on this block device with
a $100 Amazon.com gift certificate usable by the first
person to discover and use it.
Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:
http://alestic.com/2009/09/ec2-public-ebs-danger
This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).
As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:
XXXX-XXXXXX-XXXX
If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.
If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!
For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc

After attaching the volume at /dev/sdh, I started out trying file recovery using ext3grep.

I tried various ways:

$ ext3grep --deleted --dump-names /dev/sdh

$ ext3grep --print --deleted /dev/sdh

$ ext3grep --restore-file README-2.txt /dev/sdh

or

$ ext3grep --restore-all /dev/sdh

.. which I felt should have worked, but they didn't. I'm going to have to read about the nuts-n-bolts of ext3 ( http://www.xs4all.nl/~carlo17/howto/undelete_ext3.html ) to understand why.

After failing to get results with ext3grep, I turned to regular grep, which is pretty similar to the 'strings' approach (although a little less elegant). I figured that the success message would say "Congratulations", so I grepped for 'ongrat', in much the same way an attacker may look for plaintext passwords by grepping for 'assword'.

$ grep --binary-files=text -B4 -A27 ongrat /dev/sdh

After some iteration with -B and -A values, this got me enough text to discover I'd been beaten :) Nice job Pete - and thanks Eric for making sure people are aware of this issue.

I did it using photorec. I know, probably overkill, but useful for more complicated restores.

Dave: Sorry, evil-me didn't immediately approve Pete's blog comment in order to give others the chance to have fun trying to recover the secret info.

Just as I suspected, I'm learning some new tricks for recovering disk data. Like Shlomo, I used photorec, which I'll describe a bit more here. This assumes Ubuntu:

sudo apt-get install -y testdisk
sudo photorec /d /mnt/recovered /dev/sdX

All the files (deleted and not) which are found on /dev/sdX will be placed in a directory named /mnt/recovered.1/ without touching the original content.

photorec has a slightly awkward curses-style UI which requires a 25 line terminal to function, but it does an amazing job of finding deleted files and restoring them. The file names are given extensions matching their respective content.

Anecdote: I once purchased a 4GB SD card at Best Buy only to discover somebody had slit open the package and substituted a much smaller size card. I doubt the thief intended to leave evidence in the form of explicit photos of his girlfriend, but photorec wasn't phased by simple file deletion.

Great post and a very brilliant idea for the challenge!
Congratulations ;)

I also used ext3grep.

$ ext3grep --search 'amazon' // Gives me blocks that had 'amazon' in them
$ ext3grep --block XXXXX

Voila, hex dump of README-2.txt

Is creating an EBS boot AMI from a snapshot a block-level operation or file-level operation? If I delete sensitive files from a running EBS boot instance, create a snapshot, and then create a public AMI from the snapshot, is the risk the same as for public snapshots? Thanks!

Ben: The answer is in the name. "EBS" = Elastic *Block* Store. EC2 knows nothing about the file system you are storing on the block device even for AMIs, so deleted files are easy to undelete. There are many deleted files on existing public AMIs.

This is one reason I recommend building public AMIs using a mounted file system instead of from a running instance.

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


More Entries

EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
When you need an AWS command line toolset not provided by Ubuntu packages, you can download the tools directly from Amazon and install them locally. In a previous article I…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…
You Should Use EBS Boot Instances on Amazon EC2
EBS boot vs. instance-store If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store…
Retrieve Public ssh Key From EC2
A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as…
Running EC2 Instances on a Recurring Schedule with Auto Scaling
Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time? Would you like to…
AWS Virtual MFA and the Google Authenticator for Android
Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking…