Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.
You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.
For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.
The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.
Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.
This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.
To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.
$100 Reward
To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:
snap-d53484bc
This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.
The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.
The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.
Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).
Good luck!
Follow Eric Hammond on Twitter
Hi, Eric, I recovered the gift certificate code and redeemed it. Thank you!
Here's what I did:
1. create a volume from your snapshot using ElasticFox
2. mount the volume to confirm it was the correct volume
3. unmount the volume
4. "strings /dev/sdb >/mnt/alestic"
5. check the file size of /mnt/alestic to see how manageable it is to examine; it turned out to be 1.4Kb
6. examine the file with emacs, recover the following deleted information:
Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:
http://alestic.com/2009/09/ec2-public-ebs-danger
This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).
As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:
XXXX-XXXXXX-XXXX
If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.
If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!
For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc
Tweeting that I found it and writing up this reply took longer than recovering the data.
So, the moral of this story is to overwrite the empty space on an EBS volume before sharing it, right?
Pete
Amazon gives me a different message than the normal one that it lists when a gift certificate has already been claimed: "There's something wrong with the gift certificate/card claim code you entered. Please check for transposed digits, omitted digits, and similar errors." I tested it with another already used code I have and it provided a different result.
In any case, here's how I got the code:
root@ip-10-250-131-99:/# echo dump_unused | debugfs /dev/sdg
debugfs 1.41.4 (27-Jan-2009)
debugfs:
Unused block 30721 contains non-zero data:
Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:
http://alestic.com/2009/09/ec2-public-ebs-danger
This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).
As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:
XXXX-XXXXXX-XXXX
If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.
If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!
For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc
Pete: Congratulations! That was fast.
EBS snapshots only store modified blocks, so one problem with overwriting all unused blocks on the source volume is the increased size and resulting cost of the snapshot.
There are various tools like "wipe" ("apt-get install wipe" on Ubuntu) which can be used to delete files overwriting the contents, then again this might still leave data in the file system journal, depending on the file system being used.
My recommendation would be to always create a fresh EBS volume, copy the desired files onto it in a single, clean operation, then create the shared EBS snapshot.
Mark: I don't know why Amazon is giving a different error than the usual "already been redeemed". Perhaps it has to do with different accounts trying to use the same code.
Damn. I was too late. Although I read this 30-45mins, and there were NO comments :(
# convert snapshot to volume, and attach to instance at /dev/sdf
$ strings /dev/sdf
lost+found
README-1.txt
README-2.txt
This is a public EBS snapshot created as described in:
http://alestic.com/2009/09/ec2-public-ebs-danger
There is a deleted file on this block device with
a $100 Amazon.com gift certificate usable by the first
person to discover and use it.
Congratulations! You have successfully recovered
this file which was stored on an Amazon EC2 EBS
volume, then deleted. A public snapshot was then
created of this volume and publicized at:
http://alestic.com/2009/09/ec2-public-ebs-danger
This demonstrates the danger of releasing public
EBS snapshots without knowing exactly what files
have ever been on the original volume(s).
As a tangible example of value in a deleted file,
here is an Amazon.com gift certificate for $100:
XXXX-XXXXXX-XXXX
If you were the first person to recover this file
and enter the code into your Amazon.com account,
then your balance will be increased by $100.
If somebody else beat you to it, then Amazon will
notify you that it has already been redeemed, but
you still get credit for solving the puzzle!
For more information on using an Amazon gift
certificate: http://www.amazon.com/help/gc
After attaching the volume at /dev/sdh, I started out trying file recovery using ext3grep.
I tried various ways:
$ ext3grep --deleted --dump-names /dev/sdh
$ ext3grep --print --deleted /dev/sdh
$ ext3grep --restore-file README-2.txt /dev/sdh
or
$ ext3grep --restore-all /dev/sdh
.. which I felt should have worked, but they didn't. I'm going to have to read about the nuts-n-bolts of ext3 ( http://www.xs4all.nl/~carlo17/howto/undelete_ext3.html ) to understand why.
After failing to get results with ext3grep, I turned to regular grep, which is pretty similar to the 'strings' approach (although a little less elegant). I figured that the success message would say "Congratulations", so I grepped for 'ongrat', in much the same way an attacker may look for plaintext passwords by grepping for 'assword'.
$ grep --binary-files=text -B4 -A27 ongrat /dev/sdh
After some iteration with -B and -A values, this got me enough text to discover I'd been beaten :) Nice job Pete - and thanks Eric for making sure people are aware of this issue.
I did it using photorec. I know, probably overkill, but useful for more complicated restores.
Dave: Sorry, evil-me didn't immediately approve Pete's blog comment in order to give others the chance to have fun trying to recover the secret info.
Just as I suspected, I'm learning some new tricks for recovering disk data. Like Shlomo, I used photorec, which I'll describe a bit more here. This assumes Ubuntu:
sudo apt-get install -y testdisk
sudo photorec /d /mnt/recovered /dev/sdX
All the files (deleted and not) which are found on /dev/sdX will be placed in a directory named /mnt/recovered.1/ without touching the original content.
photorec has a slightly awkward curses-style UI which requires a 25 line terminal to function, but it does an amazing job of finding deleted files and restoring them. The file names are given extensions matching their respective content.
Anecdote: I once purchased a 4GB SD card at Best Buy only to discover somebody had slit open the package and substituted a much smaller size card. I doubt the thief intended to leave evidence in the form of explicit photos of his girlfriend, but photorec wasn't phased by simple file deletion.
Great post and a very brilliant idea for the challenge!
Congratulations ;)
I also used ext3grep.
$ ext3grep --search 'amazon' // Gives me blocks that had 'amazon' in them
$ ext3grep --block XXXXX
Voila, hex dump of README-2.txt
Is creating an EBS boot AMI from a snapshot a block-level operation or file-level operation? If I delete sensitive files from a running EBS boot instance, create a snapshot, and then create a public AMI from the snapshot, is the risk the same as for public snapshots? Thanks!
Ben: The answer is in the name. "EBS" = Elastic *Block* Store. EC2 knows nothing about the file system you are storing on the block device even for AMIs, so deleted files are easy to undelete. There are many deleted files on existing public AMIs.
This is one reason I recommend building public AMIs using a mounted file system instead of from a running instance.