Alestic.com - A Personal AWS Blog by Eric Hammond

A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as it demonstrates a few handy features of EC2.

Challenge

The basic need is to get the public ssh key from a keypair that exists inside of EC2. You don’t have access to the private key at the moment (but somebody else does or you will at a different location).

The AWS console and EC2 API do not let you ask for the public ssh key associated with a keypair. However, EC2 does pass the public ssh key to a new EC2 instance when you run it with a specific keypair.

The problem is that we don’t currently have the private key, so we can’t log in to the EC2 instance to get the public key. (Besides, if we did have the private key, we could extract the public key from it directly.)

Solution

I proposed creating a user-data script that sends the public ssh key to the EC2 instance console output. You can retrieve the console output without logging in to the EC2 instance.

Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.

You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.

For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.

The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.

Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.

This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.

To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.

$100 Reward

To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:

snap-d53484bc

This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.

The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.

The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.

Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).

Good luck!