Alestic.com - A Personal AWS Blog by Eric Hammond

The ssh protocol uses two different keys to keep you secure:

1. The user ssh key is the one we normally think of. This authenticates us to the remote host, proving that we are who we say we are and allowing us to log in.

2. The ssh host key gets less attention, but is also important. This authenticates the remote host to our local computer and proves that the ssh session is encrypted so that nobody can be listening in.

Every time you see a prompt like the following, ssh is checking the host key and asking you to make sure that your session is going to be encrypted securely.

The authenticity of host 'ec2-...' can't be established.
ECDSA key fingerprint is ca:79:72:ea:23:94:5e:f5:f0:b8:c0:5a:17:8c:6f:a8.
Are you sure you want to continue connecting (yes/no)? 

If you answer “yes” without verifying that the remote ssh host key fingerprint is the same, then you are basically saying:

I don’t need this ssh session encrypted. It’s fine for any man-in-the-middle to intercept the communication.

Ouch! (But a lot of people do this.)

Amazon published a tutorial about best practices in creating public AMIs for use on EC2 last week:

How To Share and Use Public AMIs in A Secure Manner

Though the general principles put forth in the tutorial are good, some of the specifics are flawed in how to accomplish those principles. (Comments here relate to the article update from June 7, 2011 3:45 AM GMT.)

The primary message of the article is that you should not publish private information on a public AMI. Excellent advice!

Unfortunately, the article seems to recommend or at least to assume that you are building the public AMI by taking a snapshot of a running instance. Though this method seems an easy way to build an AMI and is fine for private AMIs, it is is a dangerous approach for public AMIs because of how difficult it is to identify private information and to clear that private information from a running system in such a way that it does not leak into the public AMI.

In a thread on the EC2 forum, Marko describes a situation where an outbound firewall prevents the ability to ssh to port 22, which is the default port on all EC2 instances.

In that thread, Shlomo Swidler proposes creating a user-data script that changes sshd to listen on a port the firewall permits.

Here’s a simple example of a user-data script that does just that. Most outbound firewalls allow traffic to port 80 (web/HTTP), so I use it in this example.

The first step is to create a file containing the user-data script:

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on https://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.

You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.

For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.

The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.

Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.

This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.

To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.

$100 Reward

To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:

snap-d53484bc

This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.

The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.

The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.

Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).

Good luck!

Perhaps you are behind a corporate firewall which does not allow you to access certain types of resources on the Internet. Or, perhaps you are accessing the Internet over an open wifi where you do not trust your network traffic to your fellow wifi users or the admins running the local network.

These instructions guide you in setting up an OpenVPN server on an EC2 instance, sending all your network traffic through a secure channel to port 80 on the EC2 instance and from there out to the Internet.

EC2 Instance

Run the latest Ubuntu 9.10 Karmic image. You can find the most current AMI id in a table on https://alestic.com

ec2-run-instances --key <KEYPAIR> ami-1515f67c

Make a note of the instance id (e.g., i-6fceba06). Watch the status using a command like this (replace with your own instance id):

ec2-describe-instances <INSTANCEID>

Repeat the describe instances command until it shows that the instance is “running” and make a note of the external hostname (e.g., ec2-75-101-179-94.compute-1.amazonaws.com).

Connect to the instance using the external hostname you noted.

remotehost=<HOSTNAME>
ssh -i <KEYPAIR>.pem ubuntu@$remotehost

OpenVPN Server

Upgrade the EC2 instance and install the necessary OpenVPN software:

sudo apt-get update &&
sudo apt-get upgrade -y &&
sudo apt-get install -y openvpn
sudo modprobe iptable_nat
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables -t nat -A POSTROUTING -s 10.4.0.1/24 -o eth0 -j MASQUERADE

Generate a secret key for secure communication with OpenVPN:

sudo openvpn --genkey --secret ovpn.key

Start the OpenVPN server on the EC2 instance. We are (ab)using port 80 because most closed networks will allow traffic to this port ;-)

sudo openvpn                     \
  --proto tcp-server             \
  --port 80                      \
  --dev tun1                     \
  --secret ovpn.key              \
  --ifconfig 10.4.0.1 10.4.0.2   \
  --daemon

OpenVPN Client

Back on the local (non-EC2) workstation, set up the software:

sudo apt-get install -y openvpn
sudo modprobe tun # if necessary
sudo iptables -I OUTPUT -o tun+ -j ACCEPT
sudo iptables -I INPUT -i tun+ -j ACCEPT

Download the secret key from the EC2 instance:

ssh -i <KEYPAIR>.pem ubuntu@$remotehost 'sudo cat ovpn.key' > ovpn.key
chmod 600 ovpn.key

Start the OpenVPN client:

sudo openvpn                    \
  --proto tcp-client            \
  --remote $remotehost          \
  --port 80                     \
  --dev tun1                    \
  --secret ovpn.key             \
  --redirect-gateway def1       \
  --ifconfig 10.4.0.2 10.4.0.1  \
  --daemon

Edit /etc/resolv.conf and set it so that DNS is resolved by the EC2 name server:

sudo mv /etc/resolv.conf /etc/resolv.conf.save
echo "nameserver 172.16.0.23" | sudo tee /etc/resolv.conf

You should now be able to access any Internet resource securely and without restriction.

Teardown

When you are done with this OpenVPN tunnel, remember to shut down the EC2 instance and restore the DNS configuration:

sudo killall openvpn
ec2-terminate-instances <INSTANCEID>
sudo mv /etc/resolv.conf.save /etc/resolv.conf

If you have ways to improve this approach, please leave a comment.

Disclaimer

These instructions are not intended to assist in illegal activities. If you are breaking the laws or rules of your government or college or company or ISP, then you should understand the security implications of the above steps better than I do and be willing to accept consequences of your actions.

[Update 2009-01-15: Upgrade instructions for Karmic]