Alestic.com - A Personal AWS Blog by Eric Hammond

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on https://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

Amazon EC2 currently has a limit of 1,000 GB (1 TB) for EBS volumes (Elastic Block Store). It is possible to create file systems larger than this limit using RAID 0 across multiple EBS volumes. Using RAID 0 can also improve the performance of the file system reducing total IO wait as demonstrated in a number of published EBS performance tests.

The following instructions walk through one way to set up RAID 0 across multiple EBS volumes. Note that there is a limit on the size of a file system on 32-bit instances, but 64-bit instances can get unreasonably large. This test was run with 40 EBS volumes of 1,000 GB each for a total of 40,000 GB (40 TB) in the resulting file system.

Actual command line output showing the size of the RAID:

# df /vol
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             41942906368      1312 41942905056   1% /vol

# df -h /vol
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               40T  1.3M   40T   1% /vol

These commands can run in less than 10 minutes and this could probably be reduced further by parallelizing the creation and attaching of the EBS volumes.

Note that the default limit is 20 EBS volumes per EC2 account. You can request an increase from Amazon if you need more.

Caution: 40 TB of EBS storage on EC2 will cost $4,000 per month plus usage charges.

Instructions

Start a 64-bit instance (say, Ubuntu 8.04 Hardy from https://alestic.com). Use your own KEYPAIR:

ec2-run-instances \
  --key KEYPAIR \
  --instance-type c1.xlarge \
  --availability-zone us-east-1a \
  ami-0772946e

Configurable parameters (set on both local host and on EC2 instance):

instanceid=i-XXXXXXXX
volumes=40
size=1000
mountpoint=/vol

On the local host (with EC2 API tools installed)…

Create and attach EBS volumes:

devices=$(perl -e 'for$i("h".."k"){for$j("",1..15){print"/dev/sd$i$j\n"}}'|
           head -$volumes)
devicearray=($devices)
volumeids=
i=1
while [ $i -le $volumes ]; do
  volumeid=$(ec2-create-volume -z us-east-1a --size $size | cut -f2)
  echo "$i: created  $volumeid"
  device=${devicearray[$(($i-1))]}
  ec2-attach-volume -d $device -i $instanceid $volumeid
  volumeids="$volumeids $volumeid"
  let i=i+1
done
echo "volumeids='$volumeids'"

On the EC2 instance (after setting parameters as above)…

Install software:

sudo apt-get update &&
sudo apt-get install -y mdadm xfsprogs

Set up the RAID 0 device:

devices=$(perl -e 'for$i("h".."k"){for$j("",1..15){print"/dev/sd$i$j\n"}}'|
           head -$volumes)

yes | sudo mdadm \
  --create /dev/md0 \
  --level 0 \
  --metadata=1.1 \
  --chunk 256 \
  --raid-devices $volumes \
  $devices

echo DEVICE $devices       | sudo tee    /etc/mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

Create the file system (pick your preferred file system type)

sudo mkfs.xfs /dev/md0

Mount:

echo "/dev/md0 $mountpoint xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir $mountpoint
sudo mount $mountpoint

Check it out:

df -h $mountpoint

When you’re done with it and want to destroy the data and stop paying for storage, tear it down:

sudo umount $mountpoint
sudo mdadm --stop /dev/md0

Terminate the instance:

sudo shutdown -h now

On the local host (with EC2 API tools installed)…

Detach and delete volumes:

for volumeid in $volumeids; do
  ec2-detach-volume $volumeid
done

for volumeid in $volumeids; do
  ec2-delete-volume $volumeid
done

Credits

This article was originally posted on the EC2 Ubuntu group.

Thanks to M. David Peterson for the basic mdadm instructions:

[Update 2012-01-21: Added –chunk 256 based on community recognized best practices.]