October 2009 Archives

WARNING!

Though most Ubuntu 9.04 Jaunty systems can upgrade to 9.10 Karmic in place, this is not possible on EC2 and should not be attempted. If you do try this, your system will become unusable on reboot and there will be no recovery and no access to any of the data on the boot disk or ephemeral storage.

Here’s why:

  • Ubuntu 9.10 Karmic has a version of udev which requires a newer kernel than you would be running for Ubuntu 9.04 Jaunty (especially on EC2).

  • You cannot upgrade the kernel used by a running instance on Amazon EC2 (not even rebooting).

  • When an EC2 instance cannot boot (as in the case of the udev/kernel mismatch) your only option is to terminate it, losing the local storage.

How To Upgrade

In order to upgrade to Karmic you will need to start a new EC2 instance running a fresh copy of the appropriate Karmic AMI. I post the latest AMI ids for Karmic in the second table on http://alestic.com/.

Keep your old instance(s) running while you configure and test the new Karmic instances. EC2 makes it easy to have multiple sets of servers running in parallel instead of upgrading in place. When you are confident your new servers are functioning properly, you can discard the old ones.

The Ubuntu 9.10 Karmic AMIs released by Canonical have a number of differences from the community Ubuntu AMIs which have been published on http://alestic.com.

One of the biggest differences is that you will ssh to ubuntu@ instead of to root@ on your instance. You can then sudo to perform commands as the root user. Back in April I wrote a guide about Using sudo, ssh, rsync on the Official Ubuntu Images for EC2.

The Ubuntu server team has put a lot of work into making Ubuntu 9.10 Karmic function beautifully on Amazon EC2 and it’s been a pleasure to have a small part in the process. I’m already using the Karmic AMIs on EC2 for one of my production processes. Please give these AMIs a spin and give feedback.

Amazon Web Services just announced the release of two new instance types for EC2. These new types have 34.2 GB and 68.4 GB of RAM with a decent amount of CPU capacity on modern CPUs to go along with it.

Others have already done a great job of describing the instance types:

Jeff Barr’s AWS blog

RightScale’s blog

but when it comes to flexing the raw power at my fingertips with AWS, sometimes I can’t help myself. So…

sitting on my couch with my laptop watching an episode of “Lie to me” on TiVo I just typed:

ec2-run-instances              --instance-type m2.4xlarge   --key KEYPAIR                --instance-count 19          ami-e6f6158f

and in under a minute and about $45 later, I had ssh access to well over 1 TB (1,000 GB) of free memory. To be sure, it was spread over 19 Ubuntu servers, but still, there’s gotta be something I can do with that, no?

Here are the results on a single one of these servers running Ubuntu 8.04 Hardy:

root@domU-12-31-39-08-7F-51:~# free
             total       used       free     shared    buffers     cached
Mem:      71687580    1521464   70166116          0       2632      17704
-/+ buffers/cache:    1501128   70186452
Swap:            0          0          0

root@domU-12-31-39-08-7F-51:~# free -g
             total       used       free     shared    buffers     cached
Mem:            68          1         66          0          0          0
-/+ buffers/cache:          1         66
Swap:            0          0          0

Wait, I’d better do whatever I’m gonna do quick or I’m going to be charged another $45.60 for the next hour’s worth of fun!

Ok, time to cut my losses:

ec2-describe-instances | 
  egrep m2.4xlarge | 
  cut -f2 | 
  xargs ec2-terminate-instances

In case you didn’t feel like spending $2.40 to find out the CPUs on one of these beasts, here’s one of the ones I ran:

vendor_id   : GenuineIntel
cpu family  : 6
model       : 26
model name  : Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping    : 5
cpu MHz     : 2666.760
cache size  : 8192 KB
bogomips    : 5203.00

And remember that there’s 8 of these on the m2.4xlarge instance size. (Exact CPUs not guaranteed, your results my vary, etc.)

Amazon Web Services seems to keep releasing new features in advance of when our growing startup needs them. As we start to think about whether we are going to need to trim some tables or split up the database, here comes an instance type that will let us grow a lot longer just focusing on our core business challenges instead of on the infrastructure.

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

http://alestic.com

The following notes apply to this release:

  • The images have been upgraded to use the newest 2.6.21 kernel, ramdisk, and kernel modules from Amazon. This fixes a serious security hole in the previous 2.6.21 kernel.

  • The Alestic PPA (personal package archive) has been added to the Ubuntu AMIs. This makes it easy to install software packages listed in this PPA, including ec2-consistent-snapshot.

  • The runurl package from the Alestic PPA has been pre-installed on the Ubuntu AMIs. This can be a handy tool for setting up new instances with user-data scripts.

  • The EC2 AMI tools have been upgraded to version 1.3-34544.

  • The ec2-ami-tools package version has been pinned so it does not get downgraded if the official Ubuntu archives still have older versions.

  • All packages have been upgraded to their respective latest versions.

  • The Ubuntu Karmic images were not updated and have been removed from the listings at the top of http://alestic.com. If you would like to use Ubuntu Karmic Beta, please test with the AMIs published by Canonical listed a bit lower down on the page.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

  1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

  2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

  3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on http://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

Ubuntu AMIs

Ubuntu AMIs for EC2: