Results matching “tools”

Note: I do not recommend that new users start with these AMIs. These AMIs run with older versions of Amazon’s Fedora 8 kernel which have some incompatibilities with Ubuntu and Debian (e.g., XFS is broken). My strong recommendation is for all users to convert to one of the Ubuntu 9.10 Karmic or Ubuntu 8.04 Hardy AMIs which run with a modern, compatible Ubuntu kernel.

I have released updates of the following legacy Ubuntu and Debian S3-based AMIs published in the “alestic” buckets:

• Ubuntu: 8.04 Hardy, 8.10 Intrepid, 9.04 Jaunty
• Debian: 4.0 Etch, 5.0 Lenny

I welcome testing and feedback from folks who are already using the older versions of these AMIs.

In addition to upgraded software packages, the following enhancements are in this release:

• Wait for meta-data before deciding whether to generate ssh host key (Thanks to Dmitry for catching)

• Patch from Tom White to support compressed user-data scripts

• Allow user-data script to remove the “been run” file so that it is run on every boot instead of just the first

• Upgrade EC2 AMI tools to 1.3-34544

• Debian: Add Alestic PPA to apt sources

• Debian: Install runurl from Alestic PPA

I would like to reiterate that these AMIs are not recommended for anybody except folks who are already using older versions of these AMIs and I would encourage you to upgrade to an Ubuntu 9.10 Karmic AMI or the upcoming Ubuntu 10.04 Lucid AMI which will be released in April. The base Ubuntu image building responsibilities have been transfered to Canonical and I don’t have any plans to release new AMIs in the older Ubuntu or Debian series beyond what I am announcing here.

I have not yet posted the AMI ids on http://alestic.com pending testing and feedback, but here are the ids for those who need to run them:

us-east-1

ami-1116f978 Ubuntu 9.04 Jaunty   server  32-bit
ami-e116f988 Ubuntu 9.04 Jaunty   server  64-bit
ami-cd16f9a4 Ubuntu 9.04 Jaunty   desktop 32-bit
ami-c316f9aa Ubuntu 9.04 Jaunty   desktop 64-bit

ami-1316f97a Ubuntu 8.10 Intrepid server  32-bit
ami-e316f98a Ubuntu 8.10 Intrepid server  64-bit
ami-cb16f9a2 Ubuntu 8.10 Intrepid desktop 32-bit
ami-c116f9a8 Ubuntu 8.10 Intrepid desktop 64-bit

ami-e916f980 Ubuntu 8.04 Hardy    server  32-bit
ami-e716f98e Ubuntu 8.04 Hardy    server  64-bit
ami-1716f97e Ubuntu 8.04 Hardy    desktop 32-bit
ami-e516f98c Ubuntu 8.04 Hardy    desktop 64-bit

ami-eb16f982 Ubuntu 6.06 Dapper   server  32-bit
ami-f916f990 Ubuntu 6.06 Dapper   server  64-bit

ami-ed16f984 Debian 5.0 Lenny     server  32-bit
ami-fb16f992 Debian 5.0 Lenny     server  64-bit
ami-cf16f9a6 Debian 5.0 Lenny     desktop 32-bit
ami-c516f9ac Debian 5.0 Lenny     desktop 64-bit

ami-ef16f986 Debian 4.0 Etch      server  32-bit
ami-fd16f994 Debian 4.0 Etch      server  64-bit

us-west-1

ami-197a2b5c Ubuntu 9.04 Jaunty   server  32-bit
ami-237a2b66 Ubuntu 9.04 Jaunty   server  64-bit
ami-357a2b70 Ubuntu 9.04 Jaunty   desktop 32-bit
ami-c97a2b8c Ubuntu 9.04 Jaunty   desktop 64-bit

ami-257a2b60 Ubuntu 8.10 Intrepid server  32-bit
ami-2d7a2b68 Ubuntu 8.10 Intrepid server  64-bit
ami-377a2b72 Ubuntu 8.10 Intrepid desktop 32-bit
ami-d57a2b90 Ubuntu 8.10 Intrepid desktop 64-bit

ami-277a2b62 Ubuntu 8.04 Hardy    server  32-bit
ami-2f7a2b6a Ubuntu 8.04 Hardy    server  64-bit
ami-cf7a2b8a Ubuntu 8.04 Hardy    desktop 32-bit
ami-d77a2b92 Ubuntu 8.04 Hardy    desktop 64-bit

ami-217a2b64 Ubuntu 6.06 Dapper   server  32-bit
ami-e77a2ba2 Ubuntu 6.06 Dapper   server  64-bit

ami-d37a2b96 Debian 5.0 Lenny     server  32-bit
ami-df7a2b9a Debian 5.0 Lenny     server  64-bit
ami-e37a2ba6 Debian 5.0 Lenny     desktop 32-bit
ami-db7a2b9e Debian 5.0 Lenny     desktop 64-bit

ami-d17a2b94 Debian 4.0 Etch      server  32-bit
ami-dd7a2b98 Debian 4.0 Etch      server  64-bit

eu-west-1

ami-a798b3d3 Ubuntu 9.04 Jaunty   server  32-bit
ami-af98b3db Ubuntu 9.04 Jaunty   server  64-bit
ami-9798b3e3 Ubuntu 9.04 Jaunty   desktop 32-bit
ami-9d98b3e9 Ubuntu 9.04 Jaunty   desktop 64-bit

ami-a198b3d5 Ubuntu 8.10 Intrepid server  32-bit
ami-a998b3dd Ubuntu 8.10 Intrepid server  64-bit
ami-9198b3e5 Ubuntu 8.10 Intrepid desktop 32-bit
ami-9998b3ed Ubuntu 8.10 Intrepid desktop 64-bit

ami-a398b3d7 Ubuntu 8.04 Hardy    server  32-bit
ami-ab98b3df Ubuntu 8.04 Hardy    server  64-bit
ami-9398b3e7 Ubuntu 8.04 Hardy    desktop 32-bit
ami-8798b3f3 Ubuntu 8.04 Hardy    desktop 64-bit

ami-ad98b3d9 Ubuntu 6.06 Dapper   server  32-bit
ami-9598b3e1 Ubuntu 6.06 Dapper   server  64-bit

ami-8398b3f7 Debian 5.0 Lenny     server  32-bit
ami-8f98b3fb Debian 5.0 Lenny     server  64-bit
ami-8998b3fd Debian 5.0 Lenny     desktop 32-bit
ami-8b98b3ff Debian 5.0 Lenny     desktop 64-bit

ami-8198b3f5 Debian 4.0 Etch      server  32-bit
ami-8d98b3f9 Debian 4.0 Etch      server  64-bit

Here are a few little-publicized benefits that were launched with Amazon EC2’s new EBS boot instances: You can lock them from being accidentally terminated; you can prevent termination even when you try to shutdown the server from inside the instance; and you can automatically save your data storage when they are terminated.

In discussions with other EC2 users, I’ve found that it is a common feeling of near panic when you go to terminate an instance and you check very carefully to make sure that you are deleting the right instance instead of an active production system. Slightly less common but even worse is the feeling of dread when you realize you just casually terminated the wrong EC2 instance.

It is always recommended to set up your AWS architecture so that you are able to restart production systems on EC2 easily, as they could, in theory, be lost at any point due to hardware failure or other actions. However, new features released with the EBS boot instances help reduce the risks associated with human error.

The following examples will demonstrate with the EC2 API command line tools ec2-run-instances, ec2-modify-instance-attribute, and ec2-terminate-instances. Since AWS is Amazon’s “web service” all of these features are also available through the API and should be coming available (if they aren’t already) in other tools, packages, and interfaces using the web service API.

1. Shutdown Behavior

First, let’s look at what happens when you run a command like the following in an EC2 instance:

sudo shutdown -h now
# or, equivalently and much easier to type:
sudo halt

Using the legacy S3 based AMIs, either of the above terminates the instance and you lose all local and ephemeral storage (boot disk and /mnt) forever. Hope you remembered to save the important stuff elsewhere!

A shutdown from within an EBS boot instance, by default, will initiate a “stop” instead of a “terminate”. This means that your instance is not in a running state and not getting charged, but the EBS volumes still exist and you can “start” the same instance id again later, losing nothing.

You can explicitly set or change this with the --instance-initiated-shutdown-behavior option in ec2-run-instances. For example:

ec2-run-instances                             \
  --instance-initiated-shutdown-behavior stop \
  [...]

This is the first safety precaution and, as mentioned, should already be built in, though it doesn’t hurt to document your intentions with an explicit option.

2. Delete on Termination

Though EBS volumes created and attached to an instance at instantiation are preserved through a “stop”/”start” cycle, by default they are destroyed and lost when an EC2 instance is terminated. This behavior can be changed with a delete-on-termination boolean value buried in the documentation for the --block-device-mapping option of ec2-run-instances.

Here is an example that says “Don’t delete the root EBS volume when this instance is terminated”:

ec2-run-instances                          \
  --block-device-mapping /dev/sda1=::false \
  [...]

If you are associating other EBS snapshots with the instance at run time, you can also specify that those created EBS volumes should be preserved past the lifetime of the instance:

  --block-device-mapping /dev/sdh=SNAPSHOTID::false

When you use these options, you’ll need to manually clean up the EBS volume(s) if you no longer want to pay for the storage costs after an instance is gone.

Note: EBS volumes attached to an instance after it is already running are, by default, left alone on termination (i.e., not deleted). The default rules are: If the EBS volume is created by the creation of the instance, then the termination of the instance deletes the volumes. If you created the volume explicitly, then you must delete it explicitly.

3. Disable Termination

This is my favorite new safety feature. Years ago, I asked Amazon for the ability to lock an EC2 instance from being accidentally terminated, and with the launch of EBS boot instances, this is now possible. Using ec2-run-instances, the key option is:

ec2-run-instances           \
  --disable-api-termination \
  [...]

Now, if you try to terminate the instance, you will get an error like:

Client.OperationNotPermitted: The instance 'i-XXXXXXXX' may not be terminated.
Modify its 'disableApiTermination' instance attribute and try again.

Yay!

To unlock the instance and allow termination through the API, use a command like:

ec2-modify-instance-attribute     \
  --disable-api-termination false \
  INSTANCEID

Then end it all with:

ec2-terminate-instances INSTANCEID

Oh, wait! did you make sure you unlocked and terminated the right instance?! :)

Note: disableApiTermination is also available when you run S3 based AMIs today, but since the instance can still be terminated from inside (shutdown/halt) I am moving towards EBS based instances for all around security.

Put It Together

Here’s a command line I just used to start up an EC2 instance of an Ubuntu 9.10 Karmic EBS boot AMI which I intend to use as a long-term production server with strong uptime and data safety requirements. I wanted to add all the protection available:

ec2-run-instances                                    \
  --key $keypair                                     \
  --availability-zone $availabilityzone              \
  --user-data-file $startupscript                    \
  --block-device-mapping /dev/sda1=::false           \
  --block-device-mapping /dev/sdh=$snapshotid::false \
  --instance-initiated-shutdown-behavior stop        \
  --disable-api-termination                          \
  $amiid

With regular EBS snapshots of the volumes, copies to off site backups, and documented/automated restart processes, I feel pretty safe.

Runtime Modification

If you have a valuable running EC2 instance, but forgot to specify the above options to protect it when you started it, or you are now ready to turn a test system into a production system, you can still lock things after the fact using the ec2-modify-instance-attribute command (or equivalent API call).

For example:

ec2-modify-instance-attribute --disable-api-termination true INSTANCEID
ec2-modify-instance-attribute --instance-initiated-shutdown-behavior stop INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sda1=::false INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sdh=::false INSTANCEID

Notes:

• Only one type of option can be specified with each invocation.

• The --disable-api-termination option has no argument value when used in in ec2-run-instances, but it takes a value of true or false in ec2-modify-instance-attribute.

• You don’t specify the snapshot id when changing the delete-on-terminate state of an attached EBS volume.

• You may change the delete-on-terminate to “true” for an EBS volume you created and attached after the instance was running. By default it will not be deleted since you created it.

• The above --block-device-mapping options currently generate a Server.InternalError for me. I reported it to Amazon.

You can find out the current state of the options using ec2-describe-instance-attribute, which only takes one option at a time:

ec2-describe-instance-attribute --disable-api-termination INSTANCEID
ec2-describe-instance-attribute --instance-initiated-shutdown-behavior INSTANCEID
ec2-describe-instance-attribute --block-device-mapping INSTANCEID

Unfortunately, the block-device-mapping output does not currently show the state of delete-on-termination value, but thanks to Andrew Lusk for pointing out in a comment below that it is available through the API. Here’s a hack which extracts the information from the debug output:

ec2-describe-instance-attribute -v --block-device-mapping INSTANCEID | 
  perl -0777ne 'print "$1\t$2\t$3\n" while 
  m%<deviceName>(.*?)<.*?<volumeId>(.*?)<.*?<deleteOnTermination>(.*?)<%sg'

While we’re on the topic of EC2 safety, I should mention the well known best practice of separating development and production systems by using a different AWS account for each. Amazon lets you create and use as many accounts as you’d like even with the same credit card as long as you use a unique email address for each. Now that you can share EBS snapshots between accounts, this practice is even more useful.

What additional safety precautions do you take with your EC2 instances and data?

In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.

In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.

Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.

1. Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:

   # 32-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     ami-1515f67c |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   
   
   # 64-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     --instance-type m1.large       \
     ami-ab15f6c2 |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   

   Wait for the instance to move to the “running” state, then note the public hostname:

   while host=$(ec2-describe-instances "$instanceid" | 
     egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
   echo host=$host
   

   Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

   rsync                            \
     --rsh="ssh -i YOURKEYPAIR.pem" \
     --rsync-path="sudo rsync"      \
     ~/.ec2/{cert,pk}-*.pem         \
     ubuntu@$host:/mnt/
   

   Connect to the instance:

   ssh -i YOURKEYPAIR.pem ubuntu@$host
   

2. Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

   export DEBIAN_FRONTEND=noninteractive
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
   sudo apt-get update &&
   sudo -E apt-get dist-upgrade -y &&
   sudo -E apt-get install -y ec2-api-tools
   

3. Set up some parameters:

   codename=karmic
   release=9.10
   tag=server
   if [ $(uname -m) = 'x86_64' ]; then
     arch=x86_64
     arch2=amd64
     ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
     ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
   else
     arch=i386
     arch2=i386
     ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
     ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
   fi
   

4. Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.

   imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
   image=/mnt/$codename-$tag-uec-$arch2.img
   imagedir=/mnt/$codename-uec-$arch2
   wget -O- $imagesource |
     sudo tar xzf - -C /mnt
   sudo mkdir -p $imagedir
   sudo mount -o loop $image $imagedir
   

5. [OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.

   In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.

   # Allow network access from chroot environment
   sudo cp /etc/resolv.conf $imagedir/etc/
   # Fix what I consider to be a bug in vmbuilder
   sudo rm -f $imagedir/etc/hostname
   # Add multiverse
   sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
     $imagedir/etc/ec2-init/templates/sources.list.tmpl
   # Add Alestic PPA for runurl package (handy in user-data scripts)
   echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
     sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
   sudo chroot $imagedir \
     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
   # Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
   sudo chroot $imagedir \
     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
   # Upgrade the system and install packages
   sudo chroot $imagedir mount -t proc none /proc
   sudo chroot $imagedir mount -t devpts none /dev/pts
   cat <<EOF > $imagedir/usr/sbin/policy-rc.d
   #!/bin/sh
   exit 101
   EOF
   chmod 755 $imagedir/usr/sbin/policy-rc.d
   DEBIAN_FRONTEND=noninteractive
   sudo chroot $imagedir apt-get update &&
   sudo -E chroot $imagedir apt-get dist-upgrade -y &&
   sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
   sudo chroot $imagedir umount /proc
   sudo chroot $imagedir umount /dev/pts
   rm -f $imagedir/usr/sbin/policy-rc.d
   

   Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.

6. Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

   size=15 # root disk in GB
   now=$(date +%Y%m%d-%H%M)
   prefix=ubuntu-$release-$codename-$tag-$arch-$now
   description="Ubuntu $release $codename $tag $arch $now"
   export EC2_CERT=$(echo /mnt/cert-*.pem)
   export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
   volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
     cut -f2)
   instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
   ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
   while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
   sudo mkfs.ext3 -F /dev/sdi
   ebsimage=$imagedir-ebs
   sudo mkdir $ebsimage
   sudo mount /dev/sdi $ebsimage
   sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
   sudo umount $ebsimage
   ec2-detach-volume "$volumeid"
   snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
   ec2-delete-volume "$volumeid"
   while ec2-describe-snapshots "$snapshotid" | grep -q pending
     do echo -n .; sleep 1; done
   ec2-register                   \
     --architecture $arch         \
     --name "$prefix"             \
     --description "$description" \
     $ebsopts                     \
     --snapshot "$snapshotid"
   

7. Depending on what you want to keep from the above process, there are various things that you might want to clean up.

   If you no longer want to use an EBS boot AMI:

   ec2-deregister $amiid
   ec2-delete-snapshot $snapshotid
   

   When you’re done with the original instance:

   ec2-terminate-instance $instanceid
   

In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]

[Update 2010-01-22: New location for downloadable Ubuntu images.]

[Update 2010-03-26: Path tweak, thanks to paul.]

Here’s my current recipe for how to build an Ubuntu 9.10 Karmic AMI, either the new EBS boot or the standard S3 based, using the Ubuntu vmbuilder software. The Ubuntu vmbuilder utility replaces ec2ubuntu-build-ami for building EC2 images and it can build images for a number of other virtual machine formats as well.

There is a lot of room for simplification and scripting in the following instructions, but I figured I’d publish what is working now so others can take advantage of the research to date. Happy New Year!

Some sections are marked [For EBS boot AMI] or [For S3 based AMI] and should only be followed when you are building that type of AMI. The rest of the sections apply to either type. It is possible to follow all instructions to build both types of AMIs at the same time.

1. Run an instance of Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. I prefer the c1.* instance types to speed up the builds, but you can get by cheaper with the m1.* instance types. Make a note of the resulting instance id:

   # 32-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     --instance-type c1.medium      \
     ami-1515f67c |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   
   
   # 64-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     --instance-type c1.xlarge      \
     ami-ab15f6c2 |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   

   Wait for the instance to move to the “running” state, then note the public hostname:

   while host=$(ec2-describe-instances "$instanceid" | 
     egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
   echo host=$host
   

2. Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

   rsync                            \
     --rsh="ssh -i YOURKEYPAIR.pem" \
     --rsync-path="sudo rsync"      \
     ~/.ec2/{cert,pk}-*.pem         \
     ubuntu@$host:/mnt/
   

3. Connect to the instance:

   ssh -i YOURKEYPAIR.pem ubuntu@$host
   

4. Install the image building software. We install the python-vm-builder package from Karmic, but we’re going to be using the latest vmbuilder from the development branch in Launchpad because it has good bug fixes. We also use the EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

   export DEBIAN_FRONTEND=noninteractive
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
   sudo apt-get update &&
   sudo -E apt-get upgrade -y &&
   sudo -E apt-get install -y \
     python-vm-builder ec2-ami-tools ec2-api-tools bzr &&
   bzr branch lp:vmbuilder
   

   You can ignore the “Launchpad ID” warning from bzr.

5. Fill in your AWS credentials:

   export AWS_USER_ID=...
   export AWS_ACCESS_KEY_ID=...
   export AWS_SECRET_ACCESS_KEY=...
   export EC2_CERT=$(echo /mnt/cert-*.pem)
   export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
   

   Set up parameters and create files to be used by the build process. The bucket value is only required for S3 based AMIs:

   bucket=...
   codename=karmic
   release=9.10
   tag=server
   if [ $(uname -m) = 'x86_64' ]; then
     arch=x86_64
     arch2=amd64
     pkgopts="--addpkg=libc6-i386"
     kernelopts="--ec2-kernel=aki-fd15f694 --ec2-ramdisk=ari-c515f6ac"
     ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
     ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
   else
     arch=i386
     arch2=i386
     pkgopts=
     kernelopts="--ec2-kernel=aki-5f15f636 --ec2-ramdisk=ari-0915f660"
     ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
     ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
   fi
   cat > part-i386.txt <<EOM
   root 10240 a1
   /mnt 1 a2
   swap 1024 a3
   EOM
   cat > part-x86_64.txt <<EOM
   root 10240 a1
   /mnt 1 b
   EOM
   

6. Create a script to perform local customizations to the image before it is bundled. This is passed to vmbuilder below using the --execscript option:

   cat > setup-server <<'EOM'
   #!/bin/bash -ex
   imagedir=$1
   # fix what I consider to be bugs in vmbuilder
   perl -pi -e "s%^127.0.1.1.*\n%%" $imagedir/etc/hosts
   rm -f $imagedir/etc/hostname
   # Use multiverse
   perl -pi -e 's%(universe)$%$1 multiverse%' \
     $imagedir/etc/ec2-init/templates/sources.list.tmpl
   # Add Alestic PPA for runurl package (handy in user-data scripts)
   echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
     tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
   chroot $imagedir \
     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
   # Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
   chroot $imagedir \
     sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
   # Install packages
   chroot $imagedir apt-get update
   chroot $imagedir apt-get install -y runurl
   chroot $imagedir apt-get install -y ec2-ami-tools
   EOM
   chmod 755 setup-server
   

7. Build the image:

   now=$(date +%Y%m%d-%H%M)
   dest=/mnt/dest-$codename-$now
   prefix=ubuntu-$release-$codename-$arch-$tag-$now
   description="Ubuntu $release $codename $arch $tag $now"
   sudo vmbuilder/vmbuilder xen ubuntu       \
     --suite=$codename                       \
     --arch=$arch2                           \
     --dest=$dest                            \
     --tmp=/mnt                              \
     --ec2                                   \
     --ec2-version="$description"            \
     --manifest=$prefix.manifest             \
     --lock-user                             \
     --part=part-$arch.txt                   \
     $kernelopts                             \
     $pkgopts                                \
     --execscript ./setup-server             \
     --debug
   

   [For S3 based AMI] include the following options in the vmbuilder command above. This does not preclude you from also building an EBS boot AMI with the same image. Make a note of the resulting AMI id output by vmbuilder:

     --ec2-bundle                            \
     --ec2-upload                            \
     --ec2-register                          \
     --ec2-bucket=$bucket                    \
     --ec2-prefix=$prefix                    \
     --ec2-user=$AWS_USER_ID                 \
     --ec2-cert=$EC2_CERT                    \
     --ec2-key=$EC2_PRIVATE_KEY              \
     --ec2-access-key=$AWS_ACCESS_KEY_ID     \
     --ec2-secret-key=$AWS_SECRET_ACCESS_KEY \
   

8. [For EBS boot AMI] Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

   size=15 # root disk in GB
   volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
     cut -f2)
   instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
   ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
   while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
   sudo mkfs.ext3 -F /dev/sdi
   ebsimage=$dest/ebs
   sudo mkdir $ebsimage
   sudo mount /dev/sdi $ebsimage
   imageroot=$dest/root
   sudo mkdir $imageroot
   sudo mount -oloop $dest/root.img $imageroot
   sudo tar -cSf - -C $imageroot . | sudo tar xvf - -C $ebsimage
   sudo umount $imageroot $ebsimage
   ec2-detach-volume "$volumeid"
   snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
   ec2-delete-volume "$volumeid"
   while ec2-describe-snapshots "$snapshotid" | grep -q pending
     do echo -n .; sleep 1; done
   ec2-register                   \
     --architecture $arch         \
     --name "$prefix"             \
     --description "$description" \
     $ebsopts                     \
     --snapshot "$snapshotid"
   

9. Depending on what you want to keep from the above process, there are various things that you might want to clean up.

   If you no longer want to use an S3 based AMI:

   ec2-deregister $amiid
   ec2-delete-bundle                     \
     --access-key $AWS_ACCESS_KEY_ID     \
     --secret-key $AWS_SECRET_ACCESS_KEY \
     --bucket $bucket                    \
     --prefix $prefix
   

   If you no longer want to use an EBS boot AMI:

   ec2-deregister $amiid
   ec2-delete-snapshot $snapshotid
   

   When you’re done with the original instance:

   ec2-terminate-instance $instanceid
   

In the above instructions I stray a bit from the defaults. For example, I add the runurl package from the Alestic PPA so that it is available for use in user-data scripts on first boot. I enable multiverse for easy access to more software, and I install ec2-ami-tools which works better for me than the current euca2ools.

I also set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance.

Explore and set options as you see fit for your applications. Go wild with the --execscript feature (similar to the ec2ubuntu-build-ami --script option) to customize your image.

The following vmbuilder options do not currently work with creating EC2 images: --mirror, --components, --ppa. I have submitted bug 502490 to track this.

As with much of my work here, I’m simply explaining how to use software that others have spent a lot of energy building. In this case a lot of thanks go to the Ubuntu server team for developing vmbuilder, the EC2 plugin, the ec2-init startup software, and the code which builds the official Ubuntu AMIs; especially Søren Hansen, Scott Moser, and Chuck Short. I also appreciate the folks who reviewed early copies of these instructions and provided feedback including Scott Moser, Art Zemon, Trifon Trifonov, Vaibhav Puranik, and Chris.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

Amazon EC2’s new EBS Boot feature not only provides persistent root disks for instances, but also supports root disks larger than the previous limit of 10GB under S3 based AMIs.

Since EBS boot AMIs are implemented by creating a snapshot, the AMI publisher controls the default size of the root disk through the size of the snapshot. There are a number of factors which go into deciding the default root disk size of an EBS boot AMI and some of them conflict.

On the one hand, you want to give users enough free space to run their applications, but on the other hand, you don’t want to increase the cost of running the instance too much. EBS volumes run $0.10 to $0.11 per GB per month depending on the region, or about $10/month for 100GB and $100/month for 1TB.

I suspect the answer to this problem might be for AMI publishers to provide a reasonable low default, perhaps 10GB as per the old standard or 15GB following in the footsteps of Amazon’s first EBS Boot AMIs. This would add $1.00 to $1.50 per month to running the instance which seems negligible for most purposes. Note: There are also IO charges and charges for EBS snapshots, but those are more affected by usage and less by the size of the original volume.

For applications where the EBS boot AMI’s default size is not sufficient, users can increase the root disk size at run time all the way up to 1 TB. Here’s a quick overview of how to do this.

Example

The following demonstrates how to run Amazon’s getting-started-with-ebs-boot AMI increasing the root disk from the default of 15GB up to 100GB.

Before we start, let’s check to see the default size of the root disk in the target AMI and what the device name is:

$ ec2-describe-images ami-b232d0db
IMAGE   ami-b232d0db    amazon/getting-started-with-ebs-boot    amazon  available   public      i386    machine aki-94c527fd    ari-96c527ff        ebs
BLOCKDEVICEMAPPING  /dev/sda1       snap-a08912c9   15

We can see the EBS snapshot id snap-a08912c9 and the fact that it is 15 GB attached to /dev/sda1. If we start an instance of this AMI it will have a 15 GB EBS volume as the root disk and we won’t be able to change it once it’s running.

Now let’s run the EBS boot AMI, but we’ll override the default size, specifying 100 GB for the root disk device (/dev/sda1 as seen above):

ec2-run-instances                       \
  --key KEYPAIR                         \
  --block-device-mapping /dev/sda1=:100 \
  ami-b232d0db

If we check the EBS volume mapped to the new instance we’ll see that it is 100GB, but when we ssh to the instance and check the root file system size we’ll notice that it is only showing 15 GB:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G  1.6G   13G  12% /

There’s one step left. We need to resize the file system so that it fills up the entire 100 GB EBS volume. Here’s the magic command for ext3. In my early tests it took 2-3 minutes to run:

$ sudo resize2fs /dev/sda1
resize2fs 1.40.4 (31-Dec-2007)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 7
Performing an on-line resize of /dev/sda1 to 26214400 (4k) blocks.
The filesystem on /dev/sda1 is now 26214400 blocks long.

Finally, we can check to make sure that we’re running on a bigger file system:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              99G  1.6G   92G   2% /

Note: The output reflects “99” instead of “100” because of slight differences in how df and EBS calculate “GB” (e.g., 1024 MB vs 1000 MB).

XFS

If it were possible to create an EBS boot AMI with an XFS root file system, then the resizing would be near instantaneous using commands like

sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /

The Ubuntu kernels built for EC2 by Canonical have XFS support built in, so XFS based EBS boot AMIs might be possible. This would also allow for more consistent EBS snapshots.

Toolset

Make sure you are running the latest version of the ec2-run-instances command. The current version can be determined with the command

ec2-version

To use EBS boot features, the version should be at least 1.3-45772.

Updated 2009-12-11: Switch instructions to default us-east-1 since all regions now support this feature.

The ec2-consistent-snapshot software tries its best to flush and lock a MySQL database on an EC2 instance while it initiates the EBS snapshot, and for many environments it does a pretty good job.

However, there are situations where the database may spend time performing crash recovery from the log file when it is started from a copy of the snapshot. We are seeing this behavior at CampusExplorer.com where the database is constantly active and we have innodb_log_file_size set (probably too) high. The delay is doubtless exacerbated by the fact that the blocks on the new EBS volume are being recovered from S3 as it is being built from the snapshot.

Google has created an innodb_disallow_writes MySQL patch which I think points out the problem we may be hitting.

“Note that it is not sufficient to run FLUSH TABLES WITH READ LOCK as there are background IO threads used by InnoDB that may still do IO.”

It would be very nice to have this patch incorporated in MySQL on Ubuntu. It looks like the OurDelta folks have already incorporated the patch. [Update: See rsimmons’ comment below which explains why this particular patch might not be the answer.]

In any case, when we bring up a database using an EBS volume created from an EBS snapshot of an active database, it can take up to 45 minutes recovering before it lets normal clients connect. This is too long for us so we’re trying a new approach.

The ec2-consistent-snapshot now has a --mysql-stop option which shuts down the MySQL server, initiates the snapshot, and then restarts the database. Our hope is that this will get us a snapshot which can be restored and run without delay. If any MySQL experts can point out the potential flaws in this, please do.

Since we obviously can’t stop and start our production database every hour, we are performing this snapshot activity on a replication slave that is dedicated to snapshots and backups.

We continue to perform occasional snapshots on the production database EBS volume just to help keep it reliable per Amazon’s instructions, but we don’t expect to be able to restore it without crash recovery.

If you’d like to test the new --mysql-stop option, please upgrade your ec2-consistent-snapshot package from the Alestic PPA and let me know how it goes.

Amazon Web Services (AWS) has a dizzying proliferation of credentials, keys, ids, usernames, certificates, passwords, and codes which are used to access and control various account and service features and functionality. I have never met an AWS user who, when they started, did not have trouble figuring out which ones to use when and where, much less why.

Amazon is fairly consistent across the documentation and interfaces in the specific terms they use for the different credentials, but nowhere have I found these all listed in one place. (Update: Shlomo pointed out Mitch Garnaat’s article on this topic which, upon examination, may even have been my subconscious inspiration for this. Mitch goes into a lot of great detail in his two part post.)

Pay close attention to the exact names so that you use the right credentials in the right places.

(1) AWS Email Address and (2) Password. This pair is used to log in to your AWS account on the AWS web site. Through this web site you can access and change information about your account including billing information. You can view the account activity. You can control many of the AWS services through the AWS console. And, you can generate and view a number of the other important access keys listed in this article. You may also be able to order products from Amazon.com with this account, so be careful. You should obviously protect your password. What you do with your email address is your business. Both of these values may be changed as needed.

(3) MFA Authentication Code. If you have ordered and activated a multi-factor authentication device, then parts of the AWS site will be protected not only by the email address and password described above, but also by an authentication code. This is a 6 digit code displayed on your device which changes every 30 seconds or so. The AWS web site will prompt you for this code after you successfully enter your email address and password.

(4) AWS Account Number. This is a 12 digit number separated with dashes in the form 1234-5678-9012. You can find your account number under your name on the top right of most pages on the AWS web site (when you are logged in). This number is not secret and may be available to other users in certain circumstances. I don’t know of any situation where you would use the number in this format with dashes, but it is needed to create the next identifier:

(5) AWS User ID. This is a 12 digit number with no dashes. In fact, it is simply the previously mentioned AWS Account Number with the dashes removed (e.g., 12345678912). Your User ID is needed by some API and command line tools, for example when bundling a new image with ec2-bundle-vol. It can also be entered in to the ElasticFox plugin to help display the owners of public AMIs. Again, your User ID does not need to be kept private. It is shown to others when you publish an AMI and make it public, though it might take some detective work to figure out who the number really belongs to if you don’t publicize that, too.

(6) AWS Access Key ID and (7) Secret Access Key. This is the first of two pairs of credentials which can be used to access and control basic AWS services through the API including EC2, S3, SimpleDB, CloudFront, SQS, EMR, RDS, etc. Some interfaces use this pair, and some use the next pair below. Pay close attention to the names requested. The Access Key ID is 20 alpha-numeric characters like 022QF06E7MXBSH9DHM02 and is not secret; it is available to others in some situations. The Secret Access Key is 40 alpha-numeric-slash-plus characters like kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct and must be kept very secret.

You can change your Access Key ID and Secret Access Key if necessary. In fact, Amazon recommends regular rotation of these keys by generating a new pair, switching applications to use the new pair, and deactivating the old pair. If you forget either of these, they are both available from AWS.

(8) X.509 Certificate and (9) Private Key. This is the second pair of credentials that can be used to access the AWS API (SOAP only). The EC2 command line tools generally need these as might certain 3rd party services (assuming you trust them completely). These are also used to perform various tasks for AWS like encrypting and signing new AMIs when you build them. These are the largest credentials, taking taking the form of short text files with long names like cert-OHA6ZEBHMCGZ66CODVHEKKVOCYWISYCS.pem and pk-OHA6ZEBHMCGZ66CODVHEKKVOCYWISYCS.pem respectively.

The Certificate is supposedly not secret, though I haven’t found any reason to publicize it. The Private Key should obviously be kept private. Amazon keeps a copy of the Certificate so they can confirm your requests, but they do not store your Private Key, so don’t lose it after you generate it. Two of these pairs can be associated with your account at any one time, so they can be rotated as often as you rotate the Access Key ID and Secret Access Key. Keep records of all historical keys in case you have a need for them, like unencrypting an old AMI bundle which you created with an old Certificate.

(10) Linux username. When you ssh to a new EC2 instance you need to connect as a user that already exists on that system. For almost all public AMIs, this is the root user, but on Ubuntu AMIs published by Canonical, you need to connect using the ubuntu user. Once you gain access to the system, you can create your own users.

(11) public ssh key and (12) private ssh key. These are often referred to as a keypair in EC2. The ssh keys are used to make sure that only you can access your EC2 instances. When you run an instance, you specify the name of the keypair and the corresponding public key is provided to that instance. When you ssh to the above username on the instance, you specify the private key so the instance can authenticate you and let you in.

You can have multiple ssh keypairs associated with a single AWS account; they are created through the API or with tools like the ec2-add-keypair command. The private key must be protected as anybody with this key can log in to your instances. You generally never see or deal with the public key as EC2 keeps this copy and provides it to the instances. You download the private key and save it when it is generated; Amazon does not keep a record of it.

(13) ssh host key. Just to make things interesting, each EC2 instance which you run will have its own ssh host key. This is a private file generated by the host on first boot which is used to protect your ssh connection to the instance so it cannot be intercepted and read by other people. In order to make sure that your connection is secure, you need to verify that the (14) ssh host key fingerprint, which is provided to you on your first ssh attempt, matches the fingerprint listed in the console output of the EC2 instance.

At this point you may be less or more confused, but here are two rules which may help:

A. Create a file or folder where you jot down and save all of the credentials associated with each AWS account. You don’t want to lose any of these, especially the ones which Amazon does not store for you. Consider encrypting this information with (yet another) secure passphrase.

B. Pay close attention to what credentials are being asked for by different tools. A “secret access key” is different from a “private key” is different from a “private ssh key”. Use the above list to help sort things out.

Use the comment section below to discuss what I missed in this overview.

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

http://alestic.com

The following notes apply to this release:

• The images have been upgraded to use the newest 2.6.21 kernel, ramdisk, and kernel modules from Amazon. This fixes a serious security hole in the previous 2.6.21 kernel.

• The Alestic PPA (personal package archive) has been added to the Ubuntu AMIs. This makes it easy to install software packages listed in this PPA, including ec2-consistent-snapshot.

• The runurl package from the Alestic PPA has been pre-installed on the Ubuntu AMIs. This can be a handy tool for setting up new instances with user-data scripts.

• The EC2 AMI tools have been upgraded to version 1.3-34544.

• The ec2-ami-tools package version has been pinned so it does not get downgraded if the official Ubuntu archives still have older versions.

• All packages have been upgraded to their respective latest versions.

• The Ubuntu Karmic images were not updated and have been removed from the listings at the top of http://alestic.com. If you would like to use Ubuntu Karmic Beta, please test with the AMIs published by Canonical listed a bit lower down on the page.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on http://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

http://alestic.com

The following notes apply to this release:

• The EC2 AMI tools have been upgraded to version 1.3-34544. Note that an “apt-get upgrade” will downgrade the EC2 AMI tools because the versions of the ec2-ami-tools package currently in the Ubuntu Hardy, Intrepid, Jaunty archives are outdated. If you have an easy solution to this, please let us know.

• The ssh host key regeneration has been moved to run after the RNG is seeded. This improves security, especially for folks who are verifying the ssh host key on the first connect by comparing the fingerprint to the value in the instance console output. Thanks to Andrew Becherer for suggesting this improvement.

• The Ubuntu Karmic Alpha images were not updated. Due to a new kernel requirement, Karmic will no longer run on Amazon’s 2.6.21 kernel. Once Canonical releases an Ubuntu kernel with the appropriate features, the Karmic series may be resumed. Running “apt-get upgrade” on an existing Karmic AMI will cause it to be inaccessible after rebooting.

• The Debian Etch desktop images were not updated because apparently LaTeX refuses to build a format from a source file which is more than five years old. If there is anybody who wants Etch desktops and is willing to investigate, please contact me, otherwise I plan to discontinue support for this series. If you’re using Debian, I’d encourage you to upgrade to Lenny “stable” anyway.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

When you start up an instance (server) on Amazon EC2, you need to pick the image or AMI (Amazon Machine Image) to run. This determines the Linux distribution and version as well as the initial software installed and how it is configured.

There are a number of public images to choose from with EC2 including the Ubuntu and Debian image published on http://alestic.com but sometimes it is appropriate to create your own private or public images. There are two primary ways to create an image for EC2:

1. Create an EC2 image from scratch. This process lets you control every detail of what goes into the image and is the easiest way to automate image creation.

2. Rebundle a running EC2 instance into a new image. This approach is the topic of the rest of this article.

After you rebundle a running instance to create a new image, you can then run new EC2 instances of that image. Each instance starts off looking exactly like the original instance as far as the files on the disk go (with a few exceptions).

This guide is primarily written in the context of running Ubuntu on EC2, but the concepts should apply without too much changing on Debian and other Linux distributions.

To use this rebundling approach, you start by running an instance of an image that (1) is as close as possible to the image you want to create, and (2) is published by a source you trust. You then proceed to install software and configure that instance so that it contains exactly what you want to be available on new instances right down to the startup scripts.

The next step is to bundle the instance’s disk image into a new AMI, but before we get to that, it is important to understand a few things about security.

Security

If you are creating a new EC2 image, you need to be very careful what pieces of information you inadvertently leave on the image, especially if you have the goal of publishing it as a public AMI. Anybody who runs an instance of that AMI will have access to the files you included in the bundle, and there is no way to modify an AMI after it has been created (though you can delete it).

For example, you don’t want to leave your AWS certificate or private key on the disk. You’ll even want to clear out the shell history file in case you had typed secret information in commands or in setting environment variables.

You also want to consider the security concerns from the perspective of the people who run the new image. For example, you don’t want to leave any passwords active on accounts. You should also make sure you don’t include your public ssh key in authorized_keys files. Leaving a back door into other people’s servers is in poor taste even if you have no intention of ever using it.

Here are some sample commands, but only you can decide if this wipes out too much or what other files you need to exclude depending on how you set up and used the instance you are bundling:

sudo rm -f /root/.*hist* $HOME/.*hist*
sudo rm -f /var/log/*.gz
sudo find /var/log -name mysql -prune -o -type f -print | 
  while read i; do sudo cp /dev/null $i; done

Whole directories can be excluded from the image using the --exclude option of the ec2-bundle-vol command (see below).

Rebundling

Now we’re ready to bundle the actual EC2 image (AMI). To start, you need to copy your certificate and key to the instance ephemeral storage. Adjust the sample command to use the appropriate keypair file for authentication and the appropriate location of your certification and private key files. If you are not running a modern Ubuntu image, then change remoteuser to “root”.

remotehost=<ec2-instance-hostname>
remoteuser=ubuntu

rsync                           \
  --rsh="ssh -i KEYPAIR.pem"     \
  --rsync-path="sudo rsync"    \
  PATHTOKEYS/{cert,pk}-*.pem    \
  $remoteuser@$remotehost:/mnt/

Set up some environment variables for convenience in the following commands. A single S3 bucket can be used for multiple AMIs. The manifest prefix should be descriptive, especially if you plan to publish the AMI publicly, as it is the only piece of documentation many users will see when they look through AMI lists. At a minimum, I recommend including the Linux distribution (e.g, “ubuntu”), the architecture (e.g., “i386” or “32”), and the date (e.g., “20090621”), as well as some tag that indicates the special nature of the image (e.g., “desktop” or “lamp”).

bucket=<your-bucket-name>
prefix=<descriptive-image-title>

On the EC2 instance itself, you also set up some environment variables to help the bundle and upload commands. You can find these values in your EC2 account.

export AWS_USER_ID=<your-value>
export AWS_ACCESS_KEY_ID=<your-value>
export AWS_SECRET_ACCESS_KEY=<your-value>

if [ $(uname -m) = 'x86_64' ]; then
  arch=x86_64
else
  arch=i386
fi

Bundle the files on the current instance into a copy of the image under /mnt:

sudo -E ec2-bundle-vol           \
  -r $arch                       \
  -d /mnt                        \
  -p $prefix                     \
  -u $AWS_USER_ID                \
  -k /mnt/pk-*.pem               \
  -c /mnt/cert-*.pem             \
  -s 10240                       \
  -e /mnt,/root/.ssh,/home/ubuntu/.ssh

Upload the bundle to a bucket on S3:

ec2-upload-bundle                \
    -b $bucket                   \
    -m /mnt/$prefix.manifest.xml \
    -a $AWS_ACCESS_KEY_ID        \
    -s $AWS_SECRET_ACCESS_KEY

Now that the AMI files have been uploaded to S3, you register the image as a new AMI. This is done back on your local system (with the API tools installed):

ec2-register \
  --name "$bucket/$prefix" \
  $bucket/$prefix.manifest.xml

The output of this command is the new AMI id which is used to run new instances of that image.

It is important to use the same account access information for the ec2-bundle-vol and ec2-register commands even though they are run on different systems. If you don’t you’ll get an error indicating you don’t have the rights to register the image.

Public Images

By default, the new EC2 image is private, which means it can only be seen and run by the user who created it. You can share access with another individual account or with the public.

To let another EC2 user run the image without giving access to the world:

ec2-modify-image-attribute -l -a <other-user-id> <ami-id>

To let all other EC2 users run instances of your image:

ec2-modify-image-attribute -l -a all <ami-id>

Cost

AWS will charge you standard S3 charges for the stored AMI files which comes out to $0.15 per GB per month. Note, however, that the bundling process uses sparse files and compression, so the final storage size is generally very small and your resulting cost may only be pennies per month.

The AMI owner incurs no charge when users run the image in new instances. The users who run the AMI are responsible for the standard hourly instance charges.

Cleanup

Before removing any public image, please consider the impact this might have on people who depend on that image to run their business. Once you publish an AMI, there is no way to tell how many users are regularly creating instances of that AMI and expecting it to stay available. There is also no way to communicate with these users to let them know that the image is going away.

If you decide you want to remove an image anyway, here are the steps to take.

Deregister the AMI

ec2-deregister ami-XXX

Delete the AMI bundle in S3:

ec2-delete-bundle \
  --access-key $AWS_ACCESS_KEY_ID \
  --secret-key $AWS_SECRET_ACCESS_KEY \
  --bucket $bucket \
  --prefix $prefix

[Update 2009-09-12: Security tweak for running under non-root.] [Update 2010-02-01: Update to use latest API/AMI tools and work for Ubuntu 9.10 Karmic.]

Amazon EC2 currently has a limit of 1,000 GB (1 TB) for EBS volumes (Elastic Block Store). It is possible to create file systems larger than this limit using RAID 0 across multiple EBS volumes. Using RAID 0 can also improve the performance of the file system reducing total IO wait as demonstrated in a number of published EBS performance tests.

The following instructions walk through one way to set up RAID 0 across multiple EBS volumes. Note that there is a limit on the size of a file system on 32-bit instances, but 64-bit instances can get unreasonably large. This test was run with 40 EBS volumes of 1,000 GB each for a total of 40,000 GB (40 TB) in the resulting file system.

Actual command line output showing the size of the RAID:

# df /vol
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md0             41942906368      1312 41942905056   1% /vol

# df -h /vol
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               40T  1.3M   40T   1% /vol

These commands can run in less than 10 minutes and this could probably be reduced further by parallelizing the creation and attaching of the EBS volumes.

Note that the default limit is 20 EBS volumes per EC2 account. You can request an increase from Amazon if you need more.

Caution: 40 TB of EBS storage on EC2 will cost $4,000 per month plus usage charges.

Instructions

Start a 64-bit instance (say, Ubuntu 8.04 Hardy from http://alestic.com). Use your own KEYPAIR:

ec2-run-instances                \
  --key KEYPAIR                  \
  --instance-type c1.xlarge      \
  --availability-zone us-east-1a \
  ami-0772946e

Configurable parameters (set on both local host and on EC2 instance):

instanceid=i-XXXXXXXX
volumes=40
size=1000
mountpoint=/vol

On the local host (with EC2 API tools installed)…

Create and attach EBS volumes:

devices=$(perl -e 'for$i("h".."k"){for$j("",1..15){print"/dev/sd$i$j\n"}}'|
           head -$volumes)
devicearray=($devices)
volumeids=
i=1
while [ $i -le $volumes ]; do
  volumeid=$(ec2-create-volume -z us-east-1a --size $size | cut -f2)
  echo "$i: created  $volumeid"
  device=${devicearray[$(($i-1))]}
  ec2-attach-volume -d $device -i $instanceid $volumeid
  volumeids="$volumeids $volumeid"
  let i=i+1
done
echo "volumeids='$volumeids'"

On the EC2 instance (after setting parameters as above)…

Install software:

sudo apt-get update &&
sudo apt-get install -y mdadm xfsprogs

Set up the RAID 0 device:

devices=$(perl -e 'for$i("h".."k"){for$j("",1..15){print"/dev/sd$i$j\n"}}'|
           head -$volumes)

yes | sudo mdadm          \
  --create /dev/md0       \
  --level 0               \
  --metadata=1.1          \
  --raid-devices $volumes \
  $devices

echo DEVICE $devices       | sudo tee    /etc/mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

Create the file system (pick your preferred file system type)

sudo mkfs.xfs /dev/md0

Mount:

echo "/dev/md0 $mountpoint xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir $mountpoint
sudo mount $mountpoint

Check it out:

df -h $mountpoint

When you’re done with it and want to destroy the data and stop paying for storage, tear it down:

sudo umount $mountpoint
sudo mdadm --stop /dev/md0

Terminate the instance:

sudo shutdown -h now

On the local host (with EC2 API tools installed)…

Detach and delete volumes:

for volumeid in $volumeids; do
  ec2-detach-volume $volumeid
done

for volumeid in $volumeids; do
  ec2-delete-volume $volumeid
done

Credits

This article was originally posted on the EC2 Ubuntu group.

Thanks to M. David Peterson for the basic mdadm instructions:

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

http://alestic.com

The following improvements are included in this release:

• Ubuntu 9.04 Jaunty now uses an Ubuntu mirror inside of EC2 hosted by RightScale. This dramatically improves the performance of updates and upgrades. Hardy and Intrepid were already using the mirrors inside EC2.

• The Hardy, Intrepid, and Jaunty images have been enhanced to add failover for Ubuntu archive mirror hosts across availability zones (data centers). This change lets an Ubuntu instance perform package updates and upgrades even if one or two of the EC2 availability zones are completely unavailable.

• The denyhosts package is now installed on desktop images for improved security. The Amazon abuse team has identified the Ubuntu desktop images as a source of compromised systems. The cause for this is believed to be insecure passwords set by users, since the desktop images have PasswordAuthentication enabled by default so that the NX client can connect. The denyhosts package blocks ssh attacks by adding remote systems to /etc/hosts.deny if they keep failing password logins.

  The published Ubuntu and Debian server images continue to have PasswordAuthentication turned off by default for improved security. If you choose to turn this on, I recommend installing a package like denyhosts and using software like the following to generate secure passwords:

  sudo apt-get install pwgen
  pwgen -s 10 1
  

• The EC2 AMI tools have been upgraded to version 1.3-31780.

• All software packages have been updated to versions current as of 2009-06-14.

Community support for Ubuntu on EC2 is available in this group:

http://groups.google.com/group/ec2ubuntu

Community support for Debian on EC2 is available in this group:

http://groups.google.com/group/ec2debian

The 32-bit Debian squeeze images and the 32-bit Debian etch desktop image have not been updated yet due to problems with initial package installation. Images will be released when these issues are resolved.

The following enhancements have been made to the ec2ubuntu-build-ami software which is used to build Ubuntu and Debian images for EC2.

• New --kernel and --ramdisk options have been added to specify AKI and ARI. If you specify a different kernel, you should also specify kernel modules with --package or install them with the --script option.

• Support has been removed for Ubuntu Edgy, Feisty, and Gutsy. These releases have reached their end of life. To improve the clarity of the code this software no longer supports building these images.

• There has been a typo fix for $originaldir for folks who were using the --script option.

• There has been a typo fix for /dev/ptmx though it apparently had no effect given how these images are built.

Thanks to Stephen Parkes and Paul Dowman for submitting patches.

Enjoy!

Elastic IP

Amazon EC2 supports Elastic IP Addresses to implement the effect of having a static IP address for public servers running on EC2. You can point the Elastic IP at any of your EC2 instances, changing the active instance at any time, without changing the IP address seen by the public outside of EC2.

This is a valuable feature for things like web and email servers, especially if you need to replace a failing server or upgrade or downgrade the hardware capabilities of the server, but read on for an insiders’ secret way to use Elastic IP addresses for non-public servers.

Internal Servers

Not all servers should be publicly accessible. For example, you may have an internal EC2 instance which hosts your database server accessed by other application instances inside EC2. You want to architect your installation so that you can replace the database server (instance failure, resizing, etc) but you want to make it easy to get all your application servers to start using the new instance.

There are a number of design approaches which people have used to accomplish this, including:

1. Hard code the internal IP address into the applications and modify it whenever the internal server changes to a new instance (ugh and ouch).

2. Run your own DNS server (or use an external DNS service) and change the IP address of the internal hostname to the new internal IP address (extra work and potentially extra failover time waiting for DNS propagation).

3. Store the internal IP address in something like SimpleDB and change it when you want to point to a new EC2 instance (extra work and requires extra coding for clients to keep checking the SimpleDB mapping)

The following approach is the one I use and is the topic of the rest of this article:

4. Assign an Elastic IP to the internal instance and use the external Elastic IP DNS name. To switch servers, simply re-assign the Elastic IP to a new EC2 instance

This last option uses a little-known feature of the Elastic IP Address system as implemented by Amazon EC2:

When an EC2 instance queries the external DNS name of an Elastic IP, the EC2 DNS server returns the internal IP address of the instance to which the Elastic IP address is currently assigned.

You may need to read that a couple times to grasp the implications as it is non-obvious that an “external” name will return an “internal” address.

Setting Up

You can create an Elastic IP address in an number of ways including the EC2 Console or the EC2 API command line tools. For example:

$ ec2-allocate-address 
ADDRESS 75.101.137.243

The address returned at this point is the external Elastic IP address. You don’t want to use this external IP address directly for internal server access since you would be charged for network traffic.

The next step is to assign the Elastic IP address to an EC2 instance (which is going to be your internal server):

$ ec2-associate-address -i i-07612d6e 75.101.137.243
ADDRESS 75.101.137.243  i-07612d6e

Once the Elastic IP has been assigned to an instance, you can describe that instance to find the external DNS name (which will include the external Elastic IP address in it):

$ ec2-describe-instances i-07612d6e | egrep ^INSTANCE | cut -f4
ec2-75-101-137-243.compute-1.amazonaws.com

This is the permanent external DNS name for that Elastic IP address no matter how many times you change the instance to which it is assigned. If you query this DNS name from outside of EC2, it will resolve to the external IP address as shown above:

$ dig +short ec2-75-101-137-243.compute-1.amazonaws.com
75.101.137.243

However, if you query this DNS name from inside an EC2 instance, it will resolve to the internal IP address for the instance to which it is currently assigned:

$ dig +short ec2-75-101-137-243.compute-1.amazonaws.com
10.254.171.132

You can now use this external DNS name in your applications on EC2 instances to communicate with the server over the internal EC2 network and you won’t be charged for the network traffic as long as you’re in the same EC2 availability zone.

Changing Servers

If you ever need to move the service to a new EC2 instance, simply reassign the Elastic IP address to the new EC2 instance:

$ ec2-associate-address -i i-3b783452 75.101.137.243
ADDRESS 75.101.137.243  i-3b783452

and the original external DNS name will immediately resolve to the internal IP address of the new instance:

$ dig +short ec2-75-101-137-243.compute-1.amazonaws.com
10.254.171.132

Existing connections will fail and new connections to the external DNS name will automatically be opened on the new instance and

Using CNAME

It is not entirely intuitive to have your application use names like ec2-75-101-137-243.compute-1.amazonaws.com but you can make it clearer by creating a permanent entry in your DNS which points to that name with a CNAME alias. For example, using bind:

db.example.com.    CNAME    ec2-75-101-137-243.compute-1.amazonaws.com.

You can then use db.example.com to refer to the server internally and still not have to update your DNS when you change instances.

Further Notes

Even though you are using an Elastic IP address, you don’t need (and often don’t want) to allow external users to be able to access your internal servers. For example, it is just asking for trouble to expose a MySQL server to the Internet. Keep the security groups tight so that the internal servers and services can only be accessed from your other EC2 instances.

Open TCP connections to the original server will not survive when the Elastic IP address is assigned to a new EC2 instance. Some applications and clients will automatically attempt to re-open a failed connection, getting through to the new server on the new internal IP address, but other applications may need to be kicked or signaled so they attempt a new connection to the server.

When using this approach, you need one Elastic IP address for each internal server which needs to be addressed. AWS accounts default to a limit of 5 Elastic IP addresses, but you can request an increased limit.

How do you solve the problem of connecting internal EC2 servers to each other?

Update 2009-07-20: Correct example host name.

Dmitriy Samovskiy discovered that the startup time of an EC2 instance (not the latest boot time) is hidden in the “Last-Modified” header of the EC2 meta-data response. You can only query this from the instance itself, but this should perform better than querying the EC2 API, especially if you tend to use Amazon’s Java command line tools.

For example:

HEAD http://169.254.169.254/latest/meta-data/local-ipv4  | 
  egrep ^Last-Modified: | cut -f2- -d' '

Dmitriy has published a short bash script to calculate the instance run time using this trick:

http://somic.org/2009/06/04/how-long-ago-was-this-ec2-instance-started/

As he points out, this is not documented by AWS, so be careful assuming it will always behave this way.

A few hours ago, Amazon launched a monitoring service for EC2 instances which they are calling CloudWatch. The service costs 1.5 cents per hour per EC2 instance (of any size) which comes out to $10.95 per month for an instance running 24x7.

The concurrently announced Load Balancing and Auto Scaling services are powerful, but I’m not so sure that CloudWatch is going to be useful by itself.

My initial impression on using CloudWatch is that it is hard enough to set up and use that most folks are going to get lost figuring out how to get regular, useful information out of it. Some of this could be alleviated by improved documentation, but I still think the direct, raw usage has a small target audience.

Most users on EC2 should be able to get by with free monitoring packages like munin. Since munin is running on the instance itself, it has access to many more metrics than CloudWatch. Plus it provides pretty graphs which are much easier on the eye than the raw CloudWatch output.

Munin is also trivial to set up on Ubuntu. It takes one command:

sudo apt-get install munin munin-node apache2

Wait 10 minutes for it to start collecting data, then point your browser at http://HOSTNAME/munin

There is a bit more work to do if you want to collect all of your munin data for multiple servers in a central location or to create summary charts combining metrics, but you can get a lot of value from just the above.

Reasons you might end up using CloudWatch include:

1. You are using Amazon’s new EC2 Auto Scaling feature which requires CloudWatch. In this case, you shouldn’t have to worry about the gory details since Auto Scaling will take care of the monitoring for you.

2. You need access to accurate network and disk IO numeric values measured in the same way that Amazon uses to charge you. E.g., you might be running sets of instances for clients and want to pass on EC2 charges to them.

3. You are using a lot of EC2 instances in a large organization and have the time and expertise to implement data collection with CloudWatch for presenting in your own internal reports.

4. You are creating some tools to help other people use the CloudWatch service more easily and with pretty graphs.

On that last point, I think there is an interesting opportunity for somebody to write munin plugins for CloudWatch. It looks like the monitoring data is available on a near-real time basis, and with a bit of state-keeping it should be possible to get graphs which closely represent Amazon’s monitoring records.

I’ve posted some of my feedback from testing CloudWatch on the EC2 forum.

If you’ve had a chance to check out CloudWatch, what is your opinion?

The official Ubuntu images for EC2 do not allow ssh directly to the root account, but instead provide access through a normal “ubuntu” user account. This practice fits the standard Ubuntu security model available in other environments and, admittedly, can take a bit of getting used to if you are not familiar with it.

This document describes how to work inside this environment using the “ubuntu” user and the sudo utility to execute commands as the root user when necessary.

Canonical has released official Ubuntu images for EC2 for Ubuntu 9.10 Karmic.

The primary technical benefit brought by Canonical’s involvement in building official Ubuntu images is that custom kernels can be built for EC2 through a relationship with Amazon. This means that the Ubuntu images can now run on more modern Ubuntu kernels instead of on Amazon’s older, Fedora kernels.

Other differences are listed below:

Items listed are likely to change as images are enhanced. This table may or may not be updated to match. Please leave comments if you notice or question other differences.

Note: There are some older (2009-04) Canonical AMIs floating around for Hardy and Intrepid. These have not been maintained and are not recommended at this point.

Updated 2009-06-15: Alestic.com Jaunty is using an Ubuntu mirror inside EC2. Alestic.com images using load balanced mirror with failover between EC2 availability zones.

Updated 2009-06-25: Alestic.com published Karmic (Alpha) but later withdrew.

Updated 2009-10-29: Canonical released Karmic. None of the image currently have RightScale support built in, but RightScale has their own Ubuntu AMIs.

New updates have been released for all* of the Ubuntu and Debian AMIs listed on:

http://alestic.com

The primary enhancements in this release are:

• The EC2 AMI tools have been upgraded to 1.3-30748. This adds support for EC2 regions including the new eu-west-1 European region.

• AMIs have been created for Ubuntu Jaunty Jackelope alpha (planned for release 2009-04). This is alpha software and is not suitable for production use.

All of the AMIs are available in both the US and European regions.

• The Ubuntu 6.10 Edgy and 7.04 Feisty AMIs are obsolete and unsupported.

New updates have been released for all* of the Ubuntu and Debian AMIs listed on:

http://alestic.com

The primary enhancements in this release are:

• The kernel modules for Amazon’s old 2.6.16 kernel are no longer included in the AMI. The images have been running for a while on Amazon’s 2.6.21 kernel by default and kernel modules are included for this version. If you need the 2.6.16 kernel modules they are available for download or you can build a new AMI with the kernel modules using the ec2-ubuntu-build-ami script on the above site.

• The EC2 AMI tools have been upgraded to 1.3-26357. This has reduced the number of required patches down to one as Amazon continues to improve support for Ubuntu and Debian in these tools.

• A new —arch option has been added based on a patch submitted by Don Spaulding II. This may help in building images for both 32- and 64-bit on the same system.

• Note that the Ubuntu 6.10 Edgy release is marked as obsolete and is no longer being maintained or updated. The Ubuntu 7.04 Feisty AMI was able to be updated it is also past its end of life.

There have been no volunteers for building an automated AMI testing framework so not all of these 24 images have been tested. Please report quickly any problems you encounter and hold off a bit on upgrading to these in production environments.