Building EBS Boot AMIs Using Canonical's Downloadable EC2 Images

In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.

In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.

Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.

1. Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:

   # 32-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     ami-1515f67c |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   
   
   # 64-bit
   instanceid=$(ec2-run-instances   \
     --key YOURKEYPAIR              \
     --availability-zone us-east-1a \
     --instance-type m1.large       \
     ami-ab15f6c2 |
     egrep ^INSTANCE | cut -f2)
   echo "instanceid=$instanceid"
   

   Wait for the instance to move to the “running” state, then note the public hostname:

   while host=$(ec2-describe-instances "$instanceid" | 
     egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
   echo host=$host
   

   Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

   rsync                            \
     --rsh="ssh -i YOURKEYPAIR.pem" \
     --rsync-path="sudo rsync"      \
     ~/.ec2/{cert,pk}-*.pem         \
     ubuntu@$host:/mnt/
   

   Connect to the instance:

   ssh -i YOURKEYPAIR.pem ubuntu@$host
   

2. Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

   export DEBIAN_FRONTEND=noninteractive
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
   sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
   sudo apt-get update &&
   sudo -E apt-get dist-upgrade -y &&
   sudo -E apt-get install -y ec2-api-tools
   

3. Set up some parameters:

   codename=karmic
   release=9.10
   tag=server
   if [ $(uname -m) = 'x86_64' ]; then
     arch=x86_64
     arch2=amd64
     ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
     ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
   else
     arch=i386
     arch2=i386
     ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
     ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
   fi
   

4. Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.

   imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
   image=/mnt/$codename-$tag-uec-$arch2.img
   imagedir=/mnt/$codename-uec-$arch2
   wget -O- $imagesource |
     sudo tar xzf - -C /mnt
   sudo mkdir -p $imagedir
   sudo mount -o loop $image $imagedir
   

5. [OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.

   In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.

   # Allow network access from chroot environment
   sudo cp /etc/resolv.conf $imagedir/etc/
   # Fix what I consider to be a bug in vmbuilder
   sudo rm -f $imagedir/etc/hostname
   # Add multiverse
   sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
     $imagedir/etc/ec2-init/templates/sources.list.tmpl
   # Add Alestic PPA for runurl package (handy in user-data scripts)
   echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
     sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
   sudo chroot $imagedir \
     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
   # Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
   echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
     sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
   sudo chroot $imagedir \
     apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
   # Upgrade the system and install packages
   sudo chroot $imagedir mount -t proc none /proc
   sudo chroot $imagedir mount -t devpts none /dev/pts
   cat <<EOF > $imagedir/usr/sbin/policy-rc.d
   #!/bin/sh
   exit 101
   EOF
   chmod 755 $imagedir/usr/sbin/policy-rc.d
   DEBIAN_FRONTEND=noninteractive
   sudo chroot $imagedir apt-get update &&
   sudo -E chroot $imagedir apt-get dist-upgrade -y &&
   sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
   sudo chroot $imagedir umount /proc
   sudo chroot $imagedir umount /dev/pts
   rm -f $imagedir/usr/sbin/policy-rc.d
   

   Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.

6. Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

   size=15 # root disk in GB
   now=$(date +%Y%m%d-%H%M)
   prefix=ubuntu-$release-$codename-$tag-$arch-$now
   description="Ubuntu $release $codename $tag $arch $now"
   export EC2_CERT=$(echo /mnt/cert-*.pem)
   export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
   volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
     cut -f2)
   instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
   ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
   while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
   sudo mkfs.ext3 -F /dev/sdi
   ebsimage=$imagedir-ebs
   sudo mkdir $ebsimage
   sudo mount /dev/sdi $ebsimage
   sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
   sudo umount $ebsimage
   ec2-detach-volume "$volumeid"
   snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
   ec2-delete-volume "$volumeid"
   while ec2-describe-snapshots "$snapshotid" | grep -q pending
     do echo -n .; sleep 1; done
   ec2-register                   \
     --architecture $arch         \
     --name "$prefix"             \
     --description "$description" \
     $ebsopts                     \
     --snapshot "$snapshotid"
   

7. Depending on what you want to keep from the above process, there are various things that you might want to clean up.

   If you no longer want to use an EBS boot AMI:

   ec2-deregister $amiid
   ec2-delete-snapshot $snapshotid
   

   When you’re done with the original instance:

   ec2-terminate-instance $instanceid
   

In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]

[Update 2010-01-22: New location for downloadable Ubuntu images.]

[Update 2010-03-26: Path tweak, thanks to paul.]