Building EBS Boot AMIs Using Canonical's Downloadable EC2 Images

In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.

In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.

Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.

  1. Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:

    # 32-bit
    instanceid=$(ec2-run-instances   \
      --key YOURKEYPAIR              \
      --availability-zone us-east-1a \
      ami-1515f67c |
      egrep ^INSTANCE | cut -f2)
    echo "instanceid=$instanceid"
    
    # 64-bit
    instanceid=$(ec2-run-instances   \
      --key YOURKEYPAIR              \
      --availability-zone us-east-1a \
      --instance-type m1.large       \
      ami-ab15f6c2 |
      egrep ^INSTANCE | cut -f2)
    echo "instanceid=$instanceid"
    

    Wait for the instance to move to the “running” state, then note the public hostname:

    while host=$(ec2-describe-instances "$instanceid" | 
      egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
    echo host=$host
    

    Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:

    rsync                            \
      --rsh="ssh -i YOURKEYPAIR.pem" \
      --rsync-path="sudo rsync"      \
      ~/.ec2/{cert,pk}-*.pem         \
      ubuntu@$host:/mnt/
    

    Connect to the instance:

    ssh -i YOURKEYPAIR.pem ubuntu@$host
    
  2. Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:

    export DEBIAN_FRONTEND=noninteractive
    echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
      sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
    sudo apt-get update &&
    sudo -E apt-get dist-upgrade -y &&
    sudo -E apt-get install -y ec2-api-tools
    
  3. Set up some parameters:

    codename=karmic
    release=9.10
    tag=server
    if [ $(uname -m) = 'x86_64' ]; then
      arch=x86_64
      arch2=amd64
      ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
      ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
    else
      arch=i386
      arch2=i386
      ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
      ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
    fi
    
  4. Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.

    imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
    image=/mnt/$codename-$tag-uec-$arch2.img
    imagedir=/mnt/$codename-uec-$arch2
    wget -O- $imagesource |
      sudo tar xzf - -C /mnt
    sudo mkdir -p $imagedir
    sudo mount -o loop $image $imagedir
    
  5. [OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.

    In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.

    # Allow network access from chroot environment
    sudo cp /etc/resolv.conf $imagedir/etc/
    # Fix what I consider to be a bug in vmbuilder
    sudo rm -f $imagedir/etc/hostname
    # Add multiverse
    sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
      $imagedir/etc/ec2-init/templates/sources.list.tmpl
    # Add Alestic PPA for runurl package (handy in user-data scripts)
    echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
      sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
    sudo chroot $imagedir \
      apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
    # Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
    echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
      sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
    sudo chroot $imagedir \
      apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
    # Upgrade the system and install packages
    sudo chroot $imagedir mount -t proc none /proc
    sudo chroot $imagedir mount -t devpts none /dev/pts
    cat <<EOF > $imagedir/usr/sbin/policy-rc.d
    #!/bin/sh
    exit 101
    EOF
    chmod 755 $imagedir/usr/sbin/policy-rc.d
    DEBIAN_FRONTEND=noninteractive
    sudo chroot $imagedir apt-get update &&
    sudo -E chroot $imagedir apt-get dist-upgrade -y &&
    sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
    sudo chroot $imagedir umount /proc
    sudo chroot $imagedir umount /dev/pts
    rm -f $imagedir/usr/sbin/policy-rc.d
    

    Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.

  6. Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:

    size=15 # root disk in GB
    now=$(date +%Y%m%d-%H%M)
    prefix=ubuntu-$release-$codename-$tag-$arch-$now
    description="Ubuntu $release $codename $tag $arch $now"
    export EC2_CERT=$(echo /mnt/cert-*.pem)
    export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
    volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
      cut -f2)
    instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
    ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
    while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
    sudo mkfs.ext3 -F /dev/sdi
    ebsimage=$imagedir-ebs
    sudo mkdir $ebsimage
    sudo mount /dev/sdi $ebsimage
    sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
    sudo umount $ebsimage
    ec2-detach-volume "$volumeid"
    snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
    ec2-delete-volume "$volumeid"
    while ec2-describe-snapshots "$snapshotid" | grep -q pending
      do echo -n .; sleep 1; done
    ec2-register                   \
      --architecture $arch         \
      --name "$prefix"             \
      --description "$description" \
      $ebsopts                     \
      --snapshot "$snapshotid"
    
  7. Depending on what you want to keep from the above process, there are various things that you might want to clean up.

    If you no longer want to use an EBS boot AMI:

    ec2-deregister $amiid
    ec2-delete-snapshot $snapshotid
    

    When you’re done with the original instance:

    ec2-terminate-instance $instanceid
    

In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.

Community feedback, bug reports, and enhancements for these instructions are welcomed.

[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]

[Update 2010-01-22: New location for downloadable Ubuntu images.]

[Update 2010-03-26: Path tweak, thanks to paul.]