Canonical has released updated instance-store AMIs for Ubuntu 8.04 LTS Hardy on Amazon EC2. Read Ben Howard’s announcement on the ec2ubuntu Google group.
Canonical has released updated instance-store AMIs for Ubuntu 8.04 LTS Hardy on Amazon EC2. Read Ben Howard’s announcement on the ec2ubuntu Google group.
For folks still using the old, reliable Ubuntu 8.04 LTS Hardy from 2008, Canonical has released updated AMIs for use on Amazon EC2. Read Scott Moser’s announcement on the ec2ubuntu Google group.
Though Canonical publishes both EBS boot and instance-store for recent Ubuntu releases, they only publish instance-store AMIs for the older Ubuntu 8.04, so…
As steady as clockwork, Ubuntu 11.04 Natty is released on the day scheduled at least eleven months ago; and thanks to Canonical, tested AMIs for Natty are already published for use on Amazon EC2.
Ubuntu 9.04 Jaunty has reached EOL (End Of Life). It is no longer supported by Ubuntu with security updates and patches. You have known this day was coming for 1.5 years, as all non-LTS Ubuntu releases are supported for only 18 months.
I have no plans to delete the Ubuntu 9.04 Jaunty AMIs for EC2 published under the Alestic name in the foreseeable future, but I request, recommend, and urge you to please stop using them and upgrade to an officially supported, active, kernel-consistent release of Ubuntu on EC2 like 10.04 LTS Lucid or 10.10 Maverick.
Canonical has released an updated series of Ubuntu AMIs for EC2. When starting new EC2 instances, you should use the latest AMI ids to pick up kernel security fixes. If you have Ubuntu 10.04 running on a t1.micro instance type, you should at least upgrade the software packages to get the patch for the rebooting issue:
Amazon EC2 just launched the Asia Pacific region with data centers in Singapore.
The standard Ubuntu and Debian AMIs (Amazon Machine Images) from Canonical and Alestic are already available in this new region. I have listed the new AMI ids in the table at the top of Alestic.com.
To see the AMIs, simply click on the ap-southeast-1 tab in the table at the top.
If you’re just getting into using Ubuntu on Amazon EC2, consider joining the EC2 Ubuntu group for community support.
Ubuntu 10.04 Lucid was released on Amazon EC2 right on schedule today along with the rest of the normal Ubuntu image download channels.
Congrats to all the Ubuntu folks who were involved in putting this together! Though I’ve enjoyed contributing to the cause over the years, I’m happy to say that I had very little involvement with this release on EC2, leaving me more time to focus on my startup and family. Though many individuals are involved, I’d like to acknowledge the hard work of Scott Moser who has taken the point for publishing official Ubuntu EC2 images these last couple releases.
This is also the first Ubuntu release on EC2 that includes officially supported EBS boot AMIs, taking yet another task off my plate and providing a trusted source for this useful image type.
I’ve listed all of the current Ubuntu and Debian AMI ids in the table at the top of Alestic.com. Simply click on the EC2 region where you want to run your instances to find the AMI ids.
If you’re running an older release of Ubuntu on EC2, see my recommended upgrade paths.
Note: I do not recommend that new users start with these AMIs. These AMIs run with older versions of Amazon’s Fedora 8 kernel which have some incompatibilities with Ubuntu and Debian (e.g., XFS is broken). My strong recommendation is for all users to convert to one of the Ubuntu 9.10 Karmic or Ubuntu 8.04 Hardy AMIs which run with a modern, compatible Ubuntu kernel.
I have released updates of the following legacy Ubuntu and Debian S3-based AMIs published in the “alestic” buckets:
I welcome testing and feedback from folks who are already using the older versions of these AMIs.
In addition to upgraded software packages, the following enhancements are in this release:
Wait for meta-data before deciding whether to generate ssh host key (Thanks to Dmitry for catching)
Patch from Tom White to support compressed user-data scripts
Allow user-data script to remove the “been run” file so that it is run on every boot instead of just the first
Upgrade EC2 AMI tools to 1.3-34544
Debian: Add Alestic PPA to apt sources
Debian: Install runurl from Alestic PPA
I would like to reiterate that these AMIs are not recommended for anybody except folks who are already using older versions of these AMIs and I would encourage you to upgrade to an Ubuntu 9.10 Karmic AMI or the upcoming Ubuntu 10.04 Lucid AMI which will be released in April. The base Ubuntu image building responsibilities have been transfered to Canonical and I don’t have any plans to release new AMIs in the older Ubuntu or Debian series beyond what I am announcing here.
I have not yet posted the AMI ids on https://alestic.com pending testing and feedback, but here are the ids for those who need to run them:
us-east-1
ami-1116f978 Ubuntu 9.04 Jaunty server 32-bit
ami-e116f988 Ubuntu 9.04 Jaunty server 64-bit
ami-cd16f9a4 Ubuntu 9.04 Jaunty desktop 32-bit
ami-c316f9aa Ubuntu 9.04 Jaunty desktop 64-bit
ami-1316f97a Ubuntu 8.10 Intrepid server 32-bit
ami-e316f98a Ubuntu 8.10 Intrepid server 64-bit
ami-cb16f9a2 Ubuntu 8.10 Intrepid desktop 32-bit
ami-c116f9a8 Ubuntu 8.10 Intrepid desktop 64-bit
ami-e916f980 Ubuntu 8.04 Hardy server 32-bit
ami-e716f98e Ubuntu 8.04 Hardy server 64-bit
ami-1716f97e Ubuntu 8.04 Hardy desktop 32-bit
ami-e516f98c Ubuntu 8.04 Hardy desktop 64-bit
ami-eb16f982 Ubuntu 6.06 Dapper server 32-bit
ami-f916f990 Ubuntu 6.06 Dapper server 64-bit
ami-ed16f984 Debian 5.0 Lenny server 32-bit
ami-fb16f992 Debian 5.0 Lenny server 64-bit
ami-cf16f9a6 Debian 5.0 Lenny desktop 32-bit
ami-c516f9ac Debian 5.0 Lenny desktop 64-bit
ami-ef16f986 Debian 4.0 Etch server 32-bit
ami-fd16f994 Debian 4.0 Etch server 64-bit
us-west-1
ami-197a2b5c Ubuntu 9.04 Jaunty server 32-bit
ami-237a2b66 Ubuntu 9.04 Jaunty server 64-bit
ami-357a2b70 Ubuntu 9.04 Jaunty desktop 32-bit
ami-c97a2b8c Ubuntu 9.04 Jaunty desktop 64-bit
ami-257a2b60 Ubuntu 8.10 Intrepid server 32-bit
ami-2d7a2b68 Ubuntu 8.10 Intrepid server 64-bit
ami-377a2b72 Ubuntu 8.10 Intrepid desktop 32-bit
ami-d57a2b90 Ubuntu 8.10 Intrepid desktop 64-bit
ami-277a2b62 Ubuntu 8.04 Hardy server 32-bit
ami-2f7a2b6a Ubuntu 8.04 Hardy server 64-bit
ami-cf7a2b8a Ubuntu 8.04 Hardy desktop 32-bit
ami-d77a2b92 Ubuntu 8.04 Hardy desktop 64-bit
ami-217a2b64 Ubuntu 6.06 Dapper server 32-bit
ami-e77a2ba2 Ubuntu 6.06 Dapper server 64-bit
ami-d37a2b96 Debian 5.0 Lenny server 32-bit
ami-df7a2b9a Debian 5.0 Lenny server 64-bit
ami-e37a2ba6 Debian 5.0 Lenny desktop 32-bit
ami-db7a2b9e Debian 5.0 Lenny desktop 64-bit
ami-d17a2b94 Debian 4.0 Etch server 32-bit
ami-dd7a2b98 Debian 4.0 Etch server 64-bit
eu-west-1
ami-a798b3d3 Ubuntu 9.04 Jaunty server 32-bit
ami-af98b3db Ubuntu 9.04 Jaunty server 64-bit
ami-9798b3e3 Ubuntu 9.04 Jaunty desktop 32-bit
ami-9d98b3e9 Ubuntu 9.04 Jaunty desktop 64-bit
ami-a198b3d5 Ubuntu 8.10 Intrepid server 32-bit
ami-a998b3dd Ubuntu 8.10 Intrepid server 64-bit
ami-9198b3e5 Ubuntu 8.10 Intrepid desktop 32-bit
ami-9998b3ed Ubuntu 8.10 Intrepid desktop 64-bit
ami-a398b3d7 Ubuntu 8.04 Hardy server 32-bit
ami-ab98b3df Ubuntu 8.04 Hardy server 64-bit
ami-9398b3e7 Ubuntu 8.04 Hardy desktop 32-bit
ami-8798b3f3 Ubuntu 8.04 Hardy desktop 64-bit
ami-ad98b3d9 Ubuntu 6.06 Dapper server 32-bit
ami-9598b3e1 Ubuntu 6.06 Dapper server 64-bit
ami-8398b3f7 Debian 5.0 Lenny server 32-bit
ami-8f98b3fb Debian 5.0 Lenny server 64-bit
ami-8998b3fd Debian 5.0 Lenny desktop 32-bit
ami-8b98b3ff Debian 5.0 Lenny desktop 64-bit
ami-8198b3f5 Debian 4.0 Etch server 32-bit
ami-8d98b3f9 Debian 4.0 Etch server 64-bit
Scott Moser (Canonical) built and released new Ubuntu 8.04.3 LTS Hardy images and AMIs for Amazon EC2. I also published new EBS boot AMIs using the same images. I’ve listed all of the AMI ids on https://alestic.com (pick your region at the top).
These AMIs should work better in the us-west-1 region (apt sources.list) and have updated software packages so upgrades on new instances should be faster.
If you’ve been following along, you probably know that I have been recommending that folks using EC2 switch to the official Ubuntu AMIs published by Canonical (Hardy or Karmic). I have been building and publishing Ubuntu AMIs since 2007 (including Dapper, Edgy, Feisty, Gutsy, Hardy, Intrepid, Karmic), but the last year my focus on this project has been to transition these responsibilities to Canonical who have more time and resources to support the initiative.
I’m happy to say that I’ve finally followed my own advice. For my personal Amazon EC2 servers (including for the Alestic.com web site) I am using Ubuntu 9.10 Karmic images published for EC2 by Canonical.
While I was making the transition, I also switched to EBS boot AMIs. However, since it sounds like Canonical is not planning to publish EBS boot AMIs until Lucid, I decided to continue in service to the community and make available EBS boot AMIs for running Ubuntu on EC2.
I have published EBS boot AMIs for Ubuntu 9.10 Karmic and Ubuntu 8.04 Hardy, both 32- and 64-bit architectures, in all current EC2 regions, for a total of a dozen new AMIs.
I chose to use the exact Ubuntu images which Canonical built for running Ubuntu on EC2. This means that these EBS boot AMIs work exactly the same as the official Canonical AMIs including ssh to the ubuntu user. Again, even though I’m publishing the EBS boot AMIs for Karmic and Hardy, the contents of the image were built by Canonical.
The EBS boot AMIs are listed on Alestic.com. I have restructured the table to better feature Canonical AMIs, and now you need to pick an EC2 region to see the IDs.
Give the EBS boot AMIs a spin and let me know if you run into any issues.
In the last article, I described how to use the vmbuilder software to build an EBS boot AMI from scratch for running Ubuntu on EC2 with a persistent root disk.
In the ec2ubuntu Google group, Scott Moser pointed out that users can take advantage of the Ubuntu images for EC2 that Canonical has already built with vmbuilder. This can simplify and speed up the process of building EBS boot AMIs for the rest of us.
Let’s walk through the steps, creating an EBS boot AMI for Ubuntu 9.10 Karmic.
Run an instance of the Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. Make a note of the resulting instance id:
# 32-bit
instanceid=$(ec2-run-instances \
--key YOURKEYPAIR \
--availability-zone us-east-1a \
ami-1515f67c |
egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"
# 64-bit
instanceid=$(ec2-run-instances \
--key YOURKEYPAIR \
--availability-zone us-east-1a \
--instance-type m1.large \
ami-ab15f6c2 |
egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"
Wait for the instance to move to the “running” state, then note the public hostname:
while host=$(ec2-describe-instances "$instanceid" |
egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host
Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:
rsync \
--rsh="ssh -i YOURKEYPAIR.pem" \
--rsync-path="sudo rsync" \
~/.ec2/{cert,pk}-*.pem \
ubuntu@$host:/mnt/
Connect to the instance:
ssh -i YOURKEYPAIR.pem ubuntu@$host
Install EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:
export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get dist-upgrade -y &&
sudo -E apt-get install -y ec2-api-tools
Set up some parameters:
codename=karmic
release=9.10
tag=server
if [ $(uname -m) = 'x86_64' ]; then
arch=x86_64
arch2=amd64
ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
arch=i386
arch2=i386
ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi
Download and unpack the latest released Ubuntu server image file. This contains the output of vmbuilder as run by Canonical.
imagesource=http://uec-images.ubuntu.com/releases/$codename/release/unpacked/ubuntu-$release-$tag-uec-$arch2.img.tar.gz
image=/mnt/$codename-$tag-uec-$arch2.img
imagedir=/mnt/$codename-uec-$arch2
wget -O- $imagesource |
sudo tar xzf - -C /mnt
sudo mkdir -p $imagedir
sudo mount -o loop $image $imagedir
[OPTIONAL] At this point /mnt/$image contains a mounted filesystem with the complete Ubuntu image as released by Canonical. You can skip this step if you just want an EBS boot AMI which is an exact copy of the released S3 based Ubuntu AMI from Canonical, or you can make any updates, installations, and customizations you’d like to have in your resulting AMI.
In this example, we’ll perform similar steps as the previous tutorial and update the software packages to the latest releases from Ubuntu. Remember that the released EC2 image could be months old.
# Allow network access from chroot environment
sudo cp /etc/resolv.conf $imagedir/etc/
# Fix what I consider to be a bug in vmbuilder
sudo rm -f $imagedir/etc/hostname
# Add multiverse
sudo perl -pi -e 's%(universe)$%$1 multiverse%' \
$imagedir/etc/ec2-init/templates/sources.list.tmpl
# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
sudo chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
# Upgrade the system and install packages
sudo chroot $imagedir mount -t proc none /proc
sudo chroot $imagedir mount -t devpts none /dev/pts
cat <<EOF > $imagedir/usr/sbin/policy-rc.d
#!/bin/sh
exit 101
EOF
chmod 755 $imagedir/usr/sbin/policy-rc.d
DEBIAN_FRONTEND=noninteractive
sudo chroot $imagedir apt-get update &&
sudo -E chroot $imagedir apt-get dist-upgrade -y &&
sudo -E chroot $imagedir apt-get install -y runurl ec2-ami-tools
sudo chroot $imagedir umount /proc
sudo chroot $imagedir umount /dev/pts
rm -f $imagedir/usr/sbin/policy-rc.d
Again, the above step is completely optional and can be skipped to create the EBS boot AMI that Canonical would have published.
Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:
size=15 # root disk in GB
now=$(date +%Y%m%d-%H%M)
prefix=ubuntu-$release-$codename-$tag-$arch-$now
description="Ubuntu $release $codename $tag $arch $now"
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
cut -f2)
instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
sudo mkfs.ext3 -F /dev/sdi
ebsimage=$imagedir-ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage
sudo tar -cSf - -C $imagedir . | sudo tar xvf - -C $ebsimage
sudo umount $ebsimage
ec2-detach-volume "$volumeid"
snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
ec2-delete-volume "$volumeid"
while ec2-describe-snapshots "$snapshotid" | grep -q pending
do echo -n .; sleep 1; done
ec2-register \
--architecture $arch \
--name "$prefix" \
--description "$description" \
$ebsopts \
--snapshot "$snapshotid"
Depending on what you want to keep from the above process, there are various things that you might want to clean up.
If you no longer want to use an EBS boot AMI:
ec2-deregister $amiid
ec2-delete-snapshot $snapshotid
When you’re done with the original instance:
ec2-terminate-instance $instanceid
In this example, I set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance. If Canonical starts publishing EBS boot AMIs, they may or may not choose to make the same choice.
Community feedback, bug reports, and enhancements for these instructions are welcomed.
[Update 2009-01-14: Wrapped upgrade/installs inside of /usr/sbin/policy-rc.d setting to avoid starting daemons in chroot environment.]
[Update 2010-01-22: New location for downloadable Ubuntu images.]
[Update 2010-03-26: Path tweak, thanks to paul.]
Here’s my current recipe for how to build an Ubuntu 9.10 Karmic AMI, either the new EBS boot or the standard S3 based, using the Ubuntu vmbuilder software. The Ubuntu vmbuilder utility replaces ec2ubuntu-build-ami for building EC2 images and it can build images for a number of other virtual machine formats as well.
There is a lot of room for simplification and scripting in the following instructions, but I figured I’d publish what is working now so others can take advantage of the research to date. Happy New Year!
Some sections are marked [For EBS boot AMI] or [For S3 based AMI] and should only be followed when you are building that type of AMI. The rest of the sections apply to either type. It is possible to follow all instructions to build both types of AMIs at the same time.
Run an instance of Ubuntu 9.10 Karmic AMI, either 32-bit or 64-bit depending on which architecture AMI you wish to build. I prefer the c1.* instance types to speed up the builds, but you can get by cheaper with the m1.* instance types. Make a note of the resulting instance id:
# 32-bit
instanceid=$(ec2-run-instances \
--key YOURKEYPAIR \
--availability-zone us-east-1a \
--instance-type c1.medium \
ami-1515f67c |
egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"
# 64-bit
instanceid=$(ec2-run-instances \
--key YOURKEYPAIR \
--availability-zone us-east-1a \
--instance-type c1.xlarge \
ami-ab15f6c2 |
egrep ^INSTANCE | cut -f2)
echo "instanceid=$instanceid"
Wait for the instance to move to the “running” state, then note the public hostname:
while host=$(ec2-describe-instances "$instanceid" |
egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host
Copy your X.509 certificate and private key to the instance. Use the correct locations for your credential files:
rsync \
--rsh="ssh -i YOURKEYPAIR.pem" \
--rsync-path="sudo rsync" \
~/.ec2/{cert,pk}-*.pem \
ubuntu@$host:/mnt/
Connect to the instance:
ssh -i YOURKEYPAIR.pem ubuntu@$host
Install the image building software. We install the python-vm-builder package from Karmic, but we’re going to be using the latest vmbuilder from the development branch in Launchpad because it has good bug fixes. We also use the EC2 API tools from the Ubuntu on EC2 ec2-tools PPA because they are more up to date than the ones in Karmic, letting us register EBS boot AMIs:
export DEBIAN_FRONTEND=noninteractive
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee /etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list &&
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873 &&
sudo apt-get update &&
sudo -E apt-get upgrade -y &&
sudo -E apt-get install -y \
python-vm-builder ec2-ami-tools ec2-api-tools bzr &&
bzr branch lp:vmbuilder
You can ignore the “Launchpad ID” warning from bzr.
Fill in your AWS credentials:
export AWS_USER_ID=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export EC2_CERT=$(echo /mnt/cert-*.pem)
export EC2_PRIVATE_KEY=$(echo /mnt/pk-*.pem)
Set up parameters and create files to be used by the build process. The bucket value is only required for S3 based AMIs:
bucket=...
codename=karmic
release=9.10
tag=server
if [ $(uname -m) = 'x86_64' ]; then
arch=x86_64
arch2=amd64
pkgopts="--addpkg=libc6-i386"
kernelopts="--ec2-kernel=aki-fd15f694 --ec2-ramdisk=ari-c515f6ac"
ebsopts="--kernel=aki-fd15f694 --ramdisk=ari-c515f6ac"
ebsopts="$ebsopts --block-device-mapping /dev/sdb=ephemeral0"
else
arch=i386
arch2=i386
pkgopts=
kernelopts="--ec2-kernel=aki-5f15f636 --ec2-ramdisk=ari-0915f660"
ebsopts="--kernel=aki-5f15f636 --ramdisk=ari-0915f660"
ebsopts="$ebsopts --block-device-mapping /dev/sda2=ephemeral0"
fi
cat > part-i386.txt <<EOM
root 10240 a1
/mnt 1 a2
swap 1024 a3
EOM
cat > part-x86_64.txt <<EOM
root 10240 a1
/mnt 1 b
EOM
Create a script to perform local customizations to the image before it is bundled. This is passed to vmbuilder below using the --execscript option:
cat > setup-server <<'EOM'
#!/bin/bash -ex
imagedir=$1
# fix what I consider to be bugs in vmbuilder
perl -pi -e "s%^127.0.1.1.*\n%%" $imagedir/etc/hosts
rm -f $imagedir/etc/hostname
# Use multiverse
perl -pi -e 's%(universe)$%$1 multiverse%' \
$imagedir/etc/ec2-init/templates/sources.list.tmpl
# Add Alestic PPA for runurl package (handy in user-data scripts)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main" |
tee $imagedir/etc/apt/sources.list.d/alestic-ppa.list
chroot $imagedir \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
# Add ubuntu-on-ec2/ec2-tools PPA for updated ec2-ami-tools
echo "deb http://ppa.launchpad.net/ubuntu-on-ec2/ec2-tools/ubuntu karmic main" |
sudo tee $imagedir/etc/apt/sources.list.d/ubuntu-on-ec2-ec2-tools.list
chroot $imagedir \
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9EE6D873
# Install packages
chroot $imagedir apt-get update
chroot $imagedir apt-get install -y runurl
chroot $imagedir apt-get install -y ec2-ami-tools
EOM
chmod 755 setup-server
Build the image:
now=$(date +%Y%m%d-%H%M)
dest=/mnt/dest-$codename-$now
prefix=ubuntu-$release-$codename-$arch-$tag-$now
description="Ubuntu $release $codename $arch $tag $now"
sudo vmbuilder/vmbuilder xen ubuntu \
--suite=$codename \
--arch=$arch2 \
--dest=$dest \
--tmp=/mnt \
--ec2 \
--ec2-version="$description" \
--manifest=$prefix.manifest \
--lock-user \
--part=part-$arch.txt \
$kernelopts \
$pkgopts \
--execscript ./setup-server \
--debug
[For S3 based AMI] include the following options in the vmbuilder command above. This does not preclude you from also building an EBS boot AMI with the same image. Make a note of the resulting AMI id output by vmbuilder:
--ec2-bundle \
--ec2-upload \
--ec2-register \
--ec2-bucket=$bucket \
--ec2-prefix=$prefix \
--ec2-user=$AWS_USER_ID \
--ec2-cert=$EC2_CERT \
--ec2-key=$EC2_PRIVATE_KEY \
--ec2-access-key=$AWS_ACCESS_KEY_ID \
--ec2-secret-key=$AWS_SECRET_ACCESS_KEY \
[For EBS boot AMI] Copy the image files to a new EBS volume, snapshot it, and register the snapshot as an EBS boot AMI. Make a note of the resulting AMI id:
size=15 # root disk in GB
volumeid=$(ec2-create-volume --size $size --availability-zone us-east-1a |
cut -f2)
instanceid=$(wget -qO- http://instance-data/latest/meta-data/instance-id)
ec2-attach-volume --device /dev/sdi --instance "$instanceid" "$volumeid"
while [ ! -e /dev/sdi ]; do echo -n .; sleep 1; done
sudo mkfs.ext3 -F /dev/sdi
ebsimage=$dest/ebs
sudo mkdir $ebsimage
sudo mount /dev/sdi $ebsimage
imageroot=$dest/root
sudo mkdir $imageroot
sudo mount -oloop $dest/root.img $imageroot
sudo tar -cSf - -C $imageroot . | sudo tar xvf - -C $ebsimage
sudo umount $imageroot $ebsimage
ec2-detach-volume "$volumeid"
snapshotid=$(ec2-create-snapshot "$volumeid" | cut -f2)
ec2-delete-volume "$volumeid"
while ec2-describe-snapshots "$snapshotid" | grep -q pending
do echo -n .; sleep 1; done
ec2-register \
--architecture $arch \
--name "$prefix" \
--description "$description" \
$ebsopts \
--snapshot "$snapshotid"
Depending on what you want to keep from the above process, there are various things that you might want to clean up.
If you no longer want to use an S3 based AMI:
ec2-deregister $amiid
ec2-delete-bundle \
--access-key $AWS_ACCESS_KEY_ID \
--secret-key $AWS_SECRET_ACCESS_KEY \
--bucket $bucket \
--prefix $prefix
If you no longer want to use an EBS boot AMI:
ec2-deregister $amiid
ec2-delete-snapshot $snapshotid
When you’re done with the original instance:
ec2-terminate-instance $instanceid
In the above instructions I stray a bit from the defaults. For example, I add the runurl package from the Alestic PPA so that it is available for use in user-data scripts on first boot. I enable multiverse for easy access to more software, and I install ec2-ami-tools which works better for me than the current euca2ools.
I also set /mnt to the first ephemeral store on the instance even on EBS boot AMIs. This more closely matches the default on the S3 based AMIs, but means that /mnt will not be persistent across a stop/start of an EBS boot instance.
Explore and set options as you see fit for your applications. Go wild with the --execscript feature (similar to the ec2ubuntu-build-ami --script option) to customize your image.
The following vmbuilder options do not currently work with creating EC2 images: --mirror, --components, --ppa. I have submitted bug 502490 to track this.
As with much of my work here, I’m simply explaining how to use software that others have spent a lot of energy building. In this case a lot of thanks go to the Ubuntu server team for developing vmbuilder, the EC2 plugin, the ec2-init startup software, and the code which builds the official Ubuntu AMIs; especially Søren Hansen, Scott Moser, and Chuck Short. I also appreciate the folks who reviewed early copies of these instructions and provided feedback including Scott Moser, Art Zemon, Trifon Trifonov, Vaibhav Puranik, and Chris.
Community feedback, bug reports, and enhancements for these instructions are welcomed.
This article is about running a new EC2 instance with a larger boot volume than the default. You can also resize a running EC2 instance.
Amazon EC2’s new EBS Boot feature not only provides persistent root disks for instances, but also supports root disks larger than the previous limit of 10GB under S3 based AMIs.
Since EBS boot AMIs are implemented by creating a snapshot, the AMI publisher controls the default size of the root disk through the size of the snapshot. There are a number of factors which go into deciding the default root disk size of an EBS boot AMI and some of them conflict.
On the one hand, you want to give users enough free space to run their applications, but on the other hand, you don’t want to increase the cost of running the instance too much. EBS volumes run $0.10 to $0.11 per GB per month depending on the region, or about $10/month for 100GB and $100/month for 1TB.
I suspect the answer to this problem might be for AMI publishers to provide a reasonable low default, perhaps 10GB as per the old standard or 15GB following in the footsteps of Amazon’s first EBS Boot AMIs. This would add $1.00 to $1.50 per month to running the instance which seems negligible for most purposes. Note: There are also IO charges and charges for EBS snapshots, but those are more affected by usage and less by the size of the original volume.
For applications where the EBS boot AMI’s default size is not sufficient, users can increase the root disk size at run time all the way up to 1 TB. Here’s a quick overview of how to do this.
The following demonstrates how to run Amazon’s getting-started-with-ebs-boot AMI increasing the root disk from the default of 15GB up to 100GB.
Before we start, let’s check to see the default size of the root disk in the target AMI and what the device name is:
$ ec2-describe-images ami-b232d0db
IMAGE ami-b232d0db amazon/getting-started-with-ebs-boot amazon available public i386 machine aki-94c527fd ari-96c527ff ebs
BLOCKDEVICEMAPPING /dev/sda1 snap-a08912c9 15
We can see the EBS snapshot id snap-a08912c9 and the fact that it is 15 GB attached to /dev/sda1. If we start an instance of this AMI it will have a 15 GB EBS volume as the root disk and we won’t be able to change it once it’s running.
Now let’s run the EBS boot AMI, but we’ll override the default size, specifying 100 GB for the root disk device (/dev/sda1 as seen above):
ec2-run-instances \
--key KEYPAIR \
--block-device-mapping /dev/sda1=:100 \
ami-b232d0db
If we check the EBS volume mapped to the new instance we’ll see that it is 100GB, but when we ssh to the instance and check the root file system size we’ll notice that it is only showing 15 GB:
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 15G 1.6G 13G 12% /
There’s one step left. We need to resize the file system so that it fills up the entire 100 GB EBS volume. Here’s the magic command for ext3. In my early tests it took 2-3 minutes to run. [Update: For Ubuntu 11.04 and later, this step is performed automatically when the AMI is booted and you don’t need to run it manually.]
$ sudo resize2fs /dev/sda1
resize2fs 1.40.4 (31-Dec-2007)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 7
Performing an on-line resize of /dev/sda1 to 26214400 (4k) blocks.
The filesystem on /dev/sda1 is now 26214400 blocks long.
Finally, we can check to make sure that we’re running on a bigger file system:
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 99G 1.6G 92G 2% /
Note: The output reflects “99” instead of “100” because of slight differences in how df and EBS calculate “GB” (e.g., 1024 MB vs 1000 MB).
If it were possible to create an EBS boot AMI with an XFS root file system, then the resizing would be near instantaneous using commands like the following. [Update: For Ubuntu 11.04 and later, this step is performed automatically when the AMI is booted and you don’t need to run it manually.]
sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /
The Ubuntu kernels built for EC2 by Canonical have XFS support built in, so XFS based EBS boot AMIs might be possible. This would also allow for more consistent EBS snapshots.
Make sure you are running the latest version of the ec2-run-instances command. The current version can be determined with the command
ec2-version
To use EBS boot features, the version should be at least 1.3-45772.
[Updated 2009-12-11: Switch instructions to default us-east-1 since all regions now support this feature.]
[Updated 2011-12-13: Note that file system resize is done automatically on boot in Ubuntu 11.04 and later.]
New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:
The following notes apply to this release:
The images have been upgraded to use the newest 2.6.21 kernel, ramdisk, and kernel modules from Amazon. This fixes a serious security hole in the previous 2.6.21 kernel.
The Alestic PPA (personal package archive) has been added to the Ubuntu AMIs. This makes it easy to install software packages listed in this PPA, including ec2-consistent-snapshot.
The runurl package from the Alestic PPA has been pre-installed on the Ubuntu AMIs. This can be a handy tool for setting up new instances with user-data scripts.
The EC2 AMI tools have been upgraded to version 1.3-34544.
The ec2-ami-tools package version has been pinned so it does not get downgraded if the official Ubuntu archives still have older versions.
All packages have been upgraded to their respective latest versions.
The Ubuntu Karmic images were not updated and have been removed from the listings at the top of https://alestic.com. If you would like to use Ubuntu Karmic Beta, please test with the AMIs published by Canonical listed a bit lower down on the page.
Please give these new images a spin and let us know if you run into any problems.
Enjoy
New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:
The following notes apply to this release:
The EC2 AMI tools have been upgraded to version 1.3-34544. Note that an “apt-get upgrade” will downgrade the EC2 AMI tools because the versions of the ec2-ami-tools package currently in the Ubuntu Hardy, Intrepid, Jaunty archives are outdated. If you have an easy solution to this, please let us know.
The ssh host key regeneration has been moved to run after the RNG is seeded. This improves security, especially for folks who are verifying the ssh host key on the first connect by comparing the fingerprint to the value in the instance console output. Thanks to Andrew Becherer for suggesting this improvement.
The Ubuntu Karmic Alpha images were not updated. Due to a new kernel requirement, Karmic will no longer run on Amazon’s 2.6.21 kernel. Once Canonical releases an Ubuntu kernel with the appropriate features, the Karmic series may be resumed. Running “apt-get upgrade” on an existing Karmic AMI will cause it to be inaccessible after rebooting.
The Debian Etch desktop images were not updated because apparently LaTeX refuses to build a format from a source file which is more than five years old. If there is anybody who wants Etch desktops and is willing to investigate, please contact me, otherwise I plan to discontinue support for this series. If you’re using Debian, I’d encourage you to upgrade to Lenny “stable” anyway.
Please give these new images a spin and let us know if you run into any problems.
Enjoy