May 2008 Archives

Though it is not possible to compile and run your own kernels on Amazon EC2, it is possible to compile and run your own kernel modules and bundle them with your EC2 images. The trick is that the kernel modules need to be compiled with exactly the same kernel source and compiler versions as were used to build the original kernel.

The steps below were used to build the updated 2.6.21 kernel modules for the Ubuntu and Debian AMIs listed at

New updates have been released for all of the Ubuntu AMIs listed on:

Though this release is only 3 days after the previous one it is surprisingly not to fix bugs, but rather to add one of the most demanded features: startup hooks.

Thanks to code submitted by Kim Scheibel and Jorge Oliveira (with a little mangling from me) it is now easy to type a single command (or push a button in Elasticfox) and have an Ubuntu instance start up and immediately install, configure, and run software without any additional manual intervention.

Simply pass a script (starting with #!) as the instance user-data and it will be run automatically on the first boot. If you want it to be run on every boot, see the comments at the top of this file:

For example, to start a Hardy LAMP server, you could create a script named “install-lamp-server” with the contents:

export DEBIAN_PRIORITY=critical
apt-get update
apt-get upgrade -y
apt-get install -y lamp-server^
echo "Please remember to set the MySQL root password!"

Then using the latest Ubuntu 8.04 Hardy base install AMIID (from run a command like:

ec2-run-instances   --user-data-file install-lamp-server   --key IDENTITY   AMIID

A couple minutes later, you should be able to connect to the server’s external hostname with a web browser. To see the progress of your user-data script on the instance:

tail -f /var/log/syslog

Output from the user-data script is also available in the EC2 instance console output for convenient remote debugging.

You can write the user-data script in any language that happens to be on the base AMI (bash, perl, python, ruby, awk, …) as long as the program file starts with: #!

Note that there is a size limit on user-data in EC2, but the user-data script may download additional files from S3 or other locations, so this shouldn’t be too constraining.

Now let the competition begin for coolest and most useful instance user-data scripts!


New updates have been released for all of the Ubuntu AMIs listed on:

The focus of this update includes enhanced security for ssh host keys, cleaning up the boot process and making it a bit faster, and upgrading desktop AMI software. Specific changes to the AMIs include:

  • All Ubuntu packages upgraded to the latest versions as of 2008-04-14.
  • Create new ssh host keys on first boot.
  • Don’t try to set the hwclock under Xen and save 4 seconds on boot.
  • Don’t try to run apparmor as we don’t have the kernel module installed yet.
  • Silence grep warnings about missing authorized_keys file on boot.
  • Create /tmp earlier in the boot process to avoid warnings.
  • Desktop AMIs upgraded to NX Server 3.2.0 (Free Edition).

Additional changes available in the build script include:

  • Support for building 64-bit desktop AMIs.
  • Add —retry to ec2-upload-bundle.
  • AMIs built by the script will not claim they were built by Eric Hammond in /etc/motd :)

Thanks to Thomas Shealy and Hans Omli for many of the above ideas and patches.

The most likely change to cause problems is the generation of new ssh host keys on the first boot. Yesterday’s Debian/Ubuntu ssh key security alert is not directly related, but it did prompt me to reconsider the risk with the current practice of having a single ssh host key for an AMI.

Most public AMIs on EC2 use a fixed ssh host key which means that the entire world can look at it and know what the secret host key is for every instance you start. This spoils the secret and allows man-in- the-middle attacks with no warnings when you ssh to your instance.

Generating a new ssh host key on the first boot solves this problem, but it adds additional complexity in the following case: If you assign a hostname to different instances at different times (dynamic DNS or round robin) then you are more likely to get warnings from ssh about the host key changing when connecting to that hostname.

If you rebundle one of these AMIs, the ssh host key will be shared among new instances of that AMI unless you do the following before bundling:

ln -s ../init.d/ec2-ssh-host-key-gen /etc/rcS.d/S50ec2-ssh-host-key-gen

Bonus for reading this far: I occasionally hang out on IRC channel ##ec2 so swing by and chat sometime.


Ubuntu AMIs

Ubuntu AMIs for EC2: