Alestic.com - A Personal AWS Blog by Eric Hammond

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

As Thilo Maier pointed out in comments on my request for UDS input, I have been publishing both server and desktop AMIs for running Ubuntu on EC2 up through Jaunty, but the official Karmic AMIs on EC2 only support server installations by default.

Ubuntu makes it pretty easy to install the desktop software on a server, and NX from NoMachine makes it pretty easy to access that desktop remotely, with near real-time interactivity even over slowish connections.

Here’s a quick guide to setting this up, starting with an Ubuntu 9.10 Karmic AMI on Amazon EC2:

1. Create a user-data script which installs runurl (not on Karmic AMIs by default) and then runs a simple desktop and NX server installation script. Examine the desktop script to see what it’s doing to install the software.

    cat <<EOM >install-desktop
    #!/bin/bash -ex
    wget -qO/usr/bin/runurl run.alestic.com/runurl
    chmod 755 /usr/bin/runurl
    runurl run.alestic.com/install/desktop
    EOM
   

2. Start an instance on EC2 telling it to run the above user-data script on first boot. The following example uses the current 32-bit Karmic server AMI. Make sure you’re using the latest AMI id.

    ec2-run-instances                   \
      --key YOURKEY                     \
      --user-data-file install-desktop  \
      ami-1515f67c
   

3. Connect to the new instance and wait for it to complete the desktop software installation (when sshd is restarted). This takes about 30 minutes on an m1.small instance and 10 minutes on a c1.medium instance. Then generate and set a secure password for the ubuntu user using copy/paste from the pwgen output. Save the secure password so you can enter it into the NX client later.

    ssh -i YOURKEY.pem ubuntu@THEHOST
    tail -f /var/log/syslog | egrep --line-buffer user-data:
    pwgen -s 16 1
    sudo passwd ubuntu
   

   If anybody knows how to use ssh keys with NX, I’d love to do this instead of using passwords.

4. Back on your local system, install and run the NX client. For computers not running Ubuntu, download the appropriate software from NoMachine.

    wget http://64.34.161.181/download/3.4.0/Linux/nxclient_3.4.0-5_i386.deb
    sudo dpkg -i nxclient_3.4.0-5_i386.deb
    /usr/NX/bin/nxclient --wizard &
   

   Point the NX Client to the external hostname of your EC2 instance. Enter the Login “ubuntu” and the Password from above. Choose the “Gnome” desktop.

If all goes well, you should have a complete and fresh Ubuntu desktop filling most of your screen, available for you to mess around with and then throw away.

ec2-terminate-instances INSTANCEID

If you want to have a persistent desktop with protection from crashes, you’ll need to learn how to do things like placing critical directories on EBS volumes.

If you’d like to run KDE on EC2, replace the package “ubuntu-desktop” with “kubuntu-desktop” in the installation script.

New updates have been released for all of the Ubuntu AMIs listed on:

https://alestic.com

The focus of this update includes enhanced security for ssh host keys, cleaning up the boot process and making it a bit faster, and upgrading desktop AMI software. Specific changes to the AMIs include:

• All Ubuntu packages upgraded to the latest versions as of 2008-04-14.
• Create new ssh host keys on first boot.
• Don’t try to set the hwclock under Xen and save 4 seconds on boot.
• Don’t try to run apparmor as we don’t have the kernel module installed yet.
• Silence grep warnings about missing authorized_keys file on boot.
• Create /tmp earlier in the boot process to avoid warnings.
• Desktop AMIs upgraded to NX Server 3.2.0 (Free Edition).

Additional changes available in the build script include:

• Support for building 64-bit desktop AMIs.
• Add –retry to ec2-upload-bundle.
• AMIs built by the script will not claim they were built by Eric Hammond in /etc/motd :)

Thanks to Thomas Shealy and Hans Omli for many of the above ideas and patches.

The most likely change to cause problems is the generation of new ssh host keys on the first boot. Yesterday’s Debian/Ubuntu ssh key security alert is not directly related, but it did prompt me to reconsider the risk with the current practice of having a single ssh host key for an AMI.

Most public AMIs on EC2 use a fixed ssh host key which means that the entire world can look at it and know what the secret host key is for every instance you start. This spoils the secret and allows man-in- the-middle attacks with no warnings when you ssh to your instance.

Generating a new ssh host key on the first boot solves this problem, but it adds additional complexity in the following case: If you assign a hostname to different instances at different times (dynamic DNS or round robin) then you are more likely to get warnings from ssh about the host key changing when connecting to that hostname.

If you rebundle one of these AMIs, the ssh host key will be shared among new instances of that AMI unless you do the following before bundling:

ln -s ../init.d/ec2-ssh-host-key-gen /etc/rcS.d/S50ec2-ssh-host-key-gen

Bonus for reading this far: I occasionally hang out on IRC channel ##ec2 so swing by and chat sometime.

irc://irc.freenode.net/##ec2

The “desktop” images listed on https://alestic.com allow you to run a full Ubuntu desktop on an Amazon EC2 instance and access the desktop using NoMachine’s NX client from any Linux, Mac, or Windows computer.

WARNING! If you don’t absolutely need a desktop, please use one of the “server” AMIs instead. The desktop AMIs are many times larger and take many times longer to start up. The server AMIs are sleek, lean, and ready for service.