Alestic.com - A Personal AWS Blog by Eric Hammond

Update: Since this article was written, Amazon has released the ability to copy EBS boot AMIs between regions using the web console, command line, and API. You may still find information of use in this article, but Amazon has solved some of the harder parts for you.

Using Amazon EC2, you created an EBS boot AMI and it’s working fine, but now you want to run instances of that AMI in a different EC2 region. Since AMIs are region specific, you need a copy of the image in each region where instances are required.

This article presents one method you can use to copy an EBS boot from one EC2 region to another.

Ubuntu 9.04 Jaunty has reached EOL (End Of Life). It is no longer supported by Ubuntu with security updates and patches. You have known this day was coming for 1.5 years, as all non-LTS Ubuntu releases are supported for only 18 months.

I have no plans to delete the Ubuntu 9.04 Jaunty AMIs for EC2 published under the Alestic name in the foreseeable future, but I request, recommend, and urge you to please stop using them and upgrade to an officially supported, active, kernel-consistent release of Ubuntu on EC2 like 10.04 LTS Lucid or 10.10 Maverick.

Scott Moser and I were wondering if stopping and starting an EBS boot instance on EC2 would begin a new hour’s worth of charges or if AWS would not increase your costs if the stop/start were done a few minutes apart in the same hour.

For some reason, I had assumed that it would start a new hour of fees, possibly because of my experience with the somewhat unrelated terminating old instances and starting new instances. However, we decided it would be easy to test, so here are the results.

I tested with an Ubuntu 10.10 Maverick 32-bit server EBS boot AMI on the m1.small instance type in the ap-southeast-1 (Singapore) region. The AMI should have no effect on charges, so these results should apply to any OS you run on EC2.

I used an AWS account that did not have any EC2 instance fees in the Singapore region that month (Scott’s idea) so that this activity would be easy to see as the only charges on that account.

[Update 2015-06-16: Upgrade to latest aws-cli command syntax]

Amazon recently launched the ability to upload your own ssh public key to EC2 so that it can be passed to new instances when they are launched. Prior to this you always had to use an ssh keypair that was generated by Amazon.

The benefits of using your own ssh key include:

• Amazon never sees the private part of the ssh key (though they promise they do not save a copy after you downloaded it and we all trust them with this)

• The private part of the ssh key is never transmitted over the network (though it always goes over an encrypted connection and we mostly trust this)

• You can now upload the same public ssh key to all EC2 regions, so you no longer have to keep track of a separate ssh key for each region.

• You can use your default personal ssh key with brand new EC2 instances, so you no longer have to remember to specify options like -i EC2KEYPAIR in every ssh, scp, rsync command.

If you haven’t yet created an ssh key for your local system, it can be done with the command:

Does anybody really need me to tell them that you can now run a copy of the newly released Ubuntu 10.10 Maverick on Amazon EC2 with official AMIs published by Canonical?

Or, by now, perhaps you have come to expect–like I have–that the smoothly oiled machine will naturally pump out Ubuntu AMIs for EC2 on the same pre-scheduled date that the larger Ubuntu machine churns out yet another smooth launch of yet another clean Ubuntu release.

The bigger question, I guess, might be:

Should I upgrade to Ubuntu 10.10 on EC2?

ec2-consistent-snapshot version 0.35 has been released on the Alestic PPA. This software is a wrapper around the EBS create-snapshot API call and can be used to help ensure that the file system and any MySQL database on the EBS volume are in a consistent state, suitable for restoring at a later time.

The most important change in this release is a fix for a defect that has been nagging many folks for months. In rare situations, the create-snapshot API call itself took longer than 10 seconds to return from the EC2 web service at Amazon. The software did not trap the alarm correctly and exited without unfreezing the XFS file system which forced us to add an awkward unfreeze command in all cron jobs.

The Ubuntu 10.04 Lucid AMis for Amazon EC2 dated 20100923 have a known bug which causes the mountall process to spin CPU when the instance is rebooted.

You can observe this by starting a Lucid instance, running sudo reboot, and then running top after reconnecting.

Cpu(s): 38.5%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 61.5%st
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 49 root      20   0  4128 1180  920 R 38.6  0.1   0:08.57 mountall           

Canonical has released an updated series of Ubuntu AMIs for EC2. When starting new EC2 instances, you should use the latest AMI ids to pick up kernel security fixes. If you have Ubuntu 10.04 running on a t1.micro instance type, you should at least upgrade the software packages to get the patch for the rebooting issue:

I frequently fire up a temporary Ubuntu server on Amazon EC2 to test out some package feature, installation process, or other capability where I’m willing to pay a few pennies for a clean install and spare CPU.

I occasionally forget that I started an instance and leave it running for longer than I intended, turning my decision to spend ten cents into a cost of dollars. In one case, I ended up paying several hundred dollars for a super-sized instance I forgot I had running. Yes, ouch.

Because of this pain, I have a habit now of pre-scheduling the termination of my temporary instances immediately after creating them. I used to do this on the instance itself with a command like:

echo "sudo halt" | at now + 55 min

However, this only terminates the instance if its root disk is instance-store (S3 based AMI). I generally run EBS boot instances now, and a shutdown or halt only “stops” an EBS boot instance by default which leaves you paying for the EBS boot volume at, say, $1.50/month.

So, my common practice these days is to pre-schedule an instance termination call, generally from my local laptop, using a command like:

echo "ec2kill i-eb89bb81" | at now + 55 min

The at utility runs the commands on stdin with the exact same environment ($PATH, EC2 keys, current working directory, umask, etc.) as are in the current shell. There are a number of different ways to specify different times for the schedule and little documentation, but it will notify you when it plans to run the commands so you can check it.

After the command is run, at returns the output through an email. This gives you an indication of whether or not the terminate succeeded or if you will need to follow up manually.

Here’s an example email I got from the above at command:

Subject: Output from your job      114
Date: Mon, 20 Sep 2010 14:01:05 -0700 (PDT)

INSTANCE	i-eb89bb81	running	shutting-down

I already have a personal custom command which starts an instance, waits for it to move to running, waits for the ssh server to accept connections, then connects with ssh. I think I’ll add a --temporary option to do this termination scheduling for me as well.

You can get a list of the currently scheduled at jobs with

at -l

You can see the commands in a specific job id with:

at -c [JOBID]

If you decide along the way that the instance should not be temporary and you want to cancel the scheduled termination, you can delete a given at job with a command like:

at -d [JOBID]

I’ve been thinking of writing something simple that would regularly monitor my AWS/EC2 resources (instances, EBS volumes, EBS snapshots, AMIs, etc.) and alert me if it detects patterns that may indicate I am spending money where I may not have intended to.

How do you monitor and clean up temporary resources on Amazon AWS/EC2?

A few hours ago, Amazon launched a public preview of AWS Identity and Access Management (IAM) which is a powerful feature if you have a number of developers who need to access and to manage resources for an AWS account. A unique IAM user can be created for each developer and specific permissions can be doled out as needed.

You can also create IAM users for system functions, dramatically increasing the security of your AWS account in the event a server is compromised. That benefit is the focus of this article using an example frequently cited by EC2 users: Automating EBS snapshots on a local EC2 instance without putting the keys to your AWS kingdom on the file system.

Before the release of AWS IAM, if you wanted to create EBS snapshots in a local cron job on an EC2 instance, you needed to put the master AWS credentials in the file system on that instance. If those AWS credentials were compromised, the attacker could perform all sorts of havoc with resources in your AWS account and charges to your credit card.

With the launch of AWS IAM, we can create a system IAM user with its own AWS keys and all it is allowed to do is… create EBS snapshots! These keys are placed on the instance and used in the snapshot cron job. Now, an attacker can do very little damage with those keys if they are compromised, and we all feel much safer.

The AWS IAM documentation is required reading and a great reference. This article is only intended to serve as a practical introduction to one simple application of IAM.

These instructions assume you are running Ubuntu 10.04 (Lucid) on both your local system and on Amazon EC2. Adjust as appropriate for other distributions and releases.

IAM Installation

Ubuntu does not yet have an official software package for AWS IAM, so we need to download the IAM command line toolkit from Amazon. This can be done on any machine including your local desktop. The IAM command line tools require Java so we need to make sure that is installed as well.

Eventually, you’ll want to install this software somewhere more permanent, but for this demo, we’ll just use it from a subdirectory.

sudo apt-get install openjdk-6-jre unzip
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
wget http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip
unzip IAMCli.zip
export AWS_IAM_HOME=$(echo $(pwd)/IAMCli-*)
export PATH=$PATH:$AWS_IAM_HOME/bin

The AWS IAM tools require you to save your AWS account’s main access key id and AWS secret access key in yet another file format. Create this AWS credential file as, say, $HOME/.aws-credentials-master.txt in the following format (replacing the values with your own credentials):

AWSAccessKeyId=YOURACCESSKEYIDHERE
AWSSecretKey=YOURSECRETKEYHERE

Note: The above is the sample content of a file you are creating, and not shell commands to run.

Protect the above file and set an environment variable to tell IAM where to find it:

export AWS_CREDENTIAL_FILE=$HOME/.aws-credentials-master.txt
chmod 600 $AWS_CREDENTIAL_FILE

We can now use the iam-* command line tools to create and manage AWS IAM users, groups, and policies.

Create IAM User

How you manage your users and groups is sure to be a personal preference that is fine tuned over time, but for the purposes of this demo, I’ll propose that for tracking purposes we put non-human users into a new group named “system”.

iam-groupcreate -g system

Create the snapshotter system user, saving the keys to a file:

user=snapshotter
iam-usercreate -u $user -g system -k |
  tee $HOME/.aws-keys-$user.txt
chmod 600 $HOME/.aws-keys-$user.txt

You will want to have this snapshotter keys file on the EC2 instance, so copy it there:

rsync -Paz $HOME/.aws-keys-$user.txt REMOTEUSER@REMOTESYSTEM:

Allow IAM user snapshotter to create EBS snapshots of any EBS volume:

iam-useraddpolicy \
  -p allow-create-snapshot \
  -e Allow \
  -u $user \
  -a ec2:CreateSnapshot \
  -r '*'

There’s a lot of preparatory and other commands in this article, but take a second to focus on the fact that the core, functional steps are simply the iam-usercreate and iam-useraddpolicy commands above. Two commands and you have a new AWS IAM user with restricted access to your AWS account.

Create EBS Snapshot

For the purposes of this demo, we’ll assume you’re using the ec2-consistent-snapshot tool to create EBS snapshots with a consistent file system and perhaps a consistent MySQL database. (If you’re not using this tool, then you could have simply used ec2-create-snapshot from any computer without having to go through the trouble of creating a new IAM user.)

Make sure you have the latest ec2-consistent-snapshot software installed on the EC2 instance:

sudo add-apt-repository ppa:alestic/ppa
sudo apt-get install ec2-consistent-snapshot

Create the snapshot on the EC2 instance. Adjust options to fit your local EBS volume mount points and MySQL database setup.

sudo ec2-consistent-snapshot \
  --aws-credentials-file $HOME/.aws-keys-snapshotter.txt \
  --xfs-filesystem /YOURMOUNTPOINT \
  YOURVOLUMEID

Follow similar steps to create users and set policies for other system activities you perform on your EC2 instances. IAM can control access to many different AWS resource types, API calls, specific resources, and has even more fine tuned control parameters including time-based restrictions.

The release of AWS Identity and Access Management alleviates one of the biggest concerns security-conscious folks used to have when they started using AWS with a single key that gave complete access and control over all resources. Now the control is entirely in your hands.

Cleanup

If you have followed the steps in this demo and you wish to undo most of what was done, here are some steps for reference.

Delete the IAM user and the IAM group:

iam-userdel -u $user -r
iam-groupdel -g system

Wipe the credentials and keys files and remove the downloaded and unzipped IAM command line toolkit:

sudo apt-get install wipe
wipe  $HOME/.aws-credentials-master.txt \
      $HOME/.aws-keys-$user.txt
rm    IAMCli.zip
rm -r $AWS_IAM_HOME

Make sure to wipe the snapshotter key file on the remote EC2 instance as well.

Support

If you’re looking for help with AWS IAM, there is a new AWS IAM forum dedicated to the topic.

[Update 2010-11-19: Fix path where new zip file is expanded]

As Scott Moser announced, Canonical has released updated AMIs (Amazon Machine Images) for running Ubuntu 10.04 Lucid, Ubuntu 9.10 Karmic, and Ubuntu 8.04 Hardy on Amazon EC2.

The table at the top of Alestic.com has been updated to reflect the new AMI ids. In fact, I have automated this table so that it will update itself without manual intervention whenever new AMIs are released by Canonical, thanks to their AMI query API.

If you’re interested in automatically retrieving AMI ids for your own uses, here’s a sample URL for the released AMI ids for Ubuntu 10.04 Lucid (server) in an easy to parse format:

http://uec-images.ubuntu.com/query/lucid/server/released.current.txt

Other releases can be retrieved from similar URLs, or you can traverse the available results starting at the root:

http://uec-images.ubuntu.com/query/

You can get pointers to the daily AMI builds through this API, but I recommend sticking to the “released.current” lists for the best tested and most stable AMIs. These are the AMI ids that are listed in the table at the top of Alestic.com for easy browsing, along with the legacy AMIs which I published as “Alestic” for older releases not supported by Canonical.

For information on sponsoring Alestic.com, please see:

https://alestic.com/sponsors

A quick but heartfelt note of thanks to Ben Cherian at ServiceCloud, a long term supporter and faithful sponsor of Alestic.com (2 years!). Ben started sponsoring this site, supporting the public Ubuntu AWS/EC2 work I performed, back when this was just a one page site listing the Ubuntu AMI ids on EC2. Alestic.com has multiplied many times in content and organic web traffic over these two years.

Alestic.com is now open to new sponsors. If your company is interested in getting brand exposure at the top of every page on this site dedicated to running Ubuntu on Amazon EC2, please drop me an email.

I like to partner with companies like ServiceCloud and RightScale where the service or product matches the highly targeted reader audience of Alestic.com: folks using or planning to use AWS/EC2 cloud technologies.

BTW, if you are looking for a CIO, consider getting in touch with Ben Cherian who is coming available for new opportunities.

NOTE: Though this method works and there is useful information in this article about things you can do with EBS boot instances, there is a simpler way to move an instance to new hardware.

Amazon EC2 has been experiencing some power issues in a portion of one of their many data centers. Even though the relative percentage of people affected might be small, when you have as many customers as AWS does, a small fraction can still be a large absolute number of customers who are affected.

Naturally, some customers will be upset about not having access to their systems, but in the time it takes to write a complaint, you might be able to move your server to new hardware within EC2 and go on with your business

First, lets assume that you are running an EBS boot instance. If you didn’t think they were the way to go before reading this article, I expect to convince you with this one example (and there are a number of other benefits).

Setup

For this demo, I’m going to start an instance. In your situation this instance represents the currently running server that you are depending on and which has valuable software, configuration, and perhaps data on its EBS volume(s). I’m also going to drop a note on the EBS root disk on my running instance so that I know it is the one I wanted to preserve. Again, this is just setting things up for the demo:

# SKIP THIS ENTIRE SECTION IF YOU ALREADY HAVE AN INSTANCE RUNNING
keypair=YOURKEYPAIR
sshkey=.ssh/$keypair.pem # or wherever you keep it
region=us-west-1 # pick your region
zone=${region}a # pick your availability zone
type=m1.small # pick your size
amiid=ami-cb97c68e # Ubuntu 10.04 Lucid, 32-bit, EBS boot in that region
oldinstanceid=$(ec2-run-instances \
  --key $keypair \
  --region $region \
  --availability-zone $zone \
  --instance-type $type \
  $amiid |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$oldinstanceid"

while host=$(ec2-describe-instances --region $region "$oldinstanceid" | 
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

echo "save the volume" | ssh -i $sshkey ubuntu@$host tee README.txt

Moving to a New Instance

Now, we pretend that the above created instance has failed in some way and we can no longer access it. Here’s how we get our server running on a new instance:

1. If you can, stop the old instance. This increases your chance of keeping the file system consistent on the EBS volume. If the instance has really failed and this step does not work, then skip it.

    ec2-stop-instances --region $region $oldinstanceid
   

2. Run a new instance with the same startup parameters as your old instance.

    newinstanceid=$(ec2-run-instances \
      --key $keypair \
      --region $region \
      --availability-zone $zone \
      --instance-type $type \
      $amiid |
      egrep ^INSTANCE | cut -f2)
    echo "newinstanceid=$newinstanceid"
   

3. Wait until the new instance is running and then stop (not terminate) the new instance and detach the EBS boot volume from it. Delete the volume as it has nothing of importance, having just been created.

    ec2-stop-instances --region $region $newinstanceid
    newebsroot=$(ec2-describe-instances --region $region $newinstanceid |
      grep ^BLOCKDEVICE | grep /dev/sda1 | cut -f3)
    ec2-detach-volume --force --region $region $newebsroot
    ec2-delete-volume --region $region $newebsroot
   

4. Detach the old (valuable) EBS root volume from the old (broken) instance.

    oldebsroot=$(ec2-describe-instances --region $region $oldinstanceid |
      grep ^BLOCKDEVICE | grep /dev/sda1 | cut -f3)
    ec2-detach-volume --force --region $region $oldebsroot
   

5. Attach the old (valuable) EBS root volume to the new (stopped) instance.

    ec2-attach-volume \
      --region $region \
      -d /dev/sda1 \
      -i $newinstanceid \
      $oldebsroot
   

   If you had multiple EBS volumes attached to the old instance, you would move each one over in a similar manner.

6. Restart the new instance which is now going to boot with the original volume.

    ec2-start-instances --region $region $newinstanceid
   

Voila! You have moved your server from an old, perhaps broken instance to new (or at least different) hardware keeping the same file system, and it took only a few minutes! If you’d like, ssh to the new instance and make sure that your valuable information is still there.

If you had an Elastic IP address associated with the old instance, you would move it to the new instance.

Cleanup

You may terminate the old instance if you are comfortable that you won’t need it any more. If you were following this demo as an exercise, you should also terminate the new instance. Since you manually attached the old volume to the new instance yourself, it will not be deleted automatically when the instance is terminated. You can modify the instance attributes to change the delete-on-termination flag for the volume or simply delete it manually.

# BEWARE! Don't copy these blindly, but think about what you should do
ec2-terminate-instances --region $region $oldinstanceid
ec2-terminate-instances --region $region $newinstanceid
ec2-delete-volume --region $region $oldebsroot

Tips

This above process can also be used when your instance is running fine, but you want to move to a different instance type (size) of the same architecture. For example, you could move from m1.small up to c1.medium, or from m2.4xlarge down to c1.xlarge. Update: I wasn’t thinking clearly when I wrote that last sentence. It is possible to change the instance type much more easily: Simply stop the instance, use ec2-modify-instance-attribute, and start it up again. Read more about this method.

You can also resize the root disk of a running EC2 instance using the same basic principle of swapping out an EBS root volume on a running instance.

[Update 2011-02-07: Point out simpler method to move to new hardware.]
[Update 2012-01-08: Link to article on changing instance type.]

On Friday evening (2 days ago) I and a number of other high octane LA individuals came to LA StartupWeekend to participate in the process of building the beginning of a startup in 2 days. The group of eighty participants divided up organically by interest into about ten different ideas which had been pitched that very evening.

I ended up in a great team of 9 developers, designers, and product folks who were all interested in building the idea I proposed:

Connect individuals carrying cell phone cameras with people who want timely photographs at a particular location.

Less than 48 hours later, we launched the prototype which you can check out now at:

http://CrowdPhoto.net

And, we had a lot of fun in the process.

I mention this on the Alestic.com blog because our team used a number of cool technologies including:

• Ubuntu 10.04 Lucid (released one day before we started this project)
• Amazon EC2
• Amazon SimpleDB (we decided to jump on the “NoSQL” buzzword bandwagon)
• Amazon S3 (storage of uploaded photos)
• Amazon CloudFront (CDN to serve the photos)
• Perl 5.10
• Mason (templating)
• Apache (of course)
• HTML, CSS3, JavaScript
• Postfix (automatic processing of incoming photo emails)

Not shown on the site is the fact that two of the developers on the team learned Android development and built an integrated Android application which integrates with the CrowdPhoto.net service. In two days.

We also have prototype integration with Twitter. Feel free to follow us at:

@crowdphoto

Challenges we ran into and resolved included:

• Our first AWS account took a long time to get approved for SimpleDB. Solution: We used SimpleDB from another AWS account (different than the one running the EC2 instance).

• We quickly ran into the EC2 email sending limit just with cron job output and emails from our revision control system. Solution: relay email through one of our personal servers on a non-standard port.

• We found that some SimpleDB requests had too high a latency to run a large number sequentially: Solution: make the requests in parallel (not yet implemented).

Ubuntu 10.04 Lucid worked like a charm.

Take the prototype service for a spin and let us know what you think. Do you think that this type of application has potential? What would you use it for?

Feel free to be harsh on what you see. We are. But, remember that this was built in less than 2 days from concept, to design, to building development infrastructure, to implementation. This is a fun prototyping project.

We don’t know if there is going to be a future for this project as we all go back to our separate real jobs on Monday, but some of us think there might be something really cool here. Give us your ideas.

[Update 2010-05-03: Added images]

Amazon EC2 just launched the Asia Pacific region with data centers in Singapore.

The standard Ubuntu and Debian AMIs (Amazon Machine Images) from Canonical and Alestic are already available in this new region. I have listed the new AMI ids in the table at the top of Alestic.com.

To see the AMIs, simply click on the ap-southeast-1 tab in the table at the top.

If you’re just getting into using Ubuntu on Amazon EC2, consider joining the EC2 Ubuntu group for community support.