May 2010 Archives

NOTE: Though this method works and there is useful information in this article about things you can do with EBS boot instances, there is a simpler way to move an instance to new hardware.

Amazon EC2 has been experiencing some power issues in a portion of one of their many data centers. Even though the relative percentage of people affected might be small, when you have as many customers as AWS does, a small fraction can still be a large absolute number of customers who are affected.

Naturally, some customers will be upset about not having access to their systems, but in the time it takes to write a complaint, you might be able to move your server to new hardware within EC2 and go on with your business

First, lets assume that you are running an EBS boot instance. If you didn’t think they were the way to go before reading this article, I expect to convince you with this one example (and there are a number of other benefits).

Setup

For this demo, I’m going to start an instance. In your situation this instance represents the currently running server that you are depending on and which has valuable software, configuration, and perhaps data on its EBS volume(s). I’m also going to drop a note on the EBS root disk on my running instance so that I know it is the one I wanted to preserve. Again, this is just setting things up for the demo:

# SKIP THIS ENTIRE SECTION IF YOU ALREADY HAVE AN INSTANCE RUNNING
keypair=YOURKEYPAIR
sshkey=.ssh/$keypair.pem # or wherever you keep it
region=us-west-1 # pick your region
zone=${region}a # pick your availability zone
type=m1.small # pick your size
amiid=ami-cb97c68e # Ubuntu 10.04 Lucid, 32-bit, EBS boot in that region
oldinstanceid=$(ec2-run-instances   --key $keypair   --region $region   --availability-zone $zone   --instance-type $type   $amiid |
  egrep ^INSTANCE | cut -f2)
echo "instanceid=$oldinstanceid"

while host=$(ec2-describe-instances --region $region "$oldinstanceid" | 
  egrep ^INSTANCE | cut -f4) && test -z $host; do echo -n .; sleep 1; done
echo host=$host

echo "save the volume" | ssh -i $sshkey ubuntu@$host tee README.txt

Moving to a New Instance

Now, we pretend that the above created instance has failed in some way and we can no longer access it. Here’s how we get our server running on a new instance:

  1. If you can, stop the old instance. This increases your chance of keeping the file system consistent on the EBS volume. If the instance has really failed and this step does not work, then skip it.

    ec2-stop-instances --region $region $oldinstanceid
    
  2. Run a new instance with the same startup parameters as your old instance.

    newinstanceid=$(ec2-run-instances   --key $keypair   --region $region   --availability-zone $zone   --instance-type $type   $amiid |
      egrep ^INSTANCE | cut -f2)
    echo "newinstanceid=$newinstanceid"
    
  3. Wait until the new instance is running and then stop (not terminate) the new instance and detach the EBS boot volume from it. Delete the volume as it has nothing of importance, having just been created.

    ec2-stop-instances --region $region $newinstanceid
    newebsroot=$(ec2-describe-instances --region $region $newinstanceid |
      grep ^BLOCKDEVICE | grep /dev/sda1 | cut -f3)
    ec2-detach-volume --force --region $region $newebsroot
    ec2-delete-volume --region $region $newebsroot
    
  4. Detach the old (valuable) EBS root volume from the old (broken) instance.

    oldebsroot=$(ec2-describe-instances --region $region $oldinstanceid |
      grep ^BLOCKDEVICE | grep /dev/sda1 | cut -f3)
    ec2-detach-volume --force --region $region $oldebsroot
    
  5. Attach the old (valuable) EBS root volume to the new (stopped) instance.

    ec2-attach-volume   --region $region   -d /dev/sda1   -i $newinstanceid   $oldebsroot
    

    If you had multiple EBS volumes attached to the old instance, you would move each one over in a similar manner.

  6. Restart the new instance which is now going to boot with the original volume.

    ec2-start-instances --region $region $newinstanceid
    

Voila! You have moved your server from an old, perhaps broken instance to new (or at least different) hardware keeping the same file system, and it took only a few minutes! If you’d like, ssh to the new instance and make sure that your valuable information is still there.

If you had an Elastic IP address associated with the old instance, you would move it to the new instance.

Cleanup

You may terminate the old instance if you are comfortable that you won’t need it any more. If you were following this demo as an exercise, you should also terminate the new instance. Since you manually attached the old volume to the new instance yourself, it will not be deleted automatically when the instance is terminated. You can modify the instance attributes to change the delete-on-termination flag for the volume or simply delete it manually.

# BEWARE! Don't copy these blindly, but think about what you should do
ec2-terminate-instances --region $region $oldinstanceid
ec2-terminate-instances --region $region $newinstanceid
ec2-delete-volume --region $region $oldebsroot

Tips

This above process can also be used when your instance is running fine, but you want to move to a different instance type (size) of the same architecture. For example, you could move from m1.small up to c1.medium, or from m2.4xlarge down to c1.xlarge. Update: I wasn’t thinking clearly when I wrote that last sentence. It is possible to change the instance type much more easily: Simply stop the instance, use ec2-modify-instance-attribute, and start it up again. Read more about this method.

You can also resize the root disk of a running EC2 instance using the same basic principle of swapping out an EBS root volume on a running instance.

[Update 2011-02-07: Point out simpler method to move to new hardware.]
[Update 2012-01-08: Link to article on changing instance type.]

On Friday evening (2 days ago) I and a number of other high octane LA individuals came to LA StartupWeekend to participate in the process of building the beginning of a startup in 2 days. The group of eighty participants divided up organically by interest into about ten different ideas which had been pitched that very evening.

I ended up in a great team of 9 developers, designers, and product folks who were all interested in building the idea I proposed:

Connect individuals carrying cell phone cameras with people who want timely photographs at a particular location.

Less than 48 hours later, we launched the prototype which you can check out now at:

http://CrowdPhoto.net

And, we had a lot of fun in the process.

I mention this on the Alestic.com blog because our team used a number of cool technologies including:

  • Ubuntu 10.04 Lucid (released one day before we started this project)
  • Amazon EC2
  • Amazon SimpleDB (we decided to jump on the “NoSQL” buzzword bandwagon)
  • Amazon S3 (storage of uploaded photos)
  • Amazon CloudFront (CDN to serve the photos)
  • Perl 5.10
  • Mason (templating)
  • Apache (of course)
  • HTML, CSS3, JavaScript
  • Postfix (automatic processing of incoming photo emails)

Not shown on the site is the fact that two of the developers on the team learned Android development and built an integrated Android application which integrates with the CrowdPhoto.net service. In two days.

We also have prototype integration with Twitter. Feel free to follow us at:

@crowdphoto

Challenges we ran into and resolved included:

  • Our first AWS account took a long time to get approved for SimpleDB. Solution: We used SimpleDB from another AWS account (different than the one running the EC2 instance).

  • We quickly ran into the EC2 email sending limit just with cron job output and emails from our revision control system. Solution: relay email through one of our personal servers on a non-standard port.

  • We found that some SimpleDB requests had too high a latency to run a large number sequentially: Solution: make the requests in parallel (not yet implemented).

Ubuntu 10.04 Lucid worked like a charm.

Take the prototype service for a spin and let us know what you think. Do you think that this type of application has potential? What would you use it for?

Feel free to be harsh on what you see. We are. But, remember that this was built in less than 2 days from concept, to design, to building development infrastructure, to implementation. This is a fun prototyping project.

We don’t know if there is going to be a future for this project as we all go back to our separate real jobs on Monday, but some of us think there might be something really cool here. Give us your ideas.

[Update 2010-05-03: Added images]

Ubuntu AMIs

Ubuntu AMIs for EC2: