April 2011 Archives

As steady as clockwork, Ubuntu 11.04 Natty is released on the day scheduled at least eleven months ago; and thanks to Canonical, tested AMIs for Natty are already published for use on Amazon EC2.

The Ubuntu AMI table at the top of Alestic.com always reflects the latest official Ubuntu AMIs from Canonical and now includes Ubuntu 11.04 Natty.

Click on the EC2 region tab (e.g., us-west-1) to see the list of AMI ids. AMIs are listed for both 32-bit and 64-bit architectures (depending on what instance type you want to run) and for both EBS boot and instance-store. I strongly recommend EBS boot unless you have a specific situation that requires instance-store and you really know what you’re doing. EBS boot has many advantages over instance-store that you may not be aware of until after you need them.

As part of this release, I have added launch buttons to the right of each AMI id on Alestic.com. If you click on one of these buttons you will be taken to the AWS console and dropped directly into the process of launching an EC2 instance of that AMI. Thanks to Scott Moser for pointing out that AWS had implemented support for this cool feature—and for building the Ubuntu AMIs themselves :-)

I’ve been running Ubuntu Natty on my laptop for a couple weeks with good success and look forward to trying out this latest release of Ubuntu on EC2 as well.

This article is a followup to Matching EC2 Availability Zones Across AWS Accounts written back in 2009. Please read that article first in order to understand any of what I write here.

Since I wrote that article, Amazon has apparently changed the reserved instance offering ids at least once. I haven’t been tracking this, so I don’t know if this was a one time thing or if the offering ids change on a regular basis.

Interestingly, the offering ids still seem to match across accounts and still map to different availability zones, so perhaps they can still be used to map the true, underlying availability zones as the original article proposes.

To document for posterity and comparison, here are my 2009 availability zone offering ids as calculated by the procedure in the above mentioned article:

“Blue” Account (2009):

eu-west-1a 649fd0c8-75d4-4e16-88c7-1ddb83f66062
eu-west-1b 438012d3-0440-480a-9f5c-eb7e55dd5a37
us-east-1a 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1b 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1c c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

“Red” Account (2009):

eu-west-1a 438012d3-0440-480a-9f5c-eb7e55dd5a37
eu-west-1b 649fd0c8-75d4-4e16-88c7-1ddb83f66062
us-east-1a c48ab04c-c057-457e-a4d8-a0f172f4db2d
us-east-1b 4b2293b4-1e6c-4eb3-ab74-4493c0e57987
us-east-1c 60dcfab3-a56c-4092-8c90-3677e9da02b7
us-east-1d c48ab04c-7e96-4ea8-9579-d62194490546

And here are the new, 2011-05-25 availability zone offering ids.

“Blue” Account (2011):

eu-west-1a c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
eu-west-1b d586503b-7025-44e8-8487-09907b6b0e7e
us-east-1a 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1b 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1c ceb6a579-757c-474b-b09b-52c84b605767
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

“Red” Account (2011):

eu-west-1a d586503b-7025-44e8-8487-09907b6b0e7e
eu-west-1b c48ab04c-0bd0-4be9-8db5-a4bad61c6c58
us-east-1a ceb6a579-757c-474b-b09b-52c84b605767
us-east-1b 438012d3-80c7-42c6-9396-a209c58607f9
us-east-1c 60dcfab3-06bb-4b68-9503-53bf89823b5e
us-east-1d 649fd0c8-5d76-4881-a522-fe5224c10fcc

Note: I have excluded the regions and availability zones that Amazon has added since 2009 (quite a few).

Though all of the offering ids have changed since 2009, it appears that the availability zone mappings between these two accounts has remained the same across the years. This is a small sampling and I still think that Amazon could change this, especially for zones where no resources are being used (instances running, EBS volumes created), so don’t depend on it remaining so.

Please let me know if you do any further experimentation or future tracking of these ids. There’s not a ton of practical application, but I still find this interesting.

Amazon designs availability zones so that it is extremely unlikely that a single failure will take out multiple zones at once. Unfortunately, whatever happened last night seemed to cause problems in all us-east-1 zones for one of my accounts.

Of the 20+ full time EC2 instances that I am responsible for (including my startup and my personal servers) only one instance was affected. As it turns out it was the one that hosts Alestic.com along with some other services that are important to me personally.

Here are some steps I took in response to the outage:

  1. My instance was responding to pings, but no other services were responding. I tried a reboot through the EC2 API, but that did not seem to take effect as the pings never stopped responding.

  2. I then tried to stop/start the instance as this is an easy way to move an EBS boot instance to new hardware. Unfortunately, the instance stayed in a “stopping” state (for what turned out to be at least 7 hours).

  3. After the stop did not take effect in 20 minutes, I posted on the EC2 forum and noticed that a slew of others were having similar issues.

  4. I was regularly checking the AWS status page and within a short time, Amazon posted that they were having issues.

  5. With an instance wedged, my next move was to bring up a new replacement instance. This is fairly easy for me with complete documented steps and automated scripts that I have used in the past in similar situations.

    Unfortunately, my attempts to bring up new EBS boot instances failed in all four availability zones in us-east-1. In three of the zones the new instances went from pending to terminated. In the fourth zone, I got the insufficient capacity error for the c1.medium size.

    At this point, I realized that though my disaster recovery strategy for this particular server did span multiple availability zones, it did not extend beyond a single EC2 region. My databases are backed up outside of EC2, so I won’t lose key data, but there are other files that I would have to recreate in order for all systems to operate smoothly.

  6. I decided to trust Amazon to bring things back faster than I could rebuild everything in a new region and went to sleep for a few hours.

  7. Shortly after I woke up, I was able to start a new instance in a different availability zone. To move the services from the wedged instance to the new instance, I:

    a. Created snapshots of the EBS volumes on the wedged instance.

    b. Created new volumes in the new availability zone based on those snapshots.

    c. Stopped the new EBS boot instance.

    d. Detached the EBS boot volume from the new instance.

    e. Attached the new EBS volumes (created from the snapshots) to the new instance.

    f. Started the new EBS boot instance.

  8. Sure ‘nuff. The new instance looked and ran exactly like the old, wedged instance and I was able to assign my Elastic IP Address to the new instance and move on with my day.

Notes

  • Since AWS does not post their status updates to Twitter (that I’ve found), I follow ylastic. Ylastic sends out updates on Twitter as soon as Amazon updates the status page. In fact, Ylastic often sends out problem alerts before Amazon updates their status page!

  • Though I got many errors in trying to run API commands, I kept trying and some requests got through so that I was able to accomplish my recovery well before Amazon announced that things were getting better.

  • Judgment Day = 2011-04-21 (in some timelines)

Lessons learned

I’m happy with AWS and my current strategies except for one weakness: I need to keep backups of everything required to bring up my servers in a different EC2 region, not just a different availability zone. This would also work for moving to a different provider if that became necessary.

At my company, Campus Explorer, we have implemented this higher level of disaster recovery and have even tested it with a fire drill. We were able to recreate our entire service on a different hosting provider using only off-AWS backups in about 24 hours of work. Fortunately for my sleep last night, none of our company services were affected in today’s EC2 outage.

Alestic Git Server (alpha testing)

I’m working on making it easy to start a centralized Git server with an unlimited number of private Git repositories and unlimited users under your control running on an Amazon EC2 instance. I need people who can help test and provide feedback so I can improve the experience and capabilities.

Background

I’ve used a number of different services to host open source software including Launchpad.net, Google Code, and GitHub. Recently, however, I found the need to host a number of different private repositories and I decided to go with the flow and use Git as the software to manage the repositories.

After investigating a dozen Git private repository hosting options, I decided that they were too limiting for my purposes: number of repositories, number of users, size of the repositories, and/or cost. I also had an urge to keep a bit more control over my own repositories by managing them on my own server[1].

The commercial Git hosting services are going to be a good choice for many users, especially if you are hosting open source software or need the additional web-based features, but there are some folks like me who may want to run and manage our own private Git repositories.

As I started investigating and building my private Git server, I had to do research to find out what software to use to manage what users had access to repositories and how to configure the server and software. I’ve decided to share the results so far in case this helps others who have similar needs for running centralized Git servers.

The Project

The first iteration of the Alestic Git Server is an AMI (available for all architectures in all regions). When you run an instance of this AMI, it includes the basic software required to run a centralized Git server (Ubuntu, Git, gitosis) and you have control over adding new repositories, adding new users, and managing users’ access to the repositories.

Please read more on the project page:

Alestic Git Server

The Request

At this point, the project is in its early stages. What is there works for my basic needs, but I’d like to get feedback on things like:

  • Is it easy to use?

  • At what point in the documentation do new users get stuck?

  • What additional features, functionality, software would make sense to add?

Instead of posting comments on the Alestic.com blog, please join the Alestic Git Google group and kick off or join interactive discussions there.

[1] It has been pointed out to me that running a system on Amazon EC2 isn’t quite “my own server” and that theoretically Amazon has access to my data as much as a Git hosting service would have access to my private repositories. As with any technology project, each individual and company need to make their own judgements about who they trust with what.

Ubuntu AMIs

Ubuntu AMIs for EC2:


AWS Jobs

AWS Jobs