The source for ec2-conssitent-snapshot has historically been available here:
For your convenience, it is now also available here:
You are welcome to fork ec2-consistent snapshot under the liberal terms of the Apache License, Version 2.0.
I welcome patch submissions, especially if:
EBS boot vs. instance-store
If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store and accept my apology that I just mentioned it. Once you are completely comfortable with using EBS boot instances on EC2, you may (or may not) want to come back here and read why you made a good decision.
EC2 experts may find that there are specific cases, few and far between, where instance-store might make sense, but they don’t attempt to use instance-store without understanding and accounting for all the serious drawbacks and dangers that go with making this choice. For example, experts using instance-store don’t mind losing all of the data on the instance as they have designed the system so that the data is stored elsewhere and so that a new instance can easily and automatically be rebuilt from scratch.
One of the challenges for beginners is that many of the benefits of EBS boot don’t necessarily seem like something you’ll need to use right away. Then they get down the road and into situations where they realize that they would have been much better off if they had gone with EBS boot in the first place and may find it takes some work to make the transition.
Here are some of the reasons I use and recommend EBS boot instances. None of these benefits are available with instance-store, so even a single one of these can be an overriding factor for choosing EBS boot.
A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as it demonstrates a few handy features of EC2.
The basic need is to get the public ssh key from a keypair that exists inside of EC2. You don’t have access to the private key at the moment (but somebody else does or you will at a different location).
The AWS console and EC2 API do not let you ask for the public ssh key associated with a keypair. However, EC2 does pass the public ssh key to a new EC2 instance when you run it with a specific keypair.
The problem is that we don’t currently have the private key, so we can’t log in to the EC2 instance to get the public key. (Besides, if we did have the private key, we could extract the public key from it directly.)
I proposed creating a user-data script that sends the public ssh key to the EC2 instance console output. You can retrieve the console output without logging in to the EC2 instance.
Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time?
Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance?
Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing).
In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minimum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job.
Here’s a high level overview for folks familiar with Auto Scaling:
Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking up unwanted space in my pocket for two years.
Multi-factor authentication means that in order to log in to my AWS account using the AWS console or portal (including the AWS forums) you not only need my secret password, you also need access to a device that I carry around with me.
Before, this was a physical device attached to my key ring. Now, this is my smart phone which has the virtual (software) MFA device on it. I already carry my phone with me, so the software doesn’t take up any additional space.
To log in to AWS, I enter my password and then the current 6 digit access code displayed by the Android app on my phone. These digits change every 30 seconds in an unguessable pattern, so this enhances the security of my AWS account.
Canonical has released updated instance-store AMIs for Ubuntu 8.04 LTS Hardy on Amazon EC2. Read Ben Howard’s announcement on the ec2ubuntu Google group.
New AMIs have been released for the Alestic Git Server. Major upgrade points include:
The Amazon EC2 Forum has been around since the beginning of EC2 and has always been a place where you can get your EC2 questions in front of an audience of experts and Amazon employees.
Though I’m still listed as one of the top posters (scored by questioners marking my answers as helpful) I’ve slowed down my activity on that forum due to the sheer volume of support requests, many of which are commonly asked questions or things that only Amazon can help with.
When you reboot a physical computer at your desk it is very similar to shutting down the system, and booting it back up. With Amazon EC2, rebooting an instance is much the same as with a local physical computer, but a stop/start differs in a few keys ways that may cause some problems and definitely have some benefits.
When you stop an EBS boot instance you are giving up the physical hardware that the server was running on and EC2 is free to start somebody else’s instance there.
Your EBS boot volume (and other attached EBS volumes) are still preserved, though they aren’t really tied to a physical or virtual server. They are just associated with an instance id that isn’t running anywhere.
When you start the instance again, EC2 picks some hardware to run it on, ties in the EBS volume(s) and boots it up again.
Things that change when you stop/start include:
[Update: As predicted, these numbers are already out of date and Amazon has added more public IP address ranges for use by EC2 in various regions.]
Each standard Amazon EC2 instance has a public IP address. This is true for normal instances when they are first brought up and for instances which have had elastic IP addresses assigned to them. Your EC2 instance still has a public IP address even if you have configured the security group so that it cannot be contacted from the Internet, which happens to be the default setting for security groups.
Amazon has made public the EC2 IP address ranges that may be in use for each region.
From this information, we can calculate the absolute upper limit for the number of concurrently running standard EC2 instances that could possibly be supported in each region. At the time of this writing I calculate the hard upper limits to be:
I’m taking a class about using Chef with EC2 by Florian Drescher today and Florian mentioned that he noticed one of the four availability zones in us-east-1 is not currently available for starting new instances.
I’ve confirmed this in my own AWS accounts and found that one of the three availability zones in us-west-1 is also unavailable in addition to one of the four availability zones in us-east-1.
Here’s the error I get when I try to start an instance in the availability zone using an old AWS account:
Client.Unsupported: The requested Availability Zone is no longer supported. Please retry your request by not specifying an Availability Zone or choosing us-east-1d, us-east-1a, us-east-1b.
When I use an AWS account I created two days ago, I don’t even see the fourth availability zone at all:
Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.
These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.
As always, hats off to the hard working folks at AWS and thank for all the great products and services.
Original message:
If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.
For folks still using the old, reliable Ubuntu 8.04 LTS Hardy from 2008, Canonical has released updated AMIs for use on Amazon EC2. Read Scott Moser’s announcement on the ec2ubuntu Google group.
Though Canonical publishes both EBS boot and instance-store for recent Ubuntu releases, they only publish instance-store AMIs for the older Ubuntu 8.04, so…
Amazon published a tutorial about best practices in creating public AMIs for use on EC2 last week:
Though the general principles put forth in the tutorial are good, some of the specifics are flawed in how to accomplish those principles. (Comments here relate to the article update from June 7, 2011 3:45 AM GMT.)
The primary message of the article is that you should not publish private information on a public AMI. Excellent advice!
Unfortunately, the article seems to recommend or at least to assume that you are building the public AMI by taking a snapshot of a running instance. Though this method seems an easy way to build an AMI and is fine for private AMIs, it is is a dangerous approach for public AMIs because of how difficult it is to identify private information and to clear that private information from a running system in such a way that it does not leak into the public AMI.
As steady as clockwork, Ubuntu 11.04 Natty is released on the day scheduled at least eleven months ago; and thanks to Canonical, tested AMIs for Natty are already published for use on Amazon EC2.