August 2011 Archives

[Update: As predicted, these numbers are already out of date and Amazon has added more public IP address ranges for use by EC2 in various regions.]

Each standard Amazon EC2 instance has a public IP address. This is true for normal instances when they are first brought up and for instances which have had elastic IP addresses assigned to them. Your EC2 instance still has a public IP address even if you have configured the security group so that it cannot be contacted from the Internet, which happens to be the default setting for security groups.

Amazon has made public the EC2 IP address ranges that may be in use for each region.

From this information, we can calculate the absolute upper limit for the number of concurrently running standard EC2 instances that could possibly be supported in each region. At the time of this writing I calculate the hard upper limits to be:

EC2 RegionMax Instances*
us-east-1585,704
us-west-198,298
eu-west-1135,156
ap-southeast-143,000
ap-northeast-134,808


*Caveats:

  • An upper limit based on the IP address ranges does not tell you what the real number of possible instances is in a given EC2 region. It’s just an upper limit.

  • Amazon is sure to keep requesting, reserving, and announcing more IP addresses than is actively needed by EC2 at any point in time. Only they know the growth buffer percentage that they like to maintain.

  • Amazon may need to use different ranges of IP addresses for different groups of instances in different parts of their network, even in the same data center or availability zone, so they may publish and start using new ranges of IP addresses even before they are even near using up the capacity of previous ranges.

  • Amazon is free to add new IP address blocks to the list at any time as they keep growing, and they do. The specific numbers above could be out of date by the time you read this.

  • There are probably some IP addresses in each range that are reserved for various networking devices and protocols.

  • This calculation is for concurrently running EC2 instances. Many instances run for just a few minutes or hours and another instance, perhaps for another customer, can start up and use that same IP address moments later.

  • EC2 instances running inside Amazon VPC don’t necessarily use up an external IP address in Amazon’s EC2 public IP address ranges, so they are not included in the upper limits.

  • I am not a networking expert. I only play one at my startup.

Anything else I’m missing?

[Update 2012-12-27: Correct URL for Amazon’s latest IP address list.]

I’m taking a class about using Chef with EC2 by Florian Drescher today and Florian mentioned that he noticed one of the four availability zones in us-east-1 is not currently available for starting new instances.

I’ve confirmed this in my own AWS accounts and found that one of the three availability zones in us-west-1 is also unavailable in addition to one of the four availability zones in us-east-1.

Here’s the error I get when I try to start an instance in the availability zone using an old AWS account:

Client.Unsupported: The requested Availability Zone is no longer supported. Please retry your request by not specifying an Availability Zone or choosing us-east-1d, us-east-1a, us-east-1b.

When I use an AWS account I created two days ago, I don’t even see the fourth availability zone at all:

$ ec2-describe-availability-zones --region us-east-1
AVAILABILITYZONE    us-east-1b  available   us-east-1   
AVAILABILITYZONE    us-east-1c  available   us-east-1   
AVAILABILITYZONE    us-east-1d  available   us-east-1

The exact name of the unavailable availability zones will vary between EC2 accounts. You can read more about that here:

Matching EC2 Availability Zones Across AWS Accounts

The availability zones that are unavailable in my AWS accounts map to the following identifiers using the method described in the above article:

us-east-1x ceb6a579-757c-474b-b09b-52c84b605767
us-west-1x e5a2ff3b-79b4-4217-8c93-ebf1d633dd6e

If my guess is correct based on old accounts I have, I believe these may be the oldest (original) availability zones in their respective regions.

Has there been any communication from Amazon about unsupported availability zones? Is this temporary or permanent? When I searched Google for the above error, I got back one result in Japanese and it appears to be somebody asking what the error is.

No longer supporting an availability zone in EC2 is something that Amazon is allowed to do under the EC2 SLA, especially with the way that they seem to phase them out. The SLA does not kick in until two availability zones are completely unavailable and “unavailable” includes your existing instances having no external connectivity. This is one reason we try to architect services with the ability to quickly move resources from one availability zone to another.

I’d love to hear if other people are able to start instances in these availability zones. Please also mention if you already have instances running in those zones.

Update 2011-08-06: According to a post in May from Amazon this seems to be a normal part of how AWS grows in an orderly manner, and if you already have instances running in a zone, you should be able to continue running instances in that zone. It isn’t clear to me how quickly you might lose a zone after your last remaining instance is stopped or terminated, but according to one user it sounds like it might be nearly immediate.

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

They just sent me a notice that all of my old Ubuntu and Debian desktop AMIs from 2008-2010 need to be taken down, but it was a misunderstanding of how the NoMachine NX software implements login security and I’m sure those AMIs just got caught up in their automated sweeps.

I sent an email to help Amazon to help explain why a fixed public ssh key in the AMI with a publicly known “private” ssh key is not a security risk in this instance (see command=”/usr/NX/bin/nxserver —login” in the authorized_keys2 file) but they might be sending out notices to users of these AMIs before they get my response.

You can read up on how NX authentication works securely here:

http://www.nomachine.com/ar/view.php?ar_id=AR02C00150

It’s the middle of the night for me, so I’ll publish more later, but I wanted to get the world out just in case there is alarm.

Note: There may be other reasons you should not run some of those AMIs, e.g., if they are of a release that is past end of life, but they don’t have open back doors that let me access them.

I’ve written about building AMIs securely recently and have been preaching this kind of stuff with EC2 for years.

Ubuntu AMIs

Ubuntu AMIs for EC2:


AWS Jobs

AWS Jobs