Alestic.com - A Personal AWS Blog by Eric Hammond

reduce the risk of losing control of your AWS account by not knowing the root account password

As Amazon states, one of the best practices for using AWS is:

We strongly recommend that you do not use the root user for your everyday tasks, even the administrative ones. Instead, use your root user credentials only to create your IAM admin user. Then securely lock away the root user credentials and use them to perform only a few account and service management tasks.

The root account credentials are the email address and password that you used when you first registered for AWS. These credentials have the ultimate authority to create and delete IAM users, change billing, close the account, and perform all other actions on your AWS account.

You can create a separate IAM user with near-full permissions for use when you need to perform admin tasks, instead of using the AWS root account. If the credentials for the admin IAM user are compromised, you can use the AWS root account to disable those credentials to prevent further harm, and create new credentials for ongoing use.

However, if the credentials for your AWS root account are compromised, the person who stole them can take over complete control of your account, change the associated email address, and lock you out.

I have consulted for companies who lost control over the root AWS account which contained their assets. You want to avoid this.

Proposal

Given:

• The AWS root account is not required for regular use as long as you have created an IAM user with admin privileges

• Amazon recommends not using your AWS root account

• You can’t accidentally expose your AWS root account password if you don’t know it and haven’t saved it anywhere

• You can always reset your AWS root account password as long as you have access to the email address associated with the account

Consider this approach to improving security:

The ssh protocol uses two different keys to keep you secure:

1. The user ssh key is the one we normally think of. This authenticates us to the remote host, proving that we are who we say we are and allowing us to log in.

2. The ssh host key gets less attention, but is also important. This authenticates the remote host to our local computer and proves that the ssh session is encrypted so that nobody can be listening in.

Every time you see a prompt like the following, ssh is checking the host key and asking you to make sure that your session is going to be encrypted securely.

The authenticity of host 'ec2-...' can't be established.
ECDSA key fingerprint is ca:79:72:ea:23:94:5e:f5:f0:b8:c0:5a:17:8c:6f:a8.
Are you sure you want to continue connecting (yes/no)? 

If you answer “yes” without verifying that the remote ssh host key fingerprint is the same, then you are basically saying:

I don’t need this ssh session encrypted. It’s fine for any man-in-the-middle to intercept the communication.

Ouch! (But a lot of people do this.)

Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking up unwanted space in my pocket for two years.

Multi-factor authentication means that in order to log in to my AWS account using the AWS console or portal (including the AWS forums) you not only need my secret password, you also need access to a device that I carry around with me.

Before, this was a physical device attached to my key ring. Now, this is my smart phone which has the virtual (software) MFA device on it. I already carry my phone with me, so the software doesn’t take up any additional space.

To log in to AWS, I enter my password and then the current 6 digit access code displayed by the Android app on my phone. These digits change every 30 seconds in an unguessable pattern, so this enhances the security of my AWS account.

Update 2011-08-04: Amazon Security did more research and investigated the desktop AMIs. They have confirmed that their software incorrectly flagged the AMIs (false positive) and they caught it in time to stop the warning emails from going out to users.

These AMIs include the NX software for remote desktop operation and the way that NX implement login authentication with ssh is convoluted, but secure. I can easily understand why it might have looked like there were potential problems with the AMIs, and I’m glad things turned out well.

As always, hats off to the hard working folks at AWS and thank for all the great products and services.

Original message:

If Amazon AWS/EC2 contacts you with a warning that one of my AMIs you are running contains a back door security hole with ssh keys or user passwords, please don’t be alarmed.

Amazon published a tutorial about best practices in creating public AMIs for use on EC2 last week:

How To Share and Use Public AMIs in A Secure Manner

Though the general principles put forth in the tutorial are good, some of the specifics are flawed in how to accomplish those principles. (Comments here relate to the article update from June 7, 2011 3:45 AM GMT.)

The primary message of the article is that you should not publish private information on a public AMI. Excellent advice!

Unfortunately, the article seems to recommend or at least to assume that you are building the public AMI by taking a snapshot of a running instance. Though this method seems an easy way to build an AMI and is fine for private AMIs, it is is a dangerous approach for public AMIs because of how difficult it is to identify private information and to clear that private information from a running system in such a way that it does not leak into the public AMI.

In a thread on the EC2 forum, Marko describes a situation where an outbound firewall prevents the ability to ssh to port 22, which is the default port on all EC2 instances.

In that thread, Shlomo Swidler proposes creating a user-data script that changes sshd to listen on a port the firewall permits.

Here’s a simple example of a user-data script that does just that. Most outbound firewalls allow traffic to port 80 (web/HTTP), so I use it in this example.

The first step is to create a file containing the user-data script:

[Update 2015-06-16: Upgrade to latest aws-cli command syntax]

Amazon recently launched the ability to upload your own ssh public key to EC2 so that it can be passed to new instances when they are launched. Prior to this you always had to use an ssh keypair that was generated by Amazon.

The benefits of using your own ssh key include:

• Amazon never sees the private part of the ssh key (though they promise they do not save a copy after you downloaded it and we all trust them with this)

• The private part of the ssh key is never transmitted over the network (though it always goes over an encrypted connection and we mostly trust this)

• You can now upload the same public ssh key to all EC2 regions, so you no longer have to keep track of a separate ssh key for each region.

• You can use your default personal ssh key with brand new EC2 instances, so you no longer have to remember to specify options like -i EC2KEYPAIR in every ssh, scp, rsync command.

If you haven’t yet created an ssh key for your local system, it can be done with the command:

Amazon Web Services (AWS) has a dizzying proliferation of credentials, keys, ids, usernames, certificates, passwords, and codes which are used to access and control various account and service features and functionality. I have never met an AWS user who, when they started, did not have trouble figuring out which ones to use when and where, much less why.

Amazon is fairly consistent across the documentation and interfaces in the specific terms they use for the different credentials, but nowhere have I found these all listed in one place. (Update: Shlomo pointed out Mitch Garnaat’s article on this topic which, upon examination, may even have been my subconscious inspiration for this. Mitch goes into a lot of great detail in his two part post.)

Pay close attention to the exact names so that you use the right credentials in the right places.

(1) AWS Email Address and (2) Password. This pair is used to log in to your AWS account on the AWS web site. Through this web site you can access and change information about your account including billing information. You can view the account activity. You can control many of the AWS services through the AWS console. And, you can generate and view a number of the other important access keys listed in this article. You may also be able to order products from Amazon.com with this account, so be careful. You should obviously protect your password. What you do with your email address is your business. Both of these values may be changed as needed.

(3) MFA Authentication Code. If you have ordered and activated a multi-factor authentication device, then parts of the AWS site will be protected not only by the email address and password described above, but also by an authentication code. This is a 6 digit code displayed on your device which changes every 30 seconds or so. The AWS web site will prompt you for this code after you successfully enter your email address and password.

(4) AWS Account Number. This is a 12 digit number separated with dashes in the form 1234-5678-9012. You can find your account number under your name on the top right of most pages on the AWS web site (when you are logged in). This number is not secret and may be available to other users in certain circumstances. I don’t know of any situation where you would use the number in this format with dashes, but it is needed to create the next identifier:

(5) AWS User ID. This is a 12 digit number with no dashes. In fact, it is simply the previously mentioned AWS Account Number with the dashes removed (e.g., 12345678912). Your User ID is needed by some API and command line tools, for example when bundling a new image with ec2-bundle-vol. It can also be entered in to the ElasticFox plugin to help display the owners of public AMIs. Again, your User ID does not need to be kept private. It is shown to others when you publish an AMI and make it public, though it might take some detective work to figure out who the number really belongs to if you don’t publicize that, too.

(6) AWS Access Key ID and (7) Secret Access Key. This is the first of two pairs of credentials which can be used to access and control basic AWS services through the API including EC2, S3, SimpleDB, CloudFront, SQS, EMR, RDS, etc. Some interfaces use this pair, and some use the next pair below. Pay close attention to the names requested. The Access Key ID is 20 alpha-numeric characters like 022QF06E7MXBSH9DHM02 and is not secret; it is available to others in some situations. The Secret Access Key is 40 alpha-numeric-slash-plus characters like kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct and must be kept very secret.

You can change your Access Key ID and Secret Access Key if necessary. In fact, Amazon recommends regular rotation of these keys by generating a new pair, switching applications to use the new pair, and deactivating the old pair. If you forget either of these, they are both available from AWS.

(8) X.509 Certificate and (9) Private Key. This is the second pair of credentials that can be used to access the AWS API (SOAP only). The EC2 command line tools generally need these as might certain 3rd party services (assuming you trust them completely). These are also used to perform various tasks for AWS like encrypting and signing new AMIs when you build them. These are the largest credentials, taking taking the form of short text files with long names like cert-OHA6ZEBHMCGZ66CODVHEKKVOCYWISYCS.pem and pk-OHA6ZEBHMCGZ66CODVHEKKVOCYWISYCS.pem respectively.

The Certificate is supposedly not secret, though I haven’t found any reason to publicize it. The Private Key should obviously be kept private. Amazon keeps a copy of the Certificate so they can confirm your requests, but they do not store your Private Key, so don’t lose it after you generate it. Two of these pairs can be associated with your account at any one time, so they can be rotated as often as you rotate the Access Key ID and Secret Access Key. Keep records of all historical keys in case you have a need for them, like unencrypting an old AMI bundle which you created with an old Certificate.

(10) Linux username. When you ssh to a new EC2 instance you need to connect as a user that already exists on that system. For almost all public AMIs, this is the root user, but on Ubuntu AMIs published by Canonical, you need to connect using the ubuntu user. Once you gain access to the system, you can create your own users.

(11) public ssh key and (12) private ssh key. These are often referred to as a keypair in EC2. The ssh keys are used to make sure that only you can access your EC2 instances. When you run an instance, you specify the name of the keypair and the corresponding public key is provided to that instance. When you ssh to the above username on the instance, you specify the private key so the instance can authenticate you and let you in.

You can have multiple ssh keypairs associated with a single AWS account; they are created through the API or with tools like the ec2-add-keypair command. The private key must be protected as anybody with this key can log in to your instances. You generally never see or deal with the public key as EC2 keeps this copy and provides it to the instances. You download the private key and save it when it is generated; Amazon does not keep a record of it.

(13) ssh host key. Just to make things interesting, each EC2 instance which you run will have its own ssh host key. This is a private file generated by the host on first boot which is used to protect your ssh connection to the instance so it cannot be intercepted and read by other people. In order to make sure that your connection is secure, you need to verify that the (14) ssh host key fingerprint, which is provided to you on your first ssh attempt, matches the fingerprint listed in the console output of the EC2 instance.

At this point you may be less or more confused, but here are two rules which may help:

A. Create a file or folder where you jot down and save all of the credentials associated with each AWS account. You don’t want to lose any of these, especially the ones which Amazon does not store for you. Consider encrypting this information with (yet another) secure passphrase.

B. Pay close attention to what credentials are being asked for by different tools. A “secret access key” is different from a “private key” is different from a “private ssh key”. Use the above list to help sort things out.

Use the comment section below to discuss what I missed in this overview.

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

https://alestic.com

The following notes apply to this release:

• The images have been upgraded to use the newest 2.6.21 kernel, ramdisk, and kernel modules from Amazon. This fixes a serious security hole in the previous 2.6.21 kernel.

• The Alestic PPA (personal package archive) has been added to the Ubuntu AMIs. This makes it easy to install software packages listed in this PPA, including ec2-consistent-snapshot.

• The runurl package from the Alestic PPA has been pre-installed on the Ubuntu AMIs. This can be a handy tool for setting up new instances with user-data scripts.

• The EC2 AMI tools have been upgraded to version 1.3-34544.

• The ec2-ami-tools package version has been pinned so it does not get downgraded if the official Ubuntu archives still have older versions.

• All packages have been upgraded to their respective latest versions.

• The Ubuntu Karmic images were not updated and have been removed from the listings at the top of https://alestic.com. If you would like to use Ubuntu Karmic Beta, please test with the AMIs published by Canonical listed a bit lower down on the page.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

Over the years, Amazon has repeatedly recommended that customers who care about the security of their data should consider encrypting information stored on disks, whether ephemeral storage (/mnt) or EBS volumes. This, even though they take pains to ensure that disk blocks are wiped between uses by different customers, and they implement policies which restrict access to disks even by their own employees.

There are a few levels where encryption can take place:

1. File level. This includes tools like GnuPG, freely available on Ubuntu in the gnupg package. If you use this approach, make sure that you don’t store the unencrypted information on the disk before encrypting it.

2. File system level. This includes useful packages like encfs which transparently encrypt files before saving to disk, presenting the unencrypted contents in a virtual file system. This can even be used on top of an s3fs file system letting you store encrypted data on S3 with ease.

3. Block device level. You can place any file system you’d like on top of the encrypted block interface and neither your application nor your file system realize that the hardware disk never sees unencrypted data.

The rest of this article presents a simple way to set up a level of encryption at the block device level using cryptsetup/LUKS. It has been tested on the 32-bit Ubuntu 9.10 Jaunty server AMI listed on https://alestic.com and should work on other Ubuntu AMIs and even other distros with minor changes.

This walkthrough uses the /mnt ephemeral storage, but you can replace /mnt and /dev/sda2 with appropriate mount point and device for 64-bit instance types or EBS volumes.

Setup

Install tools and kernel modules:

sudo apt-get update
sudo apt-get install -y cryptsetup xfsprogs
for i in sha256 dm_crypt xfs; do 
  sudo modprobe $i
  echo $i | sudo tee -a /etc/modules
done

Before you continue, make sure there is nothing valuable on /mnt because we’re going to replace it!

sudo umount /mnt
sudo chmod 000 /mnt

Encrypt the disk and create your favorite file system on it:

sudo luksformat -t xfs /dev/sda2
sudo cryptsetup luksOpen /dev/sda2 crypt-sda2

Remember your passphrase! It is not recoverable!

Update /etc/fstab and replace the /mnt line (or create a new line for an EBS volume):

fstabentry='/dev/mapper/crypt-sda2 /mnt xfs noauto 0 0'
sudo perl -pi -e "s%^.* /mnt .*%$fstabentry%" /etc/fstab

Mount the file system on the encrypted block device:

sudo mount /mnt

You’re now to free to place files on /mnt knowing that the content will be encrypted before it is written to the hardware disk.

After reboot, /mnt will appear empty until you re-mount the encrypted partition, entering your passphrase:

sudo cryptsetup luksOpen /dev/sda2 crypt-sda2
sudo mount /mnt

Notes

See “man cryptsetup” for info on adding keys and getting information from the LUKS disk header.

It is possible to auto-mount the encrypted disk on reboot if you are willing to put your passphrase in the root partition (almost ruins the point of encryption). See the documentation on crypttab and consider adding a line like:

crypt-sda2 /dev/sda2 /PASSPHRASEFILE luks

Study the cryptsetup documentation carefully so that you understand what is going on. Keeping your data private is important, but it’s also important that you know how to get it back in the case of problems.

This article does not attempt to cover all of the possible security considerations you might need to take into account for data leakage on disks. For example, sensitive information might be stored in /tmp, /etc, or log files on the root disk. If you have swap enabled, anything in memory could be saved in the clear to disk whenever the operating system feels like it.

How do you solve your data security challenges on EC2?

This article was based on a post made on the EC2 Ubuntu group.

Amazon EC2 recently released a feature which lets you share an EBS snapshot so that other accounts can access it. The snapshot can be shared with specific individual accounts or with the public at large.

You should obviously be careful what files you put on a shared EBS snapshot because other people are going to be able to read them. What may not be so obvious to is that you also need to be wary of what files are not currently on the snapshot but once were.

For example, if you copied some files onto the EBS volume, then realized a few contained sensitive information, you might think it’s sufficient to delete the private files and continue on to create a public EBS snapshot of the volume.

The problem with this is that EBS is an elastic block store device, not an interface at the file system level. Any block which was once written to on the block device will be available on the shared EBS volume, even if it is not being used by a visible file on the file system.

Since popular Linux file systems do not generally wipe data when a file is deleted, it is often possible to recover the contents of the deleted files. Even attempting to overwrite a file may, depending on the application, leave the original content available on the disk.

This means any content that touched your EBS volume at any point may still be available to users of your shared EBS snapshot.

To be clear: I do not consider this to be a security flaw in EC2 or EBS. It is merely a security risk for people who do not understand and take precautions against the combination of interactions with file systems, block devices, EBS volumes, and snapshots.

$100 Reward

To demonstrate the security risk, I have created a simple challenge with a tangible reward. Here is a public EBS snapshot:

snap-d53484bc

This EBS snapshot contains two files. The first file is README-1.txt which has nothing sensitive in it but will let you know that you’ve got the right device mounted on your EC2 instance.

The second file created on the source EBS volume contained an Amazon.com gift certificate for $100. I deleted this second file, then took an EBS snapshot of the volume and released it to the public.

The first person who successfully recovers the deleted file on this shared EBS snapshot and enters the gift certificate code into their Amazon.com account will win the $100 prize. Subsequent solvers will get a notice from Amazon that the certificate has already been redeemed, but you still get credit for solving it and helping demonstrate the risks.

Feel free to post a comment on this blog entry if you recovered the deleted file on the shared EBS snapshot. Recipes for doing so are welcomed even if you were not the first. I tested this, so know it’s possible and that the deleted file is still accessible (but I did not redeem the gift certificate, of course).

Good luck!

Help! I can’t connect to my EC2 instance!
Woah! My box just stopped talking to me!
Hey! I can’t access the server!

These and other variations on the connectivity theme are some of the most common problems raised on the Amazon EC2 forum.

The EC2 community and Amazon employees do a valiant job helping users track down and solve these issues despite the facts that (1) there are hundreds of reasons why a server or service might not be accessible, (2) connectivity is one of the harder problems to diagnose, especially without being hands-on, and (3) users complaining about a problem generally don’t provide the clues necessary to solve the issue (because the ones who knew what those clues were probably solved it themselves and didn’t post).

This article is an attempt to provide some general assistance to folks who are experiencing connectivity issues with Amazon EC2. Please post additional help in the comments; this document will be updated over time.

Questions

First off, you should understand that it’s ok to ask for help. When you do, though, you should provide as many details as possible about what you are trying to do and what results you are seeing. It also helps if you drop some clues about your level of expertise. A person using Linux for the first time is likely to make different mistakes on EC2 than a person who is having problems connecting to a custom AMI they built from scratch.

The more specific you can be about your problem and the more information you can provide, the more likely somebody will be able to help. Here are some common questions which are important to have answered for connectivity problems on EC2:

1. When you say you “can’t connect” what application are you trying to use and on what port? For example: “ssh to port 22” or “accessing port 80 with Firefox”. If you don’t know what a port is, then provide as many details as possible about the application you’re using and what command or steps you are taking to initiate the connection.

2. What, specifically, happens when you try to connect? Does it hang for a long time and eventually time out? Do you get an error message? What is the exact text you see? (Copy and paste, don’t summarize.)

3. What is the AMI id which the instance is running? If it is not a public AMI, then what is the AMI id of the public AMI it is based on?

4. What Linux distro and release is the instance running? E.g., Ubuntu 9.04 Jaunty, Debian etch, Fedora 8, CentOS 5.

5. What is the instance id of the instance you are trying to contact? Providing this can let Amazon employees take a look at the internals of what might be going on.

6. What are the internal and external IP addresses and/or host names for the instance you are trying to reach? Providing this information is, in effect, giving permission to the community to try to contact your server over the network so that they can gather information about connectivity and help solve your problem.

7. Have you ever been able to contact this instance in the past? How recently?

8. How long has the instance been running?

9. Have you ever been able to contact another instance of the same AMI?

10. Is there a difference in connectivity when you try from another EC2 instance instead of from the Internet?

11. What were you doing when the connectivity stopped?

12. What is the console output of the instance? You can get this through an API client or a command like:

     ec2-get-console-output INSTANCE_ID
    

There are so many reasons that connectivity might be down to a remote server or service that it would be impossible to get a significant percentage of them listed in one article. I’ll start by listing some of the more common problems here; please add to the comments as you run into or remember others.

You

By far the most common cause of the problem is you (the person experiencing the problem) and that’s ok. We all make mistakes. It’s important, though, that you start with this attitude: open to the possibilities that you typed something wrong, forgot a step, or didn’t quite understand the complex instructions. Ninety percent of the people reading this paragraph think I’m talking to somebody else; oddly, they also think this sentence is not about them.

Here are some of the most common reasons folks (including me) can’t connect to their Amazon EC2 instance. Really.

• You’re not connecting to the right instance or to the instance you think you’re trying to connect to. Servers on EC2 are identified by opaque instance ids like i-ae1df2c6 and opaque host names like ec2-75-101-182-20.compute-1.amazonaws.com. It’s easy for anybody to get these confused or mistype them.

• The instance you’re trying to connect to has not completed the boot process yet. Though some AMIs are ready to connect in under a minute, others can take 10+ minutes.

• The instance you’re trying to connect to has been terminated. (Did you just shut down what you thought was a different instance?)

• The service you are trying to reach on the instance is not running on that instance.

• The service you are trying to reach on the instance is not listening on that port or that network interface.

• You did not open the port in the security group.

• You did not start the instance with the correct security group.

• You did not start the instance with the same ssh keypair as you are using to access it.

• Your local firewall is preventing you from getting out to that port on any server outside your network. Talk to your local network administrators.

• Your firewall on the instance is preventing access to the service. Try shutting down iptables temporarily to see if that helps.

You “experts” laugh when you read these, but if you’re having trouble reaching a server, I recommend you go through each one carefully and double check that your assumptions are correct and the world is really as you remember it. Remember: We all make mistakes. A lot of these come from personal experience.

If you’re not quite sure what terms like “security group” and “keypair” mean in the EC2 context, I recommend going back and reading some introductory material. These are important concepts for beginners.

ssh

The ssh connectivity problems generally fall into a couple major buckets

1. ssh is not accessible, or

2. ssh is rejecting the connection due to a failure to authenticate or authorize

You can find out which type of problem you have by using a command like

telnet HOSTNAME 22

If this connects, then ssh is running and accepting connections on port 22. (Hit [Enter] a couple times to disconnect from the telnet session). If you don’t connect, then it’s important to note if the attempt basically hung forever or if you got a “Connection refused” type of message immediately. (Hit [Ctrl]-[C] to stop the telnet command.)

If the connection attempt hangs, then there might be a problem with the security group, iptables, or your instance might not be running at that IP address.

If the telnet connection attempt gets rejected, then there might be a problem with iptables, ssh configuration, ssh not running on the instance, or perhaps it’s listening on different port if the admin likes to configure things a bit more securely. The console output can be helpful in determining if sshd was started at boot.

If you can get connected to the ssh port with telnet, then you need to start debugging why ssh is not letting you in. The most important information can be gathered by running the ssh connection attempt in verbose (-v) mode:

ssh -v -i KEYPAIR.pem USERNAME@HOSTNAME

The complete output of this command can be very helpful to post when asking for help.

The most common problems with ssh relate to:

• Forgetting to specify -i KEYPAIR.pem in the ssh command

• Not starting the instance specifying a keypair

• Using a different keypair than the one which was used to start the instance

• Not ssh’ing with the correct username. Amazon Linux instances require a first ssh connection with ec2-user@... while mages published by Canonical require a first connection with ubuntu@... and others might need you to ssh using root@...

• Not having the correct ownership or mode on the .ssh directory or authorized_keys file.

• Not having the correct Allow* or *Authentication settings in /etc/ssh/sshd_config

Apache

Web servers are much easier to connect to than other applications because there is generally no authentication and authorization involved to get a basic web page. If you can’t reach your web server on EC2, then it’s generally one of the simple problems described above like using the wrong IP address, trying to reach a terminated instance, or not having the web port opened in the security group.

MySQL

The most common problem specific to MySQL connectivity on EC2 is the fact that MySQL is configured securely by default to not allow access by remote hosts. If you need to allow a connection from your other instances running in EC2, then edit /etc/mysql/my.cnf and replace this line:

bind-address            = 127.0.0.1

with

bind-address            = 0.0.0.0

and restart the mysqld server.

IMPORTANT! You should not open the MySQL port in the EC2 security group. You only want your own EC2 instances to connect to the database and the default security group allows your EC2 instances to connect to any port on your other EC2 instances. If you open up the port to the public, then your database will be attacked by the Internet at large.

If you need to talk to your MySQL database running on EC2 from a server running outside EC2, then do it over a secure channel like an ssh tunnel or openvpn. You don’t need the MySQL port open in the security group to do this. The MySQL protocol is not by itself encrypted and your usernames and passwords would be sent in the clear for anybody else to intercept if you didn’t talk over a secure channel.

Custom AMIs

If you are building your own custom AMIs from scratch, then there are a number of complicated barriers to getting network and ssh connectivity working. Unfortunately it is nearly impossible to debug these problems since you don’t have access to the machine to see what went wrong. Console output is your only friend in these cases.

Here are some examples of odd things which others in the EC2 community have run into and solved:

• Make sure you start networking on instance boot. It should come up with DHCP on eth0.

• Make sure your Linux distro does not save the MAC address somewhere, preventing the network from functioning in the next instance. Ubuntu stores this in the /etc/udev/rules.d/70-persistent-net.rules file and Debian stores this in the /etc/udev/rules.d/z25_persistent-net.rules file.

• Make sure your image downloads the ssh keypair and installs it in authorized_keys.

• Make sure you have the right devices created and file systems mounted.

• Make sure you’re using a udev lower than v144 as higher versions are incompatible with Amazon’s 2.6.21 kernel.

• Make sure you’re using the right libc6 and related configurations including /lib/tls

Amazon

I realize this was your first thought, but it’s such a rare cause, I’ve put it here at the end. Sometimes there are problems with Amazon EC2. The hardware running your instance may fail or the networks might have temporary glitches. There are a couple different classes of problems here:

1. Small scale problems local to the hardware running your instance. Though these are rare for any single instance, they are happening all the time for some customer somewhere given that AWS has hundreds of thousands of customers. Amazon often sends you an email when they notice that an instance is starting to have problems, and you should move to a new instance as soon as possible. If the failure happens without the warning, the only solution is to move to a new instance anyway, so you should always be prepared to do this.

2. Large scale problems which affect a large number of customers simultaneously. These are very rare, and generally don’t affect more than a single availability zone given the way that Amazon has spread out the risk in their architecture.

You can check the AWS service health dashboard to see if Amazon is aware of any widespread problems with the EC2 service. If there are problems with a specific availability zone, you may want to move your servers to a different availability zone until the issues get resolved.

First Responses

For general cases where you can’t immediately figure out what went wrong with the connectivity, here are two things which are almost always recommended on EC2: reboot the instance and replace the instance.

Reboot your EC2 instance using the EC2 Console, another API client, or a command like:

    ec2-reboot-instances INSTANCE_ID

After giving it sufficient time to come up, see if that fixed the connectivity problem. Do not reboot your instance if you currently have a working ssh connection to it, but other ssh connections are failing!

If you have a production service running on Amazon EC2 and you lose connectivity to an instance, then I recommend your first reaction be to kick off a replacement instance so that it boots and configures itself while you investigate the original issue. If you don’t solve the problem by the time the replacement is ready, simply switch over to the new server. You may want to continue investigating what happened with the old server, though I generally don’t care what the problem was unless it happens more than once or twice in a short time period.

If your installation environment does not allow you to easily start replacement instances, then you should reconsider how you are using EC2 and work to improve this.

Seeking Help

If the above did not help you solve your problem reaching your EC2 instance, you may want to reach out to the community including some AWS employees on the EC2 forum.

Amazon also has premium AWS support available.

Requests for connectivity help by posting a comment on this particular thread will not be published or answered. Please only post a comment if you have corrections or additional information to share for users experiencing problems. I do occasionally receive and respond to questions posted on other articles, but for this topic, please use the EC2 forum.

[Update 2011-10-01: Amazon Linux requires ssh using ubuntu@...]

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

https://alestic.com

The following notes apply to this release:

• The EC2 AMI tools have been upgraded to version 1.3-34544. Note that an “apt-get upgrade” will downgrade the EC2 AMI tools because the versions of the ec2-ami-tools package currently in the Ubuntu Hardy, Intrepid, Jaunty archives are outdated. If you have an easy solution to this, please let us know.

• The ssh host key regeneration has been moved to run after the RNG is seeded. This improves security, especially for folks who are verifying the ssh host key on the first connect by comparing the fingerprint to the value in the instance console output. Thanks to Andrew Becherer for suggesting this improvement.

• The Ubuntu Karmic Alpha images were not updated. Due to a new kernel requirement, Karmic will no longer run on Amazon’s 2.6.21 kernel. Once Canonical releases an Ubuntu kernel with the appropriate features, the Karmic series may be resumed. Running “apt-get upgrade” on an existing Karmic AMI will cause it to be inaccessible after rebooting.

• The Debian Etch desktop images were not updated because apparently LaTeX refuses to build a format from a source file which is more than five years old. If there is anybody who wants Etch desktops and is willing to investigate, please contact me, otherwise I plan to discontinue support for this series. If you’re using Debian, I’d encourage you to upgrade to Lenny “stable” anyway.

Please give these new images a spin and let us know if you run into any problems.

Enjoy

When you ssh to a new EC2 instance, you are presented with the challenge:

The authenticity of host 'XXX' can't be established.
RSA key fingerprint is YYY.
Are you sure you want to continue connecting (yes/no)?

For optimal security, you are supposed to request the instance console output and find the ssh host key fingerprint in the log to verify that it is the same as the fingerprint presented to you by the ssh command.

New updates have been released for the Ubuntu and Debian AMIs (EC2 images) published on:

https://alestic.com

The following improvements are included in this release:

• Ubuntu 9.04 Jaunty now uses an Ubuntu mirror inside of EC2 hosted by RightScale. This dramatically improves the performance of updates and upgrades. Hardy and Intrepid were already using the mirrors inside EC2.

• The Hardy, Intrepid, and Jaunty images have been enhanced to add failover for Ubuntu archive mirror hosts across availability zones (data centers). This change lets an Ubuntu instance perform package updates and upgrades even if one or two of the EC2 availability zones are completely unavailable.

• The denyhosts package is now installed on desktop images for improved security. The Amazon abuse team has identified the Ubuntu desktop images as a source of compromised systems. The cause for this is believed to be unsecure passwords set by users, since the desktop images have PasswordAuthentication enabled by default so that the NX client can connect. The denyhosts package blocks ssh attacks by adding remote systems to /etc/hosts.deny if they keep failing password logins.

  The published Ubuntu and Debian server images continue to have PasswordAuthentication turned off by default for improved security. If you choose to turn this on, I recommend installing a package like denyhosts and using software like the following to generate secure passwords:

    sudo apt-get install pwgen
    pwgen -s 10 1
  

• The EC2 AMI tools have been upgraded to version 1.3-31780.

• All software packages have been updated to versions current as of 2009-06-14.

Community support for Ubuntu on EC2 is available in this group:

http://groups.google.com/group/ec2ubuntu

Community support for Debian on EC2 is available in this group:

http://groups.google.com/group/ec2debian

The 32-bit Debian squeeze images and the 32-bit Debian etch desktop image have not been updated yet due to problems with initial package installation. Images will be released when these issues are resolved.

The following enhancements have been made to the ec2ubuntu-build-ami software which is used to build Ubuntu and Debian images for EC2.

• New --kernel and --ramdisk options have been added to specify AKI and ARI. If you specify a different kernel, you should also specify kernel modules with --package or install them with the --script option.

• Support has been removed for Ubuntu Edgy, Feisty, and Gutsy. These releases have reached their end of life. To improve the clarity of the code this software no longer supports building these images.

• There has been a typo fix for $originaldir for folks who were using the --script option.

• There has been a typo fix for /dev/ptmx though it apparently had no effect given how these images are built.

Thanks to Stephen Parkes and Paul Dowman for submitting patches.

Enjoy!