A Simpler Way To Replace Instance Hardware on EC2

A while back I wrote an article describing a way to move the root EBS volume from one running instance to another. I pitched this as a way to replace the hardware for your instance in the event of failures.

Since then, I have come to the realization that there is a much simpler method to move your instance to new hardware, and I have been using this new method for months when I run into issues that I suspect might be attributed to underlying hardware issues.

This method is so simple, that I am almost embarrassed about having written the previous article, but I’ll point out below at least one benefit that still exists with the more complicated approach.

I now use this process as the second step–after a simple reboot–when I am experiencing odd problems like not being able to connect to a long running EC2 instance. (The zeroth step is to start running and setting up a replacement instance in the event that steps one and two do not produce the desired results.)

Here goes…

Moving an EC2 Instance to a Larger (or Smaller) Instance Type

When you discover that the entry level t1.micro instance size is simply not cutting it for your growing application needs, you may want to try upgrading it to a larger instance type, perhaps an m1.small or even a c1.medium.

Instead of starting a new instance and having to configure it from scratch, you may be able to simply resize the existing instance by asking Amazon move it to better hardware for you. Of course, since this is AWS, you don’t have to actually talk to anybody–just type a few commands and the job is done automatically.

Constraints

Before you try this approach, note that there are some conditions:

Logging user-data Script Output on EC2 Instances

Real time access to user-data script output

The early implementations of user-data scripts on EC2 automatically sent all output of the script (stdout and stderr) to /var/log/syslog as well as to the EC2 console output, to help monitor the startup progress and to debug when things went wrong.

The recent Ubuntu AMIs still send user-data script to the console output, so you can view it remotely, but it is no longer available in syslog on the instance. The console output is only updated a few minutes after the instance boots, reboots, or terminates, which forces you to wait to see the output of the user-data script as well as not capturing output that might come out after the snapshot.

Here is an example written in bash that demonstrates how to send all user-data script stdout and stderr automatically, transparently, and simultaneously to three locations:

Boot EC2 Instance With ssh on Port 80

In a thread on the EC2 forum, Marko describes a situation where an outbound firewall prevents the ability to ssh to port 22, which is the default port on all EC2 instances.

In that thread, Shlomo Swidler proposes creating a user-data script that changes sshd to listen on a port the firewall permits.

Here’s a simple example of a user-data script that does just that. Most outbound firewalls allow traffic to port 80 (web/HTTP), so I use it in this example.

The first step is to create a file containing the user-data script:

Copying EBS Boot AMIs Between EC2 Regions

Update: Since this article was written, Amazon has released the ability to copy EBS boot AMIs between regions using the web console, command line, and API. You may still find information of use in this article, but Amazon has solved some of the harder parts for you.

Using Amazon EC2, you created an EBS boot AMI and it’s working fine, but now you want to run instances of that AMI in a different EC2 region. Since AMIs are region specific, you need a copy of the image in each region where instances are required.

This article presents one method you can use to copy an EBS boot from one EC2 region to another.

Ubuntu 9.04 Jaunty End Of Life

Ubuntu 9.04 Jaunty has reached EOL (End Of Life). It is no longer supported by Ubuntu with security updates and patches. You have known this day was coming for 1.5 years, as all non-LTS Ubuntu releases are supported for only 18 months.

I have no plans to delete the Ubuntu 9.04 Jaunty AMIs for EC2 published under the Alestic name in the foreseeable future, but I request, recommend, and urge you to please stop using them and upgrade to an officially supported, active, kernel-consistent release of Ubuntu on EC2 like 10.04 LTS Lucid or 10.10 Maverick.

EBS Boot Instance Stop+Start Begins a New Hour of Charges on EC2

Scott Moser and I were wondering if stopping and starting an EBS boot instance on EC2 would begin a new hour’s worth of charges or if AWS would not increase your costs if the stop/start were done a few minutes apart in the same hour.

For some reason, I had assumed that it would start a new hour of fees, possibly because of my experience with the somewhat unrelated terminating old instances and starting new instances. However, we decided it would be easy to test, so here are the results.

I tested with an Ubuntu 10.10 Maverick 32-bit server EBS boot AMI on the m1.small instance type in the ap-southeast-1 (Singapore) region. The AMI should have no effect on charges, so these results should apply to any OS you run on EC2.

I used an AWS account that did not have any EC2 instance fees in the Singapore region that month (Scott’s idea) so that this activity would be easy to see as the only charges on that account.

Uploading Personal ssh Keys to Amazon EC2

[Update 2015-06-16: Upgrade to latest aws-cli command syntax]

Amazon recently launched the ability to upload your own ssh public key to EC2 so that it can be passed to new instances when they are launched. Prior to this you always had to use an ssh keypair that was generated by Amazon.

The benefits of using your own ssh key include:

  • Amazon never sees the private part of the ssh key (though they promise they do not save a copy after you downloaded it and we all trust them with this)

  • The private part of the ssh key is never transmitted over the network (though it always goes over an encrypted connection and we mostly trust this)

  • You can now upload the same public ssh key to all EC2 regions, so you no longer have to keep track of a separate ssh key for each region.

  • You can use your default personal ssh key with brand new EC2 instances, so you no longer have to remember to specify options like -i EC2KEYPAIR in every ssh, scp, rsync command.

If you haven’t yet created an ssh key for your local system, it can be done with the command:

Ubuntu 10.10 Maverick Released for Amazon EC2

Does anybody really need me to tell them that you can now run a copy of the newly released Ubuntu 10.10 Maverick on Amazon EC2 with official AMIs published by Canonical?

Or, by now, perhaps you have come to expect–like I have–that the smoothly oiled machine will naturally pump out Ubuntu AMIs for EC2 on the same pre-scheduled date that the larger Ubuntu machine churns out yet another smooth launch of yet another clean Ubuntu release.

The bigger question, I guess, might be:

Should I upgrade to Ubuntu 10.10 on EC2?

ec2-consistent-snapshot: New release 0.35

ec2-consistent-snapshot version 0.35 has been released on the Alestic PPA. This software is a wrapper around the EBS create-snapshot API call and can be used to help ensure that the file system and any MySQL database on the EBS volume are in a consistent state, suitable for restoring at a later time.

The most important change in this release is a fix for a defect that has been nagging many folks for months. In rare situations, the create-snapshot API call itself took longer than 10 seconds to return from the EC2 web service at Amazon. The software did not trap the alarm correctly and exited without unfreezing the XFS file system which forced us to add an awkward unfreeze command in all cron jobs.

mountall Bug and Workaround in Ubuntu 10.04 Lucid on EC2

The Ubuntu 10.04 Lucid AMis for Amazon EC2 dated 20100923 have a known bug which causes the mountall process to spin CPU when the instance is rebooted.

You can observe this by starting a Lucid instance, running sudo reboot, and then running top after reconnecting.

Cpu(s): 38.5%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 61.5%st
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 49 root      20   0  4128 1180  920 R 38.6  0.1   0:08.57 mountall           

New Release of Ubuntu AMIs Solves t1.micro Rebooting Issue

Canonical has released an updated series of Ubuntu AMIs for EC2. When starting new EC2 instances, you should use the latest AMI ids to pick up kernel security fixes. If you have Ubuntu 10.04 running on a t1.micro instance type, you should at least upgrade the software packages to get the patch for the rebooting issue:

Automatic Termination of Temporary Instances on Amazon EC2

I frequently fire up a temporary Ubuntu server on Amazon EC2 to test out some package feature, installation process, or other capability where I’m willing to pay a few pennies for a clean install and spare CPU.

I occasionally forget that I started an instance and leave it running for longer than I intended, turning my decision to spend ten cents into a cost of dollars. In one case, I ended up paying several hundred dollars for a super-sized instance I forgot I had running. Yes, ouch.

Because of this pain, I have a habit now of pre-scheduling the termination of my temporary instances immediately after creating them. I used to do this on the instance itself with a command like:

echo "sudo halt" | at now + 55 min

However, this only terminates the instance if its root disk is instance-store (S3 based AMI). I generally run EBS boot instances now, and a shutdown or halt only “stops” an EBS boot instance by default which leaves you paying for the EBS boot volume at, say, $1.50/month.

So, my common practice these days is to pre-schedule an instance termination call, generally from my local laptop, using a command like:

echo "ec2kill i-eb89bb81" | at now + 55 min

The at utility runs the commands on stdin with the exact same environment ($PATH, EC2 keys, current working directory, umask, etc.) as are in the current shell. There are a number of different ways to specify different times for the schedule and little documentation, but it will notify you when it plans to run the commands so you can check it.

After the command is run, at returns the output through an email. This gives you an indication of whether or not the terminate succeeded or if you will need to follow up manually.

Here’s an example email I got from the above at command:

Subject: Output from your job      114
Date: Mon, 20 Sep 2010 14:01:05 -0700 (PDT)

INSTANCE    i-eb89bb81  running shutting-down

I already have a personal custom command which starts an instance, waits for it to move to running, waits for the ssh server to accept connections, then connects with ssh. I think I’ll add a --temporary option to do this termination scheduling for me as well.

You can get a list of the currently scheduled at jobs with

at -l

You can see the commands in a specific job id with:

at -c [JOBID]

If you decide along the way that the instance should not be temporary and you want to cancel the scheduled termination, you can delete a given at job with a command like:

at -d [JOBID]

I’ve been thinking of writing something simple that would regularly monitor my AWS/EC2 resources (instances, EBS volumes, EBS snapshots, AMIs, etc.) and alert me if it detects patterns that may indicate I am spending money where I may not have intended to.

How do you monitor and clean up temporary resources on Amazon AWS/EC2?

Improving Security on EC2 With AWS Identity and Access Management (IAM)

A few hours ago, Amazon launched a public preview of AWS Identity and Access Management (IAM) which is a powerful feature if you have a number of developers who need to access and to manage resources for an AWS account. A unique IAM user can be created for each developer and specific permissions can be doled out as needed.

You can also create IAM users for system functions, dramatically increasing the security of your AWS account in the event a server is compromised. That benefit is the focus of this article using an example frequently cited by EC2 users: Automating EBS snapshots on a local EC2 instance without putting the keys to your AWS kingdom on the file system.

Before the release of AWS IAM, if you wanted to create EBS snapshots in a local cron job on an EC2 instance, you needed to put the master AWS credentials in the file system on that instance. If those AWS credentials were compromised, the attacker could perform all sorts of havoc with resources in your AWS account and charges to your credit card.

With the launch of AWS IAM, we can create a system IAM user with its own AWS keys and all it is allowed to do is… create EBS snapshots! These keys are placed on the instance and used in the snapshot cron job. Now, an attacker can do very little damage with those keys if they are compromised, and we all feel much safer.

The AWS IAM documentation is required reading and a great reference. This article is only intended to serve as a practical introduction to one simple application of IAM.

These instructions assume you are running Ubuntu 10.04 (Lucid) on both your local system and on Amazon EC2. Adjust as appropriate for other distributions and releases.

IAM Installation

Ubuntu does not yet have an official software package for AWS IAM, so we need to download the IAM command line toolkit from Amazon. This can be done on any machine including your local desktop. The IAM command line tools require Java so we need to make sure that is installed as well.

Eventually, you’ll want to install this software somewhere more permanent, but for this demo, we’ll just use it from a subdirectory.

sudo apt-get install openjdk-6-jre unzip
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
wget http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip
unzip IAMCli.zip
export AWS_IAM_HOME=$(echo $(pwd)/IAMCli-*)
export PATH=$PATH:$AWS_IAM_HOME/bin

The AWS IAM tools require you to save your AWS account’s main access key id and AWS secret access key in yet another file format. Create this AWS credential file as, say, $HOME/.aws-credentials-master.txt in the following format (replacing the values with your own credentials):

AWSAccessKeyId=YOURACCESSKEYIDHERE
AWSSecretKey=YOURSECRETKEYHERE

Note: The above is the sample content of a file you are creating, and not shell commands to run.

Protect the above file and set an environment variable to tell IAM where to find it:

export AWS_CREDENTIAL_FILE=$HOME/.aws-credentials-master.txt
chmod 600 $AWS_CREDENTIAL_FILE

We can now use the iam-* command line tools to create and manage AWS IAM users, groups, and policies.

Create IAM User

How you manage your users and groups is sure to be a personal preference that is fine tuned over time, but for the purposes of this demo, I’ll propose that for tracking purposes we put non-human users into a new group named “system”.

iam-groupcreate -g system

Create the snapshotter system user, saving the keys to a file:

user=snapshotter
iam-usercreate -u $user -g system -k |
  tee $HOME/.aws-keys-$user.txt
chmod 600 $HOME/.aws-keys-$user.txt

You will want to have this snapshotter keys file on the EC2 instance, so copy it there:

rsync -Paz $HOME/.aws-keys-$user.txt REMOTEUSER@REMOTESYSTEM:

Allow IAM user snapshotter to create EBS snapshots of any EBS volume:

iam-useraddpolicy \
  -p allow-create-snapshot \
  -e Allow \
  -u $user \
  -a ec2:CreateSnapshot \
  -r '*'

There’s a lot of preparatory and other commands in this article, but take a second to focus on the fact that the core, functional steps are simply the iam-usercreate and iam-useraddpolicy commands above. Two commands and you have a new AWS IAM user with restricted access to your AWS account.

Create EBS Snapshot

For the purposes of this demo, we’ll assume you’re using the ec2-consistent-snapshot tool to create EBS snapshots with a consistent file system and perhaps a consistent MySQL database. (If you’re not using this tool, then you could have simply used ec2-create-snapshot from any computer without having to go through the trouble of creating a new IAM user.)

Make sure you have the latest ec2-consistent-snapshot software installed on the EC2 instance:

sudo add-apt-repository ppa:alestic/ppa
sudo apt-get install ec2-consistent-snapshot

Create the snapshot on the EC2 instance. Adjust options to fit your local EBS volume mount points and MySQL database setup.

sudo ec2-consistent-snapshot \
  --aws-credentials-file $HOME/.aws-keys-snapshotter.txt \
  --xfs-filesystem /YOURMOUNTPOINT \
  YOURVOLUMEID

Follow similar steps to create users and set policies for other system activities you perform on your EC2 instances. IAM can control access to many different AWS resource types, API calls, specific resources, and has even more fine tuned control parameters including time-based restrictions.

The release of AWS Identity and Access Management alleviates one of the biggest concerns security-conscious folks used to have when they started using AWS with a single key that gave complete access and control over all resources. Now the control is entirely in your hands.

Cleanup

If you have followed the steps in this demo and you wish to undo most of what was done, here are some steps for reference.

Delete the IAM user and the IAM group:

iam-userdel -u $user -r
iam-groupdel -g system

Wipe the credentials and keys files and remove the downloaded and unzipped IAM command line toolkit:

sudo apt-get install wipe
wipe  $HOME/.aws-credentials-master.txt \
      $HOME/.aws-keys-$user.txt
rm    IAMCli.zip
rm -r $AWS_IAM_HOME

Make sure to wipe the snapshotter key file on the remote EC2 instance as well.

Support

If you’re looking for help with AWS IAM, there is a new AWS IAM forum dedicated to the topic.

[Update 2010-11-19: Fix path where new zip file is expanded]

Updated Ubuntu AMIs for EC2 released by Canonical

As Scott Moser announced, Canonical has released updated AMIs (Amazon Machine Images) for running Ubuntu 10.04 Lucid, Ubuntu 9.10 Karmic, and Ubuntu 8.04 Hardy on Amazon EC2.

The table at the top of Alestic.com has been updated to reflect the new AMI ids. In fact, I have automated this table so that it will update itself without manual intervention whenever new AMIs are released by Canonical, thanks to their AMI query API.

If you’re interested in automatically retrieving AMI ids for your own uses, here’s a sample URL for the released AMI ids for Ubuntu 10.04 Lucid (server) in an easy to parse format:

http://uec-images.ubuntu.com/query/lucid/server/released.current.txt

Other releases can be retrieved from similar URLs, or you can traverse the available results starting at the root:

http://uec-images.ubuntu.com/query/

You can get pointers to the daily AMI builds through this API, but I recommend sticking to the “released.current” lists for the best tested and most stable AMIs. These are the AMI ids that are listed in the table at the top of Alestic.com for easy browsing, along with the legacy AMIs which I published as “Alestic” for older releases not supported by Canonical.