December 2009 Archives

Thanks to everybody who submitted bug reports and feature requests for ec2-consistent-snapshot, software which can be used to create consistent EBS snapshots on Amazon EC2 especially for use with XFS and/or MySQL.

A new release (0.1-9) has been published to the Alestic PPA. This release provides the following fixes and enhancements:

  • Read MySQL “host” parameter from .my.cnf if provided. (closes: bug#485978)

  • Support quoted values in .my.cnf (closes: bug#454184)

  • Require Net::Amazon::EC2 version 0.11 or higher. (closes: bug#490686)

  • Require xfsprogs package. (closes: bug#493420)

  • Replace “/etc/init.d/mysql stop” with “mysqladmin shutdown”. (closes: bug#497557)

  • Document —description option which was added earlier. (closes: bug#487692)

If you already have ec2-consistent-snapshot installed, you can upgrade using commands like:

sudo apt-get update
sudo apt-get install ec2-consistent-snapshot

If you don’t yet have the Alestic PPA set up, run these commands first:

codename=$(lsb_release -cs)
echo "deb http://ppa.launchpad.net/alestic/ppa/ubuntu $codename main"|
  sudo tee /etc/apt/sources.list.d/alestic-ppa.list    
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571

If you find bugs or think of feature requests, please submit them.

The new spot instances on EC2 are a great way to get some extra compute power at a price you can live with, especially if you are flexible on exactly when the instances run. On the other hand maybe you won’t get the compute power if the spot instance price never drops to the max price you are willing to pay.

The best way to approach auction type situations like this is often to simply list the maximum price you can afford. Your instance(s) will get run if and when the spot instance price reaches that price and you will regularly get charged less depending on what other users are bidding for their instances.

Though I don’t recommend trying to chase the spot instance price around, it is natural to be curious about what others have been paying and whether or not you might have a chance to get in with your bid.

The ec2-describe-spot-price-history command lists the historical values of the spot instance price for all sizes and types of instances. This is useful for studying fluctuations over time, but sometimes you’ll just want to know what the latest spot instance price is, even though this is not necessarily the price you would get charged if you opened new spot instance requests.

The following command shows the most recent spot instance price for each size of the Linux instances:

ec2-describe-spot-price-history -d Linux/UNIX |
  sort -rk3 |
  perl -ane 'print "$F[1]\t$F[3]\n" unless $seen{$F[3]}++' |
  sort -n

To check the spot price in a different region, simply add an option like

--region us-west-1

or

--region eu-west-1

As of the writing of this article, the prices returned for the us-east-1 region are as follows:

0.026   m1.small
0.05    c1.medium
0.11    m1.large
0.25    c1.xlarge
0.265   m1.xlarge
0.443   m2.2xlarge
0.997   m2.4xlarge

The exact spot instance prices will vary significantly from these samples over time and likely even during the course of a day. In fact, the spot instance price may occasionally exceed the on demand (standard) EC2 instance price.

Who’s going to be the first person to provide handy, real time, spot price history graphs by EC2 region, instance class (Linux/Windows), and instance type with correct X-axis date/time scaling?

Good luck bidding!

This article is about running a new EC2 instance with a larger boot volume than the default. You can also resize a running EC2 instance.

Amazon EC2’s new EBS Boot feature not only provides persistent root disks for instances, but also supports root disks larger than the previous limit of 10GB under S3 based AMIs.

Since EBS boot AMIs are implemented by creating a snapshot, the AMI publisher controls the default size of the root disk through the size of the snapshot. There are a number of factors which go into deciding the default root disk size of an EBS boot AMI and some of them conflict.

On the one hand, you want to give users enough free space to run their applications, but on the other hand, you don’t want to increase the cost of running the instance too much. EBS volumes run $0.10 to $0.11 per GB per month depending on the region, or about $10/month for 100GB and $100/month for 1TB.

I suspect the answer to this problem might be for AMI publishers to provide a reasonable low default, perhaps 10GB as per the old standard or 15GB following in the footsteps of Amazon’s first EBS Boot AMIs. This would add $1.00 to $1.50 per month to running the instance which seems negligible for most purposes. Note: There are also IO charges and charges for EBS snapshots, but those are more affected by usage and less by the size of the original volume.

For applications where the EBS boot AMI’s default size is not sufficient, users can increase the root disk size at run time all the way up to 1 TB. Here’s a quick overview of how to do this.

Example

The following demonstrates how to run Amazon’s getting-started-with-ebs-boot AMI increasing the root disk from the default of 15GB up to 100GB.

Before we start, let’s check to see the default size of the root disk in the target AMI and what the device name is:

$ ec2-describe-images ami-b232d0db
IMAGE   ami-b232d0db    amazon/getting-started-with-ebs-boot    amazon  available   public      i386    machine aki-94c527fd    ari-96c527ff        ebs
BLOCKDEVICEMAPPING  /dev/sda1       snap-a08912c9   15

We can see the EBS snapshot id snap-a08912c9 and the fact that it is 15 GB attached to /dev/sda1. If we start an instance of this AMI it will have a 15 GB EBS volume as the root disk and we won’t be able to change it once it’s running.

Now let’s run the EBS boot AMI, but we’ll override the default size, specifying 100 GB for the root disk device (/dev/sda1 as seen above):

ec2-run-instances   --key KEYPAIR   --block-device-mapping /dev/sda1=:100   ami-b232d0db

If we check the EBS volume mapped to the new instance we’ll see that it is 100GB, but when we ssh to the instance and check the root file system size we’ll notice that it is only showing 15 GB:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G  1.6G   13G  12% /

There’s one step left. We need to resize the file system so that it fills up the entire 100 GB EBS volume. Here’s the magic command for ext3. In my early tests it took 2-3 minutes to run. [Update: For Ubuntu 11.04 and later, this step is performed automatically when the AMI is booted and you don’t need to run it manually.]

$ sudo resize2fs /dev/sda1
resize2fs 1.40.4 (31-Dec-2007)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 7
Performing an on-line resize of /dev/sda1 to 26214400 (4k) blocks.
The filesystem on /dev/sda1 is now 26214400 blocks long.

Finally, we can check to make sure that we’re running on a bigger file system:

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              99G  1.6G   92G   2% /

Note: The output reflects “99” instead of “100” because of slight differences in how df and EBS calculate “GB” (e.g., 1024 MB vs 1000 MB).

XFS

If it were possible to create an EBS boot AMI with an XFS root file system, then the resizing would be near instantaneous using commands like the following. [Update: For Ubuntu 11.04 and later, this step is performed automatically when the AMI is booted and you don’t need to run it manually.]

sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /

The Ubuntu kernels built for EC2 by Canonical have XFS support built in, so XFS based EBS boot AMIs might be possible. This would also allow for more consistent EBS snapshots.

Toolset

Make sure you are running the latest version of the ec2-run-instances command. The current version can be determined with the command

ec2-version

To use EBS boot features, the version should be at least 1.3-45772.

[Updated 2009-12-11: Switch instructions to default us-east-1 since all regions now support this feature.]
[Updated 2011-12-13: Note that file system resize is done automatically on boot in Ubuntu 11.04 and later.]

Ubuntu AMIs

Ubuntu AMIs for EC2:


AWS Jobs

AWS Jobs