November 2011 Archives

Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time?

Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance?

Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing).

In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minumum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job.

Here’s a high level overview for folks familiar with Auto Scaling:

  • The instance is started using a recurring schedule action that raises the min and max to 1.

  • The launch configuration is set to pass in a user-data script that runs the desired job on first boot. There’s no need to build our own AMI unless the software installation takes too long.

  • The user-data script runs shutdown at the end to move the instance to the “stopped” state, suspending hourly run charges.

  • Another recurring schedule action lowers the min and max to 0. This is done long after the job should have completed, just to clean up the stopped instance or to terminate an instance that perhaps failed to shut itself down. It also resets the min/max count so a new instance will be started the next time they are raised.

The missing key I found through experimentation is that we need to also suspend Auto Scaling’s (usually valid) desire to replace any unhealthy instances. If we don’t turn off this behavior, Auto Scaling will start another instance as soon as the first one shuts itself down, causing the job to be run over and over.

Prerequisites

The examples in this article assume that you have:

  1. Installed and set up the EC2 command line tools

  2. Installed and set up the EC2 Auto Scaling command line tools

  3. Tell EC2 about your ssh keys using the approach described here:

    Uploading Personal ssh Keys to Amazon EC2

User-data Script

The first step in setting up this process is to create a user-data script that performs the tasks you want your scheduled instance to execute each time it runs.

This user-data script can do anything you want as long as it runs this command after all the work has beeen completed:

shutdown -h now

I have created a demo user-data script for this article which you can download to your local computer now, saving it as demo-user-data-script.sh

wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh

WARNING: Do not attempt to run this user-data script on your local computer!

Edit the downloaded demo-user-data-script.sh file and change this line at the top of the script to reflect your email address:

EMAIL=youraddress@example.com

This demo user-data script:

  • Upgrades the Ubuntu instance

  • Installs Postfix with a generic configuration so that it can send email

  • Sends you a demo informative message about the instance at the email address you edited in the script.

  • Sleeps for 5 minutes giving the email a chance to be delivered

  • Shuts down the EC2 instance

The first time you run this demo, try using the script as it stands, only changing your email address. Then, try adjusting the script little by little to make it do tasks that you would find more useful.

If you’d like, you can test the user-data script by itself running an instance of Ubuntu 11.10. Please update the AMI id to use the latest release:

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
ec2-run-instances   --key $USER   --instance-type t1.micro   --instance-initiated-shutdown-behavior terminate   --user-data-file demo-user-data-script.sh   $ami_id

You should see an email from the instance and then it should terminate itself about 5 minutes later. Make sure you terminate it manually if it stays running after 10 minutes.

Auto Scaling Group

With the user-data script in hand, we are now ready to create the Auto Scaling setup.

Set some variables used in the commands below. Make sure you are using the latest release of the appropriate AMI.

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
region=us-east-1    # Region for running the demo

zone=${region}a     # A zone in that region
export EC2_URL=https://$region.ec2.amazonaws.com
export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com
launch_config=demo-launch-config
auto_scale_group=demo-auto-scale-group

This lauch configuration describes how we want our instance run each time including the AMI id, instance type, ssh key, and most importantly, our user-data script we edited above:

as-create-launch-config   --key $USER   --instance-type t1.micro   --user-data-file demo-user-data-script.sh   --image-id $ami_id   --launch-config "$launch_config"

The Auto Scaling group keeps track of many things including how many instances we want to have running, how they should be run (launch config above), and what instances are currently running.

as-create-auto-scaling-group   --auto-scaling-group "$auto_scale_group"   --launch-configuration "$launch_config"   --availability-zones "$zone"   --min-size 0   --max-size 0

Here’s a non-obvious but key part of this approach! Tell the Auto Scaling group that we don’t want it to restart our instance right after the instance intentionally shuts down (or fails):

as-suspend-processes   "$auto_scale_group"   --processes ReplaceUnhealthy

Now we’re finally ready to tell EC2 Auto Scaling when we want to run the instance launch configuration in the Auto Scaling group.

Here’s an example that starts one instance four times a day to run the above user-data script:

# UTC: 1:00, 7:00, 13:00, 19:00
as-put-scheduled-update-group-action   --name "demo-schedule-start"   --auto-scaling-group "$auto_scale_group"   --min-size 1   --max-size 1   --recurrence "0 01,07,13,19 * * *"

And, we need to create a matching action to make sure the instance is terminated at some point after the longest time the job could take. For this demo, we’ll trigger it 55 minutes later, but it could just as easily be 3 hours and 55 minutes later:

# UTC: 1:55, 7:55, 13:55, 19:55
as-put-scheduled-update-group-action   --name "demo-schedule-stop"   --auto-scaling-group "$auto_scale_group"   --min-size 0   --max-size 0   --recurrence "55 01,07,13,19 * * *"

The recurrence value is in a cron format using UTC.

You are welcome to change the specs in the above commands to any time you want to run the demo, especially if you don’t want to wait up to six hours for it to trigger.

Before setting new schedules, make sure you delete the existing schedule (see the next section). Don’t forget to make the stop time(s) match up appropriately with the start time(s).

Clean up

Once you’re done with this demo, you can delete the AWS resources we created by following these steps:

Delete the schedule start and stop actions:

as-delete-scheduled-action   --force   --name "demo-schedule-start"   --auto-scaling-group "$auto_scale_group"
as-delete-scheduled-action   --force   --name "demo-schedule-stop"   --auto-scaling-group "$auto_scale_group"

Scale the Auto Scaling group down to zero instances. This will terminate any running instances:

as-update-auto-scaling-group   "$auto_scale_group"   --min-size 0   --max-size 0

Delete the Auto Scaling group

as-delete-auto-scaling-group   --force-delete   --auto-scaling-group "$auto_scale_group"

Delete the Auto Scaling launch config:

as-delete-launch-config   --force   --launch-config "$launch_config"

You might now want to check to make sure nothing was left over. This works best in a wide terminal:

as-describe-launch-configs --headers
as-describe-auto-scaling-groups --headers
as-describe-scheduled-actions --headers
as-describe-auto-scaling-instances --headers

Timing

Everything takes a little time to filter through the system including:

  • scheduled action to raise min/max

  • triggering the start of an instance after a min/max is raised

  • starting an instance

  • booting an instance, installing software

  • running your job

  • shutting down an instance

  • scheduled action to lower min/max

  • triggering the termination of an instance after a min/max is lowered

When you set up the schedules, remember to make room for these things. For example, don’t schedule the termination of your instance too early or it could kill your job before it has a chance to complete.

Notes

Here are some great resources from Amazon for getting started with and learning about Auto Scaling:

The user-data script logging uses the approach described here:

Watching the output of the user-data script lets you monitor its progress as well as debug where things might be going wrong.

I haven’t run the above approach except in testing, and would welcome any pointers or improvements folks might have to offer.

Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking up unwanted space in my pocket for two years.

Multi-factor authentication means that in order to log in to my AWS account using the AWS console or portal (including the AWS forums) you not only need my secret password, you also need access to a device that I carry around with me.

Before, this was a physical device attached to my key ring. Now, this is my smart phone which has the virtual (software) MFA device on it. I already carry my phone with me, so the software doesn’t take up any additional space.

To log in to AWS, I enter my password and then the current 6 digit access code displayed by the Android app on my phone. These digits change every 30 seconds in an unguessable pattern, so this enhances the security of my AWS account.

I started by using Amazon’s AWS Virtual MFA app for my Android phone, but had some complaints about it including:

  • You have to click on an account name to see the current digits instead of just having them shown when the app is run. There’s nothing else for the app to do but show me these digits. Just do it!

  • The digits disappear from the screen too fast. Sometimes I want to glance back and see if I typed them in correctly, but they’re gone and I have to click again, hoping that they haven’t changed yet.

  • It’s hard to choose your own account names so that you know which entry to use for different AWS accounts.

I then noticed some cryptic information in the announcements: the new feature will work with “any application that supports the open OATH TOTP standard”.

Hmmm…

Sure ‘nuff!

I already use the Google Authenticator app on my Android phone so that my Google logins can use MFA. As it turns out, Google Authenticator also works seamlessly with AWS Virtual MFA.

  • Google Authenticator shows the codes as soon as it is run with a little timer showing me when they will change.

  • Google Authenticator lets me easily edit the displayed name so that I know at a glance which code is for my personal AWS account and which one is for my company AWS account.

This also means that I only have to run one app to get access to my devices for Google accounts and for AWS accounts. Amazon may improve their Android app over time, but by using open standards users can pick whatever works best for them at the time.

I love the fact that Amazon now supports Virtual MFA. I’ve already thrown away my hardware token and my pocket feels less full.

I love the fact that Amazon implemented this as a service based on existing standards so that I can use Google’s Android app to access my account.

I love open standards.

Update: I just found this great starting page which even links to Google Authenticator as a client for Android and iPhone:

http://aws.amazon.com/mfa/

Ubuntu AMIs

Ubuntu AMIs for EC2: