Alestic.com - A Personal AWS Blog by Eric Hammond

Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time?

Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance?

Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing).

In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minimum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job.

Here’s a high level overview for folks familiar with Auto Scaling:

• The instance is started using a recurring schedule action that raises the min and max to 1.

• The launch configuration is set to pass in a user-data script that runs the desired job on first boot. There’s no need to build our own AMI unless the software installation takes too long.

• The user-data script runs shutdown at the end to move the instance to the “stopped” state, suspending hourly run charges.

• Another recurring schedule action lowers the min and max to 0. This is done long after the job should have completed, just to clean up the stopped instance or to terminate an instance that perhaps failed to shut itself down. It also resets the min/max count so a new instance will be started the next time they are raised.

The missing key I found through experimentation is that we need to also suspend Auto Scaling’s (usually valid) desire to replace any unhealthy instances. If we don’t turn off this behavior, Auto Scaling will start another instance as soon as the first one shuts itself down, causing the job to be run over and over.

Prerequisites

The examples in this article assume that you have:

1. Installed and set up the EC2 command line tools

2. Installed and set up the EC2 Auto Scaling command line tools

3. Tell EC2 about your ssh keys using the approach described here:

   Uploading Personal ssh Keys to Amazon EC2

User-data Script

The first step in setting up this process is to create a user-data script that performs the tasks you want your scheduled instance to execute each time it runs.

This user-data script can do anything you want as long as it runs this command after all the work has beeen completed:

shutdown -h now

I have created a demo user-data script for this article which you can download to your local computer now, saving it as demo-user-data-script.sh

wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh

WARNING: Do not attempt to run this user-data script on your local computer!

Edit the downloaded demo-user-data-script.sh file and change this line at the top of the script to reflect your email address:

EMAIL=youraddress@example.com

This demo user-data script:

• Upgrades the Ubuntu instance

• Installs Postfix with a generic configuration so that it can send email

• Sends you a demo informative message about the instance at the email address you edited in the script.

• Sleeps for 5 minutes giving the email a chance to be delivered

• Shuts down the EC2 instance

The first time you run this demo, try using the script as it stands, only changing your email address. Then, try adjusting the script little by little to make it do tasks that you would find more useful.

If you’d like, you can test the user-data script by itself running an instance of Ubuntu 11.10. Please update the AMI id to use the latest release:

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
ec2-run-instances \
  --key $USER \
  --instance-type t1.micro \
  --instance-initiated-shutdown-behavior terminate \
  --user-data-file demo-user-data-script.sh \
  $ami_id

You should see an email from the instance and then it should terminate itself about 5 minutes later. Make sure you terminate it manually if it stays running after 10 minutes.

Auto Scaling Group

With the user-data script in hand, we are now ready to create the Auto Scaling setup.

Set some variables used in the commands below. Make sure you are using the latest release of the appropriate AMI.

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
region=us-east-1    # Region for running the demo

zone=${region}a     # A zone in that region
export EC2_URL=https://$region.ec2.amazonaws.com
export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com
launch_config=demo-launch-config
auto_scale_group=demo-auto-scale-group

This lauch configuration describes how we want our instance run each time including the AMI id, instance type, ssh key, and most importantly, our user-data script we edited above:

as-create-launch-config \
  --key $USER \
  --instance-type t1.micro \
  --user-data-file demo-user-data-script.sh \
  --image-id $ami_id \
  --launch-config "$launch_config"

The Auto Scaling group keeps track of many things including how many instances we want to have running, how they should be run (launch config above), and what instances are currently running.

as-create-auto-scaling-group \
  --auto-scaling-group "$auto_scale_group" \
  --launch-configuration "$launch_config" \
  --availability-zones "$zone" \
  --min-size 0 \
  --max-size 0

Here’s a non-obvious but key part of this approach! Tell the Auto Scaling group that we don’t want it to restart our instance right after the instance intentionally shuts down (or fails):

as-suspend-processes \
  "$auto_scale_group" \
  --processes ReplaceUnhealthy

Now we’re finally ready to tell EC2 Auto Scaling when we want to run the instance launch configuration in the Auto Scaling group.

Here’s an example that starts one instance four times a day to run the above user-data script:

# UTC: 1:00, 7:00, 13:00, 19:00
as-put-scheduled-update-group-action \
  --name "demo-schedule-start" \
  --auto-scaling-group "$auto_scale_group" \
  --min-size 1 \
  --max-size 1 \
  --recurrence "0 01,07,13,19 * * *"

And, we need to create a matching action to make sure the instance is terminated at some point after the longest time the job could take. For this demo, we’ll trigger it 55 minutes later, but it could just as easily be 3 hours and 55 minutes later:

# UTC: 1:55, 7:55, 13:55, 19:55
as-put-scheduled-update-group-action \
  --name "demo-schedule-stop" \
  --auto-scaling-group "$auto_scale_group" \
  --min-size 0 \
  --max-size 0 \
  --recurrence "55 01,07,13,19 * * *"

The recurrence value is in a cron format using UTC.

You are welcome to change the specs in the above commands to any time you want to run the demo, especially if you don’t want to wait up to six hours for it to trigger.

Before setting new schedules, make sure you delete the existing schedule (see the next section). Don’t forget to make the stop time(s) match up appropriately with the start time(s).

Clean up

Once you’re done with this demo, you can delete the AWS resources we created by following these steps:

Delete the schedule start and stop actions:

as-delete-scheduled-action \
  --force \
  --name "demo-schedule-start" \
  --auto-scaling-group "$auto_scale_group"
as-delete-scheduled-action \
  --force \
  --name "demo-schedule-stop" \
  --auto-scaling-group "$auto_scale_group"

Scale the Auto Scaling group down to zero instances. This will terminate any running instances:

as-update-auto-scaling-group \
  "$auto_scale_group" \
  --min-size 0 \
  --max-size 0

Delete the Auto Scaling group

as-delete-auto-scaling-group \
  --force-delete \
  --auto-scaling-group "$auto_scale_group"

Delete the Auto Scaling launch config:

as-delete-launch-config \
  --force \
  --launch-config "$launch_config"

You might now want to check to make sure nothing was left over. This works best in a wide terminal:

as-describe-launch-configs --headers
as-describe-auto-scaling-groups --headers
as-describe-scheduled-actions --headers
as-describe-auto-scaling-instances --headers

Timing

Everything takes a little time to filter through the system including:

• scheduled action to raise min/max

• triggering the start of an instance after a min/max is raised

• starting an instance

• booting an instance, installing software

• running your job

• shutting down an instance

• scheduled action to lower min/max

• triggering the termination of an instance after a min/max is lowered

When you set up the schedules, remember to make room for these things. For example, don’t schedule the termination of your instance too early or it could kill your job before it has a chance to complete.

Notes

Here are some great resources from Amazon for getting started with and learning about Auto Scaling:

• Auto Scaling Getting Started Guide

• Auto Scaling Developer Guide

The user-data script logging uses the approach described here:

• Logging user-data Script Output on EC2 Instances

Watching the output of the user-data script lets you monitor its progress as well as debug where things might be going wrong.

I haven’t run the above approach except in testing, and would welcome any pointers or improvements folks might have to offer.