Do you want to run short jobs on Amazon EC2 on a recurring schedule, but
don’t want to pay for an instance running all the time?
Would you like to do this using standard Amazon AWS services without
needing an external server to run and terminate the instance?
Amazon EC2 Auto Scaling is normally used to keep a
reasonable number of instances running to handle measured or expected
load (e.g., web site traffic, queue processing).
In this article I walk through the steps to create an Auto Scaling
configuration that runs an instance on a recurring schedule (e.g.,
four times a day) starting up a pre-defined task and letting that
instance shut itself down when it is finished. We tweak the Auto
Scaling group so that this uses the minumum cost in instance run time,
even though we may not be able to predict in advance exactly how long
it will take to complete the job.
Here’s a high level overview for folks familiar with Auto Scaling:
The instance is started using a recurring schedule action that
raises the min and max to 1.
The launch configuration is set to pass in a user-data script that
runs the desired job on first boot. There’s no need to build our
own AMI unless the software installation takes too long.
The user-data script runs shutdown at the end to move the instance
to the “stopped” state, suspending hourly run charges.
Another recurring schedule action lowers the min and max to 0.
This is done long after the job should have completed, just to
clean up the stopped instance or to terminate an instance that
perhaps failed to shut itself down. It also resets the min/max
count so a new instance will be started the next time they are
raised.
The missing key I found through experimentation is that we need to
also suspend Auto Scaling’s (usually valid) desire to replace any
unhealthy instances. If we don’t turn off this behavior, Auto Scaling
will start another instance as soon as the first one shuts itself
down, causing the job to be run over and over.
Prerequisites
The examples in this article assume that you have:
Installed and set up the EC2 command line tools
Installed and set up the EC2 Auto Scaling command line tools
Tell EC2 about your ssh keys using the approach described here:
Uploading Personal ssh Keys to Amazon EC2
User-data Script
The first step in setting up this process is to create a user-data
script that performs the tasks you want your scheduled
instance to execute each time it runs.
This user-data script can do anything you want as long as it runs this
command after all the work has beeen completed:
shutdown -h now
I have created a demo user-data script for this article
which you can download to your local computer now, saving it as
demo-user-data-script.sh
wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh
WARNING: Do not attempt to run this user-data script on your local computer!
Edit the downloaded demo-user-data-script.sh file and change this
line at the top of the script to reflect your email address:
EMAIL=youraddress@example.com
This demo user-data script:
Upgrades the Ubuntu instance
Installs Postfix with a generic configuration so that it can send email
Sends you a demo informative message about the instance at the
email address you edited in the script.
Sleeps for 5 minutes giving the email a chance to be delivered
Shuts down the EC2 instance
The first time you run this demo, try using the script as it stands,
only changing your email address. Then, try adjusting the script
little by little to make it do tasks that you would find more useful.
If you’d like, you can test the user-data script by itself running an
instance of Ubuntu 11.10. Please update the AMI id to use the latest
release:
ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
ec2-run-instances --key $USER --instance-type t1.micro --instance-initiated-shutdown-behavior terminate --user-data-file demo-user-data-script.sh $ami_id
You should see an email from the instance and then it should terminate
itself about 5 minutes later. Make sure you terminate it manually if
it stays running after 10 minutes.
Auto Scaling Group
With the user-data script in hand, we are now ready to create the Auto
Scaling setup.
Set some variables used in the commands below. Make sure you are
using the latest release of the appropriate AMI.
ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
region=us-east-1 # Region for running the demo
zone=${region}a # A zone in that region
export EC2_URL=https://$region.ec2.amazonaws.com
export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com
launch_config=demo-launch-config
auto_scale_group=demo-auto-scale-group
This lauch configuration describes how we want our instance run each
time including the AMI id, instance type, ssh key, and most
importantly, our user-data script we edited above:
as-create-launch-config --key $USER --instance-type t1.micro --user-data-file demo-user-data-script.sh --image-id $ami_id --launch-config "$launch_config"
The Auto Scaling group keeps track of many things including how many
instances we want to have running, how they should be run (launch
config above), and what instances are currently running.
as-create-auto-scaling-group --auto-scaling-group "$auto_scale_group" --launch-configuration "$launch_config" --availability-zones "$zone" --min-size 0 --max-size 0
Here’s a non-obvious but key part of this approach! Tell the Auto
Scaling group that we don’t want it to restart our instance right
after the instance intentionally shuts down (or fails):
as-suspend-processes "$auto_scale_group" --processes ReplaceUnhealthy
Now we’re finally ready to tell EC2 Auto Scaling when we want to run
the instance launch configuration in the Auto Scaling group.
Here’s an example that starts one instance four times a day to run the
above user-data script:
# UTC: 1:00, 7:00, 13:00, 19:00
as-put-scheduled-update-group-action --name "demo-schedule-start" --auto-scaling-group "$auto_scale_group" --min-size 1 --max-size 1 --recurrence "0 01,07,13,19 * * *"
And, we need to create a matching action to make sure the instance is
terminated at some point after the longest time the job could take. For
this demo, we’ll trigger it 55 minutes later, but it could just as
easily be 3 hours and 55 minutes later:
# UTC: 1:55, 7:55, 13:55, 19:55
as-put-scheduled-update-group-action --name "demo-schedule-stop" --auto-scaling-group "$auto_scale_group" --min-size 0 --max-size 0 --recurrence "55 01,07,13,19 * * *"
The recurrence value is in a cron format using UTC.
You are welcome to change the specs in the above commands to any time
you want to run the demo, especially if you don’t want to wait up to
six hours for it to trigger.
Before setting new schedules, make sure you delete the existing
schedule (see the next section). Don’t forget to make the stop
time(s) match up appropriately with the start time(s).
Clean up
Once you’re done with this demo, you can delete the AWS resources we
created by following these steps:
Delete the schedule start and stop actions:
as-delete-scheduled-action --force --name "demo-schedule-start" --auto-scaling-group "$auto_scale_group"
as-delete-scheduled-action --force --name "demo-schedule-stop" --auto-scaling-group "$auto_scale_group"
Scale the Auto Scaling group down to zero instances. This will
terminate any running instances:
as-update-auto-scaling-group "$auto_scale_group" --min-size 0 --max-size 0
Delete the Auto Scaling group
as-delete-auto-scaling-group --force-delete --auto-scaling-group "$auto_scale_group"
Delete the Auto Scaling launch config:
as-delete-launch-config --force --launch-config "$launch_config"
You might now want to check to make sure nothing was left over. This
works best in a wide terminal:
as-describe-launch-configs --headers
as-describe-auto-scaling-groups --headers
as-describe-scheduled-actions --headers
as-describe-auto-scaling-instances --headers
Timing
Everything takes a little time to filter through the system including:
scheduled action to raise min/max
triggering the start of an instance after a min/max is raised
starting an instance
booting an instance, installing software
running your job
shutting down an instance
scheduled action to lower min/max
triggering the termination of an instance after a min/max is lowered
When you set up the schedules, remember to make room for these
things. For example, don’t schedule the termination of your instance
too early or it could kill your job before it has a chance to
complete.
Notes
Here are some great resources from Amazon for getting started with
and learning about Auto Scaling:
The user-data script logging uses the approach described here:
Watching the output of the user-data script lets you monitor its
progress as well as debug where things might be going wrong.
I haven’t run the above approach except in testing, and would welcome
any pointers or improvements folks might have to offer.
Recent Comments