Running EC2 Instances on a Recurring Schedule with Auto Scaling

| 15 Comments

Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time?

Would you like to do this using standard Amazon AWS services without needing an external server to run and terminate the instance?

Amazon EC2 Auto Scaling is normally used to keep a reasonable number of instances running to handle measured or expected load (e.g., web site traffic, queue processing).

In this article I walk through the steps to create an Auto Scaling configuration that runs an instance on a recurring schedule (e.g., four times a day) starting up a pre-defined task and letting that instance shut itself down when it is finished. We tweak the Auto Scaling group so that this uses the minumum cost in instance run time, even though we may not be able to predict in advance exactly how long it will take to complete the job.

Here’s a high level overview for folks familiar with Auto Scaling:

  • The instance is started using a recurring schedule action that raises the min and max to 1.

  • The launch configuration is set to pass in a user-data script that runs the desired job on first boot. There’s no need to build our own AMI unless the software installation takes too long.

  • The user-data script runs shutdown at the end to move the instance to the “stopped” state, suspending hourly run charges.

  • Another recurring schedule action lowers the min and max to 0. This is done long after the job should have completed, just to clean up the stopped instance or to terminate an instance that perhaps failed to shut itself down. It also resets the min/max count so a new instance will be started the next time they are raised.

The missing key I found through experimentation is that we need to also suspend Auto Scaling’s (usually valid) desire to replace any unhealthy instances. If we don’t turn off this behavior, Auto Scaling will start another instance as soon as the first one shuts itself down, causing the job to be run over and over.

Prerequisites

The examples in this article assume that you have:

  1. Installed and set up the EC2 command line tools

  2. Installed and set up the EC2 Auto Scaling command line tools

  3. Tell EC2 about your ssh keys using the approach described here:

    Uploading Personal ssh Keys to Amazon EC2

User-data Script

The first step in setting up this process is to create a user-data script that performs the tasks you want your scheduled instance to execute each time it runs.

This user-data script can do anything you want as long as it runs this command after all the work has beeen completed:

shutdown -h now

I have created a demo user-data script for this article which you can download to your local computer now, saving it as demo-user-data-script.sh

wget -q https://raw.github.com/alestic/demo-ec2-schedule-instance/master/user-data/demo-user-data-script.sh

WARNING: Do not attempt to run this user-data script on your local computer!

Edit the downloaded demo-user-data-script.sh file and change this line at the top of the script to reflect your email address:

EMAIL=youraddress@example.com

This demo user-data script:

  • Upgrades the Ubuntu instance

  • Installs Postfix with a generic configuration so that it can send email

  • Sends you a demo informative message about the instance at the email address you edited in the script.

  • Sleeps for 5 minutes giving the email a chance to be delivered

  • Shuts down the EC2 instance

The first time you run this demo, try using the script as it stands, only changing your email address. Then, try adjusting the script little by little to make it do tasks that you would find more useful.

If you’d like, you can test the user-data script by itself running an instance of Ubuntu 11.10. Please update the AMI id to use the latest release:

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
ec2-run-instances   --key $USER   --instance-type t1.micro   --instance-initiated-shutdown-behavior terminate   --user-data-file demo-user-data-script.sh   $ami_id

You should see an email from the instance and then it should terminate itself about 5 minutes later. Make sure you terminate it manually if it stays running after 10 minutes.

Auto Scaling Group

With the user-data script in hand, we are now ready to create the Auto Scaling setup.

Set some variables used in the commands below. Make sure you are using the latest release of the appropriate AMI.

ami_id=ami-a7f539ce # Ubuntu 11.10 Oneiric server
region=us-east-1    # Region for running the demo

zone=${region}a     # A zone in that region
export EC2_URL=https://$region.ec2.amazonaws.com
export AWS_AUTO_SCALING_URL=https://autoscaling.$region.amazonaws.com
launch_config=demo-launch-config
auto_scale_group=demo-auto-scale-group

This lauch configuration describes how we want our instance run each time including the AMI id, instance type, ssh key, and most importantly, our user-data script we edited above:

as-create-launch-config   --key $USER   --instance-type t1.micro   --user-data-file demo-user-data-script.sh   --image-id $ami_id   --launch-config "$launch_config"

The Auto Scaling group keeps track of many things including how many instances we want to have running, how they should be run (launch config above), and what instances are currently running.

as-create-auto-scaling-group   --auto-scaling-group "$auto_scale_group"   --launch-configuration "$launch_config"   --availability-zones "$zone"   --min-size 0   --max-size 0

Here’s a non-obvious but key part of this approach! Tell the Auto Scaling group that we don’t want it to restart our instance right after the instance intentionally shuts down (or fails):

as-suspend-processes   "$auto_scale_group"   --processes ReplaceUnhealthy

Now we’re finally ready to tell EC2 Auto Scaling when we want to run the instance launch configuration in the Auto Scaling group.

Here’s an example that starts one instance four times a day to run the above user-data script:

# UTC: 1:00, 7:00, 13:00, 19:00
as-put-scheduled-update-group-action   --name "demo-schedule-start"   --auto-scaling-group "$auto_scale_group"   --min-size 1   --max-size 1   --recurrence "0 01,07,13,19 * * *"

And, we need to create a matching action to make sure the instance is terminated at some point after the longest time the job could take. For this demo, we’ll trigger it 55 minutes later, but it could just as easily be 3 hours and 55 minutes later:

# UTC: 1:55, 7:55, 13:55, 19:55
as-put-scheduled-update-group-action   --name "demo-schedule-stop"   --auto-scaling-group "$auto_scale_group"   --min-size 0   --max-size 0   --recurrence "55 01,07,13,19 * * *"

The recurrence value is in a cron format using UTC.

You are welcome to change the specs in the above commands to any time you want to run the demo, especially if you don’t want to wait up to six hours for it to trigger.

Before setting new schedules, make sure you delete the existing schedule (see the next section). Don’t forget to make the stop time(s) match up appropriately with the start time(s).

Clean up

Once you’re done with this demo, you can delete the AWS resources we created by following these steps:

Delete the schedule start and stop actions:

as-delete-scheduled-action   --force   --name "demo-schedule-start"   --auto-scaling-group "$auto_scale_group"
as-delete-scheduled-action   --force   --name "demo-schedule-stop"   --auto-scaling-group "$auto_scale_group"

Scale the Auto Scaling group down to zero instances. This will terminate any running instances:

as-update-auto-scaling-group   "$auto_scale_group"   --min-size 0   --max-size 0

Delete the Auto Scaling group

as-delete-auto-scaling-group   --force-delete   --auto-scaling-group "$auto_scale_group"

Delete the Auto Scaling launch config:

as-delete-launch-config   --force   --launch-config "$launch_config"

You might now want to check to make sure nothing was left over. This works best in a wide terminal:

as-describe-launch-configs --headers
as-describe-auto-scaling-groups --headers
as-describe-scheduled-actions --headers
as-describe-auto-scaling-instances --headers

Timing

Everything takes a little time to filter through the system including:

  • scheduled action to raise min/max

  • triggering the start of an instance after a min/max is raised

  • starting an instance

  • booting an instance, installing software

  • running your job

  • shutting down an instance

  • scheduled action to lower min/max

  • triggering the termination of an instance after a min/max is lowered

When you set up the schedules, remember to make room for these things. For example, don’t schedule the termination of your instance too early or it could kill your job before it has a chance to complete.

Notes

Here are some great resources from Amazon for getting started with and learning about Auto Scaling:

The user-data script logging uses the approach described here:

Watching the output of the user-data script lets you monitor its progress as well as debug where things might be going wrong.

I haven’t run the above approach except in testing, and would welcome any pointers or improvements folks might have to offer.

15 Comments

Eric - thank you for continuing to push the envelope on the cloud. We (at RightScale) also felt this pain and created Scheduled Auto-Scaling Arrays. However, now that I've seen more people working around this EC2 limitation, I think it's time for us to add a schedule to just a single server. Keep it up.

- Darryl Eaton

Darryl:

Sometimes it seems like Amazon is playing catchup to match all of the features RightScale has developed on top of the AWS/EC2 infrastructure :-)

What I think a lot of folks (including me) would like to see is a way to schedule the start/stop of an EBS boot instance. The article above does a run/terminate, but in some situations it would be nice to keep the EBS boot instance around between runs so that you can preserve configuration and data.

Hi Eric,

Thanks for the post, this helped us configure our autoscaling here at http://www.istream.com

we also had similar problem that the new instance is handling a long running job (video processing) and it knows when it should shutdown itself.

however, autoscaling always tried to keep the desired capacity etc (which is not what we wanted). we used "Replace unhealthy", machine used to enter stopped state . but autoscaling was not starting new machine unless new instances are terminated. also, the cron (to set max to 0) which you suggested will not suit us as we don't know when the job is going to finish.

finally we used "as-terminate-instance-in-auto-scaling-group" on the new instance before calling shutdown to make the new instance terminate itself on shutdown and solved the problem :)

your blog has been a great help for us.

Thank you,
-Mahaboob Khan.

Mahaboob:

In the approach I describe above, the instance terminates itself when it is done with the job. Changing max-size to 0 can be done as late as you want before the next time it should trigger a new job to start. It only clears up an instance that ran too long and makes way for the next job to kick off a new instance.

Is the user-data-file supposed to be automatically run on new instances that are launched in the auto scaling group? I've followed the instructions in this article, and my script is not being run when my group scales up, but it works when I run it manually. Your stack overflow answer here (http://stackoverflow.com/questions/9764145/ec2-user-data-how-does-it-work) indicates that the user data is just something available to new instances, but you need code bundled in your AMI to retrieve and actually use it, but this article makes it seem like your script gets automatically run when the instance in the auto scaling group starts up. Are these two different things, or am I missing something? Thank you.

Nevermind my previous comment. The problem was that my auto-scaling group was using the default security group. I had to add the argument -group my-security-group to my as-create-launch-config command to use my security group with http/ssh enabled. Now it's working. Thanks for this post!

marklmiddleton:

Thanks for posting the problem and the solution. I'm leaving these up to help others who may run into similar situations.

Hi Eric Hammond,

Instead of running the check 4 times a day, do you think it's better idea to use CloudWatch to check the EC2 instance(s) (which I think can do done every minute) and fire the autoscaling process if necessary?

Eric Nguyen

Trying to figure out how I can convert this to a CloudFormation template.

Cannot find CloudFormation resources corresponding to as-suspend-processes and as-put-scheduled-update-group-action.

Eric Nguyen:

The article is talking about using Auto Scaling to trigger a scheduled process which is the opposite of using a scheduled process to trigger auto scaling.

If you want to do normal auto scaling, then yes, I recommend using CloudWatch and Auto Scaling as they were designed.

dhawalparkar:

Excellent point. I also am unable to find a way to accomplish this with CloudFormation and I would have expected recurrence to be mentioned somewhere in the four documentation pages starting here:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html

It's possible that it simply is not yet supported. I've posted a request to Amazon on the AWS Cloud Formation forum: https://forums.aws.amazon.com/thread.jspa?messageID=419172񦕤

Great article Eric!

Re. auto-termination once the job is completed.
I was wondering whether it's possible to setup ec2-api-tools inside the user data script then run ec2-terminate-instances using an IAM role.

Will Auto Scaling allow the instances to do that themselves?

AItOawkJZrYJnE4A6sKqDD4H7Mb4OTKUFHKEqws:

Sure, ec2-terminate-instances can be used to terminate an instance started by Auto Scaling and can be run anywhere including on the instance. An IAM role is a great way to give the instance only the permissions it needs to terminate itself.

Just make sure to suspend the ReplaceUnhealthy process so that Auto Scaling does not automatically start a new instance to replace the one you terminated.

Hello Eric,

Thanks for the information.

I tried the method described, and it does indeed start/stop instances at the times as specified by the tutorial. However, I noticed in the AWS web console that when an instance is started by this method, it does not get started with a key (such that you can ssh into it), and it also does not seem to have the same stuff that I installed on my micro-instance that I use as a test (I'm not a cloud expert, but I think this means that this new instance that is spun up is not connected to the EBS?) Is there a way to automatically start and stop the same instance on a time schedule?

I noticed a comment above about connecting an EBS to an instance automatically, wondering if there was any update regarding this.

Thanks!

AItOawmhOyV6XD2dKZkyBic7YkCHdRQx12FpTuo:

The instance is started with the key specified in the launch configuration. In my example, I assume the keypair is named $USER.

The instance is started using the AMI you specify in the launch configuration. You can specify that to be any AMI you build yourself if you want a custom one.

AWS does not yet have a way to specify the use of an existing EBS volume in the launch configuration. I'm not sure they ever will as it creates issues if the EBS volume is already in use, but this isn't the first time I've heard the request, so I'm sure Amazon has considered it.

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


More Entries

EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
When you need an AWS command line toolset not provided by Ubuntu packages, you can download the tools directly from Amazon and install them locally. In a previous article I…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…
You Should Use EBS Boot Instances on Amazon EC2
EBS boot vs. instance-store If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store…
Retrieve Public ssh Key From EC2
A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as…
Running EC2 Instances on a Recurring Schedule with Auto Scaling
Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time? Would you like to…
AWS Virtual MFA and the Google Authenticator for Android
Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking…