I frequently fire up a temporary Ubuntu server on Amazon EC2 to test out some package feature, installation process, or other capability where I’m willing to pay a few pennies for a clean install and spare CPU.
I occasionally forget that I started an instance and leave it running for longer than I intended, turning my decision to spend ten cents into a cost of dollars. In one case, I ended up paying several hundred dollars for a super-sized instance I forgot I had running. Yes, ouch.
Because of this pain, I have a habit now of pre-scheduling the termination of my temporary instances immediately after creating them. I used to do this on the instance itself with a command like:
echo "sudo halt" | at now + 55 min
However, this only terminates the instance if its root disk is instance-store (S3 based AMI). I generally run EBS boot instances now, and a shutdown or halt only “stops” an EBS boot instance by default which leaves you paying for the EBS boot volume at, say, $1.50/month.
So, my common practice these days is to pre-schedule an instance termination call, generally from my local laptop, using a command like:
echo "ec2kill i-eb89bb81" | at now + 55 min
The at utility runs the commands on stdin with the exact same environment ($PATH, EC2 keys, current working directory, umask, etc.) as are in the current shell. There are a number of different ways to specify different times for the schedule and little documentation, but it will notify you when it plans to run the commands so you can check it.
After the command is run, at returns the output through an email. This gives you an indication of whether or not the terminate succeeded or if you will need to follow up manually.
Here’s an example email I got from the above at command:
Subject: Output from your job 114
Date: Mon, 20 Sep 2010 14:01:05 -0700 (PDT)
INSTANCE i-eb89bb81 running shutting-down
I already have a personal custom command which starts an instance, waits for it to move to running, waits for the ssh server to accept connections, then connects with ssh. I think I’ll add a --temporary option to do this termination scheduling for me as well.
You can get a list of the currently scheduled at jobs with
at -l
You can see the commands in a specific job id with:
at -c [JOBID]
If you decide along the way that the instance should not be temporary and you want to cancel the scheduled termination, you can delete a given at job with a command like:
at -d [JOBID]
I’ve been thinking of writing something simple that would regularly monitor my AWS/EC2 resources (instances, EBS volumes, EBS snapshots, AMIs, etc.) and alert me if it detects patterns that may indicate I am spending money where I may not have intended to.
How do you monitor and clean up temporary resources on Amazon AWS/EC2?