Here are a few little-publicized benefits that were launched with Amazon EC2’s new EBS boot instances: You can lock them from being accidentally terminated; you can prevent termination even when you try to shutdown the server from inside the instance; and you can automatically save your data storage when they are terminated.
In discussions with other EC2 users, I’ve found that it is a common feeling of near panic when you go to terminate an instance and you check very carefully to make sure that you are deleting the right instance instead of an active production system. Slightly less common but even worse is the feeling of dread when you realize you just casually terminated the wrong EC2 instance.
It is always recommended to set up your AWS architecture so that you are able to restart production systems on EC2 easily, as they could, in theory, be lost at any point due to hardware failure or other actions. However, new features released with the EBS boot instances help reduce the risks associated with human error.
The following examples will demonstrate with the EC2 API command line tools ec2-run-instances, ec2-modify-instance-attribute, and ec2-terminate-instances. Since AWS is Amazon’s “web service” all of these features are also available through the API and should be coming available (if they aren’t already) in other tools, packages, and interfaces using the web service API.
1. Shutdown Behavior
First, let’s look at what happens when you run a command like the following in an EC2 instance:
sudo shutdown -h now
# or, equivalently and much easier to type:
sudo halt
Using the legacy S3 based AMIs, either of the above terminates the instance and you lose all local and ephemeral storage (boot disk and /mnt) forever. Hope you remembered to save the important stuff elsewhere!
A shutdown from within an EBS boot instance, by default, will initiate a “stop” instead of a “terminate”. This means that your instance is not in a running state and not getting charged, but the EBS volumes still exist and you can “start” the same instance id again later, losing nothing.
You can explicitly set or change this with the --instance-initiated-shutdown-behavior option in ec2-run-instances. For example:
ec2-run-instances --instance-initiated-shutdown-behavior stop [...]
This is the first safety precaution and, as mentioned, should already be built in, though it doesn’t hurt to document your intentions with an explicit option.
2. Delete on Termination
Though EBS volumes created and attached to an instance at instantiation are preserved through a “stop”/”start” cycle, by default they are destroyed and lost when an EC2 instance is terminated. This behavior can be changed with a delete-on-termination boolean value buried in the documentation for the --block-device-mapping option of ec2-run-instances.
Here is an example that says “Don’t delete the root EBS volume when this instance is terminated”:
ec2-run-instances --block-device-mapping /dev/sda1=::false [...]
If you are associating other EBS snapshots with the instance at run time, you can also specify that those created EBS volumes should be preserved past the lifetime of the instance:
--block-device-mapping /dev/sdh=SNAPSHOTID::false
When you use these options, you’ll need to manually clean up the EBS volume(s) if you no longer want to pay for the storage costs after an instance is gone.
Note: EBS volumes attached to an instance after it is already running are, by default, left alone on termination (i.e., not deleted). The default rules are: If the EBS volume is created by the creation of the instance, then the termination of the instance deletes the volumes. If you created the volume explicitly, then you must delete it explicitly.
3. Disable Termination
This is my favorite new safety feature. Years ago, I asked Amazon for the ability to lock an EC2 instance from being accidentally terminated, and with the launch of EBS boot instances, this is now possible. Using ec2-run-instances, the key option is:
ec2-run-instances --disable-api-termination [...]
Now, if you try to terminate the instance, you will get an error like:
Client.OperationNotPermitted: The instance 'i-XXXXXXXX' may not be terminated.
Modify its 'disableApiTermination' instance attribute and try again.
Yay!
To unlock the instance and allow termination through the API, use a command like:
ec2-modify-instance-attribute --disable-api-termination false INSTANCEID
Then end it all with:
ec2-terminate-instances INSTANCEID
Oh, wait! did you make sure you unlocked and terminated the right instance?! :)
Note: disableApiTermination is also available when you run S3 based AMIs today, but since the instance can still be terminated from inside (shutdown/halt) I am moving towards EBS based instances for all around security.
Put It Together
Here’s a command line I just used to start up an EC2 instance of an Ubuntu 9.10 Karmic EBS boot AMI which I intend to use as a long-term production server with strong uptime and data safety requirements. I wanted to add all the protection available:
ec2-run-instances --key $keypair --availability-zone $availabilityzone --user-data-file $startupscript --block-device-mapping /dev/sda1=::false --block-device-mapping /dev/sdh=$snapshotid::false --instance-initiated-shutdown-behavior stop --disable-api-termination $amiid
With regular EBS snapshots of the volumes, copies to off site backups, and documented/automated restart processes, I feel pretty safe.
Runtime Modification
If you have a valuable running EC2 instance, but forgot to specify the above options to protect it when you started it, or you are now ready to turn a test system into a production system, you can still lock things after the fact using the ec2-modify-instance-attribute command (or equivalent API call).
For example:
ec2-modify-instance-attribute --disable-api-termination true INSTANCEID
ec2-modify-instance-attribute --instance-initiated-shutdown-behavior stop INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sda1=:false INSTANCEID
ec2-modify-instance-attribute --block-device-mapping /dev/sdh=:false INSTANCEID
Notes:
Only one type of option can be specified with each invocation.
The
--disable-api-terminationoption has no argument value when used in inec2-run-instances, but it takes a value oftrueorfalseinec2-modify-instance-attribute.You don’t specify the snapshot id when changing the delete-on-terminate state of an attached EBS volume.
You may change the delete-on-terminate to “true” for an EBS volume you created and attached after the instance was running. By default it will not be deleted since you created it.
The above
--block-device-mappingoption requires recent ec2-api-tools to avoid a bug.
You can find out the current state of the options using ec2-describe-instance-attribute, which only takes one option at a time:
ec2-describe-instance-attribute --disable-api-termination INSTANCEID
ec2-describe-instance-attribute --instance-initiated-shutdown-behavior INSTANCEID
ec2-describe-instance-attribute --block-device-mapping INSTANCEID
Unfortunately, the block-device-mapping output does not currently show the state of delete-on-termination value, but thanks to Andrew Lusk for pointing out in a comment below that it is available through the API. Here’s a hack which extracts the information from the debug output:
ec2-describe-instance-attribute -v --block-device-mapping INSTANCEID |
perl -0777ne 'print "$1\t$2\t$3\n" while
m%<deviceName>(.*?)<.*?<volumeId>(.*?)<.*?<deleteOnTermination>(.*?)<%sg'
While we’re on the topic of EC2 safety, I should mention the well known best practice of separating development and production systems by using a different AWS account for each. Amazon lets you create and use as many accounts as you’d like even with the same credit card as long as you use a unique email address for each. Now that you can share EBS snapshots between accounts, this practice is even more useful.
What additional safety precautions do you take with your EC2 instances and data?
[Update 2011-09-13: Corrected syntax for modifying block-device-mapping on running instance (only one “:”)]
delete-on-termination does come back through the API though the client tools might not display it. If you pass '-v' to ec2-describe-instance-attribute you'll see what I mean (I tried to paste example output in but was foiled by the site trying to interpret SOAP XML as HTML).
Andrew: Excellent info, thanks!
Nice Article!!, Definitely will help a lot of users to configure their instances to put that extra check to provide an additional layer before they decide to do away with their instances/data.
I'm confused as to the "do not delete EBS on termination of instance" feature (i.e. --block-device-mapping=/dev/sda1=::false).
Obviously I understand that it won't delete the EBS, but if you terminated the instance (versus stopping it), could you use the saved EMD from the terminated instance to launch a new instance again? Or is the intention just to make sure that your data isn't destroyed so you can use it for recovery purposes (i.e. mounting EBS image as a secondary partition on a new instance)?
But then again, if you can mount it on another image, why couldn't you just boot it back up since it should (theoretically) not be corrupt?
Halp!
jesser: You can't launch a new EC2 instance directly from an EBS volume without taking a snapshot of it first and registering it as a new AMI. There is no conceptual barrier, but AWS does not provide an API for it at this time. There is a way to substitute the root EBS volume of an EBS boot instance while it is stopped, but that is more of an advanced topic.
Great info! Thanks for this article. We added this to Ylastic recently. You cannot seem to modify the block device mapping attribute via the API currently for the instance. There have been a couple of posts about this on the forums.
From the forum, it looks like for the block device mapping, you just have to be explicit about the volume ID:
This throws an error in parsing the response, but seems to actually set the parameter successfully.
Hi Eric,
For those of us dummies using ELasticfox or the AWS console, is there a way to detach and attach an EBS volume. For instance, if a EBS backed instance is stopped, Elasticfox does not show a stopped instance in the volume attach dropdown. You can attach a EBS volume to a running instance, but you cannot replace the root /dev/sda1 so therefore its not possible to boot from the newly attached volume without changing grub etc.. Is there an easier way?
Your articles are much appreciated.
I use the AWS console on occasion, but I prefer to operate in ways that can be automated so I tend to depend on the command line more heavily. I believe the AWS console has a link for feedback, so if it does not support an operation you need, go ahead and request it.
Thanks Eric as always for the great information.
Question: if I protect an instance as described and create an AMI from it, are instances created from the AMI also protected? I'm guessing not. What would be the best way to do that?
Ben:
Stop on shutdown is the default for new instances. Block device mapping parameters can be specified when you register an AMI. Disabling API termination needs to be specified for each instance you run.
Eric, I'm just getting started with EC2 and so far your blog has answered a number of questions that have come up.
I'm interested in seeing if you have any suggestions for a better way to handle my use case. I am building an EC2 server for some development work. For the data persistence perspective I would like this server to act like any other Linux machine that I own. This blog entry has taught me how to modify the root volume so that it does not get deleted, but it's a struggle to reconnect to that volume when I start my server back up. This is complicated even more by the fact that I prefer to use spot instances since they are running about 20% the price of a standard instance.
What I have been doing to date is creating a new AMI from my running spot instance just before I terminate it. I have also set the root volume to not be deleted on termination in case I forget to create the new AMI.
If I forget to create the AMI then I, using the AWS console, manually create a snapshot of the volume, start a new spot instance based on my latest AMI and override the block device to point to my new snapshot.
This seems like a lot of work just to be able to persist my data from one instance to the next. Is there a simpler way? If not I plan to script all of this up and include the old snapshot deletion and volume management.
gleehokie:
If you use standard EBS boot instances, you don't get charged while the instance is in the "stopped" state. Stopping and starting an EBS boot instance is the easiest way to reduce costs on a development system.
Spot instances are intended more for servers where you don't care when they run. I agree with you that you are making things complicated by using spot instances to start servers that you need to start and stop on your own schedule. Note also that a spot instance may be terminated without warning even if you are bidding as much as an on-demand instance.
Automating the start/stop of spot instances as if they were on-demand sounds like a challenging project, but it sounds like it could end up saving you a lot of money if you can't afford on-demand instances.