EBS boot vs. instance-store
If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store and accept my apology that I just mentioned it. Once you are completely comfortable with using EBS boot instances on EC2, you may (or may not) want to come back here and read why you made a good decision.
EC2 experts may find that there are specific cases, few and far between, where instance-store might make sense, but they don’t attempt to use instance-store without understanding and accounting for all the serious drawbacks and dangers that go with making this choice. For example, experts using instance-store don’t mind losing all of the data on the instance as they have designed the system so that the data is stored elsewhere and so that a new instance can easily and automatically be rebuilt from scratch.
One of the challenges for beginners is that many of the benefits of EBS boot don’t necessarily seem like something you’ll need to use right away. Then they get down the road and into situations where they realize that they would have been much better off if they had gone with EBS boot in the first place and may find it takes some work to make the transition.
Big benefits of EBS boot instances
Here are some of the reasons I use and recommend EBS boot instances. None of these benefits are available with instance-store, so even a single one of these can be an overriding factor for choosing EBS boot.
-
EBS boot instances store the root file system on an EBS volume which is persistent storage. If the instance hardware fails, the EBS volume remains accessible. It is also possible to request the EBS volume to persist beyond the termination of an EC2 instance. When an instance-store instance fails or is shut down, all of the data on the root disk is lost forever and can never be retrieved. Read more about protecting EC2 instances from accidental termination and loss of data.
-
EBS boot instances can be stopped and restarted at will. The “stopped” state suspends the hourly instance billing charges, preserving the information on the EBS volumes. The stopped instance can be started again a few minutes later or months later, restoring state just as if the instance was rebooted. An instance-store instance can only be left running with full charges or terminated, which causes you to lose all data on the disk. Read more about how stop/start of an EBS boot instance is similar to and different from a simple reboot.
-
When something goes wrong with an EBS boot instance so that it can’t be booted, or you lose access through ssh (e.g., lost keys, bad ssh config change), you can still view and modify or fix the EBS root volume by attaching it to another running instance. With an instance-store instance, everything on the root disk is lost and cannot be recovered. Read more about fixing files on the root EBS volume of an EC2 instance.
-
EBS boot instances can be run with a root EBS volume size from the default specified by the AMI (often 8GB) up to 1,000GB (1TB). The instance-store AMIs have a max root disk size of 10GB with no way to increase it. Read more about increasing the root disk size of an EBS boot AMI.
-
It is possible to grow the size of the root disk after an EBS boot instance has been started. An instance-store instance has no way to grow the root disk size. Read more about resizing the root disk on a running EBS boot EC2 instance.
-
It is possible to change the instance type for a running EBS boot instance without needing to start a new instance. For example, you can scale up from an m1.large to an m1.xlarge and then a few hours or days later, scale back down. An instance-store instance is stuck with the type on which it was originally run. Read more about changing the instance type of a running EBS boot instance.
-
You can easily replace the hardware for your instance if you are running an EBS boot instance. This is extremely valuable if your instance is having problems that you suspect may be related to the underlying hardware. An instance-store instance is bound to the hardware it started on and cannot be moved. Read more about using stop/start to replace EC2 hardware.
-
EBS boot AMIs are simpler and faster to create than instance-store AMIs. In fact, you can trigger the creation of an EBS boot AMI from a running instance in one command, API call, or console click. You need to copy sensitive AWS credentials to the instance when creating an instance-store AMI.
-
Amazon has stated that EBS boot AMIs boot up faster than S3 based AMIs (instance-store). In my recent experience, the difference is negligible, especially when testing popular AMIs that are likely to be cached, but we might as well chalk this up as another benefit.
-
The t1.micro instance type released recently by Amazon only supports EBS boot instances. This move is like a sign from Amazon that you really don’t want to run the legacy instance-store instances.
-
Some versions of Windows (Server 2008) only run on EBS boot instances. I believe this may be related to the disk size limitations of instance-store, but I don’t use Windows, so am not an expert in that area.
Possible benefits of instance-store instances
EBS volumes and EBS boot instances aren’t perfect. Running an instance-store instance might be preferable in some very specific cases where you don’t care so much about losing the data you are storing on the root disk.
I’m going to list some of the possible benefits of instance-store, but each of these may not be as beneficial as they appear at first glance and you must remember that running instance-store loses all of the above benefits.
-
There is a negigible cost savings with instance-store, as there is no charge for an EBS volume nor the I/O transactions. Note however, that the cost for an 8GB root EBS volume is only around 80 cents per month (depending on the region) and you get about a billion I/O requests for a dollar, billed by the penny increment. The cost savings to run instance-store is generally not worth the increased risk of losing your valuable data, especially when you compare it as a percentage of the cost of running the instance hours in the first place (tens or hundreds of dollars per month).
-
There may be some applications that perform somewhat better when running against ephemeral or instance-store disks than against EBS volumes. Some high end users have also reported inconsistency in the performance they see from EBS volumes at high I/O transaction rates for long periods of time. EBS volumes perform better for some applications and are generally good enough for most applications. If you don’t care about losing your application data, and you have tested to see that instance-store performs better than EBS volumes, then you could either run instance-store instances, or you could run EBS boot instances and drop your data on the ephemeral disks available to all instances above t1.micro. It is also becoming increasingly popular to run multiple EBS volumes in a mdadm RAID-0 or LVM striping configuration to improve performance and smooth out some experiences of performance volatility.
-
There are a couple historical failure modes that have happened with EBS volumes that could not happen for instance-store disks. You may hear people say this is a reason not to use EBS volumes, but EBS volumes are still far more reliable, persistent, and protected against failure than instance-store. The fact that it is possible for EBS to fail is not a reason to use the less persistent instance-store. It is a reason to create regular snapshots of your EBS volumes to improve reliability and act as backups.
-
If you are creating a “paid AMI” you can only do it as an instance-store AMI, not EBS boot. Only two of the thousands of people reading this article are creating paid AMIs and they already know this fact.
Why did I write this article?
I regularly provide consulting services in public communities like serverfault, the ##aws IRC channel on Freenode, the ec2ubuntu Google group, and Amazon’s EC2 forum. It’s a rare week that passes where I am not telling somebody that they would not have the problem they’re having if they had been using EBS boot instances.
The saddest response I can give to a plea for help is that the customer’s valuable data has been lost because they were running instance-store instead of EBS boot and they did not have real-time streaming backups.
Historical background
When Amazon launched EC2 in 2006, only instance-store AMIs were available (though they weren’t called instance-store at the time as they were the only kind). For years, Amazon customers learned to work with the server limitations of risking the loss of all data at any point in time.
In August of 2008, Amazon introduced the concept of EBS volumes, and there was much rejoicing. Data could finally be stored on persistent disks even though the root disk remained on ephemeral storage (another name for instance-store).
In December of 2009, Amazon released the ability to launch instances with EBS volumes as the root disk, or EBS boot instances. Now, the entire server could be persistent and all of the above benefits were realized.
Notes
Just because EBS boot volumes are “persistent” does not mean that they do not ever fail. Amazon has released figures about their failure rates, which are proportional to the number of blocks modified since the last EBS snapshot was created. Regular snapshots improve the intrinsic reliability of your EBS volumes in addition to acting as backups. I also recommend creating regular off-site backups (outside of Amazon) to eliminate your AWS account as a single point of failuire.
Even if you use an EBS boot instance, I still recommend keeping your data on a separate EBS volume. This has a number of benefits that could perhaps form the core of a followup article. Read an example of how to set up a MySQL database on a separate EBS boot volume. I still follow this approach with EBS boot instances, storing data on a second volume.
[Update 2011-01-04: Added benefit #11 (Windows Server 2008). Added note about potential (in)consistency of EBS IO at sustained, high rates.]