In a previous article I described how to run an EBS boot AMI with a larger root disk size than the default. That’s fine if you know the size you want before running the instance, but what if you have an EC2 instance already running and you need to increase the size of its root disk without running a different instance?
As long as you are ok with a little down time on the EC2 instance (few minutes), it is possible to change out the root EBS volume with a larger copy, without needing to start a new instance.
Let’s walk through the steps on a sample Ubuntu 9.10 Karmic EBS boot instance. I tested this with ami-6743ae0e but check Alestic.com for the latest AMI ids.
On the instance we check the initial size of the root file system (15 GB):
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 15G 675M 14G 5% /
The following commands are all run on a system other than the one we are resizing. Pick a new size (in GB) that is larger than the current size:
instanceid=<YOURINSTANCEID>
size=20
Get the root EBS volume id and availability zone for this instance:
oldvolumeid=$(ec2-describe-instances $instanceid |
egrep "^BLOCKDEVICE./dev/sda1" | cut -f3)
zone=$(ec2-describe-instances $instanceid | egrep ^INSTANCE | cut -f12)
echo "instance $instanceid in $zone with original volume $oldvolumeid"
Stop (not terminate!) the instance:
ec2-stop-instances $instanceid
Detach the original volume from the instance:
while ! ec2-detach-volume $oldvolumeid; do sleep 1; done
Create a snapshot of the original volume:
snapshotid=$(ec2-create-snapshot $oldvolumeid | cut -f2)
while ec2-describe-snapshots $snapshotid | grep -q pending; do sleep 1; done
echo "snapshot: $snapshotid"
Create a new volume from the snapshot, specifying a larger size:
newvolumeid=$(ec2-create-volume --availability-zone $zone --size $size --snapshot $snapshotid |
cut -f2)
echo "new volume: $newvolumeid"
Attach the new volume to the instance:
ec2-attach-volume --instance $instanceid --device /dev/sda1 $newvolumeid
while ! ec2-describe-volumes $newvolumeid | grep -q attached; do sleep 1; done
Start the instance and find its new public IP address/hostname. (If you were using an elastic IP address, re-assign it to the instance.)
ec2-start-instances $instanceid
while ! ec2-describe-instances $instanceid | grep -q running; do sleep 1; done
ec2-describe-instances $instanceid
Connect to the instance with ssh (not shown) and resize the root file system to fill the new EBS volume. This step is done automatically at boot time on modern Ubuntu AMIs:
# ext3 root file system (most common)
sudo resize2fs /dev/sda1
#(OR)
sudo resize2fs /dev/xvda1
# XFS root file system (less common):
sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /
Show that the root file system is the new, larger size (20 GB):
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 679M 19G 4% /
Delete the old EBS volume and snapshot if you no longer need them, though I recommend you at least keep the snapshot for a while just in case:
ec2-delete-volume $oldvolumeid
ec2-delete-snapshot $snapshotid
Note: Since you manually created the new volume and attached it to the instance yourself, it will not be deleted automatically when the instance is terminated. You can modify the instance attributes to change the delete-on-termination flag for the volume if you wish.
[Update 2011-11-21: Note that the resize2fs step is done automatically on modern Ubuntu AMIs.] [Update 2011-11-21: Modern Ubuntu AMIs now call it /dev/xvda1 instead of /dev/sda1]
Follow Eric Hammond on Twitter
Thanks to Justin for the (unintended) challenge to which this article was a response.
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=42501
A related but general question: is it necessary to "stop" an EBS-based instance before snapshotting the boot volume (not when increasing the size, but just for a usual backup)?
jedwood: Check out Shlomo's article on that topic: http://www.shlomoswidler.com/2010/01/creating-consistent-snapshots-of-live.html
Excellent. Thanks Eric!
Would the same technique work for _shrinking_ the size of the EBS volume? Thanks. Tom
Tom: You can't create an EBS volume that is smaller than the snapshot on which it is based. You would have to create a smaller volume and then copy the files from the larger volume to the smaller one. Note: A 15 GB EBS volume only costs about $1.50 per month and EBS snapshots are charged by content and deltas, not by the size of the volume.
Thanks for this! Just what I has looking for!
Have tried to have home on a different volume so it would be easier to make it larger when necessary, but it made php (could be suphp spesific) so cannot use that way of doing it. This really helped me a lot!
Maybe its possible to make it a script so it could be done faster?
@Eric, RE: Tom's request for Shrinking the size.
There a couple of reasons that one would want to do this. First, creating an AMI requires a snapshot and the first snapshot of a volume in an account is full sized (e.g. 15GB). Second, to the run the instance, a volume is created and stored. So, the real cost is, generally, doubled.
But, third, and my reason, is to create a stable snapshot for booting, but put data on a separate volume. There would be the instance volume from the AMI referenced snapshot (ideally about 5GB or less, so uses about 10GB on the account) and the data volume (whatever size needed).
Back to the question: Can a running instance simply be duplicated to a separate drive by "just" copying the root ('/') with, say, rsync? If so, shrinking would be trivial and very fast, since (if I recall correctly) the initial installed system isn't even a gigabyte. Are there issues to be aware of (since it's a running system, and all)?
Actually, to answer myself...
I tried a copy, but couldn't boot the new, smaller volume. So, instead, I loaded up an instance, attached the volume I wanted to shrink (/dev/sdf), attached an empty, unformatted volume of the target size (5GB, /dev/sdg).
Performed a check of /dev/sdf (Ubuntu forced me to)
Resized the fs (resize2fs /dev/sdf 5G)
Copied the old to the new with dd (dd if=/dev/sdf of=/dev/sdg)
Checked the new volume
Detached it
Attached it to a stopped instance
Started the instance to check it
And, at this point, the new volume is ready for a snapshot and AMI creation.
It was all pretty simple and safe, just required the use of an extra instance so I was dealing with only unmounted file systems. Being an extra micro instance, the extra cost was just two cents...
So there you have it ... my two cents on shrinking an Ubuntu EC2 EBS boot disk. :)
kf6nvr: dd of the raw device is not a safe way to copy a file system from a larger to a smaller disk. Best to mount both volumes and do an rsync -PazSHAX of the file system.
kf6nvr: EBS snapshots only save the blocks that were touched and it compresses the data. This means that your EBS boot AMI should cost very little in storage.
Hello Eric,
I am new to EC2 and your posts on alestic.com have been immensely helpful. Thank you very much for that.
I am trying to shrink the EBS root volume of the latest official Ubuntu LTS (Lucid 10.04) AMI (ami-480df921) and save it as my own image. The default EBS volume attached as /dev/sda1 is 15GB. On starting the instance and logging in I see that only about 1GB is actually in use.
So if EBS snapshots store only the written/modified parts of the partition how did the snapshot become a 15GB one?
om.brahmana:
1) There is no easy way to shrink an EBS volume nor to start an AMI with a smaller volume than the AMI. This article describes how to resize *larger*, not smaller.
2) The AMI you list is no longer the latest one. In fact, the latest Ubuntu AMIs from Canonical have 8GB root volumes by default.
3) The displayed size of an EBS snapshot is the number of GB that a new volume would be if created from that snapshot. It is not the size of the snapshot in storage as the snapshots only store used/modified blocks and compress the data.
Eric,
Just wanted to say thanks, this article and others of yours have been extremely helpful as I start to dip my (companies) toes into the cloud.
And congratulations. I have been doing a lot of googling on cloud issues the past few weeks and Alestic articles come up over and over again. You are becoming quite the authority.
@Eric!
First of all thanks a lot for the time and effort you put in to that tutorial! Greatly appreciated!
I have a problem and i hope you can point me in the right direction.
I followed your steps but not via the command line tools, instead i used the AWS Management Console
Basically the steps were
1. Stop the instance
2. detach the volume from the instance
3. Create snapshot of the original volume
4. Create a new volume from the snapshot with a larger size
5. attach the new volume to the instance
6. Start the instance
This is where i got stuck. I couldn't be able to SSH to the instance with the new volume mounted.
A quick look at the system log (via the Management Console) revealed the following
"Mounting root filesystem.
VFS: Can't find ext3 filesystem on dev sda1.
mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Booting has failed
"
I mounted the volume to the /dev/sda1 just like the original volume.
I'd appreciate a pointer ..
Thanks in advance!
mfractal:
With GUIs it sure is difficult to explain exactly what you did, isn't it? That's why I prefer command lines as it makes it easier to reproduce correct steps and to debug incorrect steps.
Excellent article !
For Europe users, note that you need to specify --region for ec2-describe-instances command
Hi, thanks for writing this. In the line about attaching the new volume, you want to change $volumeid to $newvolumeid!
while ! ec2-describe-volumes $volumeid | grep -q attached; do sleep 1; done
narula:
Fixed, thanks!
Eric I tried to copy the contents of the volume over with the rsync options you listed (-PazSHAX) but the resulting volume contains substantially less data.
Any thoughts?
Any more details on why dd of a shrunken source volume to an equal sized destination volume wouldn't work? As long as you resize the source first and then only dd the (smaller) destination volume's worth of data it seems like that would be a perfect way to clone the disk.
therealcmj:
I would be just as confused/suspicous if you told me that you used "cp" and the file size was different. "dd" can be used, but copies deleted files and causes snapshots to be larger because every block is touched.
Eric,
Having never used Amazon's command line API before, the one thing that got me was having to add -K -C on almost all the commands. I read through Amazon's docs and they mention that you could put these in environment variables but couldn't make this work so I just edited your lines one by one. Not that much work but definitely a pain. Great article, though!
AItOawlj2S8IGYyPUL2o5zBuDWnggVtmaETMbUo:
Yeah, in all my articles I assume that folks have the environment variables set up. I suppose I should write an article that explains how to do some of this groundwork.
I also followed this guide and got stuck at the point where I had to re-connect to my instance.
I took me hours to figure out that the public dns had changed when I attached the new volume!
I had overseen this very important bit of the tutorial:
Start the instance and find its new public IP address/hostname. (If you were using an elastic IP address, re-assign it to the instance.)
This should be printed in RED :)
Thank you for sharing the tutorial!
AItOawkIeuTSzmjmrQnKCvGws9-JRvlBVLzgoFM:
Excellent point. Please consider this entire blog to be written in red. It's a rare sentence I leave in that was intended to be skipped.
Eric, I can confirm that this works for Ubuntu 11.10 server.
I've hardly come across something like this before, following each step to the letter it just works!
AItOawnO_RUPKBDsTEzne5WCAsAcobMl7lq4JXc:
Glad to hear it works for you.
I do a fair amount of testing of the commands in each article before publishing.
Hi,
I have followed the instructions up to the point that I had to resize the /dev/xvda2 partition.
The command:
resize2fs /dev/xvda2
reports:
The filesystem is already XXXXXXX blocks long. Nothing to do!
Any help on that?
AItOawkDGgFU4BlT8gR1HHaGacQR2o8kooLADEQ:
If XXXXXXX is the size you want, then something else (perhaps cloud-init) already resized the file system on boot and there's nothing else for you to do.
It you have a swap partition on the disk, then the ext3 filesystem is in a partition too, and the partition must be resized before the filesystem within it can be resized by resize2fs.
To find out if it is, run
# fdisk -l
Disk /dev/xvde: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000eb738
Device Boot Start End Blocks Id System
/dev/xvde1 1 621 4988151 83 Linux
/dev/xvde2 1215 1245 249007+ 82 Linux swap / Solaris
The number of blocks is the size in Mb. So this disk has a 5Gb partition, and a 256Mb partition, and then 5Gb of free space. Do this to fix this:
1 - "free" to make sure you have some free memory
2 - "swapoff /dev/xvde2"
3 - Use fdisk to delete /dev/xvde2
4 - Follow http://raftaman.net/?p=228 to resize /dev/xvde1
5 - Use fdisk to recreate /dev/xvde2
6 - Reboot
7 - Run fdisk -1 again. You should now see your partition is larger.
8 - Use resize2fs
After this, the free space should be available to you.
realflash:
Repartitioning a disk with a mounted file system makes me a bit uncomfortable. I also don't think there's any need to have swap on EBS, so the partitions seem useless in the first place on EBS volumes.
Hello Eric
Sorry But I am missing something :
>.
The following commands are all run on a system other than the one we are resizing. Pick a new size (in GB) that is larger than the current size:
instanceid=
size=20
DO I have to ssh to the instance and run
instanceid=
size=20
??
and all other commands or else
Please explain
ssharoni:
The commands after "The following commands are all run on a system other than the one we are resizing" are run on your local computer (not the EC2 instance).
The commands after "Connect to the instance with ssh" are run on the EC2 instance.