Resizing the Root Disk on a Running EBS Boot EC2 Instance

| 33 Comments

In a previous article I described how to run an EBS boot AMI with a larger root disk size than the default. That’s fine if you know the size you want before running the instance, but what if you have an EC2 instance already running and you need to increase the size of its root disk without running a different instance?

As long as you are ok with a little down time on the EC2 instance (few minutes), it is possible to change out the root EBS volume with a larger copy, without needing to start a new instance.

Let’s walk through the steps on a sample Ubuntu 9.10 Karmic EBS boot instance. I tested this with ami-6743ae0e but check Alestic.com for the latest AMI ids.

On the instance we check the initial size of the root file system (15 GB):

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              15G  675M   14G   5% /

The following commands are all run on a system other than the one we are resizing. Pick a new size (in GB) that is larger than the current size:

instanceid=<YOURINSTANCEID>
size=20

Get the root EBS volume id and availability zone for this instance:

oldvolumeid=$(ec2-describe-instances $instanceid |
  egrep "^BLOCKDEVICE./dev/sda1" | cut -f3)
zone=$(ec2-describe-instances $instanceid | egrep ^INSTANCE | cut -f12)
echo "instance $instanceid in $zone with original volume $oldvolumeid"

Stop (not terminate!) the instance:

ec2-stop-instances $instanceid

Detach the original volume from the instance:

while ! ec2-detach-volume $oldvolumeid; do sleep 1; done

Create a snapshot of the original volume:

snapshotid=$(ec2-create-snapshot $oldvolumeid | cut -f2)
while ec2-describe-snapshots $snapshotid | grep -q pending; do sleep 1; done
echo "snapshot: $snapshotid"

Create a new volume from the snapshot, specifying a larger size:

newvolumeid=$(ec2-create-volume   --availability-zone $zone   --size $size   --snapshot $snapshotid |
  cut -f2)
echo "new volume: $newvolumeid"

Attach the new volume to the instance:

ec2-attach-volume   --instance $instanceid   --device /dev/sda1   $newvolumeid
while ! ec2-describe-volumes $newvolumeid | grep -q attached; do sleep 1; done

Start the instance and find its new public IP address/hostname. (If you were using an elastic IP address, re-assign it to the instance.)

ec2-start-instances $instanceid
while ! ec2-describe-instances $instanceid | grep -q running; do sleep 1; done
ec2-describe-instances $instanceid

Connect to the instance with ssh (not shown) and resize the root file system to fill the new EBS volume. This step is done automatically at boot time on modern Ubuntu AMIs:

# ext3 root file system (most common)
sudo resize2fs /dev/sda1
#(OR)
sudo resize2fs /dev/xvda1

# XFS root file system (less common):
sudo apt-get update && sudo apt-get install -y xfsprogs
sudo xfs_growfs /

Show that the root file system is the new, larger size (20 GB):

$ df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  679M   19G   4% /

Delete the old EBS volume and snapshot if you no longer need them, though I recommend you at least keep the snapshot for a while just in case:

ec2-delete-volume $oldvolumeid
ec2-delete-snapshot $snapshotid

Note: Since you manually created the new volume and attached it to the instance yourself, it will not be deleted automatically when the instance is terminated. You can modify the instance attributes to change the delete-on-termination flag for the volume if you wish.

[Update 2011-11-21: Note that the resize2fs step is done automatically on modern Ubuntu AMIs.] [Update 2011-11-21: Modern Ubuntu AMIs now call it /dev/xvda1 instead of /dev/sda1]

33 Comments

Thanks to Justin for the (unintended) challenge to which this article was a response.
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=42501

A related but general question: is it necessary to "stop" an EBS-based instance before snapshotting the boot volume (not when increasing the size, but just for a usual backup)?

Would the same technique work for _shrinking_ the size of the EBS volume? Thanks. Tom

Tom: You can't create an EBS volume that is smaller than the snapshot on which it is based. You would have to create a smaller volume and then copy the files from the larger volume to the smaller one. Note: A 15 GB EBS volume only costs about $1.50 per month and EBS snapshots are charged by content and deltas, not by the size of the volume.

Thanks for this! Just what I has looking for!
Have tried to have home on a different volume so it would be easier to make it larger when necessary, but it made php (could be suphp spesific) so cannot use that way of doing it. This really helped me a lot!

Maybe its possible to make it a script so it could be done faster?

@Eric, RE: Tom's request for Shrinking the size.

There a couple of reasons that one would want to do this. First, creating an AMI requires a snapshot and the first snapshot of a volume in an account is full sized (e.g. 15GB). Second, to the run the instance, a volume is created and stored. So, the real cost is, generally, doubled.

But, third, and my reason, is to create a stable snapshot for booting, but put data on a separate volume. There would be the instance volume from the AMI referenced snapshot (ideally about 5GB or less, so uses about 10GB on the account) and the data volume (whatever size needed).

Back to the question: Can a running instance simply be duplicated to a separate drive by "just" copying the root ('/') with, say, rsync? If so, shrinking would be trivial and very fast, since (if I recall correctly) the initial installed system isn't even a gigabyte. Are there issues to be aware of (since it's a running system, and all)?

Actually, to answer myself...

I tried a copy, but couldn't boot the new, smaller volume. So, instead, I loaded up an instance, attached the volume I wanted to shrink (/dev/sdf), attached an empty, unformatted volume of the target size (5GB, /dev/sdg).

Performed a check of /dev/sdf (Ubuntu forced me to)
Resized the fs (resize2fs /dev/sdf 5G)
Copied the old to the new with dd (dd if=/dev/sdf of=/dev/sdg)
Checked the new volume
Detached it
Attached it to a stopped instance
Started the instance to check it

And, at this point, the new volume is ready for a snapshot and AMI creation.

It was all pretty simple and safe, just required the use of an extra instance so I was dealing with only unmounted file systems. Being an extra micro instance, the extra cost was just two cents...

So there you have it ... my two cents on shrinking an Ubuntu EC2 EBS boot disk. :)

kf6nvr: dd of the raw device is not a safe way to copy a file system from a larger to a smaller disk. Best to mount both volumes and do an rsync -PazSHAX of the file system.

kf6nvr: EBS snapshots only save the blocks that were touched and it compresses the data. This means that your EBS boot AMI should cost very little in storage.

Hello Eric,

I am new to EC2 and your posts on alestic.com have been immensely helpful. Thank you very much for that.

I am trying to shrink the EBS root volume of the latest official Ubuntu LTS (Lucid 10.04) AMI (ami-480df921) and save it as my own image. The default EBS volume attached as /dev/sda1 is 15GB. On starting the instance and logging in I see that only about 1GB is actually in use.

So if EBS snapshots store only the written/modified parts of the partition how did the snapshot become a 15GB one?

om.brahmana:

1) There is no easy way to shrink an EBS volume nor to start an AMI with a smaller volume than the AMI. This article describes how to resize *larger*, not smaller.

2) The AMI you list is no longer the latest one. In fact, the latest Ubuntu AMIs from Canonical have 8GB root volumes by default.

3) The displayed size of an EBS snapshot is the number of GB that a new volume would be if created from that snapshot. It is not the size of the snapshot in storage as the snapshots only store used/modified blocks and compress the data.

Eric,

Just wanted to say thanks, this article and others of yours have been extremely helpful as I start to dip my (companies) toes into the cloud.

And congratulations. I have been doing a lot of googling on cloud issues the past few weeks and Alestic articles come up over and over again. You are becoming quite the authority.

@Eric!
First of all thanks a lot for the time and effort you put in to that tutorial! Greatly appreciated!

I have a problem and i hope you can point me in the right direction.

I followed your steps but not via the command line tools, instead i used the AWS Management Console
Basically the steps were

1. Stop the instance
2. detach the volume from the instance
3. Create snapshot of the original volume
4. Create a new volume from the snapshot with a larger size
5. attach the new volume to the instance
6. Start the instance

This is where i got stuck. I couldn't be able to SSH to the instance with the new volume mounted.

A quick look at the system log (via the Management Console) revealed the following

"Mounting root filesystem.
VFS: Can't find ext3 filesystem on dev sda1.

mount: error mounting /dev/root on /sysroot as ext3: Invalid argument
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Booting has failed
"

I mounted the volume to the /dev/sda1 just like the original volume.

I'd appreciate a pointer ..


Thanks in advance!

mfractal:

With GUIs it sure is difficult to explain exactly what you did, isn't it? That's why I prefer command lines as it makes it easier to reproduce correct steps and to debug incorrect steps.

Excellent article !

For Europe users, note that you need to specify --region for ec2-describe-instances command

Hi, thanks for writing this. In the line about attaching the new volume, you want to change $volumeid to $newvolumeid!

while ! ec2-describe-volumes $volumeid | grep -q attached; do sleep 1; done

narula:

Fixed, thanks!

Eric I tried to copy the contents of the volume over with the rsync options you listed (-PazSHAX) but the resulting volume contains substantially less data.

Any thoughts?

Any more details on why dd of a shrunken source volume to an equal sized destination volume wouldn't work? As long as you resize the source first and then only dd the (smaller) destination volume's worth of data it seems like that would be a perfect way to clone the disk.

therealcmj:

I would be just as confused/suspicous if you told me that you used "cp" and the file size was different. "dd" can be used, but copies deleted files and causes snapshots to be larger because every block is touched.

Eric,

Having never used Amazon's command line API before, the one thing that got me was having to add -K -C on almost all the commands. I read through Amazon's docs and they mention that you could put these in environment variables but couldn't make this work so I just edited your lines one by one. Not that much work but definitely a pain. Great article, though!

AItOawlj2S8IGYyPUL2o5zBuDWnggVtmaETMbUo:

Yeah, in all my articles I assume that folks have the environment variables set up. I suppose I should write an article that explains how to do some of this groundwork.

I also followed this guide and got stuck at the point where I had to re-connect to my instance.

I took me hours to figure out that the public dns had changed when I attached the new volume!

I had overseen this very important bit of the tutorial:


Start the instance and find its new public IP address/hostname. (If you were using an elastic IP address, re-assign it to the instance.)

This should be printed in RED :)

Thank you for sharing the tutorial!

AItOawkIeuTSzmjmrQnKCvGws9-JRvlBVLzgoFM:

Excellent point. Please consider this entire blog to be written in red. It's a rare sentence I leave in that was intended to be skipped.

Eric, I can confirm that this works for Ubuntu 11.10 server.
I've hardly come across something like this before, following each step to the letter it just works!

AItOawnO_RUPKBDsTEzne5WCAsAcobMl7lq4JXc:

Glad to hear it works for you.

I do a fair amount of testing of the commands in each article before publishing.

Hi,

I have followed the instructions up to the point that I had to resize the /dev/xvda2 partition.

The command:

resize2fs /dev/xvda2

reports:

The filesystem is already XXXXXXX blocks long. Nothing to do!

Any help on that?

AItOawkDGgFU4BlT8gR1HHaGacQR2o8kooLADEQ:

If XXXXXXX is the size you want, then something else (perhaps cloud-init) already resized the file system on boot and there's nothing else for you to do.

It you have a swap partition on the disk, then the ext3 filesystem is in a partition too, and the partition must be resized before the filesystem within it can be resized by resize2fs.

To find out if it is, run

# fdisk -l
Disk /dev/xvde: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000eb738

Device Boot Start End Blocks Id System
/dev/xvde1 1 621 4988151 83 Linux
/dev/xvde2 1215 1245 249007+ 82 Linux swap / Solaris

The number of blocks is the size in Mb. So this disk has a 5Gb partition, and a 256Mb partition, and then 5Gb of free space. Do this to fix this:

1 - "free" to make sure you have some free memory
2 - "swapoff /dev/xvde2"
3 - Use fdisk to delete /dev/xvde2
4 - Follow http://raftaman.net/?p=228 to resize /dev/xvde1
5 - Use fdisk to recreate /dev/xvde2
6 - Reboot
7 - Run fdisk -1 again. You should now see your partition is larger.
8 - Use resize2fs

After this, the free space should be available to you.

realflash:

Repartitioning a disk with a mounted file system makes me a bit uncomfortable. I also don't think there's any need to have swap on EBS, so the partitions seem useless in the first place on EBS volumes.

Hello Eric
Sorry But I am missing something :
>.
The following commands are all run on a system other than the one we are resizing. Pick a new size (in GB) that is larger than the current size:

instanceid=
size=20
DO I have to ssh to the instance and run
instanceid=
size=20
??
and all other commands or else
Please explain

ssharoni:

The commands after "The following commands are all run on a system other than the one we are resizing" are run on your local computer (not the EC2 instance).

The commands after "Connect to the instance with ssh" are run on the EC2 instance.

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


AWS Jobs

AWS Jobs

More Entries

Throw Away The Password To Your AWS Account
reduce the risk of losing control of your AWS account by not knowing the root account password As Amazon states, one of the best practices for using AWS is Don’t…
AWS Community Heroes Program
Amazon Web Services recently announced an AWS Community Heroes Program where they are starting to recognize publicly some of the many individuals around the world who contribute in so many…
EBS-SSD Boot AMIs For Ubuntu On Amazon EC2
With Amazon’s announcement that SSD is now available for EBS volumes, they have also declared this the recommended EBS volume type. The good folks at Canonical are now building Ubuntu…
EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
This article describes how to install the old generation of AWS command line tools. For the most part, these have been replaced with the new AWS cli that is…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…
You Should Use EBS Boot Instances on Amazon EC2
EBS boot vs. instance-store If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store…