Copying EBS Boot AMIs Between EC2 Regions

| 26 Comments

Update: Since this article was written, Amazon has released the ability to copy EBS boot AMIs between regions using the web console, command line, and API. You may still find information of use in this article, but Amazon has solved some of the harder parts for you.

Using Amazon EC2, you created an EBS boot AMI and it’s working fine, but now you want to run instances of that AMI in a different EC2 region. Since AMIs are region specific, you need a copy of the image in each region where instances are required.

This article presents one method you can use to copy an EBS boot from one EC2 region to another.

Setup

Define the region from which we are copying the EBS boot AMI (source) and the region to which we are copying (target). Define the EBS boot AMI that we are copying from the source region.

We also need to determine which ids to use in the target region for the AKI (kernel image) and ARI (ramdisk image). These must correspond to the AKI and ARI in the source region or the new AMI may not work correctly. This is probably the trickiest step of the process and one which is not trivial to automate for the general case.

source_region=us-west-1  # replace with your regions
target_region=eu-west-1
source_ami=[AMI_ID_TO_COPY]
target_aki=[AKI_ID]
target_ari=[ARI_ID]

To make things easier, we’ll upload our own ssh public key to both regions. We could also do this with ssh keys generated by EC2, but that it is slightly more complex as EC2 generates unique keys for each region.

ssh_key_file=$HOME/.ssh/id_rsa
tmp_keypair=copy-ami-keypair-$$
ec2-import-keypair --region $source_region    --public-key-file $ssh_key_file.pub $tmp_keypair
ec2-import-keypair --region $target_region    --public-key-file $ssh_key_file.pub $tmp_keypair

Find the Ubuntu 10.04 LTS Lucid AMI in each of our regions of interest using the REST API provided by Ubuntu. Pick up some required information about the EBS boot AMI we are going to copy.

instance_type=c1.medium
source_run_ami=$(wget -q -O-   http://uec-images.ubuntu.com/query/lucid/server/released.current.txt |
  egrep "server.release.*ebs.i386.$source_region" | cut -f8)
target_run_ami=$(wget -q -O-   http://uec-images.ubuntu.com/query/lucid/server/released.current.txt |
  egrep "server.release.*ebs.i386.$target_region" | cut -f8)
architecture=$(ec2-describe-images --region $source_region   $source_ami | egrep ^IMAGE | cut -f8)
ami_name=$(ec2-describe-images --region $source_region   $source_ami | egrep ^IMAGE | cut -f3 | cut -f2 -d/)
source_snapshot=$(ec2-describe-images --region $source_region   $source_ami | egrep ^BLOCKDEVICEMAPPING | cut -f4)
ami_size=$(ec2-describe-snapshots --region $source_region   $source_snapshot | egrep ^SNAPSHOT | cut -f8)

Image Copy

Start an instance in each region. Have EC2 create a new volume from the AMI to copy and attach it to the source instance. Have EC2 create a new, blank volume and attach it to the target instance.

dev=/dev/sdi
xvdev=/dev/sdi # On modern Ubuntu, you will need to use: xvdev=/dev/xvdi
mount=/image

source_instance=$(ec2-run-instances   --region $source_region   --instance-type $instance_type   --key $tmp_keypair   --block-device-mapping $dev=$source_snapshot::true   $source_run_ami |
  egrep ^INSTANCE | cut -f2)

target_instance=$(ec2-run-instances   --region $target_region   --instance-type $instance_type   --key $tmp_keypair   --block-device-mapping $dev=:$ami_size:true   $target_run_ami |
  egrep ^INSTANCE | cut -f2)

while ! ec2-describe-instances --region $source_region   $source_instance | grep -q running; do sleep 1; done
while ! ec2-describe-instances --region $target_region   $target_instance | grep -q running; do sleep 1; done

source_ip=$(ec2-describe-instances --region $source_region   $source_instance | egrep "^INSTANCE" | cut -f17)
target_ip=$(ec2-describe-instances --region $target_region   $target_instance | egrep "^INSTANCE" | cut -f17)
target_volume=$(ec2-describe-instances --region $target_region   $target_instance | egrep "^BLOCKDEVICE.$dev" | cut -f3)

Copy the file system from the EBS volume in the source region to the to the EBS volume in the target region.

ssh -i $ssh_key_file ubuntu@$source_ip "sudo mkdir -m 000 $mount && sudo mount $xvdev $mount"

ssh -i $ssh_key_file ubuntu@$target_ip "sudo mkfs.ext3 -F -L cloudimg-rootfs $xvdev &&
 sudo mkdir -m 000 $mount && sudo mount $xvdev $mount"

ssh -A -i $ssh_key_file ubuntu@$source_ip "sudo -E rsync    -PazSHAX    --rsh='ssh -o \"StrictHostKeyChecking no\"'    --rsync-path 'sudo rsync'    $mount/    ubuntu@$target_ip:$mount/"

ssh -i $ssh_key_file ubuntu@$target_ip "sudo umount $mount"

The cloudimg-rootfs file system label is required to boot correctly on EC2. It can be left off for other distributions.

AMI Creation

Snapshot the target EBS volume and register it as a new AMI in the target region. If the source AMI included parameters like block device mappings for ephemeral storage, then add these options to the ec2-register command.

target_snapshot=$(ec2-create-snapshot --region $target_region   $target_volume | egrep ^SNAPSHOT | cut -f2)

target_ami=$(ec2-register   --region $target_region   --snapshot $target_snapshot   --architecture $architecture   --name "$ami_name"   --kernel $target_aki   --ramdisk $target_ari |
  cut -f2)

echo "Make a note of the new AMI id in $target_region: $target_ami"

Make a note of the new AMI id.

Cleanup

Terminate the EC2 instances that were used to copy the AMI. Since we let EC2 create the EBS volumes on instance run, EC2 will automatically delete those volumes when the instances terminate. Delete the temporary keypairs we used to access the instances. Clean up the temporary files we created on the local system.

ec2-terminate-instances --region $source_region $source_instance
ec2-terminate-instances --region $target_region $target_instance

ec2-delete-keypair --region $source_region $tmp_keypair
ec2-delete-keypair --region $target_region $tmp_keypair

rm $tmp_private_key $user_data

OPTIONAL: If you don’t want to keep the AMI you created, you can remove it with commands like:

ec2-deregister --region $target_region $target_ami
ec2-delete-snapshot --region $target_region $target_snapshot

Other People’s AMIs

In order to follow this procedure you need to have read access to the snapshot associated with the source AMI, which generally means it must be an EBS boot AMI that you created and registered.

If you want to copy a public EBS boot AMI that somebody else created and you don’t have read access to the EBS snapshot for that AMI (the common case) then you can’t create a volume directly from the snapshot.

However, you should be able to play a little trick where you run an instance of that AMI and immediately stop the instance. Detach the EBS root volume from that instance, and attach it to another instance to perform the copy as above.

The new AMI might not be exactly the same if it had a chance to start the actual boot processes, but it should be pretty close.

Software: migrate-ebs-image

Lincoln Stein has written a program named migrate-ebs-image.pl which migrates an EBS boot AMI from one region to another using the above approach. You can read about it here:

http://search.cpan.org/~lds/VM-EC2/bin/migrate-ebs-image.pl

Cost

The AWS fees for copying an EBS boot AMI following the instructions in this article will include a couple hours of instance time, a small amount for the temporary EBS volumes, and some EBS I/O request charges. You will also be charged a nominal amount each month for the S3 storage of the EBS snapshot for the new AMI in the target region.

[Update 2010-11-01: Added uec-rootfs label for Ubuntu 10.10 thanks to Scott Moser]
[Update 2011-11-30: Updated to use /dev/xvdi for modern Ubuntu.]
[Update 2012-06-25: Replaced uec-rootfs with cloudimg-rootfs for Ubuntu 11.04 and up]
[Update 2012-07-30: Added note about Lincoln Stein’s migrate-ebs-image software]

26 Comments

Let me point yuo to our cloudyscripts-service that provides the same script (and some others) as a web-service with GUI: https://cloudyscripts.com/tool/show/5

elastic-security.com: Thanks for the link; looks like a useful set of tools. Unfortunately, many folks including me are going to be extremely reluctant or forbidden to give a third party the secret keys to the entire AWS account just to get an AMI copied.

It's not that we don't trust you, but.. we don't trust you :-)

How about publishing the code as open source so that people can use it securely on their own systems? Sure this take some work to package as a tool and maintain, but perhaps the community would pitch in to help.

Note: The web tool currently uses the term "availability zone" when it means "region". There are multiple availability zones inside each EC2 region.

Hi Eric, I completely agree with you and understand your concerns of trust. That's why we made the actually scripting code available as Open Source (http://rubygems.org/gems/CloudyScripts) and also provide an AMI to start the service within your own AWS account. Would be happy if guys like you would be involved into this...

Cheers,
Matthias

Matthias: Sweet! You're way ahead of me :)

Based on discussions at UDS (Ubuntu Developer Summit) this week, I think the Ubuntu project may want to include some tools that make it easier to work with EC2 images. I'll point folks your way to see if there might be some opportunity for collaboration to speed things along.

Hi Eric,

Since rsync might list the file permission under ntfs-3g mount type. Any suggestion for Windows NTFS boot disk?

Cheers,
Ping Wu

Hi Eric,

I have a need to copy my EBS AMI based on Amazon Linux AMI using the kernel id aki-9ba0f1de in the us-west-1 region to all the other 3 regions.

You said "We also need to determine which ids to use in the target region for the AKI (kernel image) and ARI (ramdisk image). These must correspond to the AKI and ARI in the source region or the new AMI may not work correctly. This is probably the trickiest step of the process and one which is not trivial to automate for the general case."

Can you please help me out showing me "the trickiest step" how I can find a compatible one in the other 3 regions? Thanks for the great help.

PSK

PSK: You'll need to find the corresponding AMIs in the other regions and use their associated AKI and ARI (if any).

PSK: I used the ec2dim command to find the corresponding AKIs and ARIs in each region. (The description should be same in each region.)

ec2-describe-images --region us-west-1 -o self -o amazon

Eric, thanks for helpful post!

Ping Wu:

I'm afraid I don't know anything about Windows. You'll want to ask your questions on a different forum.

Hi Eric, thanks for a great article, as always.

I created my own AMI, which has a 8 GB boot ebs volume.
I am having problems after starting the source instance with

ec2-run-instances --region $source_region --instance-type $instance_type --key $tmp_keypair --block-device-mapping $dev=$source_snapshot::true $source_run_ami

The command gives back the instance id, which is running.
I can see
BLOCKDEVICE /dev/sdi vol-52dbbb3b 2011-06-06T08:59:56.000Z
which is created from my AMI-snapshot. $source_snapshot exists and is snap-e05b7889.

But the new instance never fully boots. The log says

[ 6.560020] EXT3 FS on sdi, internal journal
...
The disk drive for /vol1 is not ready yet or not present
Continue to wait; or Press S to skip mounting or M for manual recovery

So it looks like the /dev/sdi can not mount als /vol1.
I already tried adding the availability zone on the above command. But had no luck with that. Would you have any ideas?

TIA

Christian

This seems to only address moving an individual partition. Any thoughts on moving partitioned EBS volumes (such as those sometimes used with pv-grub kernels)

For example;
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 2497 20051968 83 Linux
/dev/sdb2 2497 2611 916481 5 Extended
/dev/sdb5 2497 2611 916480 82 Linux swap / Solaris

Do I just have to manually partition and format, then move over the /dev/sdb1 contents?

Man, this is overcomplicated.. it'd be nice if Amazon could do something to address these shortcomings.

AItOawm0FLL0oS3ypXSfxhvasGJBoKVAopxGtfM:

Since I get to pick the size of my EBS volumes, I no longer use partitions. To me, partitioning was for the old days when we bought hardware disks of a fixed size and then had to customize their configuration to meet our specific needs. In modern times, if I want a couple "partitions" for different purposes, I simply create multiple (unpartitioned) EBS volumes, each exactly the size that I want.

In your case, it appears that your boot EBS volume is partitioned with swap space on it. Besides making it more complex to resize the volume, it seems like an inefficient design as every time you swap it goes over the network and charges you EBS IO transactions. Do you really need swap contents to survive a stop/start or could swap be placed on the ephemeral storage?

In fact, I don't recommend using swap on servers in general. In my experience, if a server starts swapping, it is often a sign that things are not designed well. I'd rather have server software crash quickly running out of memory than bog down in swapping so that it's unusable for minutes or hours before it crashes. I do use swap on my laptop, but just so that I can hibernate to disk.

The Ubuntu AMIs use an unpartitioned EBS root volume even with PV, making it easy to start an instance with a bigger root disk:

http://alestic.com/2009/12/ec2-ebs-boot-resize

In fact, the latest Ubuntu releases on EC2 automatically perform a "resize2fs" so that it detects and adjusts to the new size of the EBS volume without manual intervention.

All that said, if you're stuck with an AMI that uses a partitioned EBS root volume, you should still be able to extend the size. You'll probably need to use low level tools to turn off swap, repartition the drive, extend the file system, and turn swap back on. It looks like you might be using Solaris which I haven't used since the 80's, so I can't offer much more help.

I realize that this article is pretty old.

However I totally do not understand why you do not do:

* Create volume of snapshot (or start / stop)
* Attach Volume to instance in source region
* Create Target Volume
* Attach target volume to instance in target region

And then do something like (insecure way, only for public AMIs):

Target:

nc -l 10000 > /dev/[target] # new nc syntax

Source:

dd if=/dev/[source] | nc 10000

DONE and you have a 1:1 block copy of the volume.

Secure version:

Target:

nc -l 10000 > /dev/[target] # new nc syntax

Source:

ssh -N 10000:localhost:10000 -i &
dd if=/dev/[source] | nc localhost 10000

And again done.

AItOawnGMHi_s2Rs0ZFPvNLJuA4aFfzV01XKbfg:

The article is old, but the technique is still recommended for copying EBS boot AMIs.

The nc approach only replaces the mount/rsync/umount commands, so it's not saving much. Here's what you're losing by using nc on the raw block devices:

1. rsync only copies the data that exists in the file system. This is much less than all of the blocks on the raw device, saving you transfer time.

2. rsync compresses the data significantly, saving you transfer time.

3. rsync will only modify the minimum number of blocks on the target EBS volume required to save the file contents. The nc to raw block device will touch every block on the target EBS volume. This increases the size of the EBS snapshot, increasing storage cost, and possibly increasing instance startup time from the resulting AMI, not to mention EBS snapshot size of EBS volumes on the new instances, and so on.

To the benefit of posterity (at to make sense of half day of swearing):

if, like me you're still on 10.4 LTS EBS, you have to change the label for mkfs.ext3 from uec-rootfs to cloudimg-rootfs.

Thanks Eric

YMMV but for me the "big rysnc command" that the whole article hinges on didn't work. I found that the rysnc wants to use 'root' as the account even though I thought I was using the same version of ubuntu as this article recommends. I was logged in as ubuntu, so I'm not sure it made this choice.

I discovered this by adding -vv to all the ssh commands and watching what keys were offered to the remote side, including during the setup of the rysnc.

Once I put the correct keys (pub/private) in place for root, everything went ok.

ianster:

My best guess as to what is going on given your description is that you did not upload the correct ssh key to EC2, or specify it in starting the temporary EC2 instances, or you are using a different local account from the ssh key when you run the ssh command.

You can certainly get the copy done by copying ssh keys around to the various instances in .ssh/authorized_keys, but the goal of this article is to have that all set up for you by pre-loading your default ssh key into EC2.

Thanks for this article, Eric.

I'm getting a problem with the two Ubuntu 10.04 images though; I am unable to log in to them. I'm getting an "ssh: Connection timed out" error message. Up until that point, everything seemed to work fine.

Both instances (source and target) are running.

Any ideas what might be causing this? Changes in the Ubuntu 10.04 AMI's or something on my side?

Steinar Haugen:

It's near impossible for me to be able to debug a connection problem without having hands on and seeing every step you took. If you copied and pasted the instructions here, then make sure that you are using the correct matching kernel AKI id.

"The uec-rootfs file system label is required for Ubuntu 10.10 and above to boot correctly on EC2. It can be left off for other distributions and earlier versions of Ubuntu."

As I am using the Ubuntu Server 12.04 LTS AMI, I took the above quote to mean that the uec-rootfs label was the correct one to use. This proved not to be the case - the cloudimg-rootfs label is the one to go for.

When booting the source instance with a copy of the AMI as an extra volume, I've come across a difficulty using the current Ubuntu 12.04 AMIs. The extra volume /dev/xvdi is getting mounted as the root partition rather than /dev/xvda1. I suspect it is because both filesystems have the label cloudimg-rootfs, as they are both root volumes. I suspect the choice is deterministic, but probably liable to change again in the future.
It's easy to demonstrate with the current AMI:
ec2-run-instances --instance-type m1.small --key --block-device-mapping /dev/sdi=snap-67aa4c19::true ami-82fa58eb
At the moment I'm thinking of creating the /dev/sdi volume separately, and attaching it after the base AMI has booted.

jasonjhooper: Thanks for pointing this out. Your workaround of attaching after boot sounds perfect.

Thanks for the information. Based on this post I recently wrote a Perl script that automatically copies the AMI root and other associated volumes between regions, and then registers the image in the new region using the correct corresponding kernel and ramdisks. The script also handles oddball cases such as AMIs that contain LVM and/or RAID disks.

The command line looks like this:

migrate-ebs-image.pl --from us-east-1 --to ap-southeast-1 ami-123456

and can be found here:

http://search.cpan.org/~lds/VM-EC2-1.10/bin/migrate-ebs-image.pl

Lincoln Stein:

Sweet! I've updated the article to include a pointer since nobody is going to read this far down in the comments.

Version 1.23 of migrate-ebs-image.pl now supports migrating Windows and compute cluster AMIs from one region to another

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


More Entries

EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
When you need an AWS command line toolset not provided by Ubuntu packages, you can download the tools directly from Amazon and install them locally. In a previous article I…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…
You Should Use EBS Boot Instances on Amazon EC2
EBS boot vs. instance-store If you are just getting started with Amazon EC2, then use EBS boot instances and stop reading this article. Forget that you ever heard about instance-store…
Retrieve Public ssh Key From EC2
A serverfault poster had a problem that I thought was a cool challenge. I had so much fun coming up with this answer, I figured I’d share it here as…
Running EC2 Instances on a Recurring Schedule with Auto Scaling
Do you want to run short jobs on Amazon EC2 on a recurring schedule, but don’t want to pay for an instance running all the time? Would you like to…
AWS Virtual MFA and the Google Authenticator for Android
Amazon just announced that the AWS MFA (multi-factor authentication) now supports virtual or software MFA devices in addition to the physical hardware MFA devices like the one that’s been taking…