Creating Consistent EBS Snapshots with MySQL and XFS on EC2

Thanks for this script. Is there something like this for postgres ? Thanks would be awesome;)

bwaltherb: As I understand it, the folks using this program to snapshot PostgreSQL databases are depending on the natural crash recovery in that software when restoring EBS snapshots. MySQL InnoDB has similar features, but the FLUSH/LOCK reduces recovery time. If there are specific steps to take to improve the reliability and recovery of PostgreSQL, I'd be happy to add the option to this package.

bwaltherb: An EBS snapshot of a volume with the binary database files can be the source for a new EBS volume which can be used by a new database server. You don't need to wait for the database to be loaded from a dump file. An EBS snapshot is near instantaneous to initiate.

You can use EBS to store anything which should be persistent beyond the lifetime of a single instance. If the instance crashes, you can generally attach the EBS volume to a new instance and resume your work.

Resizing a file system residing on an EBS volume involves creating a new EBS volume. There are ways to speed up this process and reduce downtime.

Thanks for sharing your considerable experience!

A minor problem I had in the instructions above. The key identifier didn't work:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys BE09C571
gpg: requesting key BE09C571 from hkp server keyserver.ubuntu.com

I need to interrupt it after about 30 seconds.
Subsequently apt-get update complains of a missing key:

W: GPG error: http://ppa.launchpad.net hardy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0EC7E508BE09C571

But I can recover by replacing the BE09C571 in the original command with the 0EC7E508BE09C571 in the error message:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0EC7E508BE09C571
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys 0EC7E508BE09C571
gpg: requesting key BE09C571 from hkp server keyserver.ubuntu.com
gpg: key BE09C571: public key "Launchpad PPA for Alestic" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)

Eric, does the key id shown in your code BE09C571 work for you? I still get the same "stuck" behavior. When I use the full key id (from the subsequent error message) 0EC7E508BE09C571 then the keyserver works.

Or, am I reading too deeply into the key id, and both ids should work but one doesn't (for me)?

Shlomo: Both key ids work for me.

Please report bugs (no matter how small) here so they can be tracked:

https://bugs.launchpad.net/ec2-consistent-snapshot/+filebug

My installation stucks at this message launching cpan:

(1) Africa
(2) Asia
(3) Australasia
(4) Central America
(5) Europe
(6) North America
(7) Oceania
(8) South America
Select your continent (or several nearby continents) []
Sorry! since you don't have any existing picks, you must make a
geographic selection.

I'm from spain, so my availability zone is eu-west-1, maybe this has something to do with.

eduardo: What AMI id or Ubuntu release are you running?

Hi,

I am just curious -- why have you built packages for different archictectures? It looks like this is a perl script, so you should be able to make a single package for architecture "all" and have it install on every architecture, right? That's how most perl/CPAN packages are built...

Eric,

By architecture I mean "i386", "amd64", etc. Since your package is essentially plain text, you don't need to assign any architecture to it and don't have to publish separate .deb's for each release -- make the package for the architecture "all". It's just a matter of adding a line to your "debian/control" file;

http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/#AEN92

A CPAN package would be great as well!

I'm using a customized AMI over one of yours (but now I can't remember) Uname -a tells mee "Linux version 2.6.21.7-2.fc8xen (mockbuild@xenbuilder1.fedora.redhat.com) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) #1 SMP Fri
Feb 15 12:39:36 EST 2008"
If further info required, just tell me what should I type

In case anyone is curious about installation on Debian, I was able to install without any hitches by using

dpkg -i ec2-consistent-snapshot_0.1-3_all.deb

apt-get -f install

brandon: The ec2-consistent-snapshot software adds the most value if you are using MySQL and/or XFS. Otherwise, the main differences between this and the ec2-create-snapshot command include:

1) ec2-consistent-snapshot uses the AWS access key id and AWS secret access key (short strings) while ec2-create-snapshot uses the AWS certificate and AWS private key files.

2) ec2-consistent-snapshot tends to run faster than ec2-create-snapshot (1+ second vs. 4+ seconds).

>>Reply to "eduardo: lsb_release -a"

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.3 LTS
Release: 8.04
Codename: hardy

Maybe I've not explainde properly my situatio. The message asking me for a Continent, the messages repeats quickly filling my terminal windows. It's in a loop asking me for the continent.

Thanks in advance

myles: There has been a lot of discussion about how to handle operations on an EC2 instance which require secret keys and no single satisfying answer. Folks seem to use one or more of: store keys in a private AMI; pass keys in through user-data; scp keys in from an outside host; perform operations on an outside host (perhaps at the request of the EC2 instance using a different secret); using different keys to start the instance and to perform the operations, if possible (SimpleDB, SQS, S3).

Yeah, I was trying to avoid things like rebundling AMIs because:

a) leads to higher cost for me
b) makes it harder for me to just switch AMIs when they're released.

I also wanted to avoid doing anything that required shipping OTHER keys somehow, like we would need for grabbing them from S3, etc.

Passing them in through user-data seems like the best idea. Thanks!

Yer, I'm currently building a similar set up for a Postgres database and Rails application volumes. Most people will be taking simple LVM or similar snapshots with a Postgres data volume and allowing the Write Ahead Log system in Postgres to do a recovery. You will still want to do a freeze with this if you have a plain XFS partition though as a LVM snapshot will do this internally. Don't use xfs_freeze with LVM snapshots. I don't know of any other filesystem that does an explicit freeze with a userland tool like XFS. Correct me if I'm wrong.

This works pretty well, it's something I've been doing for a long time now and it's a great Postgres feature. It's even meant on one occasion that there has been a serious hardware crash and I've simply remounted the volume to another system, started Postgres and got working. You can also throw a compatible system together, connect the snapshot and have a throwaway test system.

Do you know what the state of play is with XFS and various AMIs Eric? I had heard a while back that there was some trouble with EBS, XFS and various kernels AMIs were using. However, I've got some modified Gentoo and CentOS images with the XFS module already bundled and these all seem to have standardised on a FC8, 2.6.21 Xen kernel. Your Ubuntu images seem to use the same.

This script is a beautiful thing. A minor detail: the script chokes if you have quotes around the username and password in .my.cnf. This took a little while to figure out, but of course there's no reason the quotes have to be there.

Eric,

I think the information you provide here is wonderful. Have you looked at the AWS Perl program by Tim Kay?

The only dependency is Curl.. that's it. I remember when I was banging around on AWS a few years ago and trying to use the Amazon Perl packages.. the dependencies were an immense headache (somehow I overlooked his package.. or maybe it was too new).

He has it up on GitHub as well.. would be great if you joined forces to make the most dominant AWS package on the planet (It currently supports many S3, ec2, elb and sqs commands). I'm trying to read through the Perl code, but it is way beyond my Perl reading abilities:

http://github.com/timkay/aws

Here's the amazon page for it:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=739&categoryID=85

Some of his documentation is lagging the features he's adding.. so it's hard to tell what can be done.. but most ec2 commands are supported.. you can do s3put s3mkdir, s3rm etc.. you can do elastic load balance stuff.. you can make snapshots..

Eric,

Does any tool come close to covering everything or is one sort of stuck installing multiple tools to manage everything from EC2 to S3 to Elastic Block Stores?

It would be nice to see something rise to the top and provide all the commands needed to manage all of one's Amazon Web Services stuff..

Also, I can see that there are two ways to approach managing your aws environment..

Command Line (using the Tim Kay "aws" code for example) and treating all of AWS as a direct extension of the OS.. deleting S3 buckets.. cranking up AMI instances.. attaching EBS volumes.. making snapshots.. in the same way one might mkdir and ls (short commands, easy to type.. etc).

Or the more powerful (but also more complex) method of writing full code using Ruby, Python, Perl or Java.

Or well.. three ways.. the third being a mixture of both. :)

Anyway, thanks for the great information you create.. you are a tribute to the Internet.

Hello, I just read your comment post and that is exactly what I still have on my todo list. I am fairly new to the postgres admin and looking for references to learn how to do consistent snapshotting with postgres. Can you maybe help me out maybe with some references where to start? Cheers

I was expecting the mysql host to be picked up from the .my.cnf, in the same way it picked up the mysql user and pass from there. So for now I'm just passing the host as a parameter.

Just thought you should know.

Thanks for a great script.

@marsrc: How did you pass in the host name? I added the switch --mysql-host localhost but still get the same error.

Eric,
When you have RAID'd EBS volumes underlying the filesystem holding your data files, how do you keep track of which snapshot ID corresponds to which RAID segment and which snapshots go together to reconstitute a filesystem?

Right now, it looks like the snapshot ID gets printed to STDOUT (unless suppressed by $Quiet) and that the onus is on the caller to capture that output. Perhaps it'd be preferable to pass the ebs_snapshot sub a handler class with callbacks for success and fall cases. The default handler class could be implemented with the current STDOUT but it'd provide users with the ability to specify implementations that store snapshot outcomes to a file uploaded to S3, a SimpleDB domain or whatever.

Since the last update of ec2-consistent-snapshot I'm getting this error when it runs:

ec2-consistent-snapshot: ERROR: create_snapshot: The following parameter was passed in the call to Net::Amazon::EC2::create_snapshot but was not listed in the validation options: Description
at /usr/local/share/perl/5.10.0/Net/Amazon/EC2.pm line 707
Net::Amazon::EC2::create_snapshot(undef, 'VolumeId', 'vol-9d1ce1f4', 'Description', 'ec2-consistent-snapshot') called at /usr/bin/ec2-consistent-snapshot line 136
eval {...} called at /usr/bin/ec2-consistent-snapshot line 134
main::ebs_snapshot('ARRAY(0x9e64e50)', undef, 'ec2-consistent-snapshot') called at /usr/bin/ec2-consistent-snapshot line 90

Any thoughts on what might cause this?

Thx.

V

Awesome Eric, thank you.

Eric,

Someone at amazon ec2 forums reported seeing data loss on their mysql db where the data folder was resident on an ebs volume. The relevant thread is http://developer.amazonwebservices.com/connect/thread.jspa?threadID=39448&tstart=0

Have you heard of this? I was planning on following your instructions and setting up mysql on the ebs volume but now I am wondering if this is an isolated incident (perhaps engendered by operator error) or something that you and others perhaps have noticed?

Thanks,

Bill

Bill: Yes, I saw that thread, but it is difficult to diagnose a report of "missing data", especially when I know how easy it is to accidentally not have a volume attached, or not mounted, or the wrong volume, or on the wrong instance, or... In any case, I believe in multiple backup strategies not just to protect against potential EBS issues (and they do have a failure rate as a block device) but also against human error (which has an even higher failure rate).

Couple requests.

1. Support for more than 1 EBS mount point. We have 10 volumes mounted on /data and /mnt2 has some additional info. Both are XFS filesystems and can be locked. Maybe ability to Lock the database and run an external script? That would allow us to change /etc/my.cnf too.

2. Support for yum for CentOS 5?

3. Email on error?

4. Ability to store in the description the mounted partition. I.e. "/dev/sdi", "/dev/sdi1", "/dev/sdi2". etc. That helps a lot when bringing up a new version.

Eric, thanks for the script (and your previous response to my question); we've been using it for a month or so now, and it seems o be doing great.

Have you considered adding the ability to do grandfather-father-son (as it's apparently called) rotations on the snapshots? Something like: http://www.bermita.com/snapback.html

That's what we're looking at doing now, and I could file a wishlist bug if that's something that sounds interesting to you.

Thanks again!

I am complete newbie when talkin about amazon. Still, i have a quick question.

Can i use this to backup an partition(ebs) even if it has more than db data? Like images, or mp3s.

I installed the ec2-consistent-snapshot program and have a question related to the filesystem. I am using ext3 filesystem and would like to know of any risks using this program to backup the EBS volume.