Creating Consistent EBS Snapshots with MySQL and XFS on EC2

| 134 Comments

In the article Running MySQL on Amazon EC2 with Elastic Block Store I describe the principles involved in using EBS on EC2. Though originally published in 2008, it is still relevant today and is worth reviewing to get context for this article.

In the above tutorial, I included a sample script which followed the basic instructions in the article to initiate EBS snapshots of an XFS file system containing a MySQL database. For the most part this script worked for basic installations with low volume.

Over the last year as I and my co-workers have been using this code in production systems, we identified a number of ways it could be improved. Or, put another way, some serious issues came up when the idealistic world view of the original simplistic script met the complexities which can and do arise in the brutal real world.

We gradually improved the code over the course of the year, until the point where it has been running smoothly on production systems with no serious issues. This doesn’t mean that there aren’t any areas left for improvement, but does seem like it’s ready for the general public to give it a try.

The name of the new program is ec2-consistent-snapshot.

Features

Here are some of the ways in which the ec2-consistent-snapshot program has improved over the original:

  • Command line options for passing in AWS keys, MySQL access information, and more.

  • Can be run with or without a MySQL database on the file system. This lets you use the command to initiate snapshots for any EBS volume.

  • Can be used with or without XFS file systems, though if you don’t use XFS, you run the risk of not having a consistent file system on EBS volume restore.

  • Instead of using the painfully slow ec2-create-snapshot command written in Java, this Perl program accesses the EC2 API directly with orders of magnitude speed improvement.

  • A preliminary FLUSH is performed on the MySQL database before the FLUSH WITH READ LOCK. This preparation reduces the total time the tables are locked.

  • A preliminary sync is performed on the XFS file system before the xfs_freeze. This preparation reduces the total time the file system is locked.

  • The MySQL LOCK now has timeouts and retries around it. This prevents horrible blocking interactions between the database lock, long running queries, and normal transactions. The length of the timeout and the number of retries are configurable with command line options.

  • The MySQL FLUSH is done in such a way that the statement does not propagate through to slave databases, negatively impacting their performance and/or causing negative blocking interactions with long running queries.

  • Cleans up MySQL and XFS locks if it is interrupted, if a timeout happens, or if other errors occur. This prevents a number of serious locking issues when things go wrong with the environment or EC2 API.

  • Can snapshot EBS volumes in a region other than the default (e.g., eu-west-1).

  • Can initiate snapshots of multiple EBS volumes at the same time while everything is consistently locked. This has been used to create consistent snapshots of RAIDed EBS volumes.

Installation

On Ubuntu, you can install the ec2-consistent-snapshot package using the new Alestic PPA (personal package archive) hosted on Launchpad.net. Here are the steps to set up access to packages in the Alestic PPA and install the software package and its dependencies:

sudo add-apt-repository ppa:alestic &&
sudo apt-get update &&
sudo apt-get install -y ec2-consistent-snapshot

Now you can read the documentation using:

man ec2-consistent-snapshot

and run the ec2-consistent-snapshot command itself.

Feedback

If you find any problems with ec2-consistent-snapshot, please create bug reports in launchpad. The same mechanism can be used to submit ideas for improvement, which are especially welcomed if you include a patch.

Other questions and feedback are accepted in the comments section for this article. If you’re reading this on a planet, please click through on the title to read the comments.

[Update 2011-02-09: Simplify install instructions to not require use of CPAN.]

134 Comments


Thanks a lot for a great script and easy to follow instructions. It worked for me out of the box. The script returns very fast, I had to list the snapshots to make sure it actually worked.

Can you share some thoughts:
1. on how frequently you run
2. how many snapshots to keep
3. is it ok to delete any snapshots (they are incremental, no?)

Thanks again for all your contributions to the community.

Marius

Thanks for this script. Is there something like this for postgres ? Thanks would be awesome;)

lov2cod: Glad it's working for you. The frequency and expiration of snapshots is a personal decision based on a number of factors I can't cover in a comment. There is a limit of snapshots per account which can be raised by request. Deleting snapshots is fine and won't cause any problems even though they are in a sense incremental.

bwaltherb: As I understand it, the folks using this program to snapshot PostgreSQL databases are depending on the natural crash recovery in that software when restoring EBS snapshots. MySQL InnoDB has similar features, but the FLUSH/LOCK reduces recovery time. If there are specific steps to take to improve the reliability and recovery of PostgreSQL, I'd be happy to add the option to this package.

Thxs for the answer.

Another general question: Why do you not just do a mysqldump and then store the dump in to the EBS volume? After that run the snapshot.

Is there a big time advantage to do it with storing the data files directyl in the EBSEBSvolume? When the machine crashes you need to recover from a snapshot anyway and not from the EBS right?

Besides I am generally wondering what data are best stored in EBSbesides database data, e.g. /var/log ? /home ?

And another one: As far as I saw you cannot increase an attached EBS - you would have to reattach a new one an move the stuff correct?

Thanks a bunch

bwaltherb: An EBS snapshot of a volume with the binary database files can be the source for a new EBS volume which can be used by a new database server. You don't need to wait for the database to be loaded from a dump file. An EBS snapshot is near instantaneous to initiate.

You can use EBS to store anything which should be persistent beyond the lifetime of a single instance. If the instance crashes, you can generally attach the EBS volume to a new instance and resume your work.

Resizing a file system residing on an EBS volume involves creating a new EBS volume. There are ways to speed up this process and reduce downtime.

You may need to install 'make' before installing the Net::Amazon::EC2 perl module.

sudo apt-get install make
sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2

Thanks for sharing your considerable experience!

A minor problem I had in the instructions above. The key identifier didn't work:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BE09C571
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys BE09C571
gpg: requesting key BE09C571 from hkp server keyserver.ubuntu.com

I need to interrupt it after about 30 seconds.
Subsequently apt-get update complains of a missing key:

W: GPG error: http://ppa.launchpad.net hardy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0EC7E508BE09C571

But I can recover by replacing the BE09C571 in the original command with the 0EC7E508BE09C571 in the error message:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0EC7E508BE09C571
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys 0EC7E508BE09C571
gpg: requesting key BE09C571 from hkp server keyserver.ubuntu.com
gpg: key BE09C571: public key "Launchpad PPA for Alestic" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)

Shlomo: Thanks for reporting this. The keyserver.ubuntu.com server has been having a bit of trouble the last couple days, not specific to any particular key id. It looks like you caught it at a bad time then tried it again when it was working.

Eric, does the key id shown in your code BE09C571 work for you? I still get the same "stuck" behavior. When I use the full key id (from the subsequent error message) 0EC7E508BE09C571 then the keyserver works.

Or, am I reading too deeply into the key id, and both ids should work but one doesn't (for me)?

The help message in the script is missing a description of the --mysql-host option. Something like this:

Name of the MySQL host whose data directory is on the specified volume(s). Defaults to "localhost".

Shlomo: Both key ids work for me.

Please report bugs (no matter how small) here so they can be tracked:

https://bugs.launchpad.net/ec2-consistent-snapshot/+filebug

I'm having a lot of problems installing this wonderful (at least it appears to be) util.

yesterday I had some problems with the key. Today I've the key but when I launch PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2
It stucks at various points. Sometimes searching for a ftp file and sometimes selecting a country. ┬┐This has something to do with I'm launching it from Europe (Spain)?

Thanks in advance

My installation stucks at this message launching cpan:

(1) Africa
(2) Asia
(3) Australasia
(4) Central America
(5) Europe
(6) North America
(7) Oceania
(8) South America
Select your continent (or several nearby continents) []
Sorry! since you don't have any existing picks, you must make a
geographic selection.

I'm from spain, so my availability zone is eu-west-1, maybe this has something to do with.

How does one retrieve and install this on Debian?

eduardo: What AMI id or Ubuntu release are you running?

konsl: I don't use Debian, but since this tool is written in Perl it should work without modification. You might be able to download the .deb file from the PPA and use "dpkg -i" to install. Or you could just copy the single script file from the Bazaar source code repository and install the dependencies from CPAN. If anybody figures out a simple recipe to install on Debian or any other distro, help out the community by posting it in a comment here.

Hi,

I am just curious -- why have you built packages for different archictectures? It looks like this is a perl script, so you should be able to make a single package for architecture "all" and have it install on every architecture, right? That's how most perl/CPAN packages are built...

zarzavin: In the Launchpad PPA I publish packages for different Ubuntu releases (Dapper, Hardy, Intrepid, Jaunty, Karmic) not different architectures. I'm new to PPAs, but this seemed like the preferred approach until somebody points me in a different direction.

If by "architecture" you meant Ubuntu vs. Debian vs. CentOS, then I'm not aware of any single way to release packages which integrate easily with all of their package management systems.

I would like to see support for Debian added to the Launchpad PPA facility since Ubuntu and Debian use the same package structure. There is a bug report requesting this, but I don't know if the Launchpad developers have any incentive to implement it.

CPAN is not easy to use for non-Perl programmers and the goal of ec2-consistent-snapshot is to be usable by everybody, even non-programmers.

I'm considering turning this into a CPAN package as well, since I'm already a CPAN author. The goal there would be put all of the serious logic into a Perl package which programmers could plug into their applications. Then the ec2-consistent-snapshot command would be a simple wrapper to call the package methods.

Eric,

By architecture I mean "i386", "amd64", etc. Since your package is essentially plain text, you don't need to assign any architecture to it and don't have to publish separate .deb's for each release -- make the package for the architecture "all". It's just a matter of adding a line to your "debian/control" file;

http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/#AEN92

A CPAN package would be great as well!

zarzavin: I see what you mean now. This is being done automatically by Launchpad, so I didn't even notice it.
http://ppa.launchpad.net/alestic/ppa/ubuntu/pool/main/e/ec2-consistent-snapshot/
I'm changing debian/control architecture from "any" to "all" and perhaps the next build will take care of it.

I'm using a customized AMI over one of yours (but now I can't remember) Uname -a tells mee "Linux version 2.6.21.7-2.fc8xen (mockbuild@xenbuilder1.fedora.redhat.com) (gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)) #1 SMP Fri
Feb 15 12:39:36 EST 2008"
If further info required, just tell me what should I type

In case anyone is curious about installation on Debian, I was able to install without any hitches by using

dpkg -i ec2-consistent-snapshot_0.1-3_all.deb
followed by
apt-get -f install
to install the necessary dependencies. The Net::Amazon::EC2 CPAN module also had to be installed.

Is this script a good choice/usable if I'm not using MySQL on the EBS?

brandon: The ec2-consistent-snapshot software adds the most value if you are using MySQL and/or XFS. Otherwise, the main differences between this and the ec2-create-snapshot command include:

1) ec2-consistent-snapshot uses the AWS access key id and AWS secret access key (short strings) while ec2-create-snapshot uses the AWS certificate and AWS private key files.

2) ec2-consistent-snapshot tends to run faster than ec2-create-snapshot (1+ second vs. 4+ seconds).

Forgive me for what may be a simplistic question -- I'm new at this. Since this script a) requires your AWS credentials, and b) must be run from an EC2 instance, what's the standard way of automatically getting your credentials installed on the instance? Assume that ec2-consistent-snapshot will be run automatically on an instance that was provisioned, perhaps by autoscaling.

>>Reply to "eduardo: lsb_release -a"

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 8.04.3 LTS
Release: 8.04
Codename: hardy

Maybe I've not explainde properly my situatio. The message asking me for a Continent, the messages repeats quickly filling my terminal windows. It's in a loop asking me for the continent.

Thanks in advance

eduardo: I was able to reproduce your problem on Ubuntu Hardy. The documentation in the ec2-consistent-snapshot package has been updated with a fix. It basically comes down to using the following on Hardy:

sudo apt-get install -y build-essential
sudo cpan Net::Amazon::EC2

myles: There has been a lot of discussion about how to handle operations on an EC2 instance which require secret keys and no single satisfying answer. Folks seem to use one or more of: store keys in a private AMI; pass keys in through user-data; scp keys in from an outside host; perform operations on an outside host (perhaps at the request of the EC2 instance using a different secret); using different keys to start the instance and to perform the operations, if possible (SimpleDB, SQS, S3).

Eric, does this script write to a log when the snapshot is complete? If not, does anyone know if there's a way I can check for a successful snapshot and then write to a log?

Yeah, I was trying to avoid things like rebundling AMIs because:

a) leads to higher cost for me
b) makes it harder for me to just switch AMIs when they're released.

I also wanted to avoid doing anything that required shipping OTHER keys somehow, like we would need for grabbing them from S3, etc.

Passing them in through user-data seems like the best idea. Thanks!

vaughnhannon: The program has various levels of output which you are welcome to parse and/or send to a log file with standard Linux redirection or pipes.

Yer, I'm currently building a similar set up for a Postgres database and Rails application volumes. Most people will be taking simple LVM or similar snapshots with a Postgres data volume and allowing the Write Ahead Log system in Postgres to do a recovery. You will still want to do a freeze with this if you have a plain XFS partition though as a LVM snapshot will do this internally. Don't use xfs_freeze with LVM snapshots. I don't know of any other filesystem that does an explicit freeze with a userland tool like XFS. Correct me if I'm wrong.

This works pretty well, it's something I've been doing for a long time now and it's a great Postgres feature. It's even meant on one occasion that there has been a serious hardware crash and I've simply remounted the volume to another system, started Postgres and got working. You can also throw a compatible system together, connect the snapshot and have a throwaway test system.

Do you know what the state of play is with XFS and various AMIs Eric? I had heard a while back that there was some trouble with EBS, XFS and various kernels AMIs were using. However, I've got some modified Gentoo and CentOS images with the XFS module already bundled and these all seem to have standardised on a FC8, 2.6.21 Xen kernel. Your Ubuntu images seem to use the same.

This script is a beautiful thing. A minor detail: the script chokes if you have quotes around the username and password in .my.cnf. This took a little while to figure out, but of course there's no reason the quotes have to be there.

Eric,

I think the information you provide here is wonderful. Have you looked at the AWS Perl program by Tim Kay?

The only dependency is Curl.. that's it. I remember when I was banging around on AWS a few years ago and trying to use the Amazon Perl packages.. the dependencies were an immense headache (somehow I overlooked his package.. or maybe it was too new).

He has it up on GitHub as well.. would be great if you joined forces to make the most dominant AWS package on the planet (It currently supports many S3, ec2, elb and sqs commands). I'm trying to read through the Perl code, but it is way beyond my Perl reading abilities:

http://github.com/timkay/aws

Here's the amazon page for it:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=739&categoryID=85

Some of his documentation is lagging the features he's adding.. so it's hard to tell what can be done.. but most ec2 commands are supported.. you can do s3put s3mkdir, s3rm etc.. you can do elastic load balance stuff.. you can make snapshots..

eli: Yes, I use Tim Kay's "aws" program regularly. There are a lot of different ways do do things with AWS/EC2/S3 and many different tools. I've tried to stay away from favoring one over another in the base install AMIs I publish. The easier the tool publishers make it to add them to a running system, the better for the users. For example s3cmd is already an Ubuntu package so it's very easy to install.

I would like for Jeff Kim's Net::Amazon::EC2 Perl package to be simpler to install. I hear that it's a matter of asking the right Debian developers to add it to the list of supported CPAN packages; pointers to how to do this would be appreciated.

Eric,

Does any tool come close to covering everything or is one sort of stuck installing multiple tools to manage everything from EC2 to S3 to Elastic Block Stores?

It would be nice to see something rise to the top and provide all the commands needed to manage all of one's Amazon Web Services stuff..

Also, I can see that there are two ways to approach managing your aws environment..

Command Line (using the Tim Kay "aws" code for example) and treating all of AWS as a direct extension of the OS.. deleting S3 buckets.. cranking up AMI instances.. attaching EBS volumes.. making snapshots.. in the same way one might mkdir and ls (short commands, easy to type.. etc).

Or the more powerful (but also more complex) method of writing full code using Ruby, Python, Perl or Java.

Or well.. three ways.. the third being a mixture of both. :)

Anyway, thanks for the great information you create.. you are a tribute to the Internet.


eli: One cool thing about AWS implementing everything as a "web service" protocol is that it opens the field for competition making the best tools/UIs to control the resources. Different types of interfaces work for different people. In fact, I use a handful of different ways of interacting with AWS depending on the particular task I'm trying to accomplish. California wouldn't be where it is today without a bit of the Wild West chaos.

Hello, I just read your comment post and that is exactly what I still have on my todo list. I am fairly new to the postgres admin and looking for references to learn how to do consistent snapshotting with postgres. Can you maybe help me out maybe with some references where to start? Cheers

I get an error message when running the program, but the snapshot seems to be created. Any thoughts on what's causing this, how to avoid it, or if it's a real problem?

ec2-consistent-snapshot: master_log_file="mysql-bin.000459", 
master_log_pos=106
ec2-consistent-snapshot: ERROR: create_snapshot: Attribute (description) 
does not pass the type constraint because: 
Validation failed for 'Maybe[Str]' failed 
with value HASH(0x2268380) at /usr/local/share/perl/5.10.0/Net/
Amazon/EC2.pm 
line 731
Net::Amazon::EC2::create_snapshot
(undef, 'VolumeId', 'vol-da999999') 
called at /usr/bin/ec2-consistent-snapshot
line 134
eval {...} called at /usr/bin/ec2-consistent-snapshot
line 132
main::ebs_snapshot('ARRAY(0x1811748)', undef) 
called at /usr/bin/ec2-consistent-snapshot 
line 88
Snapshots:  1

I was expecting the mysql host to be picked up from the .my.cnf, in the same way it picked up the mysql user and pass from there. So for now I'm just passing the host as a parameter.

Just thought you should know.

Thanks for a great script.

I'd also like to let you know I have the same problem as the poster at Nov 18 7:06. Here is the output I get (but yes, the snapshot appears to be created):

ec2-consistent-snapshot: ERROR: create_snapshot: Attribute (description) does not pass the type constraint because: Validation failed for 'Maybe[Str]' failed with value HASH(0xa6cd410) at /usr/local/share/perl/5.10.0/Net/Amazon/EC2.pm line 731
Net::Amazon::EC2::create_snapshot(undef, 'VolumeId', 'vol-9999999') called at /usr/bin/ec2-consistent-snapshot line 134
eval {...} called at /usr/bin/ec2-consistent-snapshot line 132
main::ebs_snapshot('ARRAY(0xa0eafd0)', undef) called at /usr/bin/ec2-consistent-snapshot line 88

@marsrc: How did you pass in the host name? I added the switch --mysql-host localhost but still get the same error.

marsrc: The warning message problem has been corrected in the latest release of ec2-consistent-snapshot if you upgrade from the Alestic PPA.

Eric,
When you have RAID'd EBS volumes underlying the filesystem holding your data files, how do you keep track of which snapshot ID corresponds to which RAID segment and which snapshots go together to reconstitute a filesystem?

Right now, it looks like the snapshot ID gets printed to STDOUT (unless suppressed by $Quiet) and that the onus is on the caller to capture that output. Perhaps it'd be preferable to pass the ebs_snapshot sub a handler class with callbacks for success and fall cases. The default handler class could be implemented with the current STDOUT but it'd provide users with the ability to specify implementations that store snapshot outcomes to a file uploaded to S3, a SimpleDB domain or whatever.

eli,
My favorite API for consistent AWS usage is boto (see http://code.google.com/p/boto/). It's python but as long as you're OK with that, I think it's nicer than ad-hoc libraries. For ruby, RightScale's right_aws library is similar. I don't know of a perl analog though.

Since the last update of ec2-consistent-snapshot I'm getting this error when it runs:

ec2-consistent-snapshot: ERROR: create_snapshot: The following parameter was passed in the call to Net::Amazon::EC2::create_snapshot but was not listed in the validation options: Description
at /usr/local/share/perl/5.10.0/Net/Amazon/EC2.pm line 707
Net::Amazon::EC2::create_snapshot(undef, 'VolumeId', 'vol-9d1ce1f4', 'Description', 'ec2-consistent-snapshot') called at /usr/bin/ec2-consistent-snapshot line 136
eval {...} called at /usr/bin/ec2-consistent-snapshot line 134
main::ebs_snapshot('ARRAY(0x9e64e50)', undef, 'ec2-consistent-snapshot') called at /usr/bin/ec2-consistent-snapshot line 90

Any thoughts on what might cause this?

Thx.

V

veeyawn: Please upgrade the Net::Amazon::EC2 package using a command like:

sudo cpan Net::Amazon::EC2

Hi, Eric:

With amazon announcing rds and making a key selling point of it that it offers snapshotting capabilities, how would you compare and contrast your ec2-consistent-snapshot from what amazon offers within its RDS offering?

Currently I have my own ec2 instance running mysql5.1 and am thinking of deploying your script.

But if there were a significant advantage to going in one direction or another, I am able to do so while we are still in our alpha stage....

Thanks in advance,

- Bill

Eric,

Someone at amazon ec2 forums reported seeing data loss on their mysql db where the data folder was resident on an ebs volume. The relevant thread is http://developer.amazonwebservices.com/connect/thread.jspa?threadID=39448&tstart=0

Have you heard of this? I was planning on following your instructions and setting up mysql on the ebs volume but now I am wondering if this is an isolated incident (perhaps engendered by operator error) or something that you and others perhaps have noticed?

Thanks,

Bill

Bill: RDS certainly looks simpler than maintaining your own instances, EBS volumes, and snapshots. I haven't tried it yet since it looked like it required selecting a two hour window each week for possible maintenance and I don't want my sites to go down that often. I also run in master/slave configurations which are not yet supported by RDS. I do a number of configuration changes to make MySQL perform better for my applications and don't know if Amazon's DBAs are going to take quite as much interest in my particular databases, if they configure everything to perform far better than I could. If I were starting fresh, I might consider RDS, but since everything is already in place, I don't have a lot of incentive to switch yet.

Bill: Yes, I saw that thread, but it is difficult to diagnose a report of "missing data", especially when I know how easy it is to accidentally not have a volume attached, or not mounted, or the wrong volume, or on the wrong instance, or... In any case, I believe in multiple backup strategies not just to protect against potential EBS issues (and they do have a failure rate as a block device) but also against human error (which has an even higher failure rate).

Eric....Thanks for your responses! Much appreciated!

Couple requests.

1. Support for more than 1 EBS mount point. We have 10 volumes mounted on /data and /mnt2 has some additional info. Both are XFS filesystems and can be locked. Maybe ability to Lock the database and run an external script? That would allow us to change /etc/my.cnf too.

2. Support for yum for CentOS 5?

3. Email on error?

4. Ability to store in the description the mounted partition. I.e. "/dev/sdi", "/dev/sdi1", "/dev/sdi2". etc. That helps a lot when bringing up a new version.

billnbell: Thanks for the feedback.

1. I run ec2-consistent-snapshot once for each EBS volume mounted. Does this not work for you? Do you have multiple file systems per volume?

2. I'm not a CentOS/Fedora/RHEL developer, but the ec2-consistent-snapshot is published under the Apache 2 license, so feel free to package it up for your favorite distro and make it available. It would be nice to always keep it up to date with the latest releases, though.

3. If you're running ec2-consistent-snapshot from cron, you can use the --quiet option and it will only send you cron email on an error. Otherwise, I'd recommend creating a generic wrapper which can implement this functionality with any shell command, perhaps through pipe input.

4. I'll think about adding this in the default. In the meantime, you can specify your own --description string when invoking ec2-consistent-snapshot.

Eric, thanks for the script (and your previous response to my question); we've been using it for a month or so now, and it seems o be doing great.

Have you considered adding the ability to do grandfather-father-son (as it's apparently called) rotations on the snapshots? Something like: http://www.bermita.com/snapback.html

That's what we're looking at doing now, and I could file a wishlist bug if that's something that sounds interesting to you.

Thanks again!

brandon: I've tried to stay out of the snapshot expiration business with ec2-consistent-snapshot, focusing on doing one thing well and letting folks implement their own expiration policies and procedures. If there are specific hooks that would help implement these, then I'd be interested in adding them. I think a lot can be done with passing in an appropriate snapshot --description value and/or saving the snapshot id returned from the program.

I am complete newbie when talkin about amazon. Still, i have a quick question.

Can i use this to backup an partition(ebs) even if it has more than db data? Like images, or mp3s.

Marcelo: Absolutely, you can store anything on an EBS volume and the snapshot backs it all up.

I installed the ec2-consistent-snapshot program and have a question related to the filesystem. I am using ext3 filesystem and would like to know of any risks using this program to backup the EBS volume.

John: The risk with EBS snapshots of ext3 is that you never know if your application and file system changes have been flushed out the the EBS volume block device or not. This means that you don't know if you will have a consistent file system or the latest data when you restore. Most of the time it will work just fine because changes are generally flushed to disk fairly quickly. I like a little more certainty which is why I use XFS on EC2 EBS volumes.

I'm running a Fedora Core 8 instance and not Ubuntu, so the steps above to install can't be used. Fortunately, it's quite easy to install manually:


yum install cpan
cpan Net::Amazon::EC2
cpan File::Slurp

Finally visit http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/annotate/head%3A/ec2-consistent-snapshot and copy the download file link

wget [download file url]

The script is excellent Eric - thanks

I was wondering the challenges I would be facing by trying to backup my RAID arrays with S3...

Imagine my happiness when I found this script.

You rock Eric. Thanks.

Nice work as always Eric.

I'm trying to figure out the best way to back up an EBS root instance running MySQL and a number of db's. would using --mysql be sufficient?

What would the restore procedure look like? Let's say I want to recover a point in time consistent snapshot, and create an instance from it? Make Volume from snapshot, then register the ebs volume as a new AMI, then launch an instance of it?

Francois: If you are not running on a file system that can be flushed and locked (e.g., XFS) then there is some risk of inconsistencies, though I'm not sure how large. It is possible to build a boot EBS volume on XFS.

Francois: You would register the EBS snapshot as an EBS boot AMI.

I have this running with no problems on two servers, but for some reason I can't get past this part on my new setup (EC2 Canonical 9.10 from EBS):

sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2

Here's the error. Googling hasn't yielded much. Any ideas?:

CPAN: Storable loaded ok (v2.18)
Going to read /home/ubuntu/.cpan/sources/authors/01mailrc.txt.gz
CPAN: Compress::Zlib loaded ok (v2.008)
............................................................................DONE
Going to read /home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz
Warning: Your /home/ubuntu/.cpan/sources/modules/02packages.details.txt.gz
contains a Line-Count header of 76708 but I see 75306 lines there. Please
check the validity of the index file by comparing it to more than one
CPAN mirror. I'll continue but problems seem likely to happen.
Database was generated on Wed, 07 Apr 2010 10:27:16 GMT
CPAN: HTTP::Date loaded ok (v5.831)
...............
New CPAN.pm version (v1.9402) available.
[Currently running version is v1.9205]
You might want to try
install CPAN
reload cpan
to both upgrade CPAN.pm and run the new version without leaving
the current session.

CPAN: Time::HiRes loaded ok (v1.9711)

.............................................................Can't use string ("CPAN::Index") as a HASH ref while "strict refs" in use at /usr/share/perl/5.10/CPAN.pm line 5075.

I have been running into the following error when I try to run ec2-consistent-snapshot:

ec2-consistent-snapshot: master_log_file="mysql-bin.000027", master_log_pos=4310
xfs_freeze: cannot freeze filesystem at /mnt/mysql_data: Operation not permitted
ec2-consistent-snapshot: ERROR: xfs_freeze -f /mnt/mysql_data: failed(256)
snap-def437b6
xfs_freeze: cannot unfreeze filesystem mounted at /mnt/mysql_data: Operation not permitted
ec2-consistent-snapshot: ERROR: xfs_freeze -u /mnt/mysql_data: failed(256)

I'm running the command as the following:

ec2-consistent-snapshot --aws-access-key-id {our key} --aws-secret-access-key {our secret key} --mysql --xfs-filesystem /mnt/mysql_data --mysql-user root vol-1f1db676

The root user does not have a password and the host value should be "localhost." I'm guessing the problem might be some sort of permissions issue where it cannot access the log files properly to flush them. I can go through freeze and locking process manually with the root user with no issue.

Any idea of what my issue might be?

Thanks for the help.

vuzit: If you are going to use --xfs-filesystem, then the ec2-consistent-snapshot program should be executed as root or with sudo.

Thanks Eric.

That was indeed it.

I have another question. I'm trying to install ec2-consistent-snapshot as part of a Capistrano build process.

Everything is fine up until the "sudo PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2" command.

I am installing the "build-essential" package beforehand (this is the solution you gave Eduardo above and I am having the same issue), but it still gets to the prompt about choosing a continent and then fails out.

(1) Africa
(2) Asia
(3) Australasia
(4) Central America
(5) Europe
(6) North America
(7) Oceania
(8) South America
Select your continent (or several nearby continents) []
Sorry! since you don't have any existing picks, you must make a
geographic selection.

Just doing "sudo cpan Net::Amazon::EC2" won't work in Capistrano either as this prompts the user for input several times.

Is there a way to fix this so the module installation will not require input throughout?

If worse comes to worse, we can always install the module seperately on the box after the Capistrano build is done.

vuzit: I've only seen this problem on 8.04 Hardy and I don't know how to solve it. It works for me on 9.10 Karmic.

Apologies in advance for what is likely a simple question.

I've installed your software successfully, and from what I can tell it has run successfully. I am not using the XFS file system, and (at this point) am only trying to snap the file system, not MySQL. So I run this command:
ec2-consistent-snapshot vol-11e15b78
and it returns the id of the snapshot created. I see it in AWS console, and I create a volume from it, attach it to an instance, and mount it. But when I ls -l in the mount point it shows 0 files. What am I doing wrong?

Thanks for your help!

Dan: If you are not using XFS, then there is a chance that the files were not yet written to the block device by the time you snapshotted it. However, it is much more likely that the EBS volume was not mounted where you thought it was when you wrote the files, or you may have mixed up volume or snapshot ids. This stuff works when you do everything right, but it is easy to make mistakes.

Thanks for your quick reply! I bit of confusion on my part: "However, it is much more likely that the EBS volume was not mounted where you thought it was when you wrote the files"

When I run the command I did not pass a parameter containing the path to the mount point. When I did it gave me an error, don't remember the exact message I think it said "no xfs filesystem". Anyway, 2 part question: 1) must I pass the path (if so, how without generating an error) and 2) must the volume to snap be attached to the instance from which the command is run?

Eric,
I ran it a couple more times. This time I used
ec2-consistent-snapshot --xfs-filesystem /var/mansion vol-11e15b78
it complained at the lack of xfs filesystem, but when I mounted the snapshot it generated, I see data. So I think I need to specify the path and ignore the XFS error.

Thanks!

Dan: I don't think I'll be able to diagnose remotely what you did wrong. If the volume is not attached and mounted, then there is no need to use ec2-consistent-snapshot. Just use ec2-create-snapshot and be done with it. The ec2-consistent-snapshot software is a wrapper around the functionality which takes pains to make sure that the mounted file system (and database) is consistent when the create-snapshot is initiated.

Dan: If you aren't using XFS, then this program can't guarantee a consistent snapshot and isn't doing a whole lot of good for you. Your file system may be in an inconsistent state when the snapshot is initiated.

Thanks for the help Eric. I had some success with your tool, but perhaps ec2-create-snapshot is really what I need. I'll try that.

Hi Eric,
When I create a volume from the snapshot and start mysql, I get messages during the mysql startup like
"InnoDB: Database was not shut down normally!
"Starting crash recovery."

Is this normal when starting mysql from a snapshot with ec2-consistent-snapshot or is something wrong? I'm running the original ec2-consistent-snapshot command with --mysql on an xfs volume.

My mysql does eventually startup with my database from the snapshot, but i'd say it takes about 10 minutes to start up due to the innodb recovery process. This long startup tends to play havoc with with my chef scripts since they expect mysql to startup much faster.

I see there's an experimental stop-mysql function, is that for getting around this?

Chris: Yes, the --stop-mysql option helps enforce even more consistency and eliminates the recovery time when starting from a snapshot. Unfortunately, it does require a longer outage, so I only use it on a slave database dedicated to backups. One problem I have found with stopping a replication slave is that it may cause problems if you use temporary tables a lot. The temporary table may not be in existence when the database comes back up.

Hi Eric,

Thanks for a fantastic script! I have been running ec2-consistent-backup as an hourly cron on my production database for a few weeks now, and everything was working great until recently.

From the logging output the script generates, the last thing I see is the ec2-create-snapshot call, followed by an "Alarm clock" message, and no further execution.

I'm guessing this means that the ec2-create-snapshot call timed out and triggered ualarm, but its strange that the END block never seems to get executed, leaving the xfs volume frozen, and the mysql database locked.

Sorry, Perl is definitely not by strong suit, but I don't know how its even possible for the END block *not* to execute, even if the program dies.

Could you offer any insight?

Hi again Eric,

I was able to mostly answer my previous question after some experimentation.

It seems that the eval{} wrapping the ec2-create-snapshot call was not trapping the SIGALRM coming from ualarm. So, the system Alarm Clock was halting the entire script without executing anything further.

My solution was to add the following as the first line to all of the eval{} blocks that call ualarm:

eval {
local $SIG{ALRM} = sub { die "timeout\n" };

ualarm($timeout);
....

};

This problem may have something to do with my specific server environment. I'm running it on a less-recent RH/Fedora/CentOs flavour with Perl 5.8.8.

(As an aside: It required some modifications and trickery to get ec2-consistent-snapshot working at all in this environment. Let me know if you're at all interested in hearing about the details. It might be helpful for a non-Ubuntu version.)

Still not answered is: what would cause "ec2-create-snapshot" to take more than 10 seconds to execute? Typically that call is quite fast.

disbelief: Your comment got lost for a month, but I'm happy it resurfaced. Based on my preliminary tests, I believe you have found and solved the sporadic problem that has been plaguing ec2-consistent-snapshot users for months.

I will be releasing a new version soon with this fix. Thanks!

Hi,

I'm curious why freezing the FS is necessary to ensure a consistent backup.

The Amazon docs seem to say that EBS snapshots are "point in time" consistent. If that's true, shouldn't it be enough to simply run the EBS snapshot process without freezing the FS or locking the tables.

On restore, the DB server should behave as though it is coming back after a power failure. After replaying the FS and InnoDB journals, everything should be good to go, right?

Am I missing something?


-- Andrew

Andrew: I believe that freezing the file system is often (almost always?) not necessary. However, since the order in which modified blocks to the block device is not guaranteed, there is a small chance that you could snapshot the block device at a point in time when the file system is not consistent. Modern file systems can do a pretty good job trying to recover some problems, but I prefer to have that last bit of confidence that things are clean in the again-rare event that I need to depend on a snapshot for disaster recovery.

MySQL should also do pretty well if the database is snapshotted when it is not locked, but again, there may be rare cases where it speeds things up or avoids corruption issues.

I've never been in the habit of powering down my physical servers by unplugging them from the wall, either. I like to use UPS's and shut them down safely and consistently.

This looks great. Do you know if I can make it work on CentOS?

-tom

tom: The ec2-consistent-snapshot software should work just fine on CentOS once you get it installed. I've packaged it for easy install on Ubuntu, but it's just a Perl program that uses other Perl packages.

Thanks for this script, it looks promising.

I am looking for a simple way to create automatic rolling backups of my ec2 server. It is just a single micro instance based on ami-4a0df923 with the one ebs drive attached that has everything on it.

I am mostly running LAMP and Java and want to know if I can switch the file system over to xfs or if I am better off just sticking with ext3. I read the ext3 has the ability to freeze but could not find any documentation of it.

Also, since I am not a coder, I am not sure how to handle deleting old snapshots.

chrowe: XFS gets you the ability to freeze the file system while the snapshot is initiated which may avoid some filesystem inconsistency issues. Not sure how likely they are, so it's hard to just if it's worth the work. There are a number of approaches to do expiration, so I've left this out of the program for now, focusing instead just on doing one job well (consistent snapshot).

folks,

i used the LTS ubuntu ami:
ami-a403f7cd

and followed the directions here:
http://groups.drupal.org/node/36750

to create a RAID0 array with 8 disks of 10gb each... (basically it uses xfs and mdadm to create the raid device)...

i then mounted it and added an entry to the fstab (according to that page) and then rebooted... and now the instance is locked up...

any ideas what i might have done wrong?

thanks,

john

john: Probably not the best place to ask for support on RAID issues. Try the EC2 forum or ec2ubuntu Google group, and provide more details on commands used and fstab contents.

When I run the program and I do show slave status it does appear that it's locked MySQL.

But as the snapshop continues to be pending in AWS web console, MySQL get unlocked before the snapshot is finished.

Also, when I do lsof /mount it shows a lot of open files -- yes the volume is XFS

The EBS volume is 750 GB, not sure if this has ay bearing.

Is this normal behavior or is it not freezing XFS and locking MySQL properly?

Here is the command:

ec2-consistent-snapshot --xfs-filesystem /vol --mysql --mysql-user XXX --mysql-password XXX vol-XXX

berube: The database and file system only need to be locked for the *initiation* of the EBS snapshot. The snapshot is point-in-time which means that it will be consistent as of the time it was started. It doesn't matter that you change the disk after you start the snapshot. None of those changes are reflected in the snapshot itself.

I'm trying to run this command:
$ sudo ec2-consistent-snapshot --mysql --xfs-filesystem /vol --description "test" --aws-credentials-file /home/ubuntu/.awssecret vol-VOLUMEID

I replace VOLUMEID with the correct value.
My .awssecret file contains the both the Amazon AWS access key and secret access key on seprate lines and in that order.

I'm getting the following error on the instance:
ec2-consistent-snapshot: ERROR: AWS was not able to validate the provided access credentials

I have re-read the tutorial and searched Google, but I'm still not sure what I'm missing. Do I need to scp the instance keypair onto the instance itself? That's just a random guess because I'm completely lost.

Do the ec2-api-tools need to be installed on the instance that will run ec2-consistent-snapshot?

Thank you!

P.S. Also, thanks for http://alestic.com/2009/11/ec2-credentials
(Unfortunately, I'm still stuck with "ec2-consistent-snapshot: ERROR: AWS was not able to validate the provided access credentials".)

mountainx:

No, ec2-consistent-snapshot uses the EC2 API directly, so there's no need to install the EC2 API command line tools.

mountainx:

Sounds like you have the right approach. It would be difficult for me to debug from here. Perhaps try using the alternative command line options to specify the access key id and secret access key. This might help figure out if the problem is the values themselves or the format of the file.

Eric, thanks a lot for this great utility. I had some trouble initially getting it installed on CentOS, but now it is working without a hitch. For any one else that wants to install ec2-consistent-snapshot on CentOS, I put together a blog post with the installation steps that worked for us:

http://pojo-mojo.blogspot.com/2011/03/ec2-consistent-snapshot-for-centos.html

peccatorimpius:

Glad to hear it's working for you on CentOS and thanks for posting the install instructions. As a somewhat busy individual I can't hope to support every distro personally, but I've released this software under an open license and welcome contributions from others who can help improve the state of things in their own corners of the world. If any distro needs a different license, just let me know and I'll add it.

Looks like a great script, and just what I've been looking for.

Been working with this all day, (I'm not that familiar with Perl but reviewing the code) and having an issue - there seems to be an issue for me connecting with socket.

If I leave out the socket parameter, the error message is:

DBI connect(';host=localhost:36129','mybackupuser',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /usr/bin/ec2-consistent-snapshot line 226

because that's not the socket. It's defined as /tmp/mysqld.sock (in my.cnf, and confirmed with a "show variables like 'socket';"). Yet when I pass in --mysql-socket "/tmp/mysql.sock" or --mysql-socket '/tmp/mysql.sock' or --mysql-socket /tmp/mysql.sock the error message is:

Value "/tmp/mysql.sock" invalid for option mysql-socket (number expected)

Any ideas?

Larry:

I think you're running into this bug: https://bugs.launchpad.net/ec2-consistent-snapshot/+bug/712210

I have not yet released the easy fix in that ticket, but you could probably tweak your local copy.

Eric - thanks for the tip. I'll give it a try and see what happens!

That was it! Thanks.

Using Ubuntu server 10.10.

When running the command, I get the following error message:

Can't locate DBI.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /usr/bin/ec2-consistent-snapshot line 204.

It doesn't appear to be installed: find / -name DBI.pm doesn't locate anything. After a little googling, this also fails with the same error except for the "at line":
perl -e 'use DBI;'

So, after a little more Googling, and a fearless sense of adventure, I tried:
perl -MCPAN -e shell
cpan[1]> install DBI

And got tons of output and installs. And still get the same error message, even after a reboot.

Help!

Larry:

Unfortunately, debugging your Perl installation is a little beyond the scope of this comment section.

Eric - for what it's worth this may help someone else.

Found the solution starting with: http://ubuntuforums.org/showthread.php?t=203573 and http://forums.mysql.com/read.php?51,71920,84949#msg-84949

sudo apt-get install libc6-dev
cpan -i DBI
cpan -i DBD::mysql

Works like a champ! Thanks so much!

Larry:

Thanks for following up.

Hi Eric

I've found this page after reading your tutorial about "Running MySQL on Amazon EC2 with EBS". Amazing stuff, I learnt a lot from it. Thanks so much for this unvaluable source of info.

I've beginning working with the AWS platform two weeks ago and I choose the AMI Linux platform because of the free tier. I've seen that you prefer the Ubuntu platform, so I had to adapt the code (just a little bit) from your article.

My question is: Is it possible to install the ec2-consistent-snapshot from your package? If not, what would be the easiest way if any?

Best Regards :)

Hi again Eric:

I've finally managed to install it my AMI Linux, following the procedure explained by peccatorimpius at his pojomojo web site.

The whole thing took me more than one hour... tons of output.

The only trouble in my machine was Time::HiRes. I'm going to explain how I solved just in case it could help other people using the AMI platform:

wget http://cpan.perl.org/authors/id/J/JH/JHI/Time-HiRes-1.9721.tar.gz
tar -zxvf Time-HiRes-1.9721.tar.gz
cd Time-HiRes-1.9721.tar.gz
perl Makefile.PL
make
make test
make install

I still have to learn how to use the script and test it but it's a good start. Thanks a lot for all the amazing work you are doing for the aws community :)

jorge-martinez:

I don't use Amazon Linux, so I'm not sure what exact steps are necessary to install the software.

However, if you want to try Ubuntu, it works just fine in the Amazon Free Tier.

Eric:

>>> However, if you want to try Ubuntu, it works just fine in the Amazon Free Tier.

I didn't know that. I began with the AWS platform about three weeks ago, when I launched my first instance from the QuickStart tab the options for the free tier were the AMI Linux 32 or 64 bits. I didn't see any option for Ubuntu.

At home, I use Ubuntu for my laptop as well as Mac OSX for my desktop. Anyway, I've found AMI Linux to work ok.

In your opinion, do you think is better to use Ubuntu instead of AMI Linux? (talking about the AWS platform)

Regards.


Hi again Eric, I'm gonna give Ubuntu a try, I chose

099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-amd64-server-20110201.1

from your home page.

Thanks for the info.

jorge-martinez:

At the moment, I think Ubuntu vs. Amazon Linux comes down to a personal preference. I am more comfortable in Ubuntu.

Hi Eric

I keep my data and binary logs on separate EBS volumes, and snapshot the data daily and logs hourly (or possibly more frequently).

When the data snapshot is done, it would be great if I could get the current binary log position (show by the command "show master status") so that I can restore the data and then replay the logs from that position.

I thought the --mysql-master-status-file option was what I was looking for, but this is a master server in a multi master setup, and --mysql-master-status-file gives me the "show slave status" information instead.

Is this possible at all?

James:

The code is currently written to run "SHOW SLAVE STATUS" first. If that does not return results, then it runs "SHOW MASTER STATUS". I have not run multi-master configurations with MySQL, but if you want to submit a clean patch that will work in all three configurations, I'd be happy to review.

I just wanted to point out that IIRC as of kernel 2.6.29, support for the fs_freeze/fs_unfreeze ioctl's has been added to ext3/4, ReiserFS, and JFS. If your Linux is of that vintage or newer, you probably also have the 'fsfreeze' command to make use of it from scripts; it's analogous to 'xfs_freeze'. This might help some people for whom XFS isn't their FS of choice.

ellenshull:

Thanks. The fact that you can freeze file systems other than XFS was clarified in ec2-consistent-snapshot as of version 0.40 thanks to Jefferson Noxon, and you can even use the option --freeze-filesystem as a synonym for --xfs-filesystem now.

See also: https://bugs.launchpad.net/ec2-consistent-snapshot/+bug/720208

Hi Eric,

Been using this for a while, now running hourly snapshots on our database, and have gone through recovery a couple of times when a customer has done something they didn't mean to.

We've been doing some tuning on mysql qcache, and notice it seems to get cleared every time the snapshot process runs, which is clearly not ideal. Any ideas on what to change so that it doesn't kill the query_cache?

Cheers

Jamie

Worked through this, but when trying to rebuild the database from a snapshot I'm getting an error in MySQL when trying to read tables (Error 1146) saying the table doesn't exist.

The original database is still functioning perfectly, I just can't create a clone of it from the snapshot. Any thoughts at all?

-Chris

Chris:

I haven't heard reports of that type of error from snapshots. I'd recommend trying another snapshot. If it continues not to work, then there must be something special about how you're working the process. Unfortunately, we're probably not going to be able to work it out in blog comments, but if you find a solution, please post a followup.

Maybe I'm just a little dense, but this article is all talking about your "database"... as if we use one server, one ebs volume, per actual database?

Our ebs volume contains many "databases" - does this tool account for that? Or does it presume everyone only ever has one database.

noadvertise:

The ec2-consistent-snapshot does not currently support multiple database servers running on a single EC2 host. However, it works fine if you are running one database server with a bunch of databases being served by that server.

Jamie:

The ec2-consistent-snapshot program currently does a "FLUSH LOCAL TABLES WITH READ LOCK" in an attempt to get all table information flushed to disk before the snapshot takes place. This does clear the query cache according to http://dev.mysql.com/doc/refman/5.0/en/flush.html

If you are aware of a way to accomplish this better, I'm open to new ideas.

I get:

sudo: add-apt-repository: command not found

When trying to install I already tried to install the python stuff as suggested in many google search threads of the issue. What else could this be?

I'm on Ubuntu 8.04


thx

hydn79:

Ubuntu 8.04 LTS Hardy is old (2008) and does not include the add-apt-repository command. For installation instructions scroll down to the "INSTALLATION" section of the ec2-consistent-snapshot manpage here and read the Hardy tips:

https://github.com/alestic/ec2-consistent-snapshot

How remove this limitation if I want to use more then one database. Actually in my office have three branch and use 3 database.

Nashir:

This software does not currently support connecting to multiple MySQL database servers running on the same instance to run the flush/lock statements. This is not a common configuration, though it is possible if the servers are running on different ports.

That said, if the databases are not super-active, then you will probably get consistent snapshots most of them time even without quiescing them.

I would consider a patch that allowed ec2-consistent-snapshot to support multiple databases on a single server, but right now I'm leaning towards thinking that it adds more complexity than it's worth for how few people need it and how much of an improvement you get in the snapshots.

In using ec2-consistent-snapshot, I have some locking going on given that I have a heavier write environment.

How does the XFS freeze actually work?

From what I have read turning it off makes the snapshot inconsistent....inconsistent in what way? In that it could not be usable?

I don't need a true point in time for all the files in the entire volume I just need a backup of all the files on the volume over the course of the snapshot running (if that makes sense).

Steve:

You can think of "inconsistent" as the risk that would happen if you pulled power on an active hard drive. The applications (e.g., database) and file system might have part of the blocks written for a given transaction but other blocks may be waiting to be written. The end result might be that files have some new data and some old, and the file system may be corrupt, possibly losing data.

The risk that this will happen is probably small as applications and file systems often try to protect themselves from corruption, but it depends to some extent on how active your applications are.

Different people will have different tolerance levels for an occasional corrupt database or file system, but since a snapshot is often created as a form of backup, I like to reduce those risks as much as possible in the event I need to depend on it.

Leave a comment

Ubuntu AMIs

Ubuntu AMIs for EC2:


More Entries

When Are Your SSL Certificates Expiring on AWS?
If you uploaded SSL certificates to Amazon Web Services for ELB (Elastic Load Balancing) or CloudFront (CDN), then you will want to keep an eye on the expiration dates and…
Throw Away The Password To Your AWS Account
reduce the risk of losing control of your AWS account by not knowing the root account password As Amazon states, one of the best practices for using AWS is Don’t…
AWS Community Heroes Program
Amazon Web Services recently announced an AWS Community Heroes Program where they are starting to recognize publicly some of the many individuals around the world who contribute in so many…
EBS-SSD Boot AMIs For Ubuntu On Amazon EC2
With Amazon’s announcement that SSD is now available for EBS volumes, they have also declared this the recommended EBS volume type. The good folks at Canonical are now building Ubuntu…
EC2 create-image Does Not Fully "Stop" The Instance
The EC2 create-image API/command/console action is a convenient trigger to create an AMI from a running (or stopped) EBS boot instance. It takes a snapshot of the instance’s EBS volume(s)…
Finding the Region for an AWS Resource ID
use concurrent AWS command line requests to search the world for your instance, image, volume, snapshot, … Background Amazon EC2 and many other AWS services are divided up into various…
Changing The Default "ubuntu" Username On New EC2 Instances
configure your own ssh username in user-data The official Ubuntu AMIs create a default user with the username ubuntu which is used for the initial ssh access, i.e.: ssh ubuntu@<HOST>…
Default ssh Usernames For Connecting To EC2 Instances
Each AMI publisher on EC2 decides what user (or users) should have ssh access enabled by default and what ssh credentials should allow you to gain access as that user.…
New c3.* Instance Types on Amazon EC2 - Nice!
Worth switching. Amazon shared that the new c3.* instance types have been in high demand on EC2 since they were released. I finally had a minute to take a look…
Query EC2 Account Limits with AWS API
Here’s a useful tip mentioned in one of the sessions at AWS re:Invent this year. There is a little known API call that lets you query some of the EC2…
Using aws-cli --query Option To Simplify Output
My favorite session at AWS re:Invent was James Saryerwinnie’s clear, concise, and informative tour of the aws-cli (command line interface), which according to GitHub logs he is enhancing like crazy.…
Reset S3 Object Timestamp for Bucket Lifecycle Expiration
use aws-cli to extend expiration and restart the delete or archive countdown on objects in an S3 bucket Background S3 buckets allow you to specify lifecycle rules that tell AWS…
Installing aws-cli, the New AWS Command Line Tool
consistent control over more AWS services with aws-cli, a single, powerful command line tool from Amazon Readers of this tech blog know that I am a fan of the power…
Using An AWS CloudFormation Stack To Allow "-" Instead Of "+" In Gmail Email Addresses
Launch a CloudFormation template to set up a stack of AWS resources to fill a simple need: Supporting Gmail addresses with “-” instead of “+” separating the user name from…
New Options In ec2-expire-snapshots v0.11
The ec2-expire-snapshots program can be used to expire EBS snapshots in Amazon EC2 on a regular schedule that you define. It can be used as a companion to ec2-consistent-snapshot or…
Replacing a CloudFront Distribution to "Invalidate" All Objects
I was chatting with Kevin Boyd (aka Beryllium) on the ##aws Freenode IRC channel about the challenge of invalidating a large number of CloudFront objects (35,000) due to a problem…
Email Alerts for AWS Billing Alarms
using CloudWatch and SNS to send yourself email messages when AWS costs accrue past limits you define The Amazon documentation describes how to use the AWS console to monitor your…
Cost of Transitioning S3 Objects to Glacier
how I was surprised by a large AWS charge and how to calculate the break-even point Glacier Archival of S3 Objects Amazon recently introduced a fantastic new feature where S3…
Running Ubuntu on Amazon EC2 in Sydney, Australia
Amazon has announced a new AWS region in Sydney, Australia with the name ap-southeast-2. The official Ubuntu AMI lookup pages (1, 2) don’t seem to be showing the new location…
Save Money by Giving Away Unused Heavy Utilization Reserved Instances
You may be able to save on future EC2 expenses by selling an unused Reserved Instance for less than its true value or even $0.01, provided it is in the…
Installing AWS Command Line Tools from Amazon Downloads
This article describes how to install the old generation of AWS command line tools. For the most part, these have been replaced with the new AWS cli that is…
Convert Running EC2 Instance to EBS-Optimized Instance with Provisioned IOPS EBS Volumes
Amazon just announced two related features for getting super-fast, consistent performance with EBS volumes: (1) Provisioned IOPS EBS volumes, and (2) EBS-Optimized Instances. Starting new instances and EBS volumes with…
Which EC2 Availability Zone is Affected by an Outage?
Did you know that Amazon includes status messages about the health of availability zones in the output of the ec2-describe-availability-zones command, the associated API call, and the AWS console? Right…
Installing AWS Command Line Tools Using Ubuntu Packages
See also: Installing AWS Command Line Tools from Amazon Downloads Here are the steps for installing the AWS command line tools that are currently available as Ubuntu packages. These include:…
Ubuntu Developer Summit, May 2012 (Oakland)
I will be attending the Ubuntu Developer Summit (UDS) next week in Oakland, CA. ┬áThis event brings people from around the world together in one place every six months to…
Uploading Known ssh Host Key in EC2 user-data Script
The ssh protocol uses two different keys to keep you secure: The user ssh key is the one we normally think of. This authenticates us to the remote host, proving…
Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the…
CloudCamp
There are a number of CloudCamp events coming up in cities around the world. These are free events, organized around the various concepts, technologies, and services that fall under the…
Use the Same Architecture (64-bit) on All EC2 Instance Types
A few hours ago, Amazon AWS announced that all EC2 instance types can now run 64-bit AMIs. Though t1.micro, m1.small, and c1.medium will continue to also support 32-bit AMIs, it…
ec2-consistent-snapshot on GitHub and v0.43 Released
The source for ec2-conssitent-snapshot has historically been available here: ec2-consistent-snapshot on Launchpad.net using Bazaar For your convenience, it is now also available here: ec2-consistent-snapshot on GitHub using Git You are…