Logging user-data Script Output on EC2 Instances

Real time access to user-data script output

The early implementations of user-data scripts on EC2 automatically sent all output of the script (stdout and stderr) to /var/log/syslog as well as to the EC2 console output, to help monitor the startup progress and to debug when things went wrong.

The recent Ubuntu AMIs still send user-data script to the console output, so you can view it remotely, but it is no longer available in syslog on the instance. The console output is only updated a few minutes after the instance boots, reboots, or terminates, which forces you to wait to see the output of the user-data script as well as not capturing output that might come out after the snapshot.

Here is an example written in bash that demonstrates how to send all user-data script stdout and stderr automatically, transparently, and simultaneously to three locations:

1. /var/log/syslog - In case you’d like to have the results in a standard logging location with automatic timestamps (real time).

2. /var/log/user-data.log - In case you’d like to have a single place to get just the output and nothing but the output (real time).

3. Console output - In case you’d like to be able to see the results through an EC2 API call or the ec2-get-console-output command line program without logging in to the instance (snapshot available a few minutes after boot).

The magic line to put towards the top of your bash script is:

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

From that point on, all output from the bash script and from programs it invokes will go to the above three locations.

Take care to put a space between the two > > characters at the beginning of the statement.

Here’s a complete user-data script as an example:

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
echo END

I started a new Ubuntu 10.10 Maverick instance with this script specified as the user-data:

ec2-run-instances --user-data-file user-data.sh --key $USERNAME ami-508c7839

I waited a bit for the instance to come up, made a note of the instance id and IP address, and then looked for the output in all three places:

$ ssh ubuntu@50.16.102.155 grep user-data: /var/log/syslog
Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo BEGIN
Dec 22 05:00:31 ip-10-117-73-160 user-data: BEGIN
Dec 22 05:00:31 ip-10-117-73-160 user-data: + date '+%Y-%m-%d %H:%M:%S'
Dec 22 05:00:31 ip-10-117-73-160 user-data: 2010-12-22 05:00:31
Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo END
Dec 22 05:00:31 ip-10-117-73-160 user-data: END

$ ssh ubuntu@50.16.102.155 cat /var/log/user-data.log
+ echo BEGIN
BEGIN
+ date '+%Y-%m-%d %H:%M:%S'
2010-12-22 05:00:31
+ echo END
END

$ ec2-get-console-output i-0c9e9461 | grep user-data:
user-data: + echo BEGIN
user-data: BEGIN
user-data: + date '+%Y-%m-%d %H:%M:%S'
user-data: 2010-12-22 05:00:31
user-data: + echo END
user-data: END

And, don’t forget to clean up your running test instance:

ec2kill i-0c9e9461

Notes:

1. The “+” lines showed up in the output because I used the -x option in the bash script. This outputs each statement before executing it. The -e option tells bash to stop running the script as soon as one of the commands returns an error. Combined, these assist in debugging and figuring out what went wrong if there are problems.

2. My ssh command examples do not use the -i option to specify a private key file because I followed my instructions on uploading your personal ssh key to EC2.

3. user-data scripts can be written in any language as long as you can start the program file with shabang (#!). Bash is popular for writing startup scripts since it’s so easy to run other programs and has basic instructions for program flow, which is why I demonstrate it here. Redirecting output for other languages is left as an exercise for the reader.*