Real time access to user-data script output
The early implementations of user-data scripts on EC2 automatically sent all output of the script (stdout and stderr) to
/var/log/syslog as well as to the EC2 console output, to help monitor the startup progress and to debug when things went wrong.
The recent Ubuntu AMIs still send user-data script to the console output, so you can view it remotely, but it is no longer available in syslog on the instance. The console output is only updated a few minutes after the instance boots, reboots, or terminates, which forces you to wait to see the output of the user-data script as well as not capturing output that might come out after the snapshot.
Here is an example written in bash that demonstrates how to send all user-data script stdout and stderr automatically, transparently, and simultaneously to three locations:
/var/log/syslog- In case you’d like to have the results in a standard logging location with automatic timestamps (real time).
/var/log/user-data.log- In case you’d like to have a single place to get just the output and nothing but the output (real time).
Console output - In case you’d like to be able to see the results through an EC2 API call or the
ec2-get-console-outputcommand line program without logging in to the instance (snapshot available a few minutes after boot).
The magic line to put towards the top of your bash script is:
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
From that point on, all output from the bash script and from programs it invokes will go to the above three locations.
Take care to put a space between the two
> > characters at the beginning of the statement.
Here’s a complete user-data script as an example:
#!/bin/bash -ex exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 echo BEGIN date '+%Y-%m-%d %H:%M:%S' echo END
I started a new Ubuntu 10.10 Maverick instance with this script specified as the user-data:
ec2-run-instances --user-data-file user-data.sh --key $USERNAME ami-508c7839
I waited a bit for the instance to come up, made a note of the instance id and IP address, and then looked for the output in all three places:
$ ssh email@example.com grep user-data: /var/log/syslog Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo BEGIN Dec 22 05:00:31 ip-10-117-73-160 user-data: BEGIN Dec 22 05:00:31 ip-10-117-73-160 user-data: + date '+%Y-%m-%d %H:%M:%S' Dec 22 05:00:31 ip-10-117-73-160 user-data: 2010-12-22 05:00:31 Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo END Dec 22 05:00:31 ip-10-117-73-160 user-data: END $ ssh firstname.lastname@example.org cat /var/log/user-data.log + echo BEGIN BEGIN + date '+%Y-%m-%d %H:%M:%S' 2010-12-22 05:00:31 + echo END END $ ec2-get-console-output i-0c9e9461 | grep user-data: user-data: + echo BEGIN user-data: BEGIN user-data: + date '+%Y-%m-%d %H:%M:%S' user-data: 2010-12-22 05:00:31 user-data: + echo END user-data: END
And, don’t forget to clean up your running test instance:
The “+” lines showed up in the output because I used the
-xoption in the bash script. This outputs each statement before executing it. The
-eoption tells bash to stop running the script as soon as one of the commands returns an error. Combined, these assist in debugging and figuring out what went wrong if there are problems.
sshcommand examples do not use the
-ioption to specify a private key file because I followed my instructions on uploading your personal ssh key to EC2.
user-data scripts can be written in any language as long as you can start the program file with shabang (#!). Bash is popular for writing startup scripts since it’s so easy to run other programs and has basic instructions for program flow, which is why I demonstrate it here. Redirecting output for other languages is left as an exercise for the reader.*