December 2010 Archives

Real time access to user-data script output

The early implementations of user-data scripts on EC2 automatically sent all output of the script (stdout and stderr) to /var/log/syslog as well as to the EC2 console output, to help monitor the startup progress and to debug when things went wrong.

The recent Ubuntu AMIs still send user-data script to the console output, so you can view it remotely, but it is no longer available in syslog on the instance. The console output is only updated a few minutes after the instance boots, reboots, or terminates, which forces you to wait to see the output of the user-data script as well as not capturing output that might come out after the snapshot.

Here is an example written in bash that demonstrates how to send all user-data script stdout and stderr automatically, transparently, and simultaneously to three locations:

  1. /var/log/syslog - In case you’d like to have the results in a standard logging location with automatic timestamps (real time).

  2. /var/log/user-data.log - In case you’d like to have a single place to get just the output and nothing but the output (real time).

  3. Console output - In case you’d like to be able to see the results through an EC2 API call or the ec2-get-console-output command line program without logging in to the instance (snapshot available a few minutes after boot).

The magic line to put towards the top of your bash script is:

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

From that point on, all output from the bash script and from programs it invokes will go to the above three locations.

Take care to put a space between the two > > characters at the beginning of the statement.

Here’s a complete user-data script as an example:

#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
echo END

I started a new Ubuntu 10.10 Maverick instance with this script specified as the user-data:

ec2-run-instances --user-data-file user-data.sh --key $USERNAME ami-508c7839

I waited a bit for the instance to come up, made a note of the instance id and IP address, and then looked for the output in all three places:

$ ssh ubuntu@50.16.102.155 grep user-data: /var/log/syslog
Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo BEGIN
Dec 22 05:00:31 ip-10-117-73-160 user-data: BEGIN
Dec 22 05:00:31 ip-10-117-73-160 user-data: + date '+%Y-%m-%d %H:%M:%S'
Dec 22 05:00:31 ip-10-117-73-160 user-data: 2010-12-22 05:00:31
Dec 22 05:00:31 ip-10-117-73-160 user-data: + echo END
Dec 22 05:00:31 ip-10-117-73-160 user-data: END

$ ssh ubuntu@50.16.102.155 cat /var/log/user-data.log
+ echo BEGIN
BEGIN
+ date '+%Y-%m-%d %H:%M:%S'
2010-12-22 05:00:31
+ echo END
END

$ ec2-get-console-output i-0c9e9461 | grep user-data:
user-data: + echo BEGIN
user-data: BEGIN
user-data: + date '+%Y-%m-%d %H:%M:%S'
user-data: 2010-12-22 05:00:31
user-data: + echo END
user-data: END

And, don’t forget to clean up your running test instance:

ec2kill i-0c9e9461

Notes:

  1. The “+” lines showed up in the output because I used the -x option in the bash script. This outputs each statement before executing it. The -e option tells bash to stop running the script as soon as one of the commands returns an error. Combined, these assist in debugging and figuring out what went wrong if there are problems.

  2. My ssh command examples do not use the -i option to specify a private key file because I followed my instructions on uploading your personal ssh key to EC2.

  3. user-data scripts can be written in any language as long as you can start the program file with shabang (#!). Bash is popular for writing startup scripts since it’s so easy to run other programs and has basic instructions for program flow, which is why I demonstrate it here. Redirecting output for other languages is left as an exercise for the reader.*

In a thread on the EC2 forum, Marko describes a situation where an outbound firewall prevents the ability to ssh to port 22, which is the default port on all EC2 instances.

In that thread, Shlomo Swidler proposes creating a user-data script that changes sshd to listen on a port the firewall permits.

Here’s a simple example of a user-data script that does just that. Most outbound firewalls allow traffic to port 80 (web/HTTP), so I use it in this example.

The first step is to create a file containing the user-data script:

cat <<'EOM' >user-data-ssh-port-80.txt
#!/bin/bash -ex
perl -pi -e 's/^#?Port 22$/Port 80/' /etc/ssh/sshd_config
service sshd restart || service ssh restart
EOM

The first statement changes the sshd config to listen on port 80 instead of port 22, and the second statement restarts sshd so it will start using this new configuration.

Now you can run a new instance on Amazon EC2, passing in this user-data script. Since the AWS APIs use standard web ports, most outbound firewalls will let through these types of requests. In this example, I use a 64-bit Ubuntu 10.04 Lucid AMI from Canonical which was current as of this article. Please use the most recent AMIs available.

ec2-run-instances --key YOURKEYPAIR --region us-east-1 --instance-type t1.micro --user-data-file user-data-ssh-port-80.txt ami-4a0df923

Save the instance id created and make a note of the IP address once it starts. Now you can ssh in to port 80 using the keypair you specified:

ssh -p 80 -i YOURKEYPAIR.pem ubuntu@IPADDRESS

If you don’t get through right away, make sure your EC2 security group allows access on port 80, and wait long enough for the instance to boot. If you’re connecting to an Amazon Linux AMI, ssh to ec2-user@ instead.

To avoid having to include “-i YOURKEYPAIR.pem” in every ssh, upload your personal ssh key to EC2.

After testing this, don’t forget to terminate your EC2 instance.

See also: Escaping Restrictive/Untrusted Networks with OpenVPN on EC2

Disclaimer

These instructions are not intended to assist in illegal activities. If you are breaking the laws or rules of your government or college or company or ISP, then you should understand the implications and be willing to accept consequences of your actions.

[Update 2010-12-06: Enhanced user-data script to work on both Ubuntu 10.04+ and Amazon Linux (CentOS). Updated AMI id.]

Ubuntu AMIs

Ubuntu AMIs for EC2: