The Ubuntu 10.04 Lucid AMis for Amazon EC2 dated 20100923 have a known bug which causes the mountall process to spin CPU when the instance is rebooted.
You can observe this by starting a Lucid instance, running sudo reboot, and then running top after reconnecting.
Cpu(s): 38.5%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 61.5%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
49 root 20 0 4128 1180 920 R 38.6 0.1 0:08.57 mountall
You’ll see that mountall is using all available CPU. The top command may show this as a low number like 38%, but if you also look at the %st or “percent steal”, you’ll see that the VM host is claiming the remaining real CPU cycles as reserved for other instances on that hardware, resulting in zero percent idle for your instance.
This problem comes from a minor change made to /etc/fstab in the latest AMIs along with an esoteric bug in mountall. Looking at /etc/fstab you’ll see the line:
/dev/sda2 /mnt auto defaults,nobootwait,comment=cloudconfig 0 0
The bug manifests itself because “nobootwait” is not the last option in the list. Until it is fixed, as a temporary workaround, we can remedy this with a command like:
sudo perl -pi -e 's/(nobootwait),(\S+)/$2,$1/' /etc/fstab
resulting in the new line looking like:
/dev/sda2 /mnt auto defaults,comment=cloudconfig,nobootwait 0 0
Reboot again, and you’ll see that mountall is now behaving itself.
This fix only needs to be applied once per instance of the current Lucid AMI.
Don’t forget to terminate the instance if you were just following along to test the procedure.
Eventually, we should see the mountall package fixed in Ubuntu 10.04 Lucid, at which point a simple apt update/upgrade should fix it for new instances. Then, when Ubuntu publishes new Ubuntu 10.04 AMIs for EC2 we won’t have to worry about this workaround ever again.
To follow the progress of the fix for this defect, subscribe to the Launchpad bug #649591.
Thanks to Simon de Boer for alerting folks to the problem on EC2, Scott Moser for submitting the bug report (and publishing updated AMIs), and Colin Watson for making sure the bug gets squashed in mountall in Ubuntu.
Follow Eric Hammond on Twitter
Thank you for posting Eric. I've updated the bug with your suggested perl fix. I'll point out, that you do not need to reboot, but only to stop mountall.
So, if you find mountall spinning, the following can fix:
sudo perl -pi -e 's/(nobootwait),(\S+)/$2,$1/' /etc/fstab &&
sudo stop mountall
Its OK to stop mountall, it normally would have run to completion. If you had other mounts in /etc/fstab that haven't been mounted you could run a then run 'mount -a' to get them mounted.
The other option is to avoid this bug on instance launch, by launching with user data like:
#cloud-config
mounts:
- [ ephemeral0 ]
Thanks for the tips, Scott!
Thanks, this already caused some latenight/earlymorning frustrations. There is another irritation I ran into with the latest Ubuntu builds, and that has to do with the console. This is what I see in /var/log/syslog
Sep 30 08:36:25 ip-10-228-61-242 console-kit-daemon[606]: WARNING: Could not determine active console
Sep 30 08:36:25 ip-10-228-61-242 console-kit-daemon[606]: WARNING: Error waiting for native console 3 activation: Invalid argument
Sep 30 08:36:25 ip-10-228-61-242 console-kit-daemon[606]: WARNING: Error waiting for native console 4 activation: Invalid argument
Sep 30 08:36:25 ip-10-228-61-242 console-kit-daemon[606]: WARNING: Error waiting for native console 5 activation: Invalid argument
Sep 30 08:36:25 ip-10-228-61-242 console-kit-daemon[606]: WARNING: Error waiting for native console 6 activation: Invalid argument
Booting takes way too long, obviously. I googled for some solution, and saw that for EC2 something with console=hvc0 should do the trick. But I have no idea how to do that.
You have a solution for this problem as well? :)
jurg: You might want to report a bug on launchpad.net to get this looked at. Here's a place to start with the process: http://alestic.com/2010/01/ec2-ubuntu-bug
Eric: it is already there (https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/606373) I just have no clue how to fix this, or how long to wait for this to propagate.
Hey guys, fstab is getting rewritten after creating an AMI from an instance that has this fix applied, making this a difficult change to persist :) Any tips? Thanks!
AItOawkzJWy1IVYYlnddJqgSGG6R3mrhN7rwUlk: Please seek help for issues like this on the ec2ubuntu Google group or ubuntu-cloud mailing list.