The Ubuntu 10.04 Lucid AMis for Amazon EC2 dated 20100923 have a known bug which causes the mountall process to spin CPU when the instance is rebooted.
You can observe this by starting a Lucid instance, running sudo reboot, and then running top after reconnecting.
Cpu(s): 38.5%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 61.5%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
49 root 20 0 4128 1180 920 R 38.6 0.1 0:08.57 mountall
You’ll see that mountall is using all available CPU. The top command may show this as a low number like 38%, but if you also look at the %st or “percent steal”, you’ll see that the VM host is claiming the remaining real CPU cycles as reserved for other instances on that hardware, resulting in zero percent idle for your instance.
This problem comes from a minor change made to /etc/fstab in the latest AMIs along with an esoteric bug in mountall. Looking at /etc/fstab you’ll see the line:
/dev/sda2 /mnt auto defaults,nobootwait,comment=cloudconfig 0 0
The bug manifests itself because “nobootwait” is not the last option in the list. Until it is fixed, as a temporary workaround, we can remedy this with a command like:
sudo perl -pi -e 's/(nobootwait),(\S+)/$2,$1/' /etc/fstab
resulting in the new line looking like:
/dev/sda2 /mnt auto defaults,comment=cloudconfig,nobootwait 0 0
Reboot again, and you’ll see that mountall is now behaving itself.
This fix only needs to be applied once per instance of the current Lucid AMI.
Don’t forget to terminate the instance if you were just following along to test the procedure.
Eventually, we should see the mountall package fixed in Ubuntu 10.04 Lucid, at which point a simple apt update/upgrade should fix it for new instances. Then, when Ubuntu publishes new Ubuntu 10.04 AMIs for EC2 we won’t have to worry about this workaround ever again.
To follow the progress of the fix for this defect, subscribe to the Launchpad bug #649591.
Thanks to Simon de Boer for alerting folks to the problem on EC2, Scott Moser for submitting the bug report (and publishing updated AMIs), and Colin Watson for making sure the bug gets squashed in mountall in Ubuntu.