mountall Bug and Workaround in Ubuntu 10.04 Lucid on EC2

The Ubuntu 10.04 Lucid AMis for Amazon EC2 dated 20100923 have a known bug which causes the mountall process to spin CPU when the instance is rebooted.

You can observe this by starting a Lucid instance, running sudo reboot, and then running top after reconnecting.

Cpu(s): 38.5%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 61.5%st
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 49 root      20   0  4128 1180  920 R 38.6  0.1   0:08.57 mountall           

You’ll see that mountall is using all available CPU. The top command may show this as a low number like 38%, but if you also look at the %st or “percent steal”, you’ll see that the VM host is claiming the remaining real CPU cycles as reserved for other instances on that hardware, resulting in zero percent idle for your instance.

This problem comes from a minor change made to /etc/fstab in the latest AMIs along with an esoteric bug in mountall. Looking at /etc/fstab you’ll see the line:

/dev/sda2       /mnt    auto    defaults,nobootwait,comment=cloudconfig 0       0

The bug manifests itself because “nobootwait” is not the last option in the list. Until it is fixed, as a temporary workaround, we can remedy this with a command like:

sudo perl -pi -e 's/(nobootwait),(\S+)/$2,$1/' /etc/fstab

resulting in the new line looking like:

/dev/sda2       /mnt    auto    defaults,comment=cloudconfig,nobootwait 0       0

Reboot again, and you’ll see that mountall is now behaving itself.

This fix only needs to be applied once per instance of the current Lucid AMI.

Don’t forget to terminate the instance if you were just following along to test the procedure.

Eventually, we should see the mountall package fixed in Ubuntu 10.04 Lucid, at which point a simple apt update/upgrade should fix it for new instances. Then, when Ubuntu publishes new Ubuntu 10.04 AMIs for EC2 we won’t have to worry about this workaround ever again.

To follow the progress of the fix for this defect, subscribe to the Launchpad bug #649591.

Thanks to Simon de Boer for alerting folks to the problem on EC2, Scott Moser for submitting the bug report (and publishing updated AMIs), and Colin Watson for making sure the bug gets squashed in mountall in Ubuntu.