Unplanned VM outage 2011-09-06 16:25-17:00 CEST

Unfortunately some VMs experienced an unplanned outage yesterday (2011-09-05) between 16:25 and 17:00 CEST as their root disks  turned read-only.

To quickly recover from this state we forcedly shut down the VMs and rebooted the associated KVM server. The VMs recovered fine after this reboot.

Owners of affected VMs have been notified individually directly after the incident.

It appears that we have hit a bug in our iSCSI initialisation code that caused all VMs on this physical host to loose connectivity to their storage server.

The issue was triggered while we were bootstrapping a new virtual machine.

The bug is currently undergoing further analysis and replication in our development environment and will be fixed soon.

For the time being we have ceased to perform actions that trigger this bug and do not expect your VM to be affected from this again.