Storage server outage - few VMs affected [Update 1]

Today around 19:30 CEST one of our storage servers encountered a massive hard drive failure: 3 out of 7 disks stopped working at once and crashed the system entirely.

However, only a few number of VMs are affected by this and none of our central services are affected: if you did not experience any issue by now then you are not affected.

We are currently restoring VMs that were located on this storage server (some have been restored already). You should see your VMs and services come back within the next few hours.

We will provide an aftermath update later.

[Update 1: 2013-04-18 01:04 CEST]

We finished restoring all business critical machines and all customer services are back online. Our disaster recovery plan estimates about 24 hours for restoring a completely failed storage server. Today it took about 5:30h to perform the necessary analysis, write some skripts, answer customer inquiries, and get services running again.