On Monday, 2011-10-31 one of our VM servers died completely. This caused an unplanned outage for several customer VMs and our mail server mail.gocept.net. We were able to limit the total down times by migrating the affected VMs to other servers quickly.
It is unlikely that any data has been lost. Please excuse the service disruption.
The server started to decline network requests around 15:30 CET. Our stand-by support team was alerted immediatly. After we had come to the conclusion that we could not revive the hardware quickly, we began on 16:15 CET to move VMs to other Servers. Around 16:45 CET services have been functional again.
![gocept.net [en]](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY65wXXMC4QddrR7HTZnjU9u-41YNivbxQFTE7ptP_w-pxs5E-UwY2lF_zetWI19slZ_BIYljxRVsXdeUffNfpsoyKXOaA-jj45jrcuIAdUZiSTYDyjEAckKVIODgfGWWFqc-JuybhbgRa/s1600-r/logo.png){kind=logo}
Security breach on an internal system on 2011-10-18
On Tuesday (2011-10-18) at 22:45 CEST an attacker managed to gain access to one of our internal systems running, among others, the external service monitoring, the Redmine project management tool, and an internal mailing list server. The compromised system is not automatically managed.
Customer VMs were not affected.
We needed to take the machine partially off the network between Wednesday (2011-10-19) 13:20 and 21:00 CEST to perform an analysis and fix security holes. The above mentioned services had only limited availability during this period. User-generated data has most likely not been compromised. We decided to take the machine back online to make the services available again as quickly as possible.
Nevertheless we will move the services to newly installed machines shortly and erase the compromised machine. We will review and improve our security practices to avoid similar incidents in the future.
Customer VMs were not affected.
We needed to take the machine partially off the network between Wednesday (2011-10-19) 13:20 and 21:00 CEST to perform an analysis and fix security holes. The above mentioned services had only limited availability during this period. User-generated data has most likely not been compromised. We decided to take the machine back online to make the services available again as quickly as possible.
Nevertheless we will move the services to newly installed machines shortly and erase the compromised machine. We will review and improve our security practices to avoid similar incidents in the future.
Subscribe to:
Posts (Atom)