New storage and application server hardware deployed

Yesterday we deployed 4 new application servers, 2 new storage servers and another switch to increase the capacity of our virtual machine cluster in our computing center in Oberhausen. The storage servers now feature 8 SAS 15k  600 GB disks in RAID 5 which should give a good performance boost over the previous SATA-based setup.


Here's pictures of thew newly deployed hardware. Note the neatly arranged cabling!




Infrastructure maintenance in computing center Oberhausen, 2010-08-26 04:00 - 06:00 CEST

Our computing center operator announced that he will perform infrastructure maintenance on 2010-08-26 04:00 - 06:00 to improve the quality, stability, and performance of the switching services.

The work only impacts redundant devices and will only result in very short interruptions.

System update review, next try on 2010-08-24, 20:00 - 24:00 CEST

The system update yesterday went smooth in the sense that no downtime was caused and that we can deploy new hardware tomorrow.

Unfortunately we were unable to update many machines due to an unexpected slow package update preparation procedure that required most of the scheduled downtime.

We are now continuously updating the physical machines over the next days and will attempt another update of the customer-oriented virtual machines next Tuesday between 20:00 and 24:00.

Urgent system update tonight including downtime, 2010-08-18, 20:00 - 24:00 CEST

Due to a planned hardware expansion (new application and storage servers) in our hosting center that is scheduled for Friday we have to urgently deploy a general system update (updated operating system packages, as well as configuration and monitoring improvements).

This will require a restart of all physical and virtual machines as the kernel will be updated.

The update will start at 20:00 CEST tonight and we're hoping to be done by 24:00 CEST. The actual downtime of individual machines is expected to be significantly less than 1 hour.

All of our services provided from the hosting centre in Oberhausen (private and shared virtual machines, mail server, internal monitoring and infrastructure) will be affected at some point during the maintenance period.

Customers with access to the Nagios monitoring infrastructure will receive notifications about the specific service and host availability.

gocept.net blog restructuring

We'd like this blog to become more accessible to our customers and thus we spent a little time today to improve it.

The largest change is that we have split the blog into two separate blogs: one for our international customers that like to communicate in English (that's the blog you are reading right now) and one for our Germany-based customers who prefer communication in German (that's at gocepthostingde.blogspot.com). This change should improve the readability for our international customers and allows us to have two separate RSS/Atom feeds based on the language.

We also updated the design to use a current Blogspot template and also our current corporate design elements.

We hope you like the changes  and - if you have an oppinion - we'd like to hear from you!

Maintenance on Monday 2010-07-12, 12:00-18:00 CEST

On Monday 2010-07-12 we are going to install a backup router in our computing centre Oberhausen. The backup router will take over routing in a matter of seconds in the case of the main router failing.

Due to the installation connectivity to all systems will be limited for short periods between 12:00 and 18:00 CEST.

Teilweiser Dienstausfall (Mittwoch 7.7.2010)

English version follows: please see below.

Am heutigen Mittwoch dem 7.7.2010 kam es zwischen 15:00 und 16:00 zu einem Fehlverhalten eines Skripts auf einem Server, wodurch verschiedene Dienste betroffen waren:
  • Einige virtuelle Maschinen wurden neugestartet. Einige MySQL-Server haben sich nach diesem Neustart nicht automatisch erholt, konnten aber kurzfristig wieder aktiviert werden.
  • Der E-Mail-Server war ebenfalls betroffen. Abruf und Versand war zeitweise nicht möglich oder unzuverlässig. E-Mails aus dieser Zeit sollten inzwischen zugestellt worden sein.
Die endgültige Ursache des Fehlverhaltens in Zusammenhang mit einem Skript zur interaktiven Statusabfrage ist noch nicht geklärt, wir werden dies jedoch weiterverfolgen.

Wir entschuldigen uns für die enstandenen Unannehmlichkeiten. Sollten Sie Fragen oder Anregungen haben, würden wir uns freuen von Ihnen zu hören. Sie erreichen uns per E-Mail unter support@gocept.com.




On Wednesday 2010-07-07 between 15h00 and 16h00 CEST we experienced a misbehaviour of an application server hosting virtual machines causing multiple service interruptions:
  • Some virtual machines were restarted. Some MySQL servers did not recover autonomously but could be started manually and quickly.
  • Delivery of and access to emails was unreliable and sporadically unavailable. Emails from that period should have been delivered by now.
The root cause of the misbehaving script used for interactive status display still has to be identified.

We apologize for any inconvenience. If you have any question or comment we'd be happy to hear from you. You can reach us by email at support@gocept.com.