Filesystem migration to ext4

As part of our ongoing platform improvement, we will migrate ext3 filesystems to the ext4 format. The migration requires an off-line filesystem check and thus a little downtime for each machine to reboot.

The downtimes will be scheduled individually according to the agreed maintenance windows. We will inform the technical contacts about the exact times for their VMs in advance. Please write to our support if any reboot time does not fit.

Benefits of the new filesystem

The ext4 filesystem offers a number of advantages over the old ext3 filesystem:

• Reduced filesystem check time: this means that both planned machine reboots and error recovery after outages will be sped up significantly.
• Large file handling: access to large files like ZODB databases require less system overhead.
• Large directory handling: file lookup in directories that contain a lot of files is more efficient.
• More subtle run-time optimizations like delayed allocation and others.

After the VMs' disks have been migrated to the new format, applications will benefit from the performance optimizations incrementally as their files are modified. ZODB databases are usually packed once a week, so they will pick up the new on-disk format automatically. Blob directories and PostgreSQL databases will use the new on-disk format only for newly written data. We can assist users with the conversion of old data on request.

Except for the reboot, the change will be completely transparent.

Maintenance on 2012-10-09 22:00 CEST - update 2

To finally finish the exchange of the faulty switch we will perform a series of hardware maintenance steps tomorrow (2012-10-09) night between 22:00 and 24:00 CEST.

The following tasks will be performed:

• Finish the migration to a new power distribution system in our racks
• Move our standby switch next to the faulty switch, verify correct operation
• Move the existing network connections from the faulty switch to the replacement
• Install a new standby switch

As our switches do not have a redundant power supply there will be a short outage of the whole network for about 1-2 minutes. We do not expect any failures in operation but existing connections may hang for this period.

Also, when moving the cables from the faulty switch to its replacement there will be short lags in storage or server network connectivity for a few seconds but no outages. 

We are sorry that this preventive measure has taken multiple attempts to implement. We think that our decisions to support a stable environment with careful small adjustments is in the interest of your operational needs.

Update 1 [2012-10-08 12:26]

The previous version of this post mentioned 2012-10-08 as the date of the maintenance. The actual scheduled date is 2012-10-09. The text above was corrected.

Update 2 [2012-10-09 23:32]

The faulty switch has, finally, been successfully replaced. In a window of about 5 minutes the redundant routers where in an inconsistent state causing some outgoing connections to fail. Otherwise all interruptions where short and intermediate without further consequences.

Switch maintenance on 2012-10-07 - update 1

On Sunday, 2012-10-07 between 22:00 and 24:00 CET we will replace a switch which has accumulated defect ports in the last months. We do not expect any services to have any visible interruption due to the change.

The exchange will be performed by adding the new switch and slowly migrating all connections from the old switch to the new one causing only small, intermittent interruptions (time needed to move the cable plus a few seconds for RSTP to enable the port).

Individual affected services will probably show a temporary increase in response times but no actual failures.

Update 2012-10-08

Unfortunately we had a cascade of technical difficulties that stopped us again from exchanging the faulty switch. We are currently reviewing the needed steps to perform at the data center and are planning another attempt in the next days, probably tomorrow evening (Tuesday, 2012-10-09).

We intend to stick to a tight schedule currently as we want to avoid any further issues of this switch to cause actual issues in the operations.

We will write another announcement once the details are fixed.

Switch maintenance on 2012-08-24 – cancelled

The maintenance has been canceled due to organizational reasons. A new date will be announced separately.

On Friday, 2012-08-24 between 22:00 and 24:00 CEST we will replace a switch which has accumulated defect ports in the last months. We do not expect any services to have any visible interruption due to the change.

The exchange will be performed by adding the new switch and slowly migrating all connections from the old switch to the new one causing only small, intermittent interruptions (time needed to move the cable plus a few seconds for RSTP to enable the port).

Individual affected services should show a temporary increase in response times but no actual failures.

Intermittent connection problems with various hosted web sites

Since Monday (2012-07-09), several users report connection problems with web sites hosted at gocept.net. Typical symptoms include painfully slow pages and browser timeouts.

Until now we have a hard time to diagnose the problems as we cannot consistently reproduce them. It looks like only some users and some sites are affected. To get this issue fixed soon, we would appreciate user feedback.

So if you are experiencing problems like intermittent hangs or connection timeouts with gocept.net hosted websites right now, please go to http://supportdetails.com/?recipient=support@gocept.com, please fill in your name and e-mail address and send a report. This would help us greatly to gather relevant data and work towards a solution.

Security update: beware of broken shared library dependencies

Today (2012-06-28) we are rolling out an unusually large security update. The update is necessary to fix known vulnerabilities. Unfortunately, the change has a small chance of breaking customer-compiled C extensions. We did our best to retain binary compatibility but we cannot guarantee it in every single case.

We ask all developers that have installed C extensions to verify that their programs are still starting up correctly. If not, a simple recompile should be sufficient to fix the problem.

Maintenance of switching infrastructure on 2012-07-05

On 2012-07-05 between 22:00pm and 23:00pm CEST we will change the power connection of one of our switches. This will cause a short interruption of network connectivity for some VMs. We expect the interruption to take less than 5 minutes.

This change is necessary to modernize the power distribution system within our racks. We try to perform this using redundant power connections without interruption of service as far as possible. During the next weeks we will also update our second rack which will require another short switch interruption. We will announce the required maintenance window separately.