tl;dr: The Flying Circus is not affected by the Heartbleed bug
As reported by several media there is a serious bug in the OpenSSL library, widely known as the Heartbleed bug. The bug was introduced in the OpenSSL development tree on January 1st, 2012 and was finally released with OpenSSL version 1.0.1.
The Flying Circus platform makes use of the Gentoo Linux distribution. The OpenSSL version maintained for the Flying Circus is 1.0.0j, so it is not affected by the Heartbleed bug.
To be sure we are not affected by the bug, for example by possible backports from the Gentoo maintainer, we also audited the OpenSSL sources that are in use to not implement the vulnerable heartbeat function.
The Flying Circus is not affected by Heartbleed and there was no time in the past when we had rolled out a vulnerable version.
There is no need to replace your certificates or keys.
![gocept.net [en]](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY65wXXMC4QddrR7HTZnjU9u-41YNivbxQFTE7ptP_w-pxs5E-UwY2lF_zetWI19slZ_BIYljxRVsXdeUffNfpsoyKXOaA-jj45jrcuIAdUZiSTYDyjEAckKVIODgfGWWFqc-JuybhbgRa/s1600-r/logo.png){kind=logo}
Retiring this status blog - introducing our new status page
The last incidents have shown that using this blog for status and maintenance announcement does not deliver reliable information to our customers in time and is also quite cumbersome for us.
In an effort to improve this part of our service, we're now announcing our official status page:
http://status.flyingcircus.io
This page should give you a quick overview of the current state of things in our hosting environment, lets you see past incidents and some performance data. It also allows you to subscribe not only via RSS but also by email and SMS.
This is the last post in this blog - see you on the other side!
In an effort to improve this part of our service, we're now announcing our official status page:
http://status.flyingcircus.io
This page should give you a quick overview of the current state of things in our hosting environment, lets you see past incidents and some performance data. It also allows you to subscribe not only via RSS but also by email and SMS.
This is the last post in this blog - see you on the other side!
Performance problems during storage maintenance
Today we are moving some of the storage servers within our data center to balance power consumption, performance, and failure zones. We are also adding new routers and remove the
(now unused) old servers.
Although all of this was not expected to have user-visible impact, the migration did not go as smooth as planned. The main reason is that hardware defects hit us in a critical phase.
As a result, some VMs suffer currently from reduced storage performance. This also affects our mail server.
We are working hard to restore our storage cluster to its full I/O capacity by the end of the day. Spare parts have already arrived at the data center and we are putting them in service now.
Finally, once all this is over we'll give a more comprehensive review of the events and make a plan on how to communicate ongoing low-level maintenance tasks we perform better. We apologize for any inconvenience.
Short outage during storage server roll-out
Unfortunately we had a short partial outage today around 16:20 CET for roughly 20 minutes.
We are currently rolling out new storage hardware: bigger and faster machines with SSDs, better CPUs, and a lot more RAM.
As we finished migrating to CEPH a while ago those servers can be integrated into our live cluster without needing to take the system down.
Unfortunately, when integrating the first server, the whole system got stuck. We diagnosed the issue being a conflicting setting on switch ports on the storage cluster. We took the new server out of the cluster again, getting everything unstuck. We resolved the setting and were able to continue integrating the new servers after that. Over the next days you should see reduced "iowait" on virtual machines with a lot of throughput.
Technical details
The conflict consisted of switch ports configured with both "jumbo frames" and "flow control" which caused partial packet loss on the storage backend network. "flow control" is known to interfere with jumbo frames, but the ProCurve switches tend to be safe regarding conflicting settings. Today we changed ports that were previously used on a different (non jumbo frames) VLAN to a jumbo frame VLAN. When noticing that the cluster was stuck we investigated on multiple levels and found the switches to emit warnings about this setting. Disabling it, removed the package loss.
As we do not require flow-control in our network at all, we disabled this setting on all ports and will integrate this into our default configuration.
We are currently rolling out new storage hardware: bigger and faster machines with SSDs, better CPUs, and a lot more RAM.
As we finished migrating to CEPH a while ago those servers can be integrated into our live cluster without needing to take the system down.
Unfortunately, when integrating the first server, the whole system got stuck. We diagnosed the issue being a conflicting setting on switch ports on the storage cluster. We took the new server out of the cluster again, getting everything unstuck. We resolved the setting and were able to continue integrating the new servers after that. Over the next days you should see reduced "iowait" on virtual machines with a lot of throughput.
Technical details
The conflict consisted of switch ports configured with both "jumbo frames" and "flow control" which caused partial packet loss on the storage backend network. "flow control" is known to interfere with jumbo frames, but the ProCurve switches tend to be safe regarding conflicting settings. Today we changed ports that were previously used on a different (non jumbo frames) VLAN to a jumbo frame VLAN. When noticing that the cluster was stuck we investigated on multiple levels and found the switches to emit warnings about this setting. Disabling it, removed the package loss.
As we do not require flow-control in our network at all, we disabled this setting on all ports and will integrate this into our default configuration.
IPv4 and IPv6 addressing now fully enabled
As announced in the previous post, we are going to enable dual-stack address resolution in the server network (SRV) now for all VMs.
The new scheme goes as follows (as already announced):
However, the private IPv4 addresses can be used for connections inside our data center. When queried from the inside, our DNS servers will happily provide private IPv4 addresses as well. Thus, there is no need to use hard-coded IP addresses instead of proper DNS names in service configurations.
The new scheme goes as follows (as already announced):
- vm.gocept.net resolves to both IPv4 and IPv6 addresses,
- vm.ipv4.gocept.net resolves only to IPv4 addresses,
- vm.ipv6.gocept.net resolves only to IPv6 addresses.
However, the private IPv4 addresses can be used for connections inside our data center. When queried from the inside, our DNS servers will happily provide private IPv4 addresses as well. Thus, there is no need to use hard-coded IP addresses instead of proper DNS names in service configurations.
IPv4 and IPv6 addressing in the backend (SRV) network
We are going to add IPv6 addresses to the canonical VM names in DNS. This will improve backend reachability and does not require configuration changes in most cases.
Our VMs used to have IPv4 and IPv6 addresses for a long time, but the canonical VM name (VM.gocept.net) used to resolve to the IPv4 address only. The IPv6 address used to be reachable only via VM.ipv6.gocept.net.
We already started to introduce private IPv4 addresses from the 172.16.0.0/12 space in the SRV network. This makes it harder to reach the SRV interface from the outside.
Thus, we will introduce a DNS change during the next days that includes both the IPv4 and IPv6 address with a VM's canonical name. It will look like this:
What does this mean for SSH logins and awstats reachability?
SSH logins and awstats usually go to the backend (SRV) interface. This means that you will be able to login to your VM at VM.gocept.net via IPv6 as well. Logins to VMs with private IPv4 backend addresses must use IPv6 or go to the frontend network.
We are currently working on an alternative solution to keep awstats reachable on VMs with private SRV addresses for customers without IPv6 connectivity.
What does this mean for inter-service communication within resource groups?
In a typical scenario, the frontend web server (nginx) talks to application instances, which talk to a database server in turn. In the vast majority of cases, just keep using the canonical VM name to address a backend service (for example, use "proxy_pass VM.gocept.net:8080" in nginx.conf). Usually the internal services will be reached regardless of the IP address family. If this does not work for some reason, state the IP address family explicitly (using VM.ipv6.gocept.net or VM.ipv4.gocept.net).
Our VMs used to have IPv4 and IPv6 addresses for a long time, but the canonical VM name (VM.gocept.net) used to resolve to the IPv4 address only. The IPv6 address used to be reachable only via VM.ipv6.gocept.net.
We already started to introduce private IPv4 addresses from the 172.16.0.0/12 space in the SRV network. This makes it harder to reach the SRV interface from the outside.
Thus, we will introduce a DNS change during the next days that includes both the IPv4 and IPv6 address with a VM's canonical name. It will look like this:
- VM.gocept.net resolves to both the IPv4 and IPv6 address
- VM.ipv4.gocept.net resolves only to the IPv4 address
- VM.ipv6.gocept.net resolves only to the IPv6 address.
What does this mean for SSH logins and awstats reachability?
SSH logins and awstats usually go to the backend (SRV) interface. This means that you will be able to login to your VM at VM.gocept.net via IPv6 as well. Logins to VMs with private IPv4 backend addresses must use IPv6 or go to the frontend network.
We are currently working on an alternative solution to keep awstats reachable on VMs with private SRV addresses for customers without IPv6 connectivity.
What does this mean for inter-service communication within resource groups?
In a typical scenario, the frontend web server (nginx) talks to application instances, which talk to a database server in turn. In the vast majority of cases, just keep using the canonical VM name to address a backend service (for example, use "proxy_pass VM.gocept.net:8080" in nginx.conf). Usually the internal services will be reached regardless of the IP address family. If this does not work for some reason, state the IP address family explicitly (using VM.ipv6.gocept.net or VM.ipv4.gocept.net).
New SSL certificate for mail.gocept.net
We updated the SSL certificate for our mail server today. SSL encryption is used to send and receive mail and to access the management frontend https://mail.gocept.net in a secure way. The fingerprint of the new certificate is:
SHA1 Fingerprint
06:7C:19:B3:CD:3C:08:0A:A5:E6:73:B2:55:9A:8F:BD:E4:E7:A9:84
SHA1 Fingerprint
06:7C:19:B3:CD:3C:08:0A:A5:E6:73:B2:55:9A:8F:BD:E4:E7:A9:84
Happy holidays! (and how we handle support)
Christmas holidays are around the corner as is 2014. We thank all of our customers for their
trust and wish everyone a restful Christmas.
Our offices will be closed between 2013-12-23 and 2014-01-06.
Our support team will be watching out for major incidents and we will respond on regular German working days to your mails at support@gocept.com. Feature-requests and non-urgent issues will be delayed until 2013-01-07 when we will resume our regular work schedules.
Here's a schedule of our support for the next weeks:
trust and wish everyone a restful Christmas.
Our offices will be closed between 2013-12-23 and 2014-01-06.
Our support team will be watching out for major incidents and we will respond on regular German working days to your mails at support@gocept.com. Feature-requests and non-urgent issues will be delayed until 2013-01-07 when we will resume our regular work schedules.
Here's a schedule of our support for the next weeks:
| 2013-12-21 | Saturday | SLA-covered emergency support only |
| 2013-12-22 | Sunday | SLA-covered emergency support only |
| 2013-12-23 | Monday | regular support, limited to major incidents |
| 2013-12-24 | Tuesday | regular support, limited to major incidents |
| 2013-12-25 | Wednesday | SLA-covered emergency support only |
| 2013-12-26 | Thursday | SLA-covered emergency support only |
| 2013-12-27 | Friday | regular support, limited to major incidents |
| 2013-12-28 | Saturday | SLA-covered emergency support only |
| 2013-12-29 | Sunday | SLA-covered emergency support only |
| 2013-12-30 | Monday | regular support, limited to major incidents |
| 2013-12-31 | Tuesday | regular support, limited to major incidents |
| 2014-01-01 | Wednesday | SLA-covered emergency support only |
| 2014-01-02 | Thursday | regular support, limited to major incidents |
| 2014-01-03 | Friday | regular support, limited to major incidents |
| 2014-01-04 | Saturday | SLA-covered emergency support only |
| 2014-01-05 | Sunday | SLA-covered emergency support only |
| 2014-01-06 | Monday | SLA-covered emergency support only |
Upcoming OS update will break old lxml installations
The upcoming Flying Circus platform update will introduce a libxml2 version that is not compatible with lxml
2.x anymore. We ask all users who compiled their own lxml library to check if this is lxml 2.x or 3.x.
All applications should switch to lxml 3.x now, which supports both old and new libxml2 versions.
All applications should switch to lxml 3.x now, which supports both old and new libxml2 versions.
Slow response times last night
In the night from 2013-11-26 21:00 and 2013-11-27 07:00 we temporarily experienced increased response times for services hosted with us.
According to our data center operator this was caused due to an issue at DECIX (Frankfurt) where maintenance operations caused temporary outages and reduced peering sessions.
This caused packet loss and slowed down aspects of various processes involved in delivering your services, like DNS lookups, downloads, etc.
We apologize for any inconvenience - unfortunately there is nothing we can directly do about the availability of major internet exchanges. However, the people at DECIX are aware of their importance and we are sure they will respond to this incident accordingly.
According to our data center operator this was caused due to an issue at DECIX (Frankfurt) where maintenance operations caused temporary outages and reduced peering sessions.
This caused packet loss and slowed down aspects of various processes involved in delivering your services, like DNS lookups, downloads, etc.
We apologize for any inconvenience - unfortunately there is nothing we can directly do about the availability of major internet exchanges. However, the people at DECIX are aware of their importance and we are sure they will respond to this incident accordingly.
[Solved] Some VMs affected by network issues [update 5]
After the kernel upgrade applied yesterday to our KVM servers we are experiencing compatibility issues on some VMs in our cluster causing them to loose network connectivity.
All VMs that have been affected by the issue have been configured with a watchdog process to reboot them cleanly and immediately the next time the issue occurs - this should keep service downtimes to a minimum while we're working on a long term fix.
Likely, we will have to provide a newer kernel for virtual machines within the next days. We will review this tomorrow and hopefully get back to a completely stable environment quickly.
Unfortunately we did not discover this incompatibility in our extensive staging and test environments and we have not found a pattern to trigger this reliably.
We apologize for any inconvenience and kindly ask for your patience while we we're getting everything back to normal.
Update 1 (2013-10-07 09:19)
We have started to provision an updated virtual machine kernel in our development environment. Our goal is to have the kernel available for selected virtual machines until noon to gather data about its stability on the VMs affected until now. Based on that data we will make a decision how to proceed later today.
Update 2 (2013-10-07 12:02)
We have successfully prepared an updated kernel (3.10) for virtual machines in our development environment. We have not found any adverse affects regarding application compatibility and are now rolling it out in our staging environment and in selected production VMs that run data-center internal services. If those prove stable we will roll out updates to all previously affected VMs in the next 2 hours.
Update 3 (2013-10-07 16:00)
We deployed the new kernel to selected machines but have not achieved the desired result. However, additionally we acquired information that a specific checksum offloading feature tends to break under certain conditions. We were able to reproduce the problem and also appear to have worked around it by disabling this specific offloading feature. We are currently monitoring a few affected VMs for stability and will continue rolling out the prepared kernel and offload disabling later this evening.
Update 4 (2013-10-08 01:48)
The combination of a new kernel and disabling the troublesome offloading feature appears to have fixed the problem (or at least resulted in a reliable workaround). We have not seen a VM network crash since about 7 hours now and all virtual machines have been updated.
We are currently in contact with upstream developers to make a root cause analysis how this problem came into existence. Hopefully this will result in an upstream open source bugfix that will also become available to other parties.
Update 5 (2013-10-08 15:40)
Since yesterday evening, we have been continuing monitoring our infrastructure closely and did not see any VM network crash again. We would therefore consider the problem as solved.
In the meantime we have started debugging the issue together with the upstream developers in an isolated environment. This will hopefully help to find the root cause and support the upstream developers in providing a fix.
All VMs that have been affected by the issue have been configured with a watchdog process to reboot them cleanly and immediately the next time the issue occurs - this should keep service downtimes to a minimum while we're working on a long term fix.
Likely, we will have to provide a newer kernel for virtual machines within the next days. We will review this tomorrow and hopefully get back to a completely stable environment quickly.
Unfortunately we did not discover this incompatibility in our extensive staging and test environments and we have not found a pattern to trigger this reliably.
We apologize for any inconvenience and kindly ask for your patience while we we're getting everything back to normal.
Update 1 (2013-10-07 09:19)
We have started to provision an updated virtual machine kernel in our development environment. Our goal is to have the kernel available for selected virtual machines until noon to gather data about its stability on the VMs affected until now. Based on that data we will make a decision how to proceed later today.
Update 2 (2013-10-07 12:02)
We have successfully prepared an updated kernel (3.10) for virtual machines in our development environment. We have not found any adverse affects regarding application compatibility and are now rolling it out in our staging environment and in selected production VMs that run data-center internal services. If those prove stable we will roll out updates to all previously affected VMs in the next 2 hours.
Update 3 (2013-10-07 16:00)
We deployed the new kernel to selected machines but have not achieved the desired result. However, additionally we acquired information that a specific checksum offloading feature tends to break under certain conditions. We were able to reproduce the problem and also appear to have worked around it by disabling this specific offloading feature. We are currently monitoring a few affected VMs for stability and will continue rolling out the prepared kernel and offload disabling later this evening.
Update 4 (2013-10-08 01:48)
The combination of a new kernel and disabling the troublesome offloading feature appears to have fixed the problem (or at least resulted in a reliable workaround). We have not seen a VM network crash since about 7 hours now and all virtual machines have been updated.
We are currently in contact with upstream developers to make a root cause analysis how this problem came into existence. Hopefully this will result in an upstream open source bugfix that will also become available to other parties.
Update 5 (2013-10-08 15:40)
Since yesterday evening, we have been continuing monitoring our infrastructure closely and did not see any VM network crash again. We would therefore consider the problem as solved.
In the meantime we have started debugging the issue together with the upstream developers in an isolated environment. This will hopefully help to find the root cause and support the upstream developers in providing a fix.
E-mail server maintenance on Sun 2013-10-06, 21:00-22:00 CEST
On Sunday, 2013-10-06, we are going to increase the disk space of our mail system. Because of that, mail delivery will be delayed between 21:00 and 22:00 CEST. Sending of emails as well as the access to the mailboxes may not be possible during this time.
Reboot of all machines on Sat 2013-10-05 21:00-23:30 CEST
We are updating to a new version of the Linux kernel on VM hosts and need to reboot all hosts. The reboot is scheduled for Saturday, 2013-10-05 between 21:00 and 23:30 CEST (19:00-21:30 UTC). During this period, expect one or several short service interruptions. We try to minimize outages but during a reboot of the hosts, the virtual machines must be restarted as well. We apologize for any inconvenience.
Storage server outage - few VMs affected [Update 1]
Today around 19:30 CEST one of our storage servers encountered a massive hard drive failure: 3 out of 7 disks stopped working at once and crashed the system entirely.
However, only a few number of VMs are affected by this and none of our central services are affected: if you did not experience any issue by now then you are not affected.
We are currently restoring VMs that were located on this storage server (some have been restored already). You should see your VMs and services come back within the next few hours.
We will provide an aftermath update later.
[Update 1: 2013-04-18 01:04 CEST]
We finished restoring all business critical machines and all customer services are back online. Our disaster recovery plan estimates about 24 hours for restoring a completely failed storage server. Today it took about 5:30h to perform the necessary analysis, write some skripts, answer customer inquiries, and get services running again.
[Update 1: 2013-04-18 01:04 CEST]
We finished restoring all business critical machines and all customer services are back online. Our disaster recovery plan estimates about 24 hours for restoring a completely failed storage server. Today it took about 5:30h to perform the necessary analysis, write some skripts, answer customer inquiries, and get services running again.
2013-03-05 00:00 Urgent network maintenance
Following up on the recent IPv6 outage our provider has received updates for their Cisco equipment that will be deployed tonight between 2013-03-05 00:00 (CET) and 06:00.
Due to the bugs in the existing firmware a "hot update" is not possible and the provider expects multiple outages of about 10 minutes in connectivity to the data center.
We do not expect any network issues within our cluster but expect to see unavailability from and to the internet.
Due to the bugs in the existing firmware a "hot update" is not possible and the provider expects multiple outages of about 10 minutes in connectivity to the data center.
We do not expect any network issues within our cluster but expect to see unavailability from and to the internet.
IPv6 connectivity outage [2013-02-26 10:17 CET, update 2, solved]
There is currently an unplanned outage of our IPv6 connectivity in the data center ongoing.
We are working with our uplink providers to restore connectivity.
IPv4 connectivity is not affected, however, services that rely on outside services that provide double-stack networking (IPv4+IPv6) may experience timeouts and delays.
We will post updates here as we work towards a solution.
We are sorry for any inconvenience.
[Update 1 - 11:52 CET]
The cause seems to have been in the data center upstream router infrastructure. We are getting improvements and see traffic on IPv6 picking up again. We do know of a couple of remaining edge cases and continue to work on restoring full connectivity soon.
[Update 2 -14:51 CET, solved]
IPv6 connectivity has been restored. Some nodes are still recovering from the outage but we are seeing continuous improvement in our monitoring.
The root cause was traced to a Cisco IOS bug involving more than 10.000 IPv6 routing entries and our upstream provider has implemented a work-around for the problem. We expect network maintenance some time in the next weeks to provide a fixed IOS version at the network operator's equipment.
We are working with our uplink providers to restore connectivity.
IPv4 connectivity is not affected, however, services that rely on outside services that provide double-stack networking (IPv4+IPv6) may experience timeouts and delays.
We will post updates here as we work towards a solution.
We are sorry for any inconvenience.
[Update 1 - 11:52 CET]
The cause seems to have been in the data center upstream router infrastructure. We are getting improvements and see traffic on IPv6 picking up again. We do know of a couple of remaining edge cases and continue to work on restoring full connectivity soon.
[Update 2 -14:51 CET, solved]
IPv6 connectivity has been restored. Some nodes are still recovering from the outage but we are seeing continuous improvement in our monitoring.
The root cause was traced to a Cisco IOS bug involving more than 10.000 IPv6 routing entries and our upstream provider has implemented a work-around for the problem. We expect network maintenance some time in the next weeks to provide a fixed IOS version at the network operator's equipment.
Major OS update roll-out starting from 2013-03-04
It has been a long time since we provided the last operating system update - longer than we expected to. However, we are happy to announce that we have finished our QA on a major set of package updates which brings updates for many packages maintained in our platform.
We will roll out the update incrementally between 2012-03-04 and 2012-03-09. All machines will receive a maintenance slot in the next days informing you of the assigned slots per email.
If you own any testing or staging environments then those will be updated at least 24 hours before the production systems.
Downtime expectations
Your VMs will be restarted at least two times:
- once during the week in their assigned slot to activate a new kernel configuration, and
- once on Saturday 2013-03-09 around 10:00-14:00 CET to restart all physical KVM hosts, this will take less than 30 minutes for any specifiv VM.
Important changes
There have been many improvements and bugfixes, most notably on our infrastructure, to make the platform more reliable and faster without you having to worry about it.
However, there are a few changes that we would like you to be aware of to avoid pitfalls:
- Varnish will be upgraded to version 3 - most configurations from Varnish 2 will continue to work correctly, but some subtle changes may break your config. The Varnish community has provided a nice upgrade document that summarises relevant changes.
- Due to a change in the license for Oracle/Sun JDK we decided to switch the Java VM to OpenJDK as we are no longer allowed to automatically distribute the Oracle/Sun JDK.
- Python 2.4 is now in "sunset" period: we will no longer install it to any additional machines and we will announce a roadmap to uninstall Python 2.4 within the next months. This version of Python is very old, unsupported and has known security issues.
- We no longer install "swftools" and will even actively uninstall them as the upstream developers do not maintain them any longer.
If you are interested in more technical details, feel free to take a look at our change log. A detailed list of updated package versions will appear there.
What's next?
Having finished this major release we are already looking forward for the next big thing. In the coming months we expect to drastically improve our storage system by introducing Ceph instead of iSCSI. Preliminary work on the new storage systems has started already and we are really excited to get this done.
If you look closely you will notice that we haven't upgraded the kernel with this update: this is currently intentional and related to the upcoming storage overhaul: the existing Linux iSCSI stack unfortunately doesn't port very well to newer kernel versions and instead of switching our iSCSI implementation while working on Ceph we decided to stay at a kernel that allows a smooth transition to Ceph.
[Update 1, 2013-02-14]
The date for the second reboot had a typo that showed 2013-02-09 for the KVM server reboot which should have read 2013-03-09
Filesystem integrity checks
During the next days, we will run a filesystem integrity check on all machines. This means that all machines will experience a minor downtime (less than 10 minutes for most machines). Technical contacts will be informed about the individual times in advance.
The global filesystem check is a preventive measure to ensure that we will not run into problems during the upcoming OS update.
A few machines in the development network showed incongruent filesystems under certain stress conditions. Although we did not notice any current problems in the production network, we identified some scenarios that could be triggered by the OS update process. The integrity check will effectively remediate any of those potential problems.
We apologize for the inconveniences caused by this safety measure.
The global filesystem check is a preventive measure to ensure that we will not run into problems during the upcoming OS update.
A few machines in the development network showed incongruent filesystems under certain stress conditions. Although we did not notice any current problems in the production network, we identified some scenarios that could be triggered by the OS update process. The integrity check will effectively remediate any of those potential problems.
We apologize for the inconveniences caused by this safety measure.
Filesystem migration to ext4
As part of our ongoing platform improvement, we will migrate ext3 filesystems to the ext4 format. The migration requires an off-line filesystem check and thus a little downtime for each machine to reboot.
The downtimes will be scheduled individually according to the agreed maintenance windows. We will inform the technical contacts about the exact times for their VMs in advance. Please write to our support if any reboot time does not fit.
Except for the reboot, the change will be completely transparent.
The downtimes will be scheduled individually according to the agreed maintenance windows. We will inform the technical contacts about the exact times for their VMs in advance. Please write to our support if any reboot time does not fit.
Benefits of the new filesystem
The ext4 filesystem offers a number of advantages over the old ext3 filesystem:- Reduced filesystem check time: this means that both planned machine reboots and error recovery after outages will be sped up significantly.
- Large file handling: access to large files like ZODB databases require less system overhead.
- Large directory handling: file lookup in directories that contain a lot of files is more efficient.
- More subtle run-time optimizations like delayed allocation and others.
Except for the reboot, the change will be completely transparent.
Maintenance on 2012-10-09 22:00 CEST - update 2
To finally finish the exchange of the faulty switch we will perform a series of hardware maintenance steps tomorrow (2012-10-09) night between 22:00 and 24:00 CEST.
The following tasks will be performed:
- Finish the migration to a new power distribution system in our racks
- Move our standby switch next to the faulty switch, verify correct operation
- Move the existing network connections from the faulty switch to the replacement
- Install a new standby switch
Also, when moving the cables from the faulty switch to its replacement there will be short lags in storage or server network connectivity for a few seconds but no outages.
We are sorry that this preventive measure has taken multiple attempts to implement. We think that our decisions to support a stable environment with careful small adjustments is in the interest of your operational needs.
Update 1 [2012-10-08 12:26]
The previous version of this post mentioned 2012-10-08 as the date of the maintenance. The actual scheduled date is 2012-10-09. The text above was corrected.
Update 2 [2012-10-09 23:32]
The faulty switch has, finally, been successfully replaced. In a window of about 5 minutes the redundant routers where in an inconsistent state causing some outgoing connections to fail. Otherwise all interruptions where short and intermediate without further consequences.
Update 1 [2012-10-08 12:26]
The previous version of this post mentioned 2012-10-08 as the date of the maintenance. The actual scheduled date is 2012-10-09. The text above was corrected.
Update 2 [2012-10-09 23:32]
The faulty switch has, finally, been successfully replaced. In a window of about 5 minutes the redundant routers where in an inconsistent state causing some outgoing connections to fail. Otherwise all interruptions where short and intermediate without further consequences.
Subscribe to:
Posts (Atom)