After a ridiculously long stretch of downtime, we have successfully resolved a hardware fault with the database server and rolled back the database to a backup from approx midnight on Saturday. Obviously this means everything posted since then has been lost, but given the amount of data corruption in the database, it was the safest option.
I can't apologise enough for this downtime. It has been a hugely frustrating couple of days. What should have been a relatively straight forward move of server hardware from one data centre to another turned into a bit of a nightmare.
Hopefully now everything is back to normal and we can forget this ever happened
As ever, you guys are my best eyes and ears so if you come across any odd behaviour - error pages, double posts etc - please post here and I'll investigate. The more detail you can provide about the problem the better.
Thanks again for your patience. Hopefully this is the last time we have such a lengthy outage.
Update:
So I spoke to soon. Obviously we're still having a lot of problems, error pages, double posts. We're actively looking into it and hopefully a solution is not far away.
Update 2:
I think we've tracked down the problem, a rogue database query that was using the wrong index which lead to long running queries that would eventually block all the database threads. As ever, can people please let me know if things are working better now - specifically when it comes to posting and editing posts. Thanks.
I can't apologise enough for this downtime. It has been a hugely frustrating couple of days. What should have been a relatively straight forward move of server hardware from one data centre to another turned into a bit of a nightmare.
Hopefully now everything is back to normal and we can forget this ever happened
As ever, you guys are my best eyes and ears so if you come across any odd behaviour - error pages, double posts etc - please post here and I'll investigate. The more detail you can provide about the problem the better.Thanks again for your patience. Hopefully this is the last time we have such a lengthy outage.
Update:
So I spoke to soon. Obviously we're still having a lot of problems, error pages, double posts. We're actively looking into it and hopefully a solution is not far away.
Update 2:
I think we've tracked down the problem, a rogue database query that was using the wrong index which lead to long running queries that would eventually block all the database threads. As ever, can people please let me know if things are working better now - specifically when it comes to posting and editing posts. Thanks.
