...and we're back

032Devil · 26th July 2011, 23:47

Quote:

Originally Posted by Niall

Ok we've made a change to a problematic query so can people let me know if things are performing better? It was specifically related to posting, so post away in here to your hearts content and let me know if you see any more errors.

Fingers crossed...

Fingers (and legs) crossed too!

032Devil · 26th July 2011, 23:48

Quote:

Originally Posted by 032Devil

Fingers (and legs) crossed too!

Much better but let see how it goes.

Solius · 26th July 2011, 23:48

Ok testing, and will edit this too.

EDIT: Helloooooo

Solius · 26th July 2011, 23:49

Seems to be better, yep.

032Devil · 26th July 2011, 23:49

Quote:

Originally Posted by 032Devil

Much better but let see how it goes.

It seems to have speeded up to usual speed in the last few seconds.

032Devil · 26th July 2011, 23:50

Quote:

Originally Posted by 032Devil

It seems to have speeded up to usual speed in the last few seconds.

Speed seems to fluctuate from a few seconds to immediate.

EDIT (about 5-10 minutes later): Better but still not running smoothly.

Big-Red · 26th July 2011, 23:53

Test bazillion and two

EDIT: bazillion and three

032Devil · 27th July 2011, 00:07

I notice that the Post Quick Reply dialogue box is still reacting temperamentally,

Niall · 27th July 2011, 00:10

How so?

032Devil · 27th July 2011, 00:29

Quote:

Originally Posted by Niall

How so?

I tried to post using the quick reply but all it did was the timer clock that sometimes appears to the left of the button kept turning without actually posting.

I also noticed that sometimes when I use the quick reply button the posting goes ahead but without a post actually appearing.

Hope I'm not leading you up the wrong path but I have experienced that in the last 30mins or so.

Niall · 27th July 2011, 00:35

Ok I'll keep an eye on it. We were tweaking things a bit over the last 30 minutes so that might have caused it, but we're done for the night now. If it keeps happening to you over the next few hours, let me know.

Red Hand Devil · 27th July 2011, 00:50

Quote:

Originally Posted by 032Devil

Fingers (and legs) crossed too!

Legs crossed! You dying for a shit dude?

SharkyMcShark · 27th July 2011, 00:50

Niall is like a benevolent god

WeasteDevil · 27th July 2011, 01:11

Everything seems to be acting normally.

WeasteDevil · 27th July 2011, 01:12

Some of it is also really fast compared to how it used to be.

WeasteDevil · 27th July 2011, 01:13

Forum response is amazing. I think the down time was worth it.

Hats off to you Niall, well done!

Phil · 27th July 2011, 01:22

I'll miss the duplicate posts, specially the guy in the Sneijder thread who insisted on letting RHD post but only if carefully moderated.

RedSky At Night · 27th July 2011, 09:39

Another hiccup?

Cina · 27th July 2011, 09:41

Well done Niall, as someone who also works with damned databases and transfers, I'm glad to hear things are back up and running!

I actually got a lot work done in the last few days. I feel dirty.

SharkyMcShark · 27th July 2011, 09:46

Yeah it was gone for about half an hour there. I was in a small amount of agony.

Niall · 27th July 2011, 09:53

Yep a brief hiccup this morning but it wasn't the site - the data centre was having some networking issues. The site should be solid from here on in.

stubie1234 · 27th July 2011, 09:55

Quote:

Originally Posted by Niall

Yep a brief hiccup this morning but it wasn't the site - the data centre was having some networking issues. The site should be solid from here on in.

Had no problems so far this morning

Cheers Niall

RedSky At Night · 27th July 2011, 10:01

Quote:

Originally Posted by Niall

Yep a brief hiccup this morning but it wasn't the site - the data centre was having some networking issues. The site should be solid from here on in.

Solius · 27th July 2011, 10:04

Just got the "She canee take the strain capt'n!" message a few times.

Niall · 27th July 2011, 10:10

What exactly were you trying to do when you got them?

Maciek · 27th July 2011, 10:22

I don't know what Solius was trying to do, but I instead of 'Mark forums read' I clicked on 'Wiki' and it doesn't work (she canee take the strain capt'n!)

Not a big problem I guess, the most important is we've got forum back.

Broyphs · 27th July 2011, 10:22

Yeah, wiki's done broken for me too.

Solius · 27th July 2011, 10:28

Quote:

Originally Posted by Niall

What exactly were you trying to do when you got them?

Just view the forums I think. Will let you know if it happens again and I'll take note of what I was doing.

mariachi-19 · 27th July 2011, 10:30

vic090482 · 27th July 2011, 10:37

kin' hell, i dont post a lot here, but I am on this site almost all the time.... it was horrible at work without redcafe.

Sunny Jim · 27th July 2011, 10:43

^^ same!

Niall · 27th July 2011, 10:47

Oh yeah, the wiki hasn't been fully setup after the maintenance. Forgot to mention that

The Taurean · 27th July 2011, 12:08

relapse of the 'canee take the strain' just awhile back.

Broyphs · 27th July 2011, 12:09

Same here

Big-Red · 27th July 2011, 12:16

I'd say the canee take the strain is genuinely from influx of people making up for lost time.

Solius · 27th July 2011, 12:23

I also just got it, after clicking on General Forum.

WeasteDevil · 27th July 2011, 13:00

I just got it. You can only take a guess as to why though as we don't know how the system architecture is set up. It could be that the php code is getting impatient for a reply from the DB server, at least it seems that way to me, as the message comes up pretty quickly - not like say a 10 second wait.

Red Indian Chief Torn Rubber · 27th July 2011, 13:39

Quote:

Originally Posted by Niall

After a ridiculously long stretch of downtime, we have successfully resolved a hardware fault with the database server and rolled back the database to a backup from approx midnight on Saturday. Obviously this means everything posted since then has been lost, but given the amount of data corruption in the database, it was the safest option.

I can't apologise enough for this downtime. It has been a hugely frustrating couple of days. What should have been a relatively straight forward move of server hardware from one data centre to another turned into a bit of a nightmare.

Hopefully now everything is back to normal and we can forget this ever happened

As ever, you guys are my best eyes and ears so if you come across any odd behaviour - error pages, double posts etc - please post here and I'll investigate. The more detail you can provide about the problem the better.

Thanks again for your patience. Hopefully this is the last time we have such a lengthy outage.

Update:
So I spoke to soon. Obviously we're still having a lot of problems, error pages, double posts. We're actively looking into it and hopefully a solution is not far away.

Update 2:
I think we've tracked down the problem, a rogue database query that was using the wrong index which lead to long running queries that would eventually block all the database threads. As ever, can people please let me know if things are working better now - specifically when it comes to posting and editing posts. Thanks.

Thanks for the great work you and your team do Niall.

Irwynwastheking · 27th July 2011, 13:43

I actually thought I might have broke the caf. I was deleted, and then when the back up was done and the caf came back, I posted and it broke again!

Niall · 27th July 2011, 14:16

Quote:

Originally Posted by WeasteDevil

I just got it. You can only take a guess as to why though as we don't know how the system architecture is set up. It could be that the php code is getting impatient for a reply from the DB server, at least it seems that way to me, as the message comes up pretty quickly - not like say a 10 second wait.

Do you remember what you were doing when you got the error Weaste?

The 'canee take the strain' error pages come from Varnish - will dig into the logs and see if they reveal anything.

RedCafe.net

The Leading Manchester United Forum

Member's Login

...and we're back

Latest Forum Topics

Jump to Forum

Not yet registered?

26th July 2011, 23:48	#83 (permalink)
Solius I am a mod, I swear Join Date: Dec 2007 Location: Everybody will be dancing and be doing it right. Posts: 49,711	Ok testing, and will edit this too. EDIT: Helloooooo

26th July 2011, 23:49	#84 (permalink)
Solius I am a mod, I swear Join Date: Dec 2007 Location: Everybody will be dancing and be doing it right. Posts: 49,711	Seems to be better, yep.

26th July 2011, 23:53	#87 (permalink)
Big-Red First Team Regular Join Date: Apr 2009 Location: The arse groove of my sofa Posts: 12,630	Test bazillion and two EDIT: bazillion and three

27th July 2011, 00:07	#88 (permalink)
032Devil First Team Regular Join Date: Oct 2004 Posts: 19,768	I notice that the Post Quick Reply dialogue box is still reacting temperamentally,

27th July 2011, 00:10	#89 (permalink)
Niall Fallible Super Being Join Date: Jun 1999 Location: Dublin Posts: 7,465	How so?

27th July 2011, 00:35	#91 (permalink)
Niall Fallible Super Being Join Date: Jun 1999 Location: Dublin Posts: 7,465	Ok I'll keep an eye on it. We were tweaking things a bit over the last 30 minutes so that might have caused it, but we're done for the night now. If it keeps happening to you over the next few hours, let me know.

27th July 2011, 00:50	#93 (permalink)
SharkyMcShark Horrified Join Date: May 2007 Location: Using location as a tagline: because I'm just not cool enough to have my own. Posts: 19,960	Niall is like a benevolent god

27th July 2011, 01:11	#94 (permalink)
WeasteDevil Redcafe's Greatest Poster Join Date: Jun 2001 Location: Salford in Castellón de la Plana Posts: 101,857	Everything seems to be acting normally.

27th July 2011, 01:12	#95 (permalink)
WeasteDevil Redcafe's Greatest Poster Join Date: Jun 2001 Location: Salford in Castellón de la Plana Posts: 101,857	Some of it is also really fast compared to how it used to be.

27th July 2011, 01:13	#96 (permalink)
WeasteDevil Redcafe's Greatest Poster Join Date: Jun 2001 Location: Salford in Castellón de la Plana Posts: 101,857	Forum response is amazing. I think the down time was worth it. Hats off to you Niall, well done!

27th July 2011, 09:39	#98 (permalink)
RedSky At Night RTA waiting to happen Join Date: Jul 2006 Location: Nottingham Forest (Soccermanager) Posts: 22,159	Another hiccup?

27th July 2011, 01:22	#97 (permalink)
Phil Reserve Team Player Join Date: Aug 2003 Posts: 4,058	I'll miss the duplicate posts, specially the guy in the Sneijder thread who insisted on letting RHD post but only if carefully moderated.

27th July 2011, 09:41	#99 (permalink)
Cina Love Nani, Hate Ando Join Date: Aug 2007 Location: The Bandwagon Posts: 24,150	Well done Niall, as someone who also works with damned databases and transfers, I'm glad to hear things are back up and running! I actually got a lot work done in the last few days. I feel dirty.

27th July 2011, 09:46	#100 (permalink)
SharkyMcShark Horrified Join Date: May 2007 Location: Using location as a tagline: because I'm just not cool enough to have my own. Posts: 19,960	Yeah it was gone for about half an hour there. I was in a small amount of agony.

27th July 2011, 09:53	#101 (permalink)
Niall Fallible Super Being Join Date: Jun 1999 Location: Dublin Posts: 7,465	Yep a brief hiccup this morning but it wasn't the site - the data centre was having some networking issues. The site should be solid from here on in.

27th July 2011, 10:04	#104 (permalink)
Solius I am a mod, I swear Join Date: Dec 2007 Location: Everybody will be dancing and be doing it right. Posts: 49,711	Just got the "She canee take the strain capt'n!" message a few times.

27th July 2011, 10:10	#105 (permalink)
Niall Fallible Super Being Join Date: Jun 1999 Location: Dublin Posts: 7,465	What exactly were you trying to do when you got them?

27th July 2011, 10:22	#106 (permalink)
Maciek Arrogant prick who is best ignored Join Date: Feb 2006 Location: 'Sometimes you look in a field and you see a cow and you think it's a better cow than the one you've got in your own field. It's a fact, right? And it never really works out that way. It’s probably the same cow and not as good as your own cow.' Posts: 10,721	I don't know what Solius was trying to do, but I instead of 'Mark forums read' I clicked on 'Wiki' and it doesn't work (she canee take the strain capt'n!) Not a big problem I guess, the most important is we've got forum back.

27th July 2011, 10:22	#107 (permalink)
Broyphs No Lifer Join Date: Nov 2006 Location: Commas are important. "Come on, Barbie, let's go party" is much less sinister than "Come on Barbie, let's go party" Posts: 33,103	Yeah, wiki's done broken for me too.

27th July 2011, 10:30	#109 (permalink)
mariachi-19 First Team Regular Join Date: Jul 2008 Location: I may be the devil, but i'm not a monster Posts: 13,344

27th July 2011, 10:37	#110 (permalink)
vic090482 Reserve Team Player Join Date: Jan 2007 Posts: 1,016	kin' hell, i dont post a lot here, but I am on this site almost all the time.... it was horrible at work without redcafe.

27th July 2011, 10:43	#111 (permalink)
Sunny Jim First Team Regular Join Date: May 2007 Location: Warsaw...that's too far away from Edinburgh... Posts: 16,267	^^ same!

27th July 2011, 10:47	#112 (permalink)
Niall Fallible Super Being Join Date: Jun 1999 Location: Dublin Posts: 7,465	Oh yeah, the wiki hasn't been fully setup after the maintenance. Forgot to mention that

27th July 2011, 12:08	#113 (permalink)
The Taurean First Team Sub Join Date: May 2009 Location: Nothing is so common as the wish to be remarkable. Posts: 6,785	relapse of the 'canee take the strain' just awhile back.

27th July 2011, 12:09	#114 (permalink)
Broyphs No Lifer Join Date: Nov 2006 Location: Commas are important. "Come on, Barbie, let's go party" is much less sinister than "Come on Barbie, let's go party" Posts: 33,103	Same here

27th July 2011, 12:16	#115 (permalink)
Big-Red First Team Regular Join Date: Apr 2009 Location: The arse groove of my sofa Posts: 12,630	I'd say the canee take the strain is genuinely from influx of people making up for lost time.

27th July 2011, 12:23	#116 (permalink)
Solius I am a mod, I swear Join Date: Dec 2007 Location: Everybody will be dancing and be doing it right. Posts: 49,711	I also just got it, after clicking on General Forum.

27th July 2011, 13:00	#117 (permalink)
WeasteDevil Redcafe's Greatest Poster Join Date: Jun 2001 Location: Salford in Castellón de la Plana Posts: 101,857	I just got it. You can only take a guess as to why though as we don't know how the system architecture is set up. It could be that the php code is getting impatient for a reply from the DB server, at least it seems that way to me, as the message comes up pretty quickly - not like say a 10 second wait.

27th July 2011, 13:43	#119 (permalink)
Irwynwastheking Skilled creeper Join Date: May 2009 Location: Manchester United - Bumrushing the box since 1878 Posts: 22,827	I actually thought I might have broke the caf. I was deleted, and then when the back up was done and the caf came back, I posted and it broke again!