Four hour downtime as mysql tumbled

General Discussions
#1

The MySQL server was janked from underneath JGO, giving you the opportunity to finally getting around to do some coding, for once. I’m sorry it took so long for me to notice a skype message by theagentd informing me of these matters, as I was drilling holes in walls and ceilings, and working on my upper body strength as I was attaching curtains to rails and moving them around for a bit.

Having learnt my lesson, I will probably setup some notification mechanism to signal my phone in case JGO goes belly up. I mean, with all the money flowing in from arguably distasteful to downright obscene ads, that’s the least I could do, right? Okay!

#2

You responded within seconds of my Skype message, so you’re not entirely correct. =P

#3

That was purely coincidental. I walked in my bedroom for the first time that day and saw a Skype message. It’s as if the universe was trying to tell me something. True story.

#4

If you don’t want the hassle of setting up nagius monitoring or similar I can recommend the free service https://uptimerobot.com/

I use it for my sites and even webservices.

#5

I took JGO down for 3 minutes to check whether the down-alert emails/SMS/app-notifications were properly harassing me. Next time JGO goes down I’ll know. Trust me, I’ll know.

#6

But last time JGO was not down,
it showed some message about a sleeping admin :wink:

I doubt uptimerobot will yield any result?
(You could ping it successfully)

-ClaasJG

#7

I do not use uptimerobot, I use Pingdom, which can be configured to check for specific words in the http-response. This ‘JGO check’ has been active since 2010, I forgot about it - yesterday I connected it to my smartphone & email.

https://share.pingdom.com/banners/7fe0bee0

https://share.pingdom.com/banners/7ce29f1b

http://stats.pingdom.com/1iyt8llwhe3z

I case you’re wondering where the dreaded 4 hour gap is: I added the http-response check only yesterday. It shows the 3min down-time, as I took MySQL down, eventhough Apache was still up. I set it up to check for availability every minute, and get notified when 5min consecutive checks fail.

#8

#9

@Drenius That made me laugh very hard (inside my head that is). You sir made my day. Thanks :point:

#10

It was a bit worrysome after the first 2 hours. The first hour went by and I was like, Riven’s working on it for sure.

2 hours went by… okay he’s still working on it…

3 hours went by… Riven… … :emo: :clue:

4 hours… …

Figured it’d come up sooner or later though, glad it wasn’t longer :P.

#11

:persecutioncomplex:

The database failed in a completely new way this time, causing the SMF template still to be loaded. Pingdom worked nicely, but as I was actually asleep, it was to no avail. Sadly, I don’t have time to investigate the underlying issue, at this moment. I simply resorted to starting the MySQL database again. Something is seriously wrong, but it’ll have to wait. I cannot even take a peek at work, as today is going to be a teambuilding day, where we’ll be riding around in jeeps and do other things like looking at 200 year old buildings from a boat… yay. :cranky:

#12

Aren’t team building exercises fun?? Don’t you enjoy building lasting relationships with your coworkers? They’re such a productive use of time! :cranky:

#13

I get along with my colleagues just fine, even after work and occasionally in the weekends. Who wudda thunk!

Unrelated: why are all MySQL log-files missing info about the crash… it’s like the mysql-process was instantly terminated. Leaving me with nothing to analyze. I’m feeling a bit hesitant towards writing a script that restarts mysql automatically once it notices it is down… :persecutioncomplex:

#14

Next time suggest some sort of painful team building exercise such as paintball. Oh yes.

Cas :slight_smile:

#15

John: Hey Bill!

Bill: What.

John: You’re OUT!

Bill gets hit in the chest with a paintball

#16

My company’s tech department loves to watch the horror spread over a new software developer’s face as they are told they need to tell a joke in front of ~50 other techies. I managed to evade mine, but other people weren’t so lucky. Does that somehow count as a cruel form of team building?

#17

Is the “X new posts” notification thing that shows up at the top broken now?

(Why am I the only one who notices when this breaks, is everybody else doing something cool that I don’t know about?)

#18

I noticed this too. So no, you’re not the only one.

#19

It’s a bug in the MySQL JDBC driver, which won’t give you a fresh connection to the database after the database crashed. When you request a brand new connection, you’ll receive a stale connection, which instantly hangs indefinitely on the first query. To work around this I have to randomize the db connection string (url), but that code isn’t yet in the push-notification service I wrote years ago. So I have to restart the service after MySQL comes back up after a serious crash.

#20

Somehow this reminds me of an xkcd comic where the developer finds out a piece of his code is completely broken, yet he’d anticipated that when he wrote it and added a nice comment about it to his future self.