JGO server folded in on itself

General Discussions
#1

In the eternal quest of figuring out why the JGO server keeps pulling the rug from under itself, the VM managed to hide its tracks surprisingly well, in an unforeseen self-inflicted denial of service attack.

10.30 PM: Arrived home from work in Belgium.
11.00 PM: I went to bed.
11.30 PM: The daily backup was made.
11.55 PM: VM instance collapsed, unreachable (no ping, no http, no ssh), constant 100% CPU usage, no disk I/O, no network I/O.
11:56 PM: Pingdom sends me down-notification texts
07:30 AM: alarm clock wakes me from my slumber
07:31 AM: I see 2 texts from Pingdom… I notice JGO is actually down
07:33 AM: VM is in lockdown, cannot even login with KVM
07:40 AM: Ensure we have backups, reboot into rescue mode.
07:42 AM: Trying to mount virtual disk, turns out to be corrupted, repairable though.
07:52 AM: Cloning JGO instance to new VM.
07:53 AM: Cloned instance has mountable disk, at least last-minute data is safe.
07:55 AM: Rolling back backup on original instance.
08:00 AM: Jumped in the shower, f**king cold - boiler gave up due to low water pressure… is there a leak? Who gives a shit! No time!
08:07 AM: (short shower…) JGO backup restored, live.
08:08 AM: java-gaming.org does not work, www.java-gaming.org does work - there seems to be some virtualhost issue
08:25 AM: off to work.
09:25 AM: working… (arrived damn late)
11:55 AM: solved virtualhost issue: despite the configuration being explicit and correct, apache2 decided that the java-gaming.org hostname became the default, and hence was routed to /var/www, which contained only an index.html with: ‘

It works!

’. 

<VirtualHost *:80>
    ServerAdmin webmaster@java-gaming.org
    DocumentRoot "/home/jgo/public_html/"
@@    ServerName java-gaming.org // ignored
    ServerAlias pastebin.java-gaming.org www.java-gaming.org
    ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
    CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>



// added dummy domain to trigger apache2 to use as 000-default
+ ServerAdmin default@default.com
+ DocumentRoot /home/websites/domains/default/
+ ServerName default

<VirtualHost *:80>
    ServerAdmin webmaster@java-gaming.org
    DocumentRoot "/home/jgo/public_html/"
    ServerName java-gaming.org
    ServerAlias pastebin.java-gaming.org www.java-gaming.org
    ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
    CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>

Open issues:

  • what happened at: 11.55 PM
  • why did apache2 change its interpretation of defined virtualhosts
#2

Much appreciated!

Your first mistake:


11.00 PM: I went to bed.

You should know by now that JGO schedules its outages for your downtimes :slight_smile:

#3

Is there some (reliable!) app that scans through your text-messages and triggers an alarm upon a pattern match?

This one seems pretty decent…
https://play.google.com/store/apps/details?id=ax.ha.it.smsalarm

So far it seems to work, survives a reboot, ignores audio-levels/mutes. It’s a shame the Pingdom app (which had an alarm set) didn’t trigger… yet the text-message from Pingdom was received.
Anybody any experience with the above app (or similar apps)?

#4

Nothing suspicious in the log files between 11:50-55?

#5

Heh, that’s the first place I looked. Nothing suspicious, just the usual jibber jabber. :emo:

#6

Wow that was a lot of info right there, timestamps even.
You didnt have to copy your twitter feed :smiley:

#7

#1 complaint after downtime of any website is lack of details :point:

#8

Your first mistake is using Apache2 :point:

#9

Just to be sure:
Are you sure you enabled the apache config before you changed it? You moved to a new VM, so you would have to (re-)enable the config before (re)starting apache.

#10

The config was alright. It had survived a handful of reboots already. To be honest, I’m not sure what ‘re-enable the config’ means. Without a config, not a single virtualhost would have been loaded, and none of the websites would have worked. Anyhoo, I went through quite a bit trial and error, and determined that basically every single change worked (even changing the main ServerName to xyz.java-gaming.org), except… java-gaming.org. Then I ran [icode]apachectl -S[/icode] and noticed java-gaming.org was picked up as default (whether it was the first or last virtualhost defined in apache2.conf…), and (as said) rerouted to /var/www/.

#11

JGO became unreachable for about 2 hours because my ISPs nameserver stopped responding to queries.

I have created a spiffy twitter account, @JavaGamingOrg, which you guys can monitor in case the regular means of communication fail.