In the eternal quest of figuring out why the JGO server keeps pulling the rug from under itself, the VM managed to hide its tracks surprisingly well, in an unforeseen self-inflicted denial of service attack.
10.30 PM: Arrived home from work in Belgium.
11.00 PM: I went to bed.
11.30 PM: The daily backup was made.
11.55 PM: VM instance collapsed, unreachable (no ping, no http, no ssh), constant 100% CPU usage, no disk I/O, no network I/O.
11:56 PM: Pingdom sends me down-notification texts
07:30 AM: alarm clock wakes me from my slumber
07:31 AM: I see 2 texts from Pingdom… I notice JGO is actually down
07:33 AM: VM is in lockdown, cannot even login with KVM
07:40 AM: Ensure we have backups, reboot into rescue mode.
07:42 AM: Trying to mount virtual disk, turns out to be corrupted, repairable though.
07:52 AM: Cloning JGO instance to new VM.
07:53 AM: Cloned instance has mountable disk, at least last-minute data is safe.
07:55 AM: Rolling back backup on original instance.
08:00 AM: Jumped in the shower, f**king cold - boiler gave up due to low water pressure… is there a leak? Who gives a shit! No time!
08:07 AM: (short shower…) JGO backup restored, live.
08:08 AM: java-gaming.org does not work, www.java-gaming.org does work - there seems to be some virtualhost issue
08:25 AM: off to work.
09:25 AM: working… (arrived damn late)
11:55 AM: solved virtualhost issue: despite the configuration being explicit and correct, apache2 decided that the java-gaming.org hostname became the default, and hence was routed to /var/www, which contained only an index.html with: ‘
It works!
’.
<VirtualHost *:80>
ServerAdmin webmaster@java-gaming.org
DocumentRoot "/home/jgo/public_html/"
@@ ServerName java-gaming.org // ignored
ServerAlias pastebin.java-gaming.org www.java-gaming.org
ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>
// added dummy domain to trigger apache2 to use as 000-default
+ ServerAdmin default@default.com
+ DocumentRoot /home/websites/domains/default/
+ ServerName default
<VirtualHost *:80>
ServerAdmin webmaster@java-gaming.org
DocumentRoot "/home/jgo/public_html/"
ServerName java-gaming.org
ServerAlias pastebin.java-gaming.org www.java-gaming.org
ErrorLog "/home/jgo/log/apache2/java-gaming.org-error_log"
CustomLog "/home/jgo/log/apache2/java-gaming.org-access_log" common
</VirtualHost>
Open issues:
- what happened at: 11.55 PM
- why did apache2 change its interpretation of defined virtualhosts