Last week, for the better part of 4.5 days, this site was offline.
Along with, of course, every other domain hosted hereon .
Here’s the timeline of my actions
- Tuesday, reboot to update kernel revs
- system did not come back online
- over the next several days, tried all kinds of diagnostic attempts, including
- verified host was pingable, tracerouteable, etc
- rescue environments to chroot and remove out of date packages, update boot menus, etc
- remote KVM (which is Java based, and wouldn’t run on my macOSÂ Sierra machine with Java 8 U121)
- late Friday (or maybe it was Saturday), received a cron-generated email – which meant the server was up
- had a bolt of inspiration, and thought to check the firewall (but couldn’t for several hours for various reasons)
- Saturday evening, using a rescue environment from my hosting provider, chroot’ed into my server, and reset firewalld
- reboot, and bingo bango! server was back
So. What happened? Short version, something enabled firewalld, and setup basic rules to block everything. And I do mean everything – ssh, http, smtp, etc etc.
Not sure exactly how the firewall rules got mucked-up, but that was the fix.