When DR fails

Lessons learned after my main server got hacked

When disaster recovery after a hack fails
Credit: www.elbpresse.de / Wikipedia

Someone hacked into my main server. I have a small organization, and the server was an old Apple Xserve 10.6.7 chosen because it’s not the usual host. Now it’s time to scratch security through obscurity off the list.

So let’s do a rudimentary recover. Forensics will have to wait.

I went to a hosting company to spin up httpd and mail. They’re already my registrar. Pretty big organization.

And they don’t have 24/7 support.

Since this happened on a Saturday, I was already in trouble. I chose one of their hosting plans. It costs a rudimentary $60 for a web server plus mail. It uses the famous CPanel hosting.

The list was: bring up the mail server, change the DNS authority to their hosting company from Expedient, which hosts the NOC resources. Their mail server kind of works; there are a few small bugs to work out. 

They have an automated cpanel-chosen script to deploy Wordpress, among many choices, including things like phpbb. Fine. I chose to deploy WordPress. The script proclaims success! (Exclamation point included.) 

But it doesn’t work. I can’t login to the WordPress instance at all.

Tomorrow—Tuesday, Nov. 8—I work the elections as an inspector. I’m gone. It turns out that their phone tech support doesn’t open until 0800 Denver time.  And, of course, the rest of my week is booked solid. 

Eventually, the website will be online, one that I hacked together. But I might have to change the site to one of the actual cloud providers, ones that have real support running 24/7 with tested scripts that work.

The talented among you will be opening up a terminal session and be running nslookup to see who I chose as the provider. Look again on Thursday to see if it’s the same one. I get the feeling it’ll change. 

Lessons learned

Lesson 1: Your backup plan has to be tested. Mine was ephemeral and thwarts me now. 

Lesson 2: If the hosting company doesn’t have 24/7 support, why are you still looking at them? 

Lesson 3: If the hosting company doesn’t QA their deployment scripts, you’ll have to do that yourself. You’re ready for that, aren’t you? 

Lesson 4: You’ll only be hacked when you’re totally booked for the next 10 days, making your allowable time to fool with such nonsense incredibly limited, so wait around until the last minute to check things out. 

Lesson 5: Once I go through the syslogs, I have my own scripts waiting. I suggest you look away once I execute them.

To comment on this article and other Network World content, visit our Facebook page or our Twitter stream.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.