The email came on Friday of last week. I got it sometime right after 6pm. Ricks Hall Primary Power Disconnect
At first I was just happy that I finally got notified this time. Last power outage to the room, I found out about it after the servers in the room all crashed. Actually it was later than that, because one of the servers that crashed was the mail server. So I actually found out about when they came back up. No one was notified actually. No one in an IT group at least - not me, not Extension IT, not the
But this we were at least notified, and I read the note two or three times, slowly dawning that this was going to be a 57hour outage. 8am Friday morning until 5pm Sunday evening, over the Thanksgiving holiday.
My first note back was thanking Rhonda for letting me know about it. And enumerating the services, more for myself than for her: primary mail server (and the MX for extension.org), secondary DNS, LDAP and Kerberos authentication services for shell access to our servers, some log processing, backups. I kept mulling over in my mind what mattered and didn’t. Or more, what would handle a 57 hour outage and what wouldn’t.
I guess then, I had my first set of epithets running through my mind. “My wife and I are supposed to be visiting family, and now I spend 2 minutes in a place with network access to shut things down. And 10 minutes after a 45 minute drive - and a cut short holiday period to bring it back up.” Plus multiple expletives
You might wonder whether I have some out-of-band management. I do on one of the servers. But not another. It’s an Apple XServe, a dual-G5. It’s not something that I can power back up remotely. I know about IP-enabled power strips. I also know that it’s on a wide-open network.
You might also wonder why a power outage is affecting things like this. Some of you data center nuts are probably freaking out about where the dual power grids and generators are. This is higher education. And I’m a small, grant-funded project. There are two or three data centers on campus that do have redundant power and adequate cooling but they are limited access and talking about them would entail a few more expletives - and my mom reads this blog.
So there’s a part of me, when not having to deal with 57 hour outages that realizes that this is only the third power interruption in three years - and those were each less than a day. And two days power downtime for services that are largely only active during the traditional business (Eastern Time) day is hard to justify hundreds of thousands in data center provisioning. We do have some services that are going hosted for the data center benefits - but not everything.
Anyway, my guilt about this is abated a bit by knowing that many of my academic brethren are thinking “Server Room? You mean that empty cube in the corner where we have those Xserves propped on concrete blocks under the desk?” Don’t laugh. It’s true. And it works actually. Works quite well. At about 1/1000th of the data center cost. Until someone needs the concrete block to hold open the conference room door.
Somewhere in between an XServe on concrete blocks, and Halon fire suppression systems is the happy medium that I find myself in - at least until it’s a 57 hour outage.
My second note was to a colleague in the
My third was to the staff. Letting them know that I had some decisions coming up. Suffer the 57 hour outage? Suffer a few hours outage to the move the servers? Try to restructure services to my primary server room to mitigate the downtime? Anyway, if anyone has an opinion, please let me know. But here’s the heads up.
And then I had to tell my wife that the holiday plans might just have gotten changed.
Elapsed time, 40 minutes. Come Monday, there’d be about 720 more (expletive filled) minutes to be had.
In the follow-on post, I’ll take you on the continued journey - from poorly designed hardware, to poorly structured documentation, to the IT flipside of Einstein’s postulation about insanity - insanity really is doing the same thing again and again, expecting the same result, but getting different ones.
And the bonus tip - why your power cords need to be cable-tied to the outlet in tight rack spaces.
Coming tomorrow, Systems Desconstructed, part deux, same bat blog, same batty author.
