The $150,000 Sunday Cook-in

The meltdown at Acuity software spurred development of low-cost server room monitors - a true story.

The equipment didn't look dead. It looked the same as it did three days ago. No burn marks, no melted faceplates. Even the Cisco logos on the routers were still shiny white. One thing was different: it was very quiet in the server room. Inside, the room normally sounded like a vacuum cleaner, plenty of fan noise. Not today.

All the gear was dead.

Our IT manager was removing some servers from the racks and placing them on wheeled metal cart that looked like the kind the food caterers delivered lunch on.

On several units, the tiny lights were fixed on, as if the electronics inside the pizza-box cases were still alive, begging to be freed. No rapid, urgent flicker, as in the past. Just steady pleading from inside of some of the units. Our server room was toast. "All the gear is dead?" I asked.

"Fried. Somebody left the thermostat turned up over the weekend and the heater cooked the gear."

"The heater blows into the server room?" I asked.

"Yes, it blows right in there," he pointed to the ceiling register.

"Any chance you'll be able to save the disks?"

"Very little. The heat sublimates the bearing oil. The drives go dry and seize up. I'm sending them to a recovery shop, just in case," he sounded half-hearted.

I called an IT friend at another company and told him what happened. "It's just a regular office heater. How did this happen?" I asked.

"The heat adds: you have the heat from all the gear plus the heat dumping in the from the ceiling grill," he answered. "I'll bet it was over 110 degrees in less than six hours. Inside the equipment cases it was over 130 degrees. Your gear is scrap, not worth repairing," he gave me no hope.

He asked what kind of software was on the equipment.

"Our internal web and data servers. E-mail, the CRM stuff, and the development guys files," I answered. "Everything except the corporate web site." The corporate web site was hosted in New Jersey.

"Tough deal there," he counseled. "Replacing the hardware will take a week or more. Installing the software could take a month."

"What if we have back-up disks?" I asked hopefully.

"Unless you have the same gear coming in as you burned up, there will be incompatibility problems. How old was the gear?"

"Maybe five years old." The depth of the problem was expanding. We had bought some of the equipment from the last start-up company occupying the building. We had not installed much of the software ourselves. We probably didn't have all the CD's. Heaven knew where all the license codes lived.

IT gear five years old was considered geriatric equipment. Probably no parts were available and they were certainly long out of warranty. It would be a fresh start on the equipment list.

The Cost of the Meltdown

I asked the lead software developer what was in the room and he gave me a thumbnail list. There were far more machines in the room than I recalled even thought the room had hall windows. I added up his guess at what was lost. Over $100,000, not counting the time needed to re-order the gear. The IT guys had a sizable project.

Cost of Replacement Equipment
Device	Quantity	Cost	Total
Routers	4	$6,000	$24,000
Servers	8	$10,000	$80,000
Switches	2	$1,000	$2,000
UPS	2	$1,000	$2,000
		Total:	$108,000

"That's what the equipment replacement is going to cost. "You have to reload the application software. You do have the CD's around the shop, right?" I answered that we must have them somewhere, but I recalled we had a complex-looking magnetic tape backup system in the room with the equipment. I wondered if the back-up unit got fried along with the gear.

"The application software is on the back-up tapes?"

Cost of Re-installing Software
Device	Applications	Days to Reinstall	Total Cost @ $1000/day
Routers	4	5	$5,000
Servers	30	40	$40,000
Switches	2	5	$5,000
UPS	2	2	$2,000
		Total:	$52,000

"No, and if it is it's probably an old version. Your guys will probably first reload the Exchange Mail Server and then work on the Pivotal CRM system. The developers can load their own stuff back. All the employee data files should be on the back up tapes." I wondered if anyone had ever tried a backup. My Power Points were on one of the disks stacked in the hallway. I had two weeks of work in those slides.

Since I didn't know salaries for anyone except myself, I used consultant prices to reinstall the software. I showed my spreadsheet around and most folks thought the cost would be twice that, but come back in two weeks and we'll have a better idea. Nobody felt like talking about the meltdown, much less the IT guys who had become almost invisible.

"Plan on a at least a month getting back up unless you have folks trained on those enterprise-size systems in-house. You'll be feeding CD's into those replacement machines for weeks. Just rewriting the router tables will take a couple of days."

I felt bad about the melt-down. I had been there that winter Saturday when one of us had jammed the thermostat setting to the highest setting, 95 degrees. The heaters must have fully kicked in after we left. We froze most of the day and forgot about the thermostat setting.

Nobody ratted on who set the thermostat, but we were all under suspicion.

I asked various technical staffers who had lost some computer capability when the room melted down how much time we lost. Weeks, easily, they said.

The thermostat cost $50. I never got my PowerPoints back. If I ever built a server room I knew this wasn't going to happen to me.

Comments by Gerry Cullen who participated in the meltdown.
He left Acuity to found NetBotz and, later, IT WatchDogs.
Both are leading providers of server room monitoring equipment.