Found this .. this morning. I must have slept through it.
This evening 4/11/2008 at 11:26 we experienced a customer impacting incident in our DC3 datacenter facility. The incident caused momentary power loss in the facility which power cycled all of the equipment. While most equipment came back online quickly, even a momentary power loss is a very disruptive event for a large datacenter. At the moment we are handling many situations where either hardware or software did not recover correctly. We are working very hard to answer all calls and respond as quickly as possible to all impacted customers. Additional staff is continuously arriving at our facilities to assist, and we will continue to bolster our staff as quickly as we can. I wanted to get this communication out to our customers as quickly as possible and we will provide more complete updates when we have 100% of our customers operational.
Specifically the main transformer feeding power to our building experienced degradation causing a low voltage condition. As best we can tell, our parallel redundant Liebert UPS drained its batteries from dealing with the prolonged voltage irregularity, and when the transfer switch attempted to take us back to street power, the UPS did not handle the load. This is a very early approximation of the situation and we will provide a full assessment as details become available.
On behalf of everyone here , we appreciate your business and understand that something like this should never happen. We are doing everything in our power to rectify the situation as quickly as possible. I take every impacted server personally and I approach every customer as if their business was my own. I know our staff does the same. Please accept our sincere apologies.
Regards,
Lou Honick, CEO
Then this evening.
We wanted to give a further update on the situation that impacted one of our Data Centers (DC-3) last evening. Our team worked through the night to ensure that we recovered from this event as fast as possible, all systems have been running normally for some time now. We sincerely apologize for the impact this has had on you and are working with the vendors of our power equipment to conduct a root cause analysis.
