Live Journal problems

[Motorboat Cruiser]

Well sure, I sign up for Live Journal and crash the whole damn thing. Story of my life.

I noticed them down earlier today. Apparently it is more serious than I thought.

Our data center (Internap, the same one we've been at for many years) lost all its power, including redundant backup power, for some unknown reason. (unknown to me, at least) We're currently dealing with bringing our 100+ servers back online. Not fun. We're not happy about this. Sorry... :-/ More details later.

Update #1, 7:35 pm PST: we have power again, and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.

Oh well, it sounds like it won't be down for too long and they at least have backups of the data to work with. Seems odd that the power backup systems would lose power as well though.