Large Scale Downtime At Rackspace Cloud

03
Nov

Large Scale Downtime At Rackspace Cloud

author: editor category: TechCrunch

Screen shot 2009-11-02 at 11.48.54 PM A large number of customers of Rackspace Cloud, including Techcrunch, have been experiencing downtime for the past 1h 20m or so. The status blog reports that the service was degraded, and other reports state that it is due to a power outage at the Dallas network operations center. Customers of both Rackspace Cloud and Slicehost are affected, putting services such as Posterous, Dailybooth, tr.im and others out of commission.

I got the first alert as I was stepping towards the door to leave (it is always like that), and when I got back to my seat found that half the web seemed to have imploded. The main Techcrunch site was still serving pages to most, due to our super-aggressive-mega-cache, but it seemed that the entire Dallas NOC was being rebooted.

From the status blog:

As of 12:35AM CST Rackspace Cloud engineers are seeing intermittent connectivity to our WC2 cluster in our Dallas – Fort Worth (DFW) and data center. We are working to resolve the issue as quickly as possible and will update the status post accordingly.

If you have any questions or concerns please contact our support via live chat or at 1-877-934-0407 international +1.210.581.040.

UPDATE: As of 1:15am CST, Rackspace Cloud engineers are still working to address the current connectivity issues. We are making significant progress and we will post another update here shortly.

UPDATE: As of 1:30am CST, service has been restored to the majority of our technology clusters in our WC2 cluster. Some sites may still be having performance issues, We are continuing to monitor and address the situation. Additional updates to follow.

From slicehost (who actually mention power outage):

DFW Interruption
November 3rd, 2009 @ 01:14 AM

UPDATE 1:16AM CDT: Power has been restored, however, we’re working to check all our systems and make sure everything comes back up correctly. Slices have not yet been restarted. We’ll try to keep you updated as much as possible.

We are currently experiencing a service interruption in our Dallas data center. Our engineers are currently working to restore connectivity. We will send an update as soon as information becomes available.

And from Scoble, on Twitter:

Screen shot 2009-11-03 at 12.03.52 AM

(the list he pointed to is actually a good one to follow if you are a Rackspace customer).

This will likely lead to many cursing the cloud, when in essence there is nothing about this problem that seems unique to being a ‘cloud problem’. What is more concerning is that the NOC seems to have run out of power (almost unimaginable) and then took so long to come back online.

So – how did you all spend the downtime? It seems most admins and devs from Rackspace hosted companies were just hanging out on Hacker News and IRC bitching about RS (first time I noticed that he shares initials with his employer).