Unexpected outage

We’re currently experiencing network problems in our hosting infrastructure which are affecting many of our systems. We are working as fast as possible to get them back online however we can’t provide you with an estimate at this stage as to when that will be. We will provide updates via this blog post & twitter as and when we know.

We apologize profusely for any inconvenience this may cause – this outage will not affect your data in any way – it is purely a network problem at this stage.

Update @ 9:10 am: All our systems are back online. We are still investigating the cause of these issues. More info to come.

Update @ 9.19 am: According to Rackspace it appears to have been a power outage rather than a network problem. Power has now been completely restored to our data center. We’ll provide more details as to the exact nature of the power failure when we hear more.

Update @ 9.59 am: According to the Rackspace blog (http://www.rackspace.com/blog/) they are still manually bringing up some services however this does not affect our systems which have all come back online (a lot of other sites are still down). We are currently running consistency checks on all our systems so you may experience some slowness – these should be finished in the next hour.

Update @ 10.32 am: We understand some customers are still having problems logging into Xero. Please email support at xero.com if you are having problems logging in. And in case you were wondering whether the problems at Rackspace this morning only affected you, even famous people were a little put out: http://twitter.com/jtimberlake/status/2392792791

Update @ 12:33 pm: Rackspace has announced that they are currently running on a mixture of utility & generator power. While all our systems are running as normal we will continue to update you as we learn more. And please let us know if you’re still experiencing problems.

Update @ 2:00 pm: Rackspace has now got all their systems back on utility power & everything at their end is running as expected (I believe there are some sites that are still unavailable but that’s not affecting us at all). As stated above we’re completely back online and have been for the last 4 hours without incident. Thank you to everyone for their patience while we worked to get our service back up and running. This will be the last update till we learn exactly what happened from Rackspace (they’re are still very busy so are only providing us with the bare essentials at this stage). Again we apologize for any inconvenience the outage caused.

Update @ 4:37 pm: Rackspace has just updated their blog with some more information but they haven’t provided any more details on the root cause yet: http://www.rackspace.com/blog/?p=355.