Xero outage – network routing issue

Earlier today we experienced an network routing issue upstream from Xero’s own network border that prevented some customers from being able to access Xero.

Xero’s production hosting environment in a Rackspace data center is connected to the Internet through multiple tier-1 carriers. However, this particular issue resulted in the  incorrect routing for some of these carriers, meaning that customers with an ISP that would normally utilize one of the affected carriers would not have been able to connect through to the Xero environment. Customers with ISP utilizing the other unaffected carriers were able to connect to and use Xero completely as normal. We understand a small number of other Rackspace customers were also impacted in a similar manner.

Xero’s monitoring systems detected the failure immediately, and our IT Operations team began working with both Rackspace and Akamai, our content delivery network (CDN) provider, to diagnose the cause of the issue and it was successfully resolved around 90 minutes after being identified.

Our apologies to any customers that were unable to access Xero over this time. It is difficult to prevent network issues which occur outside of our own hosting environment, but we will be working with Rackspace and Akamai to identify any improvements in our network connectivity that could better mitigate us from these types of issues.