Infrastructure upgrade
This morning we moved to version 3 of our hosting infrastructure.
We started planning for this move in January. This is the next natural step up in our hosting capability and puts us in an environment to continually scale and grow.
For the geeks out there, highlights of what we did last night included:
- Mixture of diagonal and vertical scaling of hardware for increased capacity and performance
- Moved our data store from attached storage to a RAID 10 SAN
- Virtualized some devices (our own dedicated virtualization environment) to allow us to better scale out our service delivery platform (the Xero application is still running on completely dedicated hardware)
- Moved to a completely 64 bit platform running on Windows 2008
- Upgraded our database management system to SQL Server 2008
Projects of this magnitude require a huge amount of planning and testing so thanks to the team for all the up front work that enabled this major platform upgrade to proceed without a hitch.
Read more about Company News
21 comments
Good news but do you have any plans to host locally or regionally i.e. NZ or Australasia as there are times when we use Xero itself where it so slow that it times out; I am talking over 2 minutes here.
All the big players Microsoft, Google host applications regionally.
From the US Xero is very fast but from NZ at times it runs like a snail.
As I am in US at moment I cannot say whether the above changes have helped speed in NZ.
A 2 minute time-out would have nothing to do with any latency at Xero’s end. Hosting in the USA only adds about 200ms to the round-trip time and, from what I have read, they are using regional content distribution for static content anyway (images, JavaScript libraries, etc). For general usage, the latency should be almost invisible. A 2 minute timeout seems like a network problem at your end, rather than a problem with Xero’s infrastructure.
Bart,
Thanks- there certainly was a lot of planning and work throughout the implementation of this project. I remember not being able to get back to sleep after the 2am Wellington Earthquake last week- double checking every detail in my mind…
Dermott,
Performance of the app is something we take very seriously. Craig has spoken about our approach in the past: http://blog.xero.com/2008/07/optimising-xero-performance/
Wayne,
You’re correct: Xero does use a regional content delivery system to improve performance in key markets. We’re also looking at a couple of new technologies that should provide even faster performance for our customers.
Hi Craig
I am interested in : “Mixture of diagonal and vertical scaling of hardware for increased capacity and performance”
Can you elaborate a little?
Thanks
JY
Sorry for the delay overseas
@Wayne
The problem is not our line we have an ISP grade fibre line high speed etc
I think the issue I raised is misunderstood by some of the people who replied; all the processors in the world etc will not overcome the lack of regionally hosted the data. There would be many advantages commercially for Xero doing this as well as speed.
@Dermott
I too from time to time have very slow access to Xero, generally the same time when Mobile Me takes an age to do anything. Australasian hosting of Xero would be fantastic, considering it is a Kiwi company.
“A 2 minute timeout seems like a network problem at your end, rather than a problem with Xero’s infrastructure.”
We used to host in the States but have since moved everything to NZ because of the following:
1. It is faster. New internet traffic (non-cached) such as DB queries and SSL traffic is normally 128 – 256Kbits through major NZ ISPs DSL so if your servers were here local customers would have 8Mbit traffic.
2. More Hops = more problems. If I do a traceroute to an NZ IP address I can expect around 10 hops. When I traceroute Xero it is 15 hops which gives the opportunity for more points of failure.
3. Local access IS better. Cloud infrastructure such as Amazon’s EC2 also appreciate these issues and clusters servers Internationally so if I access the site from the US it uses a US node; if I access the site from the UK it uses the UK node.
We’re presently structuring our servers to be EC2 compatible so when we do finally get an EC2 node in NZ we can expand our servers through their cloud.
Don’t know how this would all work with Windows though. We only use Open Source tech: http://brynnneilson.blogspot.com/2009/06/free-software-is-better-for-business.html
brynn
Local hosting is extremely expensive given the high cost of bandwidth here – i think Xero is doing the correct thing by hosting with Rackspace. They are the best hosting company worldwide and i dont believe they have a rival on this side of the atlantic.
@Brynn
As a network operator I disagree. Local hosting is actually quite cheap, and national bandwidth very much so. (It’s the international CIR that’s pricy).
I see Xero as a premium product, and hence find it supprising that it’s not sitting on the national backbone. (I haven’t trace routed the servers to verify it’s intl as I’m sitting in a restraunt on my iPhone).
Liam. (nzl@me.com).
Local hosting may start becoming a necessity for EU customers due to data protection legislation. Fundamentally this means you can’t store data about someone outside of the EU without their permission. Given that invoices frequently hold personal information it would seem that have local hosting in the EU for data would be prudent.
@jake
I am not asking for all of Xero to be hosted in NZ or even Australasia. All I think is US customers access US servers, NZ/Aus servers in this part of the world. This is not a religious argument here; it will be faster for NZ/Aus customers if it is closer; basic physics.
Local hosting is not expensive in my opinion
Craig
Rackspace claim 100% uptime but they don’t achieve it as per a 45 minute downtime they had (and Xero had) 2 months ago. No one has 100% and it is naive for anyone to claim this. Just because their SLA says this doesn’t mean they or anyone else achieves.
Hosting web based systems with high availability in NZ is not impossible; we have done it for 4.5 years.
Datacom has a $30 million dollar data centre in Albany with Multi everything.
As far as “lots” of your customers having NZ, Australian and UK accounts I find that hard to believe. Must of your customers would be in NZ surely.
The single user account concept is not as important as reasonable speed.
Brynn,
An EC2 node in NZ? That’s optimism!
Dermott,
Hosting with Rackspace is just different to any other provider in the marketplace.
I can call and speak immediately to someone at rackspace 24/7/365 – and no, not a first level tech, but a system engineer, be it a member of the Virtualisation team, Storage, Backup, Network Security or Hardware, any time, day or night.
That is a fundamental difference between hosting with Rackspace and other hosting providers- people are not on-call- they’re onsite.
I understand where you’re coming from with alot of your comments, but I side with the “Host some in NZ” team.
“What if you have a organization in NZ, Australia and the UK (which a lot of our customers do)? What if you travel? Do you expect your data to move with you on that flight to the UK?”
What if you were to let me choose where I host my data? If I were to be spending alot of time offshore I would choose to host offshore, and deal with the latency when in NZ. Since I spend 99.9% of my time in NZ, I would opt to have all the data hosted in NZ, and deal with it being in NZ while I’m overseas.
You also imply that hosting in NZ would result in more downtime, I would gladly trade a bit more downtime in return for Xero being alot more snappy. If you counted the seconds I spend waiting for pages to load that adds up for alot of downtime over a year! (Especially when I’m in Xero all day
)
It is frustrating that this is such an issue for NZ companies.
Working out how to distribute databases in different regions has nothing to do with our business and is a complex problem. The day we solve it some company will lay a few cables across the Pacific Ocean and we will be maintaining an unnecessary complex mess.
International bandwidth is the problem. More international bandwidth is required to stop NZ companies wasting time on working around this issue and save NZ customers wasting time using slow internationally hosted products.
ps. I realise I am stating the obvious and not really contributing to the debate.
James – you’re not stating the obvious
1) Latency – sure light travels fast but a NZ customer will always have a slower experience using a US hosted site – especially if there are multiple round trips (as an aside – check out aptimize for some cool web efficiency gains)
2) Jurisdiction is a major issue here. Already the Government is demanding that services it procures are domiciled locally. While not an issue for SMEs, it’s something we can’t ignore. Imagine, for example, the governemnt deciding that getting NZ SMEs onto SaaS apps was a good investment. Right now, if they were to procure those services themselves, companies like Xero would be unable to pitch for that contract…..
More here – https://www.cloudave.com/link/where-is-your-data
I would like to congratulate Xero for their transparency and willingness to discuss issues such as hosting openly. I couldn’t imagine other software suppliers being so open and this says heaps about Xero’s ethos and intent to value user opinion.
Speed is a critical factor for us. We are considering a shift to Xero and we are larger than ‘small’ with 140 staff and 9 campuses. Are there similar size organisations on Xero who can speak objectively about the speed of Xero in data input and particularly in report writing? We have outgrown QuickBoooks and wonder if Xero could cope?
As a Geek I can appreciate the amount of work that went into planning this upgrade, a good job well done – give them a raise I say!