Human judgement first: Xero and Anthropic on the future of work

I spend most of my time looking beyond the traditional planning horizon, researching emerging technologies and exploring what they might mean for people years down the line. But, with AI, capabilities are already arriving faster than most organisations can absorb them. So, the question becomes how we make deliberate choices about what comes next.

When Lina Ochman (Head of US SMB, Anthropic) and I took the stage at Xerocon US, our goal wasn’t just to talk about what AI models can do today, but to address what must be true for AI to really unlock the future for people in business, their advisors and communities. For me, that comes down to three things: the context models work from, the conditions that keep them in bounds, and a system of agreed truth.

Context makes capability useful

I describe our collective task as taking the raw AI capability on offer, including from foundation model providers like Anthropic, and refining it into genuine business utility. That refinement is driven by context – the workflows and processes behind the business, the financial data that courses through them, and the human judgement that sits on top of the numbers. Accountants and bookkeepers have always been experts in capturing and refining this context, which is just one of the reasons why the profession matters more now than ever.

Sure, AI can assemble and analyse data at speed, but it still needs a reliable foundation. For twenty years, Xero has built a platform where advisors and small businesses collaborate on financial context, permissions and workflows. Now, this gives AI capabilities a reliable system of agreed truth, grounded in real human collaboration and rich financial context, so models can execute routine work accurately while advisors and business owners remain in complete control of the decisions that follow. Capability → Utility. As an example of this in action, since launching our integration with Claude, thousands of customers have put this to work in their business context, daily.

The harness and keeping humans in the lead

In AI terminology, a ‘harness’ is everything surrounding the model that controls how it interacts with its environment: the context it accesses, the permissions it inherits, the boundaries it holds, and the deliberate pause points where it requests human validation. Those conditions become increasingly important as we ask AI to help with more tasks. For example: there are AI capabilities that can help reconcile, forecast, rank and draft, but they must stop before committing consequential actions, keeping people in control of the outcomes.

Research into cognitive offloading led me to a shorthand principle that I keep coming back to: “brain first, AI second.” People need to supply the intent, context and the purpose up front, while AI capabilities need to be deployed within those parameters and specified guardrails, making human judgement the starting point.

Because the reality is that accountability doesn’t move. For accountants and bookkeepers, this isn’t news: reviewing and standing behind the work product has always been the professional standard. So, having a reliable harness to help with focusing AI capability into utility has to be in aid of enhancing and supporting that core accountability.

Scaling governance so small businesses don’t have to

Broader AI adoption can’t depend on every small business owner figuring out how to set guardrails, build governance and manage model safety on their own. Nobody started a (non-AI consultancy) small business to become an AI governance expert – well, not any that I’ve met yet!

That’s why the harness matters, and matters at scale. For our part, the five million small businesses and advisors who trust Xero with their financial records today – and the many more we aim to support as we grow – are relying on us to build and maintain the best possible harness. As model capabilities continue to expand, we know that the ability to manage permissions, context, safety research and workflow integration have to scale with them. And alignment between the organisations building these foundations is one of the things that makes that possible.

Shaping the trajectory together

AI capabilities are already being embedded in how people work, and creating opportunities larger than any single model, product, platform or company. We think that reliably capturing and leveraging those opportunities requires the ecosystem to align around the needs of small businesses and their advisors; our partnership with Anthropic is one step in that direction.

By focusing on the context, keeping the harness in place and holding to a system of agreed truth, we’re working to shape a future where human judgement leads and AI serves people, their advisors and their communities in realising the business future they prefer.