AI Governance Is More Than a Technology Problem

Most conversations about AI governance begin with technology: which tools to approve, which vendors to trust, which controls to install. Those questions matter. But the harder questions sit elsewhere. They are about leadership, judgement and how organisations change.

AI changes who makes decisions, how quickly they are made and how well anyone understands how they were reached. That makes AI governance a leadership and change challenge first, and a technology challenge second.

Frameworks and Safeguards Are the Starting Point

Australia’s Guidance for AI Adoption, released by the National AI Centre in October 2025, sets out six practices for responsible AI: decide who is accountable, understand impacts and plan accordingly, measure and manage risks, share essential information, test and monitor, and maintain human control. In Europe, Article 14 of the EU AI Act requires human oversight of high-risk AI systems.

Boards should expect the core safeguards to be in place: a register of the AI systems in use, clear accountability for each one, risk assessments before deployment, testing and monitoring, escalation pathways, and the ability to pause or override a system when something goes wrong. These are the foundations of an AI governance framework.

But almost every framework rests on one assumption: that the people overseeing AI can, and will, exercise independent judgement. That assumption deserves far more scrutiny than it receives.

Human Involvement Is Not Human Oversight

“A human reviews every decision” has become the standard reassurance in AI governance. Yet human involvement and human oversight are not the same thing.

A person can be present at every step of a process and still not be exercising judgement. When an AI system is usually right, people learn to accept its output. Researchers call this automation bias: the tendency to over-rely on automated recommendations, even when other evidence points the other way. A peer-reviewed review of the research by Parasuraman and Manzey found that automation bias occurs in experts as well as novices, and that it was not prevented by training or instructions. The EU AI Act names it directly, requiring that people overseeing high-risk systems remain aware of the tendency to over-rely on AI output.

Process design often makes it worse. Accepting a recommendation may take one click, while disagreeing requires a written justification. Volumes rise, review time shrinks, and the reviewer gradually stops reviewing.

The risk is that human oversight becomes a procedural safeguard rather than an exercise of independent judgement. The approval is recorded and the audit trail is complete, yet no one has actually decided anything.

Researcher Ben Green’s analysis of 41 policies requiring human oversight of government algorithms reached a similar conclusion: the policies assumed people could perform that oversight effectively, while the evidence suggests they often cannot. Mandating oversight anyway can create false reassurance.

So the question for Boards is uncomfortable but necessary: are we placing too much confidence in the concept of human oversight, without examining whether it actually works in practice?

This Is a Behavioural Risk

None of this will surprise anyone who has spent time in boardrooms. Boards defer to a confident chief executive. Directors stay silent when everyone else seems to agree. Information arrives already framed, and few people question the framing.

The same dynamics can emerge when organisations rely on AI. Deference to authority becomes deference to algorithmic recommendations. Confidence in the system can discourage independent challenge, while accountability becomes blurred between those who develop, deploy and rely on its outputs. The risk is not simply that AI produces a flawed recommendation, but that organisational behaviour allows it to go unchallenged.

That is why I see AI governance as part of Behavioural Governance. The question is not only whether a control exists, but whether people use it with independent judgement when it matters. I have written previously about behavioural capture and why Decision Integrity depends on more than process. AI adds a new and persuasive voice to the room.

What Meaningful Human Oversight Looks Like

Human oversight requires more than willingness to challenge. Meaningful oversight requires people who understand the system’s limitations, have sufficient time and information to assess its outputs, and possess both the authority and confidence to challenge or override them.

• Ask for evidence that oversight works. Override rates, time spent on each review and examples where a person changed the outcome. A review process that never disagrees with the AI should prompt questions, not reassurance.
• Make disagreement as easy as agreement. If challenging a recommendation takes more effort than accepting it, people will accept it.
• Form a view before seeing the answer. For high-stakes decisions, research suggests that asking people to reach their own conclusion before seeing the AI’s recommendation reduces over-reliance.
• Match oversight to the stakes. Not every output needs human review, but those that matter need real time, context and the authority to say no.
• Keep accountability with people. “The system recommended it” is not an explanation a Board should accept.
• Look at the Board itself. Is the Board receiving AI reports it can genuinely interrogate, or simply noting them?

Redesign Roles and Build Judgement

As AI takes on more analysis and drafting, the human role shifts from producing work to evaluating it. That is a different job, and it needs different capabilities:

• Critical thinking: questioning outputs, testing assumptions and noticing what is missing.
• Judgement: weighing context the system cannot see.
• Ethical decision-making: recognising when a technically correct answer is the wrong decision.
• Confidence to challenge: being willing to disagree with a system that is usually right.

Much AI upskilling focuses on how to use the tools. Far less attention goes to how to question them. Organisations that redesign roles around judgement, and invest in ethical decision-making and judgement under pressure, will be better placed than those that simply add a review step. Organisations must also design roles, decision rights and escalation processes so that people are genuinely empowered to challenge AI outputs, rather than merely expected to approve them.

This is also a change challenge. People need to understand why their roles are changing, and to know that challenging the system is expected, not penalised. Culture will determine whether oversight is real.

Frameworks Set Expectations. Judgement Determines Whether They Hold.

AI will keep becoming more capable. The quality of governance will still depend on people who are prepared to think independently, ask uncomfortable questions and take responsibility for the outcome. As I have argued before, governance often fails even when policies exist. AI governance will be no different unless we pay as much attention to human judgement as we do to the technology.