Why AI Governance Is Failing (And What Leaders Should Do About It)

What if the biggest risk from artificial intelligence isn't the technology at all? What if it's us, and our failure to govern it properly?

That's the question I opened the first episode of The Future State with. It's a deliberately provocative one. My argument is simple: AI governance matters more than AI itself. A powerful model in a badly governed organisation is a liability. A modest model in a well-governed one can create real, lasting value.

If you lead a team, a company, a regulator or a legal practice, this is your problem now. Here's why.

Why is AI governance failing?

A few years ago, most machine learning in business was fairly contained. A regression model predicted costs. A classifier flagged a risky transaction. These systems were useful, narrow and predictable.

Generative AI changed that. Tools like ChatGPT and Claude write reports, draft documents and summarise contracts. More and more, they act on our behalf. New capabilities show up almost monthly, and AI agents are getting more autonomous. Even the people building these models don't always know what they're good or bad at until the models are out in the world.

Governance is stuck in a slower gear, for two reasons.

Software moves faster than legislation. Developers ship weekly. Parliaments don't. By the time a law has been debated, drafted and passed, the technology it was written for has moved on.

Businesses deploy faster than they govern. When something new comes out, the instinct is to find the profit in it quickly. Governance gets bolted on later, usually after something has gone wrong.

AI is like a knife. You can cook with it or you can hurt someone with it. The tool doesn't decide which. The rules around it do.

What is AI governance?

AI governance is the collection of policies, standards, accountability mechanisms and legal frameworks that make sure AI systems are used responsibly. That covers the law, but also the internal controls, approvals and review processes inside an organisation.

The easiest way I know to explain it is the road.

To drive, you pass a test. You obey traffic laws. You get insurance before the car goes anywhere. Speed limits change from 30 to 70 depending on the road, and if you break them you get a ticket. None of this makes cars less useful. It's what makes the road safe enough for everyone to use.

If you've ever lived somewhere with weakly enforced road rules, you'll know how different that feels. The cars are the same but the outcomes are very different.

AI works the same way. To get the best from it, we need the right laws, the right frameworks and the right bodies in place. Technology alone can't deliver that. People don't follow rules just because a system nudges them. They follow rules because there are consequences for breaking them, and setting those consequences is a governance job, not an engineering one.

History keeps repeating: technology first, governance later

We've been here before.

Social media was celebrated for years before we properly reckoned with its effect on young people, public debate and mental health. Now parents are rethinking it, and in Scotland, Edinburgh's council has moved to restrict phones in schools. The guardrails came long after the adoption.

Personal data was collected almost without limit for decades. GDPR now shapes how organisations handle personal data across Europe and beyond, but it only took effect in 2018. Data collection was happening long before the rules caught up.

The pattern is consistent. Technology creates capability, and governance decides the consequences. If we want AI's capability to turn into progress and not harm, the governance has to keep pace.

Three pillars of responsible AI

When I look at where AI goes wrong, it usually comes back to three areas. Get these right and most of the risk shrinks.

1. Accountability: who is responsible, and who controls the power?

Picture a driverless car approaching a junction where the traffic lights have failed. It doesn't get the signal to stop, and it hits someone crossing the road. Who's responsible? The owner, the manufacturer or whoever maintains the lights? These questions don't have settled answers yet, and courts are only starting to work through them. Before your organisation uses AI, you should know who answers when it gets things wrong.

Accountability also means asking who controls the power. Absolute power corrupts absolutely. Systems this capable can be misused, attacked or turned against the people they're meant to serve. You can't secure what you don't understand, which is one more reason leaders need to engage properly.

2. Transparency: should people understand AI decisions?

If AI is deciding on your job application, your mortgage or your credit card, should you be able to find out why? I think the answer is increasingly yes, and regulators are heading the same way.

3. Fairness: how do we prevent bias in AI?

AI reflects the data it's trained on. Some years ago, Google's image recognition mislabelled Black people, a failure widely linked to unrepresentative training data. Nobody intended that outcome. It was a governance gap. Bias testing has to be built in. You can't just assume a system is fair.

AI is a leadership issue, not a technology issue

This is the part I most want leaders to hear. AI isn't something you can hand to the IT department and forget about.

Every organisation needs to be clear on who approves the AI it uses and what risk assessment sits behind each system. It also needs to know how it protects public trust and what its regulatory exposure looks like. Those are board-level questions.

I've spent my career in data engineering, so I'll say it plainly. The leaders who do well in this era won't necessarily be the most technical. They'll be the ones who understand governance and are willing to put proper guardrails in place.

Three things leaders should do today

Build an AI inventory

Start by finding out where AI is already being used in your organisation. For each use, ask who approved it and what risks it carries. Write the answers down. Keep it as a living document that gets updated when things change. You'd be surprised how many organisations can't answer the first question.

Establish governance before you scale

Don't wait for regulation. The EU AI Act only entered into force in 2024, years after these tools went mainstream, and the UK still has no single, general AI Act. Organisations tend to formalise governance only after a crisis forces their hand. You don't want to be that case study.

Put policies in place and set up a clear accountability structure. Run regular reviews at a cadence that suits you, whether that's monthly or quarterly. Do this before scaling.

Educate your leadership team

Most people want to do the right thing, but they can't act well on something they don't understand. Make sure your senior team knows enough about how AI works, where it fails and what the law expects. Then they can ask the right questions instead of just signing off.

The bottom line

For most organisations, the AI race isn't about who builds the most powerful model. It's about who builds the best system for governing how AI is used. Get that right and AI lifts your business. Get it wrong and it can pull everything down with it.

In future episodes I'll dig into liability, algorithm audits and AI in hiring. If you want this kind of thinking in your inbox, subscribe to the newsletter at thefuturestate.net and listen to Episode 1, Why AI Governance Is Failing, in full.