Why most AI pilots never reach production (and what to do instead)

The demo worked. The pilot impressed the leadership team. Everyone agreed it was promising. And six months later, the way the organisation actually works has not changed at all.

This is the most common story in AI adoption, and it rarely has anything to do with the quality of the model. Pilots stall for organisational and engineering reasons that are visible from the start, if you know where to look.

1. The pilot was built around the technology, not a problem

Many pilots begin with a capability (“let’s see what we can do with a large language model”) rather than a problem (“our team spends two days a week answering the same supplier questions”). A capability looking for a problem produces an impressive demonstration and no owner. A problem looking for a solution produces something a specific team needs and will fight to keep.

Before building anything, write down, in one sentence, the work that should look different once the system exists, and who does that work today.

2. It never touched the real workflow

A pilot that lives in a separate tool, with its own login, copied data and a separate process, asks people to change how they work before it has proven its value. Most will politely try it and then return to what they know.

AI earns adoption when it sits inside the workflow people already use: the inbox, the CRM, the WhatsApp channel, the internal system. That makes integration the main piece of work, not an afterthought. It is also why we treat AI as one layer of an intelligent operations system rather than a standalone product.

3. Nobody decided what “good” looks like

“It seems useful” is not a standard anyone can approve a budget against. A pilot needs a small number of observable measures agreed at the start: time to respond, requests handled without escalation, errors caught, hours returned to the team. Without them, the pilot ends in a meeting where everyone has a different opinion and nobody has evidence.

4. The data wasn’t ready, and nobody planned for it

AI systems are only as reliable as the information they are given. If the knowledge they need is scattered across documents, spreadsheets and people’s heads, the pilot works on the clean sample prepared for the demo and fails on the real thing. Connecting and tidying the sources is often the first real milestone, and it deserves to be scoped as such.

5. There was no path from pilot to production

Production systems need things pilots skip: access control, monitoring, error handling, a way to review what the AI did, and someone responsible for keeping it running. If none of that is in the plan, the pilot has nowhere to go.

A better way to start

Pilots that make it to production usually share a shape:

• One problem, one team. Specific enough that success is obvious.
• Built into the existing workflow from day one, even if the first version is small.
• Measured against the way things work today, with the measures agreed up front.
• Human review where it matters, so trust grows with evidence.
• Designed for production from the start: security, monitoring and ownership included.

This is less exciting than a demo, and far more likely to change how the organisation works. It is the same discipline we apply to any system: understand, define, design, build, integrate, evolve.

What a production-ready first version looks like

Consider a common case: a team that answers a steady stream of customer or supplier questions by email and WhatsApp. A pilot might show that an AI model can draft good answers. A production-ready first version goes further:

• Incoming messages from the channels the team already uses are collected in one place.
• Each message is classified, and routine ones get a drafted reply grounded in the organisation’s own policies and records.
• A person reviews and sends the draft, or edits it, so quality stays under human control.
• Anything unusual is routed to the right person with the context attached.
• The team can see what was handled, how long it took and where the drafts needed correcting.

None of these steps is exotic. Together they turn “the AI can write answers” into “the team answers faster, with fewer mistakes, and knows it”.

Questions to ask before you approve the next pilot

1. Which team’s work will be different, and who in that team owns the outcome?
2. Where will people use it: inside an existing tool, or somewhere new?
3. What will we measure, and what does today’s baseline look like?
4. What information does it depend on, and is that information in good shape?
5. What would it take to run this every day: access, monitoring, support, review?
6. If it works, what is the plan for the week after the pilot ends?

If these questions are hard to answer, that is useful information. It usually means the pilot is still an experiment looking for a problem, and a few conversations now will save months later.

The five roles every AI project needs

Most stalled pilots are missing people, not technology. Before building, make sure each of these roles has a name next to it, even if one person covers several:

• The problem owner. The manager whose team’s work will change, and who decides whether the result is good enough.
• The workflow expert. Someone who does the work today and knows the exceptions, workarounds and edge cases that never appear in process documents.
• The data owner. The person who can say where the information lives, how reliable it is and who is allowed to see it.
• The builder. The team designing and engineering the system, including the integrations.
• The operator. Whoever will keep it running once the pilot becomes normal work: monitoring, fixing, improving.

When the operator role is empty, pilots tend to end the day the project team moves on. When the problem owner is missing, nobody is in a position to say “this is working, let’s roll it out”.

How to measure a pilot honestly

Good measures compare the new way of working with the old one, on the same kind of work. Before the pilot starts, record how the work is done today for a few weeks: how long requests take, how many are handled first time, how many errors or escalations there are. Then measure the same things during the pilot.

Three rules keep the measurement honest:

1. Measure real work, not demo cases. The pilot should handle the messy, everyday requests, not a hand-picked sample.
2. Count the human effort that remains. If people spend more time checking AI output than they saved, the pilot has not worked yet.
3. Look at failure cases. Where the system got it wrong matters as much as the average. It tells you where human review must stay.

When to stop a pilot

Stopping is a valid, useful outcome. A pilot should end, or change direction, if the work turns out to be rarer than expected, if the information it depends on cannot be made reliable, or if the team will not use it even after the workflow issues are fixed. A clear “no” after six weeks is far cheaper than a vague “maybe” that drifts on for a year.