How to tell an AI project from a project with AI written on it

A large regulated enterprise asked me to look at its AI pipeline. Proposals were arriving from business units, from vendors, and from internal teams who had all been told to have an AI plan by the end of the quarter.

Nobody could compare them. Each one was written in its own language, sized by its own sponsor, and judged mostly on who was asking. Some were substantial pieces of work. Some were an existing system with a new label on the front.

I built a rubric and ran forty of them through it. Roughly one in ten turned out not to be AI at all.

Not fraudulent. Not stupid. Just a rules engine, a database query or a lookup table that had been described in the language of the moment — because that was the language budget was being approved in.

Each one was a project that didn't go on to consume a quarter of engineering time arriving somewhere the organisation already was.

Why this happens to competent people

It is tempting to read the one-in-ten as carelessness. It isn't.

The incentives are doing exactly what you would expect. A team with a genuinely useful automation idea learns quickly that calling it automation gets a slower answer than calling it AI. The vendor in the room has a product to sell and a vocabulary that flatters it. The sponsor needs something to show at the next steering meeting.

And critically: nobody in the room is incentivised to be the person who says “this one isn't AI.” That sentence costs the speaker something and saves the organisation money it will never attribute to them.

So the label spreads, and it stops carrying information. Once everything is AI, the word can no longer help you decide anything — which is precisely when you need it to.

The question that does most of the work

Before any scoring, before feasibility, before data, one question:

Would a rule, a query or a lookup table do this better?

Better, not merely adequately. A rules engine that solves the problem is not a consolation prize. It is cheaper to build, cheaper to run, easier to test, and it does not hallucinate. It also does not need a model card, a bias assessment or a conversation with your works council.

If the honest answer is that a rule would do this better, you have found a recommendation worth paying for. That is a good outcome, not a failed assessment.

Three patterns account for most of what this question catches:

• The relabelled system. Something already in production, proposed again with a new name and a new budget line. Usually detectable by asking what it does that the current version doesn't.
• The deterministic problem. Fixed inputs, fixed outputs, a knowable set of rules. If you can write the logic down, you don't need a model to infer it.
• The search box. A retrieval problem described as understanding. People need to find a document. They do not need the document interpreted.

Write down the reasoning, not just the score

This is the part most assessment frameworks skip, and it is the part that decides whether the framework survives contact with the organisation.

A score with the reasoning written beside it can be argued with. Someone can read it, disagree with a specific line, and either change your mind or fail to. That is a conversation.

A score without reasoning is an opinion with a number attached. It survives exactly as long as the person holding it has more authority than the person challenging it.

In a regulated environment that difference is not academic. My scores had to hold up in front of security review, legal, and a works council — three groups whose job is to find the soft parts of an argument. None of them were interested in a number. All of them were interested in why.

Score the adoption, not just the idea

A technically excellent feature with an adoption problem is worth less than a modest one that lands in a habit people already have.

I have seen a genuinely good AI classifier go largely unused because reaching it meant leaving the tool its users already lived in. The model was not the problem. The distance was. Moving the same capability into the workflow people were already in changed everything about whether it got used.

So the question is not only does this work. It is who will actually use this, and where will they be standing when they do.

What to do on Monday

If you have a list of AI ideas and no way to compare them, you do not need a framework as elaborate as the one I built. You need three things:

1. The same questions, asked in the same order, of every proposal — including the ones from people you like.
2. The reasoning written down beside each answer, in language someone outside the project can follow.
3. A named person who is allowed to say no, and who is not measured on how many AI projects get started.

That third one is usually the missing piece, and it is not a tooling problem.

The organisations that get this right are not the ones with the best models. They are the ones where somebody was allowed to say “this one is a rules engine” in week one, and was thanked for it.

Common questions