Case study
A change chatbot that got smaller, and started working.
The scope everyone wanted would have shipped something nobody trusted. Cutting it in half is what made the numbers move.
The situation
Creating a change request took about fifteen minutes and produced a form that complied with policy roughly seven times in ten. Both numbers were the symptom of the same thing: the form asked for information the person filling it in often did not have to hand, so they guessed, and the guesses failed review later.
The proposal was a conversational agent covering the full change process end to end.
What happened
I cut the scope from the full use case to a limited one before building.
A conversational agent that handles every path through a change process has to be right about all of them, and being wrong in an enterprise change queue is expensive. The limited version handled the common path properly and handed the rest to the existing form, which is less impressive to describe and considerably more likely to be used twice.
The scope cut was the unpopular decision and the one the numbers came from. A smaller thing that works beats a larger thing people route around.
The result
Change creation went from about fifteen minutes to seven, and policy compliance from 72% to 89%. The compliance number matters more than the time saved: a change that fails review costs another cycle, and the agent asked for the right things in an order the person could actually answer.
The instinct with conversational AI is to cover everything, because a bot that says "I can't help with that" feels like a failure. It is not. An agent that handles the common path well and hands off cleanly is used; one that attempts everything and is unreliable at the edges gets abandoned after the second bad answer.
Scope is the feature. Deciding what the thing will refuse to do is most of the design work.
Cutting scope is the unpopular decision. A conversational agent that covers the whole process is easy to describe, easy to fund, and easy to agree to. Proposing half of it sounds like proposing to do less work, and the argument for it only lands if you can say what the smaller version will refuse to do and why that is a feature.
The other difficulty was that the original numbers were not obviously a design problem. Fifteen minutes and seven-in-ten compliance read as a training issue or a discipline issue. They were neither: the form asked for information people did not have to hand, so they guessed, and the guesses failed review later.
Common questions
Why cut the scope of a chatbot?
Because an agent that handles every path has to be right about all of them, and being wrong in an enterprise change queue is expensive. The limited version handled the common path properly and handed the rest to the existing form.
What actually improved?
Change creation went from about fifteen minutes to seven, and policy compliance from 72% to 89%. The compliance figure matters more: a change that fails review costs another cycle.
Should a chatbot ever say it cannot help?
Yes. An agent that handles the common path well and hands off cleanly gets used. One that attempts everything and is unreliable at the edges gets abandoned after the second bad answer.
Your turn
Bring me the list nobody's willing to question.
Twenty minutes, no pitch. If I'm not the right person, I'll say so and point you somewhere better.