Before the pilot starts, write down one main measure, today’s number, a target, a few limits, and what you will do if the target is met, missed, or a limit is crossed. Get your sponsor to agree to it in writing.
Why agree on the numbers first
A pilot can end with a mixed result and a long debate. It felt faster. The quality seemed fine. A few people loved it, and a few didn’t try it. Without numbers agreed in advance, the loudest opinion wins. A promising pilot gets dropped, or a weak one keeps going because nobody wants to say so.
Agreed numbers turn that debate into a check. The decision at the end becomes “did we meet what we said we would?”
How to set them, step by step
1. Pick one task
A pilot should test the AI on one task or one workflow, such as drafting support replies or summarizing meeting notes. If you test five things at once, you can’t tell which one made the difference.
2. Choose one main measure
Pick something you already measure, that the AI is meant to change, and that the people paying for the pilot care about. Common choices:
- Time per task. How long one piece of work takes.
- Output per person. How many tasks someone finishes in a week.
- Error or rework rate. How often work has to be fixed or redone.
- Cost per task. The full cost, including the AI tool and the time to review its output.
- Satisfaction. A short score from the people who use the result, such as customers or colleagues.
3. Measure the baseline first
The baseline is today’s figure, measured before anyone uses the AI and in the same way you will measure the pilot. Without it you can’t say whether the pilot helped, and it is easy to remember the old way of working as worse than it was.
4. Set the target
Say how much better the number has to get for the pilot to be worth scaling. If you’re not sure what is enough, use the AI ROI calculator to find the improvement needed to pay back the cost. Then set the target at or above that.
5. Add a few limits
Limits stop a gain in one place from hiding a loss in another. For example, quality must not drop below today’s level, and there must be no data incidents. If a limit is crossed, the pilot pauses, whatever the main number says.
6. Track who uses it
Record the share of pilot users who use the tool each week. If few people use it, a good result from those few says little about everyone else. Low use often points to a workflow or training problem, not a problem with the AI.
7. Write the scale, fix, or stop rule
Decide now, and write it down:
- Scale if the target is met, no limit was crossed, and people are using it.
- Fix and rerun if the result is better than the baseline but short of the target, or if use was low.
- Stop if the result is well short of the target at the end, or if a safety limit is crossed.
Set the length of the pilot and the date of the decision at the same time. The AI pilot success metrics builder turns your baseline and target into a sheet with the measures, limits, and rule that you can copy and share.
Before you start
The numbers are only part of a good start. Check that approvals, data rules, and the people involved are in place with the AI launch readiness checklist. To plan who to involve, and in what order, use the AI rollout plan generator.
Mistakes to avoid
- Skipping the baseline. It is the one thing you can’t go back and measure after the pilot starts.
- Tracking too many measures. With ten numbers, some will look good and some bad, and you can argue for any result. Keep one main measure.
- Ending too soon. A very short pilot can end up measuring curiosity. Let the novelty wear off.
- Picking only enthusiastic users. Include people who aren’t excited about AI, so the result isn’t skewed by the eager few.
- Writing the rule after you see the results. It is hard to be fair once you know what you hope to hear.
Common questions
- What is a good success metric for an AI pilot?
- One you already measure, that the AI is meant to change, and that matters to the people paying for it. Common choices are time per task, tasks finished per person, error or rework rate, cost per task, and a satisfaction score. Pick one main measure and a few limits.
- How long should an AI pilot run?
- Long enough for the measure to settle after the novelty wears off. For a single task that is often four to eight weeks. Set the end date and the decision date before you start.
- What if the pilot shows a small gain?
- That is what the target is for. If the gain is real but short of the target, the rule should say what to do: fix what is holding it back and run it again, or stop. A small gain can still be worth scaling if the cost is low, so check it against your ROI estimate.
I’m Courtney Stephenson, an AI strategy and implementation leader. If you’re planning a pilot and want help deciding how to measure it, get in touch.