Measurement
How to tell if an AI pilot is working
Decide what success and failure look like before the pilot starts, and measure against the initiative, not the demo.
Pilots rarely fail loudly. They drift. Usage tapers off, the champion moves on, and the renewal shows up before anyone has decided whether it worked. The way to avoid that is to make the decision rules before you start.
Before: write down three numbers
- Baseline: where the outcome metric stands today, measured the same way you will measure it later.
- Target: the number that would justify scaling, tied to the initiative.
- Kill line: the result at which you stop, no matter how good the demo felt.
During: measure the work, not the tool
Track time per task, rework and error rates, and the share of eligible work actually going through the new path. Count the exceptions people handle by hand. Logins, prompts sent and satisfaction surveys are useful context, but they do not tell you whether the initiative moved.
After: make one of three calls
- 01
Scale
You hit the target and an owner is ready to run it. Budget it and roll it out.
- 02
Fix
You are close, and you know why you missed. Change one thing and run a short second round.
- 03
Stop
You crossed the kill line. Write down what you learned and move the effort to the next item on your map.
Stopping is a good outcome. It frees budget and attention for the work that will move the initiative, and it builds trust that AI spending is being managed like any other investment.