A six-week pilot often ends in a meeting where everyone remembers something different. One person found it "quite handy". Another got little out of it. The team is enthusiastic, management wants figures, and those figures don't exist, because nobody measured beforehand how long the work took without the system. The pilot runs on for another quarter, because stopping isn't possible and neither is deciding.
That's rarely down to the tool. It's down to what wasn't agreed before the start.
The cause: the yardstick arrives afterwards
In a pilot with no yardstick fixed in advance, you choose the yardstick afterwards. And afterwards you automatically choose whatever comes out well. If the system is faster, speed counts. If it's slower but more careful, care counts. Nobody is being dishonest about it. You just have nothing to measure against.
Four things are missing, and they nearly always go missing together: a question the pilot answers, a baseline for how the work goes now, a threshold above which you carry on, and someone who makes the decision. Without those four, a pilot is a decision postponed.
The pilot card
Before the start, put six fields on one page. Fill them in before anyone opens the system, and have whoever decides read them.
1. The question. One sentence that can be answered yes or no. Not "what can AI do for us", but "can an assistant write the first draft of the proposal, so the expert review gets shorter?"
2. The baseline. How the work goes now, expressed in something countable: hours per proposal, number of correction rounds, number of errors found in review. Measure this before you begin, on at least ten real cases. Without this figure, every result afterwards is an impression.
3. The measure. What do you measure during the pilot, and how? The same unit as the baseline, otherwise you can't put them side by side. Also pick one quality measure that isn't about speed, because a system can only win on pace if the work gets worse.
4. The threshold. The result above which you carry on, and the result below which you stop. Write them as numbers, with the middle ground included: below twenty percent less time and no improvement in quality, we stop; above forty percent less time at equal quality, we continue; in between, we adjust. Those numbers are your own choice. What matters is that they're written down before the results exist.
5. The end date. A day on the calendar, not "about six weeks". On that day the pilot is over, whether or not anyone is satisfied. It can be extended only with a new pilot card.
6. The decision. Who decides, with which three outcomes (continue, adjust, stop), and on what day. Attach that decision to the meeting where the result is discussed, not to a promise to "have another look".
A made-up example of a completed card. Question: can the assistant write the first draft of a proposal? Baseline: eleven proposals from last quarter, four hours each on average, two correction rounds on average. Measure: the same two figures on ten new proposals, plus the number of claims found in review to have no source. Threshold: less than a quarter of an hour saved per proposal, or more claims without a source than now, and we stop. End date: 14 November. Decision: the director, the Monday after.
What to ask of the tool itself
A pilot only yields usable figures if the system keeps the trail as a by-product. If you have to reconstruct what happened after six weeks, you'll reconstruct it wrongly or not at all. In TRACES, that's the Trail layer: what is recorded, and does it arise by itself while someone works?
TRACES are the six layers on which you assess an AI system: Trail, Retrieval, Approval, Constraints, Expertise, Storage. Where a layer is missing, there is a risk.
Before the pilot, ask whether the system records, per task, what was asked, what the answer was and who approved it, and whether you can export that. If it can't, agree how you'll keep it yourselves, for instance a simple table where each participant fills in one line per task. Manual is fine, as long as it's agreed beforehand.
If you haven't chosen the system yet, you can start with the vendor scan on this site. It runs the public documentation past the same six layers.
What you can do this week
- Pick the pilot that's running or about to start. If it has no card yet, now is the moment.
- Fill in the six fields on one page. Start with the question and the baseline.
- Measure the baseline on at least ten real cases, before anyone uses the system.
- Have whoever decides read the page and agree to it, before the start. Put the end date and the decision date in their diary.
A pilot set up like this can still end in stop. Then it's a decision, and that's the point.