A sponsor agrees a pilot's success criteria, comparison group, and review date with the skeptics before it starts, so a good result settles the argument instead of opening a new one.
An example
For a year, a regional property insurer had argued about a model that sorted incoming claims by complexity. The claims operations director who wanted it knew the chief claims officer's deputy would dismiss any pilot she ran on her own terms. So before a single claim went through it, she sat down with him and wrote the terms together on one page: two adjuster teams in the same region, one using the model and one working as usual, ninety days, measured on cycle time, reopened claims, and customer complaints, with his staff pulling the numbers. He added a fourth measure, payment leakage, and she accepted it. At day ninety the model team was closing claims eleven days faster, reopenings were flat, and leakage was slightly better. He presented the result himself at the operating committee and asked for the rollout to start in his own region.
Why it works
The pilot makes the proposal look better because its evidence comes from a test the skeptics helped design, so they cannot discount it as the sponsor's own. It makes agreeing easier because the skeptics are judging a result against their own criteria, and changing position means keeping their word. It works when the measures are agreed in writing before any data exists, the comparison is fair, someone the skeptics trust runs the numbers, and the sponsor is prepared to lose.
The thinking behind it
Everett M. Rogers Diffusion of Innovations (1962), by Everett Rogers, names in later editions the attributes that speed adoption, including trialability, the chance to try an innovation on a limited basis, and observability, results that others can see. A pilot supplies both. Agreeing its terms with the skeptics adds what Rogers's attributes assume: that the people watching accept the trial as a fair look at the innovation.
Chip Heath and Dan Heath Chip and Dan Heath's Decisive: How to Make Better Choices in Life and Work (2013) recommends ooching, running small experiments before committing, and tripwires, signals set in advance that prompt a decision to be reconsidered. A fair-test pilot is an ooch with its tripwires written down before any data exists: the measures, the comparison group, and the review date tell everyone in advance what result will settle the question.
When it fits
Good conditions: a contested change that can be tried at small scale within a few months, and skeptics who are credible people whose agreement you need. You need a willing host unit, enough authority to run a trial, a comparison group, and the nerve to publish a bad result. Where the change cannot be tried in part, as with a merger, the move does not apply.
How to use it well
Write the terms with the skeptics before the pilot starts, and make them specific enough that nobody can argue later about what was promised.
- Agree the measures, the comparison group, the duration, and the result that counts as success or failure.
- Invite the skeptics to add a measure, and accept it unless it cannot be measured.
- Let someone they trust collect or check the numbers.
- Set the review date and the forum now, and keep them.
During the pilot, resist changing the rules when something goes wrong; log the problem and report it. At the review, present the result against the written terms first, whichever way it went.
What it costs and where it backfires
You can lose, in public, on terms you agreed to, and then you have to honor them. A fair test takes longer than a friendly one and costs a comparison group's worth of effort. Skeptics who add measures raise the bar, and you will be tempted to resist. Keep to the terms. Choosing an easy host after the fact, moving the goalposts, or presenting good conditions as typical turns the pilot into a showcase, and the skeptics who helped design it will say so.
When it is used on you
Accept an invitation to set pilot terms, and use it well: insist on a real comparison group, a measure that would catch the harm you fear, and data you can check. If a sponsor runs a pilot without asking you, ask before it ends what result would count as failure. An answer that arrives only after the data is in tells you how to read the result.