Nobody Agreed What "Working" Meant Before It Shipped
The most common reason an AI rollout cannot be evaluated is that no baseline was ever recorded. Here is how to reconstruct one after the fact, and what to do differently next time.
Ask a room what a new system was supposed to achieve and you will often get answers that sound compatible but are not. Operations wanted fewer manual hours. Finance wanted lower cost per transaction. The executive who approved it wanted to be able to say the company was using AI. Nobody wrote any of it down, and none of those three are measured the same way.
This is the quietest failure mode in the whole category, because it does not look like a failure. It looks like an ongoing disagreement.
The Problem With Measuring After the Fact
The honest difficulty is that the useful comparison — what would have happened without this system — is not observable. You only ran the world one way.
That does not make the question unanswerable. It makes it a question that requires reconstruction rather than a lookup, and reconstruction is possible more often than people assume.
Reconstructing a Baseline
Three approaches, roughly in order of how much confidence they give you.
Historical comparison. Pull the same metric for the equivalent period before the system went in. This is the easiest and the weakest, because everything else also changed — headcount, seasonality, pricing, the market. It is worth doing as a starting point and worth distrusting as a conclusion.
Segment comparison. Find part of the business the system does not touch. A region, a product line, a customer tier, a shift. If the untouched segment moved the same way over the same period, the system probably is not what moved it. This is considerably stronger than a before-and-after, and most companies have a usable segment sitting there without realizing it.
Held-out comparison. Deliberately withhold the system from a slice going forward and compare. This is the only version that gives you a genuinely clean answer, and it is the one people resist most, because it means intentionally not using something you paid for. It is usually worth two or three months of that discomfort to know for certain.
The Cost Side Is Easier and Gets Skipped
Whatever the benefit turns out to be, the cost is knowable now, and it is routinely underestimated because only the license fee gets counted.
The full number includes:
- The subscription or license
- Infrastructure it runs on
- The internal hours spent maintaining it, including the person who quietly fixes the export every week
- The hours spent reviewing or correcting its output
- Whatever integration work is still outstanding
That total is often several times the headline price. It is also the number that makes the decision easy in the cases where the benefit turns out to be small — you do not need a precise benefit estimate to conclude that a marginal improvement is not worth a large fully-loaded cost.
What to Do Before the Next One
The fix is unglamorous and takes about an hour.
Before anything is bought, write down one sentence: this system is meant to change [specific number] from [current value] to [target value] by [date], measured by [method].
If nobody can fill in the current value, stop and measure it first. That measurement is the entire difference between a system you can evaluate and a system you will argue about for two years.
If nobody can agree on the target value, that disagreement is real, and it is better to have it before the money is spent than after.
And if the honest answer is that the number cannot be measured at all, that is worth knowing too. It might still be worth buying. But you are buying it on judgment, and everyone should understand that going in rather than discovering it later when someone asks for the return figures.
The Retroactive Version
If you are already past that point — the thing is bought, running, and unevaluated — the recovery is the same sentence written backwards. Pick the number it was most plausibly meant to move. Find the best baseline available from the three methods above. Calculate the fully-loaded cost. Compare.
The answer will be less precise than if you had measured properly from the start. It will still usually be clear enough to decide on, because the cases that are genuinely too close to call are rarer than the cases where nobody has done the arithmetic at all.