The short version
A proof of concept proves the easy path works on data someone chose. Production means the messy path, on data nobody chose, with a defined answer for every failure, access controls someone approved, monitoring that notices drift, and a named owner. Budget for the second thing as a project, not as a tidy-up.
Why does the demo always work?
Because it was built against a sample that someone assembled, and assembling a sample is an act of selection whether or not anyone intended it. The obviously broken records got excluded as noise. The awkward supplier was left out because their export was a nuisance. The data was from one good month.
None of that is dishonest — it is how you build something quickly to test an idea. But it means the demo measured the clean path, and the clean path was never the expensive part. The first honest production number always looks worse than the pilot, and teams that did not expect that read it as failure rather than as the first accurate measurement they have taken.
What is actually missing?
Five things, and each is a body of work rather than a task:
- Real data, uncurated. Including the records that are wrong, duplicated, out of date or in the wrong format.
- An answer for every failure. What happens when the system is unsure, when a system it depends on is down, when the input is something nobody anticipated.
- Permissions and access. What it may read, what it may change, what it may do without approval — approved by whoever owns those controls.
- Monitoring. Something that notices when accuracy drifts, before a customer does.
- An owner. A named person whose job includes this system continuing to work.
A useful sanity check: if the proposal for productionising is much smaller than the proposal for the proof of concept, one of these has been assumed away.
How do we get from one to the other?
Run it in parallel with the existing process, on live traffic, without changing anybody’s workload. The system produces its answer, the humans produce theirs, and you compare for a full cycle.
This is the single most valuable step available and it is nearly always skipped, because it feels like paying twice. What it buys is an accuracy figure on real data with no risk attached, a labelled set of every disagreement, and a team that has watched it work before being asked to rely on it. That last one matters more than the numbers: adoption fails on trust far more often than on capability.
How long should the parallel run be?
A complete business cycle, whatever that means for the process. A month-end for a finance workflow. A full quarter if there is meaningful seasonality. Long enough to include at least one genuinely weird week, because weird weeks are what production consists of.
Resist the pressure to shorten it after two good weeks. The failure modes that matter are rare by definition, and the point of the exercise is to meet them while a human is still checking everything.
Who should own it afterwards?
Someone inside the business, named, with time in their week for it. Not the consultant, who will leave, and not “the team”, which means nobody.
The job is small and specific: run the evaluation set periodically, watch the exception rate, notice when the model version changes, and know who to call. An hour or two a month. Systems without that person degrade silently — not dramatically, just gradually, until a partner or a manager quietly stops trusting the output and reverts to doing it by hand while the licence keeps billing.
What should we insist on before signing off?
- An accuracy figure from the parallel run, on real data, not from the pilot.
- A written exception path for every category of failure, including the system being unavailable.
- Access and permissions signed off by whoever owns those controls.
- Monitoring in place and demonstrated — ask to see an alert fire.
- A named owner who has agreed, and knows what the job involves.
- Handover of the prompts, configuration and evaluation set, in your repository.
That last item is the one people concede in the interest of momentum and regret within a year. The evaluation set in particular is the asset that proves the system works; without it, the next change cannot be tested by anyone but the person who built it.
Is it ever right to stop?
Yes, and it is worth saying out loud because the sunk cost of a successful-looking pilot is substantial. If the parallel run shows the exception rate is far higher than expected, or the integration cannot be made reliable, or the saved time is genuinely smaller than the review time created — stopping is the correct decision and the pilot did its job.
A proof of concept that produces a clear “no” has saved you the production cost, which is usually several times larger. The failure is not stopping; it is continuing because stopping would be awkward to explain.
How do we handle the accuracy drop?
Expect it, name it in advance, and explain to whoever approved the pilot that the production figure is the first honest measurement rather than a regression. A pilot at ninety-four percent landing at eighty-one on real data has not got worse — it was always eighty-one, and the sample was flattering.
Then work out where the loss is. Almost always it concentrates in a few identifiable categories rather than spreading evenly, and those categories are either fixable with rules or excludable from automation entirely. A system that handles eighty percent of cases at ninety-five percent accuracy and routes the rest to a person is a good deployment. One that handles everything at eighty-one is not.
Should the same person do both phases?
Usually yes for continuity, but with a deliberate check: the person who built the proof of concept has an interest in it succeeding, and is the least likely to tell you it should stop.
The cheap safeguard is to agree the go or no-go criteria before the parallel run begins, in writing, with the numbers filled in. Then the decision is arithmetic rather than judgement, and nobody has to argue against their own work in a meeting.