Insights · · 3 min read
What a 30-day AI pilot should actually deliver
The most expensive outcome of an AI pilot is not failure. It is ambiguity: thirty days spent, a slide deck produced, and nobody able to say whether the thing worked. A pilot should be designed so that a clear answer is the only possible output.
What is the pilot actually testing?
A pilot is not testing whether language models are impressive. That question is settled. It is testing three specific things about your business: whether your data can support the workflow, whether the workflow moves a number you care about, and whether your team will use it. Write those three down before anything is built, because every deliverable below exists to answer one of them.
Deliverable 1: one workflow running on live systems
Not a sandbox, not a synthetic dataset, not a copy of last quarter's export. One workflow, connected to the real CRM or ERP or warehouse, used by the people who own the problem. Scope it narrow: exceptions on one order type, reconciliation of one metric, answers over one document set. Narrow and real beats broad and fake in every dimension that matters.
Deliverable 2: a measured baseline and a measured result
Before the pilot touches anything, measure the current state: hours per week on the report, days to catch an exception, tickets resolved per person. Thirty days later, measure again. Without the baseline there is no result, only anecvidence. The metric should be agreed in week zero and instrumented into the workflow itself, so the readout is produced by the system, not assembled by the vendor.
Deliverable 3: the governance skeleton
A pilot that skips permissions, citations, and approval gates does not de-risk anything, because the thing you would eventually ship looks nothing like the thing you tested. The pilot must include the boring parts:
- Access scoped to what each user may see in the source systems
- Citations on every answer and evidence attached to every action
- A named human approver for anything that writes to a system of record
- Logs you could show an auditor, even if no auditor ever asks
Deliverable 4: an evaluation set your team owns
During the pilot, every question the system got wrong becomes a test case. By day thirty you should own a small evaluation set: real inputs, expected outputs, and a score that reruns on every change. This is the asset that makes the system improvable after the consultants leave, and it is the first thing to ask for when a pilot ends.
Deliverable 5: a go or no-go you can defend
The final readout is one page: the metric before, the metric after, what it cost, what it would cost to run, and the risks found. Sometimes the honest answer is no-go: the data is not ready, or the workflow saves less than it costs. A pilot that can say no cheaply has done its job. A pilot that can only say yes was a sales process.
What a pilot should not include
- A custom user interface polished before the workflow is proven
- Five use cases at once, each too shallow to measure
- A model fine-tuned on your data before retrieval and definitions are fixed
- Any claim that cannot be traced to the instrumented metric
Frequently asked questions
- How much should a 30-day AI pilot cost?
- Enough that both sides take it seriously. Free proofs of concept optimize for a demo; a paid, fixed-scope pilot optimizes for a decision. The price should buy a working workflow on live data, a measured result, and an evaluation set you keep.
- Should a pilot use our production data?
- Yes, with production-grade controls: scoped access, citations, approval gates, and logging from day one. Piloting on synthetic data tests nothing except the demo, because the risks live in the real data.
- What is a reasonable success metric for a first pilot?
- One operational number the team already tracks or wishes it could: hours per week on a report, time to detect an exception, resolution time per ticket. Pick one, baseline it in week zero, and let the system itself produce the after measurement.