A better result needs a fair comparison
An AI pilot can improve a workflow and still leave an important question unanswered: how much of that improvement required AI?
Consider a hypothetical growth company preparing internal briefs for new partner applications. Today, an operator gathers details from emails, checks required information, copies it into a document, and sends the brief to a reviewer. The proposed AI workflow assembles a draft and flags missing information.
During the pilot, the team also introduces a structured intake form, names one reviewer, and agrees on a standard brief. Handling gets faster. Those changes are useful, but the pilot has changed several things at once. Comparing the new system only with the old email process gives the model credit for improvements that a clearer workflow might already deliver.
A stronger comparison includes a credible simpler alternative: the same intake, ownership, and brief template, with straightforward rules and manual judgment where needed. Then the team can test the additional value of AI on top of that foundation.
For growth companies improving systems as they scale, this distinction matters. Every new capability creates an operating responsibility. The funding decision should show where that responsibility earns its place.
Give the simpler version a fair chance
The alternative should solve the same business problem at a useful level. An intentionally weak template proves very little.
For the partner brief, a practical simpler version could:
- Collect required fields through one intake form.
- Validate that the expected information is present.
- Populate a standard brief from those fields.
- Assign review using an explicit ownership rule.
- Leave interpretation of unusual requests to the reviewer.
The AI-assisted version could use those same inputs and controls while adding a summary of free-text explanations, a proposed interpretation of unusual requests, or a draft list of follow-up questions. That is a clearer test of what the model contributes.
This principle has a long engineering history. Google's Rules of Machine Learning recommends considering a product without machine learning before adding it. The guide concerns machine learning broadly; the useful planning lesson here is to establish a simpler working baseline rather than assume a model is necessary.
The comparison also needs a time limit. Give the simpler version enough effort to be credible without turning the pilot into two full product builds. Reuse the same intake and review experience where practical, and limit both paths to one clearly defined job. Scoping an AI integration around one workflow helps keep that boundary manageable.
Compare three paths to the same accepted outcome
Keep the current process in the comparison. It shows the total opportunity. The improved simpler process shows how much of that opportunity can be captured without a model. The AI-assisted process shows whether the model adds something worth operating.
For the hypothetical partner brief, all three paths end at an accepted internal brief ready for the next decision. A generated draft is an intermediate output. Review and correction still count.
Agree on acceptance criteria before looking at results. The brief must preserve supplied facts, identify missing required information, distinguish uncertainty from a conclusion, and give the reviewer enough context to proceed. None of the versions approves a partner or makes an external commitment.
Use authorized examples that reflect routine applications, incomplete inputs, unusual requests, and conflicting information. Include the cases that make the work difficult. Keep a separate set for the final comparison so repeated tuning against familiar examples does not become the entire test.
Keep the comparison conditions visible
Use equivalent inputs and the same acceptance standard across the alternatives. If one version receives cleaner information or extra operator help, record that difference. It may be a valuable improvement, but it changes what the result demonstrates.
Reviewers can also become faster simply because they have seen a case before. Rotate the order of the alternatives or use comparable case groups, and hide which version produced a brief when practical. Record the method and its limits. A small pilot can guide a delivery decision without being presented as proof of performance at every future volume.
Track the full distribution of outcomes, including rejected drafts and unresolved cases. An average over successful examples alone can hide the work most likely to need support.
Price the additional gain honestly
Suppose the current workflow takes 12 minutes of active staff time per accepted brief, the simpler version takes 7, and the AI-assisted version takes 6. These are illustrative figures, not measured Vilcorp results or a savings forecast.
At an assumed 200 briefs per month, the total difference between the current and AI-assisted versions would be 20 staff hours. The additional difference between the simpler and AI-assisted versions would be about 3.3 hours. Those answer different questions. Neither figure is automatically cash savings or available headcount.
The model may still justify its place through better handling of varied language, fewer missed details, or more consistent briefs. Evaluate those gains against the agreed quality criteria. Avoid reducing every useful outcome to minutes, and avoid claiming quality gains that the comparison did not establish.
Consider the operating costs alongside the incremental benefit:
- Model usage and any additional retrieval or infrastructure.
- Review, corrections, and exception handling.
- Evaluation when prompts, models, or source material change.
- Monitoring, support, and recovery work.
- Implementation and maintenance differences between the alternatives.
Compare ongoing costs over the same period, and show one-time implementation effort separately. A benefit projected at much higher volume should be labeled as an assumption and checked against the team's actual demand and capacity.
Anthropic's guidance on effective agents similarly recommends starting with the simplest workable solution and adding complexity when needed. That supports a disciplined comparison; it does not establish which option will win for a particular team.
Let the result narrow the next release
The result may support different choices for different kinds of work.
If the template handles routine applications well and AI helps materially with long, varied explanations, the next release might use the template by default and offer assisted drafting for that narrower group. Test the routing rule and the handoff too. A selective approach is useful only if operators can tell which path applies and recover when it does not.
If AI improves quality across the tested cases at an acceptable operating cost, expand within the conditions the pilot actually covered. Preserve the simpler path where it provides a practical fallback, and confirm that staff can still use it.
If the simpler version meets the need, ship that improvement. The pilot has produced a useful decision and may have identified a better future AI opportunity. If neither version meets the acceptance criteria, address the missing inputs, unclear rules, or workflow boundary before widening the rollout.
Keep the examples, reviewer corrections, timing method, and decision rationale. They belong in the reusable foundation a first AI pilot should leave behind, so the next proposal can build on evidence already collected.
Decide what would change your mind
Before building, write down three decisions: what the simpler alternative will do, which result would justify adding AI, and which result would lead the team to ship the simpler version. Name the person responsible for making that call.
That is practical AI strategy and readiness. It connects the operating problem, the delivery options, and the evidence needed to fund the next step.
A credible alternative makes an AI pilot more informative. It shows which improvements come from a clearer process and which depend on model capability. The team can then invest in the parts that improve accepted outcomes enough to support their cost and complexity.
If your team is preparing an AI pilot, Start a Project to define one workflow, a fair comparison, and the evidence for a useful implementation decision.
