How to plan the first 90 days of an AI pilot

The best AI pilot is a narrow business decision with clear boundaries, usable data, and a person accountable for the output. Start with the workflow, then decide whether AI is the right tool.

Days 1–30: select and assess

Collect candidate tasks and rank them by frequency, impact, data sensitivity, and ease of checking the answer. A good first use case may be summarizing internal documents for a reviewer; an automated decision about a person's eligibility requires much stronger controls. Map the existing workflow, source systems, baseline effort, and failure cost. Assign a product owner and a subject matter reviewer.

Days 31–60: build a controlled pilot

Prepare a small evaluation set with ordinary, ambiguous, and adversarial examples. Decide what information the system may access and retain. Define what the model can draft, what a person must approve, and how users report a wrong answer. Keep prompts, retrieval sources, model versions, and evaluation results traceable.

Days 61–90: test the real workflow

Run the pilot with actual users in a limited environment. Measure task completion, correction effort, unsafe or unsupported claims, and operating cost. Review edge cases with the people affected by them. Set a threshold for expansion and a rollback path before broad release.

Governance is ongoing

The NIST AI Risk Management Framework organizes risk work around govern, map, measure, and manage. Use that structure to revisit a system as its data, model, users, and operating context change. A pilot succeeds when it helps people perform a defined job reliably, not merely when a demo looks impressive.

Choose a pilot that can be checked

Compare candidate tasks using four questions: Is the source information available and permitted for use? Can a qualified person tell whether an answer is correct? Is the cost of a wrong answer manageable? Would the workflow still be useful if a person reviewed every output? An internal document assistant is easier to evaluate than a system that makes final eligibility decisions.

Build an evaluation set

Gather examples from the real task, remove unnecessary personal information, and include incomplete requests, conflicting documents, outdated content, and questions outside scope. Ask reviewers to score accuracy, source support, completeness, and whether the assistant should have declined. Keep the set stable enough to compare model or prompt changes, then add examples as new failure modes appear.

Decide whether to continue

At day 90, compare the pilot with the current process on quality, review time, failure severity, cost, and user confidence. Document where it failed and who owns mitigation. Expand only when the workflow has a clear operator, monitoring, access controls, and a way to stop or roll back behavior that changes unexpectedly.

Compare possible approaches before building

A better search index, a rules engine, or a conventional workflow change may solve the problem with less uncertainty. Compare each approach using the same task set: quality of output, user effort, cost to operate, privacy implications, and ease of correction. AI is a fit when it adds useful flexibility that the simpler approach cannot provide. It is a poor fit when an error cannot be detected before harm or when the organization cannot maintain the evaluation process.

Example: internal knowledge assistance

A service agent might ask a system to find the current refund rule. A useful pilot must retrieve the approved policy, cite the correct section, distinguish regional variations, and refuse to invent an answer when the source is missing. The agent should see the source and make the final decision. Evaluation cases should include an outdated policy, a conflicting exception, and a customer situation outside the published rules. Measure how much review work remains; fast drafts that require extensive correction are not a win.

Plan cost and security from the start

Estimate usage from real workflow volume, not demo traffic. Include model calls, retrieval, storage, monitoring, evaluation, and human review. Check provider terms for data retention and training use, restrict access to the source material, and avoid sending personal data that is not needed for the task. Record prompt and model changes so a quality drop can be traced.

An exit memo for day 90

Summarize the original problem, pilot scope, baseline, test method, quality results, operating cost, observed risks, and user feedback. State one of three decisions: stop, revise and retest, or expand under defined controls. Give each unresolved risk an owner and a date for review. A disciplined stop can be as valuable as a successful expansion because it prevents an unreliable workflow from becoming production infrastructure.