Pilot Kill Criteria: The Rule Only 31% of CIOs Have, and a Template to Write Yours

Most AI pilots don't fail loudly. They just never end. Every review says "promising", every month gets another extension, and the bill keeps running.

What Gartner found

Gartner surveyed 2,002 CIOs and technology executives between May and July 2026. Only 13% said their enterprise has realized significant or transformational value from AI so far.

The more useful finding is about rules, not models:

• 31% have guidelines for when a pilot should not move to production. They are 2.3× more likely to be "vanguards", Gartner's name for CIOs with better business outcomes and high CEO-perceived AI value.
• 17% have guidelines for abandoning low-value AI that is already in production. They are 2.6× more likely to be vanguards.
• 4% have explicit plans to redesign roles for a human-agent workforce (2.4×).

Gartner VP Analyst Alicia Mullery summed it up in three words: "Volume is not value." And: "CIOs need business partners who are willing and ready to walk away from low value investments."

One caveat: this is self-reported and correlational. Having a stop rule doesn't guarantee value. But it is hard to see how you get value without one.

Why pilots without a stop rule never end

Without a written rule, every review drifts the same way. The demo looks good, so the pilot gets another month. Scope grows to one more use case. And because nobody measured the process before the agent arrived, there is no baseline to judge it against, so there is never a verdict.

Our view: a pilot that can't fail can't succeed either. The kill rule is what turns a demo into a decision.

The template: seven lines before day 1

Before any agent touches real work, write these down and get them signed:

• Pilot: one workflow, one agent. Example: invoice-matching agent.
• Metric: the unit finance already tracks. Example: cost per matched invoice.
• Baseline: measured for four weeks before the agent. Example: $4.10.
• Target and date: Example: $3.00 or less by day 60.
• Cost cap: model and tool spend per task. Example: $0.40 per invoice.
• Stop if: Example: target missed at day 60, or the cost cap is broken two weeks running.
• Owner: the business owner who signs the kill. Example: the accounts-payable lead.

(The numbers are illustrative. Yours come from your own baseline.)

The day-60 review: four possible outcomes

1. Was the baseline measured before the pilot? If not: reset. Measure for four weeks, then restart. Without a "before" there is no verdict.
2. Did the target metric move against the baseline? If yes, check the cost.
3. Is the cost per task under the cap? If yes: ship it to production.
4. If the target or the cap was missed: is the cause known and fixable in 30 days? If yes: fix, with one 30-day extension, once. If not: kill it and write up what you learned.

Then keep the rule running after launch. Re-run the same review every quarter in production. Only 17% of CIOs have a rule for dropping low-value AI that is already live.

How we approach it at Zylen

Zylen is an AI optimization partner. We work inside the AI systems companies already run and improve cost, security and data control one measured change at a time. In a 30-minute agent audit, we write the kill criteria for one of your pilots with you: the baseline metric and target, the cost cap per task, and the stop rule and owner.

Decide how your pilot ends before it starts: book the 30-minute agent audit →

Source: Gartner, "Gartner Survey Finds Only 13% of CIOs Say Their Enterprise Has Realized Significant or Transformational Value from AI", October 19, 2026 (survey of 2,002 CIOs and technology executives, May–July 2026).