The easiest way to make agentic engineering look successful is to run demos on isolated tasks. The easiest way to make it fail in an organization is to let those demos blur into production change without an operating model.
A good pilot is not the flashiest task. It is the task that can teach the organization something true.
Pilot selection criteria
The first pilot should be boring enough to be reversible and real enough to matter.
Good candidates usually have a clear owner, a narrow diff surface, deterministic validation, and a reviewer who understands the code. Documentation migrations, test coverage gaps, schema validation, repository hygiene, non-production tooling, and well-scoped frontend or backend fixes are often stronger first pilots than production automation.
Weak candidates have ambiguous ownership, high blast radius, hidden runtime coupling, missing tests, or political ambiguity. If nobody knows who owns the system, an AI agent will not discover the governance model by intuition.
A useful pilot scorecard
Score candidate work on five axes:
- Context clarity: Can the issue explain the task without tribal knowledge?
- Verification strength: Can the result be checked by tests, builds, contracts, or preview environments?
- Blast radius: Can the change fail safely?
- Reviewability: Can a human understand the diff in one sitting?
- Rollback quality: Is there a real reversal path, not just hope?
If a task scores poorly on more than one axis, it is probably not a phase-one pilot. It may still be valuable later, but using it too early will confuse AI maturity with organizational risk appetite.
Rollback is part of the pilot
Rollback is not a failure condition. It is part of the experiment design.
Every pilot PR should answer:
- What would make us revert this?
- How would we revert it?
- What evidence would we inspect first?
- Who owns the decision?
- Which follow-up issue records the learning?
For code-only changes, reverting the PR may be enough. For infrastructure, data, permissions, or runtime behavior, the rollback path must be more explicit. Terraform plans, database migrations, feature flags, compatibility windows, and deploy sequencing are all part of the answer.
Stop conditions
A pilot program needs stop conditions before it starts. Otherwise every failure becomes a cultural argument.
Useful stop conditions include repeated unreviewable diffs, recurring unsupported claims in PR descriptions, agents modifying forbidden areas, validation bypasses, reviewer overload, hidden manual cleanup, or degradation in incident response. These signals do not prove that AI engineering is useless. They prove that the current operating model is not ready for a higher delegation level.
What the pilot should produce
The output is not just merged code. The output is an evidence record:
- what task classes were safe;
- which instructions were insufficient;
- which tests caught real issues;
- where human review remained the bottleneck;
- which guardrails should become deterministic;
- which work should remain human-owned.
The pilot succeeds when the team can say, with evidence, what it is now safe to delegate next.