Deploying AI Is Getting Easier. Making It Operational Is Not.
When we introduced a project-support agent, the aim was straightforward: reduce the manual effort involved in collecting and reviewing project updates so the project manager could spend more time with stakeholders.
It did that. But it also did something more useful. The agent showed us where updates lacked detail, where the underlying delivery data was weak and where expectations about a good status update had never really been agreed. It did not eliminate follow-up. It helped the project manager see which follow-up needed a human conversation.
That experience shaped how I think about AI implementation. The test is not whether an agent can do a task. The test is whether the organisation can use it to improve the way work is managed.
Go-live proves the technology can run. Operations prove the organisation can rely on it.
The hard part is not the model. It is everything around it. OpenAI’s decision to create a dedicated Deployment Company is a strong industry signal.
OpenAI describes deployment as more than installing a model. Its specialists will work with business leaders, operators and frontline teams to pick the right opportunities, redesign workflows, connect models to enterprise data and controls, and build production systems that deliver measurable results in day-to-day work.
That confirms a simple point: access to capable models is only one part of enterprise implementation. A deployment partner can bring specialist engineering, implementation experience and tested patterns. It can help an organisation move from an idea to a production capability faster.
What it cannot do is permanently own the organisation’s data, decision rights, stakeholder behaviour, operational controls or benefits. Those responsibilities stay with the enterprise after the implementation team or external partner has moved on.
AI initiatives often cut across technology, data, security, risk, change and operations. Each function can complete its own tasks while no one owns whether the whole thing works in practice. A platform may be available, an agent may pass technical testing and training may be delivered, yet the operating model around it may still be unclear.
That is the gap some organisations still need to close. Getting AI into use is a delivery problem that cuts across a lot of teams. An agent can improve work without removing the human work.
The project-support agent was designed to augment the project manager, not replace the role.
It collected and collated updates, reviewed the available information and flagged where responses were too thin. That reduced administrative effort and gave the project manager more time for higher-value stakeholder discussions. The weak or incomplete answers were not just defects for the agent to work around, it told us something about the project.
Sometimes the information needed to be improved. Sometimes the person providing the update had a different view of what good looked like. Sometimes a risk needed a clearer mitigation or an owner needed to accept a commitment. In those cases, the answer was not another automated request. It was a direct conversation. The agent improved more than the speed of reporting. It showed:
where project data needed attention
where an update did not provide enough detail to support a decision
where expectations between the project manager and a stream lead were not aligned
and where judgement, accountability or relationship management was still required.
That is a more useful way to assess AI value than asking only how many manual tasks it removed.
A useful agent does not have to remove human work. It should direct human effort towards the judgement, accountability and relationships that matter most.
If the business case is limited to headcount reduction, leaders may miss improvements in data quality, decision-making, delivery conversations and management attention. In a project environment, giving a project manager better information and more time to act on it can be more valuable than removing another administrative step.
Good implementation builds an environment around the AI. At BNY, I saw a broader approach to encouraging and supporting AI use. The organisation built AI champions and working groups, encouraged teams to identify use cases, maintained a prioritised backlog and provided governance and training. Simpler opportunities could be developed through working groups, while more complex ideas could draw on regional AI support.
Not every idea became an agent. There was a pathway from interest to a supported use case, with people who could help assess complexity, governance and implementation needs. This helps when an organisation moves from a handful of pilots to a portfolio of AI use cases. Individual teams can experiment. Operational capability needs clearer ownership, controls, support and evidence.
Seven tests before an AI capability is operational
I would not approve an AI capability as operational simply because the build is complete or the agent can produce a good result in a controlled demonstration. For a material implementation, I would expect the organisation to pass seven tests.
It solves a defined operational problem
“Using AI” is not an outcome, and “improving productivity” is usually too broad to guide a release decision. The organisation should be able to identify the workflow being changed, the people affected and the result expected. It should be clear which activities the AI performs, which remain with people and how the two fit together. If that cannot be explained plainly, the capability is not ready to be governed.
It has reliable data and appropriate access
An agent can only work with the information it can reach and interpret. Data quality is therefore an operational dependency. Leaders need to know which sources the agent uses, who owns them and what happens when information is late, incomplete, inconsistent or insufficiently detailed.
Access must also pass through the organisation’s existing stage gates. Email, Teams, project systems and controlled data environments may each carry different approval requirements. Finding those constraints shortly before release is a planning failure, not an unexpected technical issue.
It has passed realistic quality assurance
A clean demonstration proves very little about live operation. Testing should include incomplete inputs, conflicting information, unusual cases and known failure modes. The team should understand when the agent can proceed, when a person must review the output and when the process must stop.
The standard should be proportionate to the consequence of error. An internal drafting assistant and an agent influencing a customer, financial or regulatory decision should not face the same release threshold.
It has an accountable operational owner
Every operational AI capability needs a named business owner. That person should be accountable for performance, exceptions, usage, cost, improvement and retirement. Technology may own the platform, but it should not inherit indefinite accountability for the business outcome.
The owner also needs authority. Naming someone without giving them the ability to resolve data issues, change the workflow, restrict access or suspend the capability creates responsibility on paper rather than control in practice.
People know how to work with it
Training the agent is only half the job. The people around it also need to be trained. Users and stakeholders should know what information is expected, what a good input looks like, how the output will be used and when they are required to validate or challenge it. They should also know when a direct conversation is the right next step.
Stakeholder behaviour is part of the operating model. If people provide weak information, ignore requests or assume someone else will handle the exceptions, the workflow will not perform as designed.
Usage and performance are reviewed
Initial interest is not sustained adoption. I would expect regular reviews of who is using the capability, whether it is being used as intended and where workarounds have returned. The review should also consider output quality, exception patterns, user feedback and operating cost.
These reviews are not only for assurance. They are how the organisation improves both the agent and the way people work with it.
Benefits can be demonstrated
The expected benefit should be defined before release and measured afterwards. Depending on the use case, this may include time saved, more complete information, faster decisions, better consistency, fewer missed actions, improved customer or stakeholder engagement, or lower operating cost.
The measure should reflect the real purpose of the capability. For the project-support agent, reduced administration mattered, but so did the improvement in data and the project manager’s ability to focus on the conversations that needed it.
If usage is low, cost keeps growing or the expected outcome does not appear, the choices are to improve the agent, redesign the workflow or retire the capability. Approval to build should not become permanent permission to operate.
What evidence should leaders expect?
These seven tests should not be answered through presentation slides and verbal assurance alone. At an operational release review, I would expect evidence that can be examined and challenged.
Operational test | Evidence expected |
Defined operational problem | Approved use-case statement; current and future workflow; clear scope and boundaries; baseline performance; agreed outcome measures |
Reliable data and access | Data-source register; named data owners; data-quality assessment; completed security and access approvals; process for missing or poor-quality inputs |
Realistic quality assurance | Test plan and results; representative test cases; failure and edge-case testing; acceptance criteria; known limitations; human intervention and stop rules |
Accountable operational owner | Named business owner; documented responsibilities and decision rights; support and escalation path; authority to change, restrict, suspend or retire the capability |
People know how to work with it | Role-based guidance; training records; examples of acceptable inputs and outputs; validation responsibilities; communications and adoption plan |
Usage and performance reviews | Agreed review cadence; usage measures; quality and exception reporting; user-feedback mechanism; improvement backlog; operating-cost reporting |
Demonstrable benefits | Pre-release baseline; benefit target; measurement method; named benefit owner; post-release results; decision criteria for improving, scaling or retiring the capability |
The evidence will vary with risk and scale. A small internal assistant should not carry the same governance burden as an agent involved in regulated decisions. The depth of evidence changes; the need for accountability does not.
Apply program-director discipline before go-live
The answer is not to appoint a program director to every agent. It is to apply program-director discipline to material AI implementations. That means treating operational readiness as part of the implementation from the beginning. Data, access, quality, adoption and benefits belong in the delivery plan as dependencies and outcomes, not as clean-up work after the build.
For a significant AI program, I would establish an operational-readiness workstream alongside the technical work. Its role would be to bring together the business owner, technology, data, security, risk, change and operations; agree the evidence required for release; and make unresolved ownership visible early enough to address it.
The release decision should then consider the complete capability:
Can it perform the task under realistic conditions?
Can people use and challenge it appropriately?
Can the organisation manage exceptions and failures?
Is someone accountable for what happens next?
Is there evidence that the capability is worth continuing?
If the answer to those questions is unclear, the organisation may have deployed AI, but it has not yet made it operational.
Operationalisation cannot be handed over at the end
OpenAI’s Deployment Company may help organisations close part of the gap by bringing specialist engineering and deployment experience into the enterprise.
But operational accountability cannot be outsourced.
External specialists can help redesign the workflow, connect the systems and establish the capability. The enterprise must still own the quality of its data, the decisions the agent can influence, the behaviour expected of its people, the controls applied in operation and the benefits used to justify continued investment.
The organisations that get value from AI will not necessarily be those that deploy the most agents. They will be those that know which agents they can rely on, what evidence supports that confidence and when human judgement still needs to take over.
An AI implementation is not complete when the agent is released. It is complete when the organisation can rely on it, control it, improve it and demonstrate the value it creates.
About Steven Gabris
Steven Gabris is an enterprise delivery and transformation leader with experience across financial services, payments, government and consulting. He works with organisations to strengthen delivery governance, operating models and the practical use of AI in project and program environments.
Through Gabris Group, Steven helps leaders align strategy, operations and execution, with a focus on delivery confidence, governance improvement and AI-supported ways of working.
Comments