Five Best Practices for Capturing AI Value

Artificial intelligence has the potential to create significant business value for organizations, but AI teams often find it challenging to realize and communicate these benefits. In fact, Gartner research found that difficulty measuring AI value and a lack of understanding around AI benefits and uses are the top barriers to its implementation.

AI technologies are unique in that they can learn and adapt in complex ways. These are powerful properties, but they can make it difficult to predict the performance of AI models. This challenge will only intensify with the advent of generative AI – an impressive technology, but one riddled with hard-to-predict failure modes.

AI benefits are also difficult to plan for because they require business actions, as well as process and behavioral changes, that are beyond the direct control of AI teams. It can also be challenging to attribute benefits specifically to AI model outputs. This often leads to a fundamental gap between AI model outputs and business benefits: Organizations can have the best AI models but still fail to deliver value.

Benefits do not come naturally. They must be actively managed and monitored before, during and after AI model deployment. So to derive tangible business benefits from AI projects, data and analytics leaders must implement five best practices for realizing benefits.

Build an AI value story

Before starting, AI projects need to secure funding from the enterprise. To sell the value of AI initiatives, data and analytics leaders must build a value story. Value stories essential to securing funding, driving adoption and creating momentum for AI projects to scale.

A value story is a narrative that illustrates progress toward business outcomes. These stories are told from the perspective of stakeholders’ key priorities and communicate both financial and non-financial benefits associated with these priorities. They are also useful in identifying the outcomes and key performance indicators that will define success for the AI ​​project. A value story is not a traditional business case, but rather a persuasive articulation of a project’s benefits.

Value stories can be supported by data, but they must be told in a compelling way that evokes emotion from the audience. They should start with the stakeholders’ priorities and end with the benefits to the organization.

Define a value hypothesis

AI teams must define a value hypothesis: an assumption about the improvement the AI ​​project will have on a specific business KPI. The hypothesis should flow from the value story, aimed at a concrete KPI that is well aligned with the top priorities of the organization. A hypothesis allows the AI ​​team to stay focused on business value and iterate toward a specific goal. It doesn’t have to be complicated: A simple format like “[AI use case] will increase/decrease [business KPI] by [X amount]” is enough.

The business KPI does not necessarily have to be financial. By focusing only on financial metrics, organizations can miss out on critical investments in important projects that have longer-term, but strategic, impacts. Indirect metrics that can affect customer success, cost efficiency and business growth can also have an impact.

Build an action plan

Capturing AI benefits will not happen automatically. AI teams must have a plan to go from an AI output to a set of actions and changes that will ultimately drive the business KPI. Develop a timeline for when and how AI will be applied to a specific business process and a forecast for how the outcome will be affected by AI.

It is crucial to avoid building this action plan in isolation, as business partners must be ready to take actions. The action plan should also include the training and incentive design needed to ensure that the business adopts the AI ​​models.

Test your value hypothesis

It is often challenging to isolate the effect an AI project has on the target business KPI because many factors outside of the AI ​​project also affect that KPI.

A/B testing is the standard approach to measuring a change’s impact on KPIs. In A/B testing of AI, the new AI solution is applied to a randomly selected subset of instances, while a control group is maintained to compare business KPI performance.

However, A/B testing is not always possible or economically feasible. Many organizations that have been successful with AI have used attribution models—such as first-touch/last-touch attribution or lag models—to assign credit to actions taken as a result of the AI ​​project.

Before fully deploying the AI ​​model, AI teams must choose one or more approaches to test their value hypothesis. By iterating quickly, they must either prove or disprove that the AI ​​use case resulted in the expected business benefit. This discipline is key to keeping the team focused on benefit realization. Testing should be done as early as possible, with the goal of failing quickly and iterating before too much time is spent on a wrong attempt.

Track leading and lagging KPIs

Deploying and releasing an AI model is only the beginning of AI benefit realization. To drive AI business value, it is essential that teams continuously monitor and respond to two types of metrics:

Lagging KPIs are metrics that assess past performance. The most important backlog KPI for AI projects is the business KPI defined in the value hypothesis. AI teams must continuously monitor the targeted business KPIs and analyze deviations from expected performance. Leading KPIs are metrics that can predict future performance and are useful early indicators of performance issues. In AI projects, the leading KPIs can measure different steps of the action plan needed to realize business benefits. Similarly, the AI ​​model performance can be seen as one leading indicator of the future business value to be created.

AI teams must establish a monitoring system that includes leading and lagging KPIs, detect deviations from expected performance, and respond to those deviations. Monitoring is especially important for AI projects because model inputs “drive”, system performance and business operations can cause business benefits to disappear.

Ultimately, this five-step process is iterative in nature. AI teams must build a value story, identify and test their hypothesis, and continuously adjust to improve performance both before and after implementing their AI models.

Leinar Ramos is a senior director analyst at Gartner Inc. covering AI, data and analytics and the creation of data-driven organizations. He wrote this article for SiliconANGLE. Gartner analysts will provide additional insights on driving business value through AI at Gartner IT Symposium/Xpowhich takes place October 16-19 in Orlando, Florida.

Image: Mohamed_Hassan/Pixabay