Measuring the business impact of AI

Artificial intelligence is in transition, both as a technology and in how it’s being used. Companies are increasingly bringing AI pilots out of the test labs and deploying them at scale, and some are seeing significant benefits as a result. Regardless of any uncertainty surrounding AI, ignoring its potential poses the risk that companies doing business the old way will go under.

For many organizations, however, deriving value from AI may be elusive. Their models might not be tuned. Their training data sets might not be big enough. Customers may be leery. There are also concerns about bias, ethics, and transparency. Pushing an AI initiative into production before it’s ready, or expanding an AI strategy beyond an initial phase before properly vetting its results can cost a company money, or worse, send it in a direction detrimental to the business.

So how do you know whether an AI project will transform or sabotage your company? Without hard ROI numbers, companies have to get creative with ways to know for certain. Here’s a look at how IT leaders and industry insiders gauge value of AI.

Mature vs. groundbreaking technologies

Measuring the business value of any initiative or technology isn’t always a linear calculation. AI is certainly no exception, especially when degrees of maturity and business potential are taken into consideration. Proven and predictive variables — like data mining, cost and training savings, investment and the ability to facilitate new uses — influence decisions when it comes to acceptable ROI, but putting a degree of trust in the technology, no matter how new or established, is essential.

At NASA’s Jet Propulsion Laboratory, for instance, the key factor to measure an AI project’s ROI is technology maturity.

Some AI use cases are at a high level of maturity, says Chris Mattmann, chief technology and innovation officer at NASA JPL. Take for example automating business processes.

“The boring stuff that every company has, we have too,” he says. “So we automate a lot of things like ticket processing, search, data mining, looking at contracts and subcontracts using AI.”

JPL uses commercially available technologies to do this, including DataRobot and Google Cloud. To determine whether a particular technology is worth investing in, the organization looks at whether it will save costs, time, and resources, Mattmann says. “It’s mature, so you should be able to show this.”

For technologies at a medium level of maturity, JPL looks at whether the technology has the ability to enable new use cases, and at what cost. “For example, we’re going to Mars, and we have a thin pipe for deep space telecom,” he says, and today, there’s enough bandwidth to send about 200 pictures a day from Mars to Earth.

“Those brilliant Mars rovers we send have pea-sized brains in them,”  he says. “They’re running iPhone 1 processors. We only put things in space that are radiation-hardened, where we’re confident they can withstand the deep space environment. The chips that we know perform well are those older chips so we don’t do advanced AI or ML on the rovers.”

But the Ingenuity helicopter, which was originally intended simply as a technology demonstration and wasn’t core to the mission, had a Qualcomm Snapdragon processor on board, an AI chip. “That demonstrated to us that it was possible to have newer chips and do more AI,” he says.

Here, the AI will enable new use cases not currently possible. For example, instead of sending back 200 images a day, the rover could analyze the images itself using AI and send a million text captions back to earth to describe, for example, that there was a dry lakebed in a particular direction. “We could get more visibility with text than we do with images today,” Mattmann says.

Finally, for the most cutting-edge, experimental AI technologies, the measure of success is whether they allow for new science to be done, and new papers to be written and published.

“There’s a cost to training and building models,” he says.

Companies like Google and Microsoft have ready access to giant volumes of training data, but at JPL, the data sets are hard to acquire and require PhD-level experts to analyze and label.

“At NASA, our costs to train a new AI model are 10 to 20 times that of commercial industry,” says Mattmann.

Here, new technologies are coming along that could allow NASA to create AI models with less manual labeling. For example, generative networks could be used to create synthetic training data, he says. Deep fakes, but for the benefit of science.

AI measurement and its spheres of influence

When there’s no direct way to measure the business impact of an AI project, companies will mine data from related key performance indicators (KPIs) instead. These proxy variables typically relate to business goals and can include customer satisfaction, time to market, or employee retention rates.

Case in point is Atlantic Health System, where patients are at the heart of every decision, says Sunil Dadlani, its senior vice president and CIO. So, in many ways, the return of investment in AI is measured by looking at the improvements to patient care. These patient-focused metrics include reduced length of stay, faster time to treatment, faster insurance eligibility verifications, and faster prior insurance authorizations, he says.

Another project involves using AI to support radiologists in examining scans. A KPI is how often radiologists are alerted to potentially abnormal findings. “As of April 2022, 99% of our radiologists have reported using AI to analyze more than 12,000 studies,” Dadlani says, adding this has triggered nearly 600 alerts. “So physicians can address potentially serious issues as quickly as possible.”

At RSM, the fifth-largest accounting firm in the US, AI investments follow two closely connected paths: one is of productivity and analyst tools that help employees to be better at their jobs. The other is those same or similar tools but used by customers, says Richard Davis, a partner in the company’s management consulting, business, and technology transformation team.

For example, when working with customers, RSM might be called on to pull in data from multiple systems — accounting, sales and marketing, HR, logistics — and bring everything together into a single pane. AI can help speed up this process, says Davis. Then, AI can be used to identify how work moves through these systems, and where the underlying challenges and barriers might be.

So how does the company know whether its AI is headed in the right direction?

“Number one, we can very clearly measure the usage of the tools,” says Davis, who declined to provide details of RSM’s investment in AI initiatives or ROI. “What we want to see over time is a more efficiently delivered engagement.”

That increased engagement, Davis says, should then lead to increased productivity. “So if something used to take us a week to do, the goal might be to bring it down to a day,” he says.

Focus on business benefits

Measuring the success of AI can be subjective as well. Evaluating an AI project is an art as much as developing the AI itself, says Eugenio Zuccarelli, an AI research scientist at MIT who also works as a data scientist in the retail industry.

Still, it’s important to be able to explain the impact AI is having on business, Zuccarelli says. “KPIs should not be set around the model itself,” he says, “but on the business and people metrics, which should be the end objectives of the project.” Otherwise, it can be too easy to pick a technical metric that seems to show success, but in reality wouldn’t translate to effective impact on the company.

Zuccarelli, who also held data science positions at BMW and Telstra, also warns against measuring progress in isolation. For example, if an AI project was designed to improve something that was already improving for other reasons, then a control group is needed to determine how much of the improvement is actually due to the AI.

Other valuable KPIs for AI projects could be, for example, a reduction of false alerts or automatic removal of excessive privileges, says Vladislav Shapiro, who has years of experience in the financial services industry and is founder of Costidity, an advisory group specializing in IT security and identity governance and administration.

In a recent AI-powered security deployment Shapiro worked at, the false positive alert rate was reduced by three-fold, he says, and many previously manual processes were automated.

“When you show these numbers to C-level management, they understand that all of the above reduce risk of being breached and increase accountability and governance,” he says.

Measuring success incrementally

Automation leading to cost reduction is the easiest and clearest way to show economic benefits of AI, says Sanjay Srivastava, chief digital strategist at Genpact, a global professional services firm. But AI can also facilitate new revenue streams, or even completely transform a company’s business model.

For example, with AI, an aircraft engine manufacturer saw it could get better at predicting failures and improving logistics so it could start offering engines as a service. “For the ultimate consumer, it’s better to buy miles flown than the engine itself,” he says. “That’s a new business model. It changes the way a company operates because the AI enables it.”

And the business impact is immediately obvious as well, he says.

So, to justify the investments in AI within that time, this particular manufacturer needed that long-term goal, but translated it into short-term projects that were measurable in other ways.

“Instead of saying, ‘In ten years, we’ll change the industry,’ say, ‘In year one, we’ll start looking at which parts we need to stock,'” he says. “You’re not yet changing the industry to air miles flown. You’re just saying, ‘We’ll need the right parts in the right quantities.’ It’s a one-year project to optimize your warehouse systems and reduce the amount you’re investing in inventory.”

In addition to supply chain optimization, other short-term measures of progress can include client satisfaction.

“If the airplane is stuck in Mumbai for five days waiting for a part, for instance, the client is going to feel that,” he says.

Alignment with strategic vision

Then there’s the reality that in the short term, some AI projects can hurt the bottom line, but still be important and transformative in the long term. For example, a company that rolls out a customer service chatbot can eliminate mundane tasks. “But chatbots can be harmful because some people are good at upselling and want to engage with people,” says Gartner analyst Whit Andrews. “So the organization might not want that.”

It goes back to what kind of company you want to be, he says. “At some point, you have to ask if you’re the kind of company that if a delivery gets screwed up, for instance, customers can call to ask where it is and you engage with them and then try to sell them on getting the product once a month.”

If the organization is committed to both AI-powered transformation with measured ROI to back it up, and has a vision of being customer-focused, then it might look past the immediate hit to the bottom line toward other potentially more meaningful indicators.

“A more fully automated organization may be more successful because it’s increasing market share,” Andrews says. “But you can develop your data so you’re able to reach out to someone at a time that’s more relevant to them. If there’s something you can point to and say, logic just tells us that this will make our customer happier and our workers more successful, then chase it.”