As the company adopts generative AI internally, CIO Chris Bedi is measuring everything — and thinks you should too.
Competition among software vendors to be “the” platform on which enterprises build their IT infrastructure is intensifying, with the focus of late on how much noise they can make about their implementation of generative AI features.
ServiceNow wasn’t the first to announce its generative AI capabilities, but it was among the first of the major platform vendors to make them generally available to all customers.
“One reason we’re releasing early is because we’re ready,” says ServiceNow CIO Chris Bedi. “We’ve been working on large language models for years.”
The Vancouver release of the company’s workflow automation Now Platform added the generative AI-powered Now Assist for IT Service Management, Customer Service Management, and HR Service Delivery: interactive chatbots with new text creation and summarization features tailored for three distinct areas of enterprise activity. Businesses can use Now Assist to triage tickets, automating responses to some and helping agents be more productive as they deal with others.
The September addition of Now Assist to the platform’s core could be bad news for the third-party software developers that had built AI-powered automations for ServiceNow’s ticketing system — although, says Bedi, “Their traction was pretty limited anyway.”
ServiceNow isn’t the only vendor introducing AI functionality previously provided by the surrounding ecosystem. More recently, OpenAI upset the business plans of smaller developers that sold specialized chatbots built on its flagship product, ChatGPT. In early November it released a tool that enables anyone to add instructions and extra knowledge to create special-purpose versions of ChatGPT that it calls “GPTs.” The simplicity with which enterprises can create them makes it less likely they will need to turn to third-party suppliers.
Bedi, naturally, sees value in the integration of functionality on a single platform, and says other CIOs are telling him, “I just want a platform. I want less complexity. I don’t want all those points solutions out there.”
Putting gen AI to work
ServiceNow doesn’t just sell its Now Platform to other enterprises; it uses it internally, too. As customer zero, Bedi had a head start in adopting the new generative AI features — albeit just a few weeks, given the lightning pace of developments in the field.
Still, Bedi and company have already leveraged Now Platform’s new generative AI capabilities to improve the user experience for customers and employees, enhance agent productivity, and accelerate digital transformation, he says, adding that the user experience improvements and productivity enhancements are a result of using generative AI to deliver self-service access to problem-solving tools.
“The understanding of the human intent is much more powerful, and the ability to take from a knowledge base article what you need to know and surface it right away is huge,” Bedi says of the advantage that generative AI has over previous generations of automated tools aimed at helping users triage their own problems.
When Bedi talks of enhancing “agent” productivity, he uses the term broadly to mean HR staff, IT service desk operatives, customer service agents, and sales staff, all of whom can benefit from generative AI’s ability to find answers in masses of documentation.
With one large language model (LLM), he says, “We indexed all of our go-to-market content and product documentation.” The goal? “Make a sales rep who joins Monday infinitely knowledgeable.” User adoption of tools like that is skyrocketing week by week, he says.
As for generative AI’s role in accelerating digital transformation, that’s all about applying the technology to deliver text-to-code and text-to-flow capabilities. The latter, Bedi says, involves building processes in low-code development platforms. In the hands of a business analyst working on a complex process, “It’ll get you 70% to 80% of the way there,” he says. “Once the flow is done, then you have somebody a little bit more technical fill in some of the blanks underneath.” Such tools can provide a way of bringing the business teams and technical teams closer together, he adds.
While some workers fear that generative AI is going to take away their jobs, Bedi sees little to no risk of that for developers. “It’s not like we need less software engineers. It’s more that we can get more done with our existing labor pool, and maybe even make our average software engineers our best software engineers,” he says.
Measurement: The key to success
Based on his experience with the internal rollout of Now Platform’s Vancouver release, Bedi has some general advice for CIOs seeking to get the most out of generative AI applications.
Measurement is key, he says. Much as carpenters advise to “measure twice, cut once” for a good fit, Bedi has been measuring generative AI’s fit using four distinct categories of metrics: sentiment, adoption, coverage, and business impact.
Sentiment can be a measure of how willing employees will be to use generative AI in their workflow. When IT agents were asked, “Are these things helping you be more productive?” he says, 58% said yes after one month of use.
Adoption, he says, is another metric entirely, “because it’s very different to say, ‘I like it,’ and then, ‘I’m adopting it,’ which means going back to it again and again.”
“Our learning has been that the adoption where it’s not incorporated into the flow of work naturally is going to wane,” he says.
Coverage is more nuanced, being a measure of “Can generative AI help with this?” For example, generative AI can reliably condense texts, so for tasks such as summarizing cases, coverage is going to be 100%, Bedi says. “But on search we defined a threshold of confidence below which we say gen AI should not contribute to an answer, and you’ll get the usual knowledge base article, so coverage is defined as the percentage of times where generative AI can actually play a role.”
Business impact is measured by more familiar KPIs, he says: speed, productivity, customer Net Promoter Score (NPS), or employee NPS.
For anybody diving into generative AI, he says, “I would think about these four categories of metrics and how you’re going to instrument them to measure exactly what’s going on.”
