(Credit: Eugene Mymrin/Getty Images)
If one topic stood out above all the others at last week's Gartner IT Symposium/Xpo, it was the difficulty most organizations have in finding return on investments from their AI projects. In a variety of sessions, Gartner analysts discussed parts of the issue, and I had the chance to talk to a number of attendees about their use of AI. What I found is a difficulty to quantify results for general productivity usage—like everyone Microsoft Copilot or ChatGPT—but some real ROI where AI can improve specific outcomes.
One thing I heard again and again was that productivity alone isn't enough. In a keynote address focused on AI readiness, analysts Alicia Mullery and Daryl Plummer shared Gartner studies that suggest that in 2025, the odds of an AI initiative achieving ROI are only in one in five, and the odds of an AI initiative achieving true transformation is just one in 50. A different study of chief financial officers showed that 74% are seeing productivity gains, but only 11% are seeing return on investment. Still, they say it is too early to be dismissive of the impact of AI, as most firms just aren't ready yet.
CEO Priorities
(Credit: Gartner)Getting value from AI is one of the biggest issues that CEOs are worried about. Discussing a recent survey of CEOs, Gartner analyst Don Scheibenreif said there are three big issues facing these execs this year: turmoil-driven growth, the AI value conundrum, and workforce mixing.
About 79% of those CEOs now say they believe AI will significantly impact their industry over the next three years. And 83% said they intend to increase investments in AI. Yet he notes that 59% of AI initiatives fail to get into production on average.
Why Gen AI Projects Fail
(Credit: Gartner)In another session, Gartner analyst Arun Chandrasekaran said that through 2025, at least 50% of generative AI (GenAI) projects will be abandoned after proof of concept due to poor data quality, inadequate risk controls, escalating costs, or unclear business value.
The first area for failure, is data. GenAI is a general-purpose technology, but "real value is created within your organization when you are able to combine your data, your context, and your domain knowledge with these models," Chandrasekaran said.
The second area is risk, though it's difficult to complete eliminate things like hallucinations, bias, and toxicity. The third area is cost, where a few cents per prompt multiplied by thousands of prompts across hundreds of users across tens of use cases gets pricey pretty quick.
And finally, there is business value. "Maybe we're expecting too much out of AI in too short a time, or we're not able to identify the right metrics or have the right processes to measure the value AI is creating within the organization," Chandrasekaran said. But perhaps we also picked the wrong use cases, overpromised on what AI could deliver, or we didn't have the right metrics.
Chandrasekaran listed 10 "failure modes" (seen in the chart above). In many cases, organizations are choosing the wrong AI technology. Or the data quality is poor and not presented in a way that is good for GenAI. It's also important to make sure you pick a composable platform architecture because models are changing so rapidly. Organizations need to think more about "responsible AI" to protect things like company confidential information and personally identifiable information (PII).
Other issues include poor change management, technology obsolescence, escalating tech, lack of AI specific roles within an organization, or escalating TCO, both with SaaS providers and by using more tokens such as using a large model when a small model would work.
Chandrasekaran suggested organizations prioritize AI use cases with high business value and technical feasibility; avoid technological obsolescence by carefully picking the right AI techniques and tools; start by planning for responsible AI and change management from the beginning of the project; invest in data and AI literacy skills; and create new roles to operationalize AI.
Getting Value From AI
(Credit: Gartner)Even Gartner's "most advanced clients...are still telling us they are struggling with" getting value from AI, analyst Nate Suda said in another session.
To get value, we have to rethink what we measure. We've all seen organizations show studies that AI can save time on one particular task or another; Suda cited a particular Goldman Sachs study that showed a 90% saving on test coverage of an application as an example.
"Time saved is not money saved," he said. It depends on what you are doing with the time you saved. In almost all cases, "productivity leakage" is real, and you need to build at least 10-30%leakage into your business case. In more recent data, they've seen up to 69% leakage.
Indeed, he cited a recent CFO study in which 74% of CFOs said they've saved time through AI but much fewer have seen money saving or increased revenues. That's because "productivity is a low-quality benefit," Suda said.
(Credit: Gartner)In order to get benefits from AI, Suda suggested that rather than focus on productivity, organizations focus on transformative goals, such as creating new products, improving customer experience, reducing losses, or using capital better capital. For instance, he talked about how Six Financial Group in Switzerland used AI to write better loans, which led to less bad debt and therefore growth; or how Verizon used traditional AI to route people to the correct customer service agent, thus decreasing churn; or how Pfizer used AI to increase vaccine yield per batch.
Suda echoed one of the big themes from previous symposiums—the difference between "everyday AI" and "game changing AI," with the latter offering a completely different return on investment. But he broke things into three categories: First, is defending a company's current position or "everyday AI," which focuses on productivity but is hard to measure. Second is extending or improving the operations of an enterprise—doing things like reengineering, catching fraud, or pricing better, which creates measurable ROI. Finally, there is Upend which changes the business through a strategic bet, which is too big to think about in ROI and instead should be thought of as akin to venture capital.
It's a mistake to just focus on ROI, he said, because it's short sighted. Your organization will need the other kinds of value that are hard to measure.
Suda talked about things for "return on employee" such as the British National Health Service using ambient voice technology to record patient visits and avoiding clinician burnout. That might not be measurable on ROI terms, but it helps keep clinicians in the system, thus improving overall care.
(Credit: Gartner)A Gartner survey of IT leaders suggests that 45% of AI projects now are focused on employee and customer improvements (classified as return on employee), 36% on process improvements (return on investment), and 19% on business model improvements (return on future). Ideally, Suda said, it would be one-third in each category.
But change management is hard. AI requires massive organizational change, and for every dollar you spend on AI implementation, you'll need to spend two dollars on organizational change. He discussed an entire AI "value journey" with different types of re-engineering, listing different companies that succeeded in changing large parts of their organizations by making the change, thus driving up the cost of operations but generating long-term value.
For instance, one firm used AI to handle 80% of customer service calls, but only cut support agents by 10 or 20%, thus getting better service. At another firm, using AI in marketing required changing the boundaries between different departments, not just speeding up a particular task.
Summing up, Suda suggested organizations leverage the different kinds of opportunities to create value, determine what kind of return to expect, and think about how to implement change to maximize that value. Then, to get some quick wins to get ROI, you should think about renegotiating contracts with your service providers, pivot your focus from generative to agentic AI (because while agentic AI is early, it probably will be faster to create returns), and don't forget traditional machine learning.
Pillars of a Successful AI Strategy
In a separate talk, Chandrasekaran talked about the pillars of a successful AI strategy. He walked through the "defend, extend, upend" list of AI initiatives, saying that it's important for organizations to think of these as part of a portfolio, with the balance between the three depending on your organization's priorities.
Chandrasekaran suggested that organizations look at projects on axes between everyday AI and game-changing AI, and between internal operations and external customer-facing ones; and he gave examples of where different kinds of initiatives fit.
(Credit: Gartner)He listed the kinds of challenges organizations face, many of which he covered in the session on why GenAI Projects Fail.
(Credit: Gartner)To overcome these challenges, he suggested developing specific roadmaps for different parts of the process. The first is how you should be organized to develop AI initiatives, including setting up an AI community of practice and appointing someone to lead AI. Other roadmaps focus on people and culture, including a focus on change management; governance; engineering; and data.
But AI strategy needs to be flexible. "The pace of innovation in AI is unprecedented, so your AI strategy has to be dynamic, it cannot be static." Recent developments include the growth of domain-specific small language models and a push for AI sovereignty, he said.
GenAI Skills You Must Master
(Credit: Gartner)In a session on generative AI skills you must master, analyst Tori Paulman said one big issue is that "we are all imposters" when it comes to AI. They quoted AI researcher Eliezer Yudkowsky, who said "By far, the greatest danger of artificial intelligence is that people conclude too early that they understand it."
Gartner research indicates the 77% of CEOs believe AI will significantly affect their business in the next three years and 87% of digital workforce managers believe AI adoption requires frequent engagement, training, and education. Yet only 21% of CIOs prioritize the impact of AI on employees' work and only 6% of individual contributors say they have received guidance on the skills they should develop.
Another disconnect: Executives are four times as likely to say they believe AI makes them more productive (40%) than individual contributors (11%), while individual contributions are five times more like to say AI makes them no more productive (33%) than executives do (7%).
(Credit: Gartner)Paulman went through four levels of skills that they said are important to make productive use of AI, with each skill building on the one below. First is identifying the correct use cases. Second is understanding how large language models generate information. Third is effective prompting. And finally, there is discernment. Paulman noted that people who use AI on a daily basis are likely more likely to be productive.
(Credit: Gartner)People who are getting the most productivity from AI are using it differently from those who are not. As an example, they said, too often the time saved by using AI is often just spent writing more emails, and then AI is used to respond and summarize the same emails, with the extra time used to just write more emails.
Better uses of AI include using it to reduce pain points by identifying repetitive, time-consuming tasks; improving customer experience; scaling knowledge by expanding your perspective; and by improving worker ingenuity by harnessing the creativity of more people in the organization.
Paulman predicted that by 2027, organizations that reward employee-driven AI experimentation will be twice as likely to achieve business value than those who don't. By 2027, Paulman expects more than half of organizations will secure funding for AI literacy programs, fueled by failure to realize expected value from generative AI.
But Paulman's focus is on urging individuals to consider use cases, understand how LLMs work, and write better prompts, and understand how to judge whether outputs are correct and useful.
What Attendees Told Me
In short, there were a lot of suggestions for getting measurable value out of AI, and that seems really important to me. I talked to a lot of attendees about what they were seeing with AI and agents and got a range of responses.
Most of the attendees I talked to thought agents are interesting, but that other than some Copilot pilots, they thought their firms aren't ready for them. Instead, the focus is on more prosaic AI uses like machine learning, data mining, and data prep, which many rightly see as a necessary step before rolling out bigger AI applications.
Still, it seems there are three big newer use cases that are getting a lot of attention. First are customer service agents. Lots of companies seem to be using agents to "augment" human agents; and a few have fully automated agents. Of course, organizations have been working on this idea for years, so it isn't all that new.
Second is software development. Lots of folks are doing code completion, even if I never heard the phrase "vibe coding" at the Gartner Symposium.
Third is meeting recording and summarization. This is pretty easy these days in applications such as Teams or Zoom and lots of people seem to be doing it and liking it.
But while it was rare, I did talk to several attendees whose firms are seeing a big return from AI. As a result, they're going all in, usually leading with applications that are specific to their industries. It's going to be fascinating to see how this develops over the next few years.