Artificial intelligence (AI) and machine learning (ML) technologies are the next set of transformative technologies in the tech sector. And while much has been prophesized about their impact, data points are few and far between. 451 Research's Voice of the Enterprise (VotE): AI & Machine Learning survey is a new biannual offering that addresses the need for quantitative metrics around these emerging technologies. It provides insight into the adoption patterns as well as benefits, barriers and applications of these critical technologies. Similar to recent reports on
industry-specific use cases and
AI adoption strategies, this Spotlight leverages
the results of the most recent survey to detail the critical challenge of measuring the impact of AI initiatives. The data comes from responses from a cohort of more than 1,600 line-of-business and IT professionals in North America, EMEA and Asia-Pacific.
The 451 Take
There is no point for enterprises to adopt technology just because it's new and interesting; there must be some measurable benefit to be gained from it. However, at the early stage of development and deployment of a technology, quantifying that benefit can be very difficult. According to new data from 451 Research's VotE: AI and Machine Learning survey, less than half of early adopters have defined key performance indicators (KPIs) around their AI and machine learning initiatives, although the vast majority plan to do so within a year. Clearly, an 'adopt now, ask questions later' mindset pervades this first stage of enterprise AI adoption. Major obstacles include defining the right success metrics, technical limitations and operationalizing data. Given the substantial benefits of well-implemented AI systems and the significant risks of poorly constructed AI, organizations plan to focus more on success metrics as the industry moves further along the adoption curve.
Adopt Now and Ask Question Later
Although AI – and machine learning in particular – is still very early in its deployment within enterprises, 451 Research survey data provides insight into how adopters are measuring the impact of these new technologies. According to the survey results presented in Figure 1 below, only 46% of AI and machine learning adoptees currently have defined KPIs to measure their initiatives. Another 42% of respondents say they plan to develop KPIs within the next year, and a 12% minority say they have no plans to produce KPIs.
Figure 1: Are early adopters of AI and machine learning defining KPIs?
Source: 451 Research Voice of the Enterprise: AI and Machine Learning, 2H 2018
The data suggests that enterprises are approaching AI and machine learning with an 'adopt now, ask questions later' mindset. For many respondents, the absence of defined KPIs does not preclude the implementation of this emerging technology. It goes without saying, but this approach is inherently high-risk. What if the 'intelligent' system performs worse than its predecessor? What if it makes one disastrous decision? The data shows many enterprises are prioritizing putative rewards over potential risks.
Digging a little deeper into the data reveals further insights about KPI definition:
- Larger companies (those with more than 1,000 employees) are more likely than smaller companies (1-999 employees) to have already defined KPIs, 52% to 43%.
- Among the smallest companies of all in our survey – those with annual revenue of less than $1m – a little over a quarter (26%) say they have no plans to define KPIs around their AI initiatives, compared with 14.9% of companies with revenue between $1 and $9.99m and 11% of respondents with revenue over $1bn.
- 62% of enterprises pursuing a primary AI strategy of vendor-provided machine learning-specific tools say they defined KPIs.
- 65% of the most tech-savvy users have defined KPIs, whereas 24% of conservative respondents say they have no plans to.
The Various Barriers to Defining KPI
Of those organizations that have yet to define KPIs around AI and machine learning, what is holding them back? The answer is, not surprisingly, a lot. Figure 2 below displays the most prominent barriers cited by respondents who had yet to implement KPIs around their AI initiatives.
Figure 2: Most prominent barriers to defining KPIs
Source: 451 Research Voice of the Enterprise: AI and Machine Learning, 2H 2018
A plurality of respondents – 38% – say they face difficulties with the initial step of the KPI creation process: describing which metrics should be collected. This statistic is honestly a bit worrisome. The best AI initiatives are those that address a narrow business outcome, and the corresponding KPIs should directly relate to this goal. For example, in the case of a customer service chatbot, a valuable KPI would be whether the system solves customer issues more efficiently than a human representative. Admittedly, there are additional interacting variables that generate complexity, but it is still disconcerting that the largest percentage of respondents cite variable definition as a challenge.
The next two leading barriers – technical limitations (31%) and operationalization (29%) – are downstream of metric definition, so it makes sense that fewer respondents report problems with these steps. Organizational inertia (26%) is a parallel, and admittedly opaque, issue that hampers many business initiatives, so it is not surprising to find that it inhibits the implementation of KPIs around AI projects.
Finally, 13% cite a completely different reason for not defining KPIs: they are so confident in the success of their AI initiatives that metrics are superfluous. Even the laziest business analysts will admit that confidence is no substitute for evidence; however, digging into the data provides some context for this statistic. Respondents whose initiatives are either security-related or employing a systems integrator are more likely to say they don't need to define KPIs. Why would that be? There are some AI applications that are categorical upgrades from the systems they replace. If your business is going from an entirely manual SOC to one with AI-assisted remediation tools, you probably feel that you don't need KPIs to tell you that it's an improvement.
Toward a KPI-first Furture?
It is important to underscore that this data is a snapshot of the current – and relatively early – state of AI and machine learning adoption in the enterprise. As these technologies become more commonplace, business processes and workflows will standardize. It's thus hard to imagine that the 'adopt now, ask questions later' mindset will persist into the future, especially as enterprises look to imbue machine intelligence into their critical systems.
