In order for the investment to generate the expected return, it is essential to be clear about the objectives of the project and to establish metrics that allow measuring its impact.
Every day more companies incorporate artificial intelligence solutions into their processes. The figures prove it: according to the Worldwide Artificial Intelligence Systems Spending Guide report, by the IDC consultancy, spending in 2023 will reach US$97.9 billion. More than two and a half times the 37.5 million expected for 2019. This represents an annual growth rate of 28.4%.
In this line, the results of a survey carried out by NewVantage Partners in the United States, 92% of the large participating companies reported that they are achieving a return on their investments in data and artificial intelligence. This represents a 48% increase compared to the same result of the survey carried out in 2017.
While many companies have embraced solutions based on artificial intelligence, data science and machine learning, it is important that organizations do so with a solid strategic plan and focus on how to measure the results and success of these initiatives.
"When measuring success, you have to differentiate between measuring business performance and the performance of the artificial intelligence model," explains Andrés Abeliuk, Academic Coordinator of the Artificial Intelligence Area of the Continuing Education Program of the Department of Computer Science at the University From Chile. Who adds: "to measure the effectiveness of an artificial intelligence solution, the performance of the model is measured with metrics aligned to the objectives of the product."
When building a model, offline metrics are used to measure performance on available historical data, says Abeliuk, "which measure the accuracy of predictions against known values." And once the model is already implemented, "online metrics measure how the model affects the users who use it."
On the other hand, to assess whether the investment generates returns, a metric must be defined to judge the success of a product or service. “This metric should be different from the model metrics, and just quantify the success of the product,” says Abeliuk.
As Jerald Murphy, Senior Vice President of Nemertes Research, explains in a Tech Target column, companies can establish appropriate key metrics to assess the efficiency of projects. For this, quantitative and qualitative KPIs can be established. “Typical AI-related KPIs include Mean Time to Repair (MTTR), or the time it takes to fix a problem, and First Contact Resolution Rate (FCRR), which indicates what percentage of problems are resolved by Tier 1 IT support (basic support) without the need to escalate. Also, the number of tickets an IT team receives per month is a tangible metric.” Ultimately, KPIs will help demonstrate a concrete return on investment (ROI), which can be time, money, or work, Murphy says.
Regarding the choice of metrics, Abeliuk adds that "while quantitative metrics are good for measuring success, qualitative metrics, such as user comments, allow us to understand the former and also incorporate them to improve the product."
Keys for a correct implementation of AI solutions
An article in the Harvard Business Review indicates that the companies that achieve the greatest success with solutions based on artificial intelligence are those that have a defined process for the implementation and evaluation of these technologies, which they follow regularly.
Deloitte's State of AI in the Enterprise study, conducted in mid-2021, identified two types of companies that are deriving value from their AI investments. 28% were classified as "Transformational", and are those companies that have implemented various artificial intelligence solutions and are obtaining great income. Something in common between them is that they have defined an AI strategy and have built an ecosystem around it.