How to measure the cost-effectiveness of generative AI? ROI metrics and often overlooked aspects explained

How to avoid one-sided ROI calculations

Many companies tend to calculate ROI based on only a limited number of factors. In this section, we will explore the problems that arise from calculating ROI without a multifaceted perspective.

The real reason why "it's introduced but not used"

The reasons for slow growth in actual use after the introduction of generative AI include not only a lack of user skills, but also operational inconvenience and low reliability of the AI's answers. In particular, in actual operation, if the AI's output is inaccurate or not suited to the target business, a single failure can cause a sudden drop in enthusiasm for its use.

When calculating ROI, if only the initial implementation costs and short-term labor reduction effects are taken into account, the long-term utilization rate and improvement process tend to be overlooked. Even if a certain amount of time is saved, if the preparation and maintenance of data that needs to be updated is neglected, the long-term return on investment will decrease. For this reason, an approach that evaluates from multiple perspectives is required, rather than using time-saving effects as the only indicator.

▼I want to know more about generative AI
⇒ Generative AI | Glossary

Overlooked "hidden variables"

Factors that affect ROI are not limited to license costs and simple time-saving effects. For example, qualitative factors such as employee proficiency, information circulation within the organization, and data quality and organization level also have a significant impact. While these "hidden variables" are difficult to quantify, they have a significant impact on results.

Many companies have high hopes for AI in order to achieve a good return on their investment, but in reality, if the data that AI handles is not properly prepared, it will not be able to fully utilize its capabilities. If low-quality data is input, the AI's answers are likely to be inaccurate, including hallucinations. This can result in a loss of trust in the field, and ultimately lead to the AI being little used.

A basic framework for calculating ROI in generative AI

The ROI of generative AI needs to be viewed from a different perspective than traditional IT investments. In this section, we will explain the differences in thinking from traditional IT investments and define ROI.

The decisive difference from traditional IT investment

The big difference is that rather than introducing functions through the purchase of hardware and software, AI is an investment that enhances the knowledge and problem-solving capabilities of personnel and organizations. Since it is not enough to acquire new functions, but also to develop the skills and knowledge to use them effectively, a process of continuous training and data enrichment is required.

Simply introducing generative AI will not produce results, and without a system and support that promotes the use of that knowledge and continuous learning, the expected results will not be achieved. Therefore, the costs of operation and training after implementation are also important factors in calculating ROI.

What's more, the true return on investment can be maximized by learning new skills and processes across the organization, such as how to manage information and digitize internal communications.

Redefining the ROI formula

In addition to the traditional ROI calculation formula (profit ÷ investment cost), it is impossible to understand the true return on investment without taking into account the "data preparation costs" and "training and ongoing operation costs" unique to generative AI. For example, the process of cleansing data and customizing models with the help of external vendors can cost more than the implementation costs. Therefore, the following formula redefines how to calculate ROI.

ROI = (Output Value - Total Cost of Ownership) ÷ Total Cost of Ownership

This formula compares the value of the results produced by AI with the total cost of ownership, expressed as a percentage. The output value includes things like reduced employee man-hours and additional profits from new business creation, while the total cost of ownership includes not only license fees but also data preparation, maintenance, and personnel training costs.

If you underestimate this total cost of ownership, you run the risk of overestimating your ROI. Because there are many uncertainties, especially in the early stages of operation, it is important to set costs conservatively at the trial calculation stage and flexibly review them.

How to visualize the results (output)

To visualize results, it is best to evaluate both easily quantifiable items (Hard ROI) such as speed and cost reduction effects, and qualitative items (Soft ROI) such as the degree of knowledge utilization in the organization and employee satisfaction. By evaluating results from both "hard" and "soft" perspectives, the true effects of introducing AI become clear.

Hard ROI (efficiency/cost)

Hard ROI is measured using indicators such as direct cost reductions and efficiency gains from implementation. It is easy to see results in a short period of time and is also an area that is easy to appeal to management.

However, care must be taken when calculating ROI from a short-term perspective, as this may overlook ongoing investment factors such as additional training costs and data maintenance costs.

Time-effectiveness: Reduction rate of labor required for searching, summarizing, and creating

By utilizing generative AI to improve the efficiency of tasks such as writing and summarizing, we can grasp in concrete numerical terms how much time has been saved compared to conventional work. In particular, for routine tasks that are repeatedly performed by employees, the more extensively AI is used, the greater the potential for labor cost reductions.

When the time-saving effect is made visible, it is easy to increase the motivation of on-site staff, and in many cases it also makes it easier to consider additional implementation.

Replacement costs: Reduction in outsourcing costs, cost reduction through tool consolidation

If AI can be used to in-house program development and translation work that previously relied on human labor or outsourcing, it will be possible to reduce the outsourcing costs that have been paid up until now.

Furthermore, if multiple functions were handled by separate tools, integrating them into a generative AI platform can be expected to reduce management costs, making these cost savings one easy-to-understand ROI indicator.

Soft ROI (quality/organizational strength)

When considering the effects of introducing generative AI, it is essential to consider aspects that are difficult to measure quantitatively. For example, improvements in employee experience (EX) and deepening information sharing within the organization are examples of soft ROI.

In particular, corporate culture and employee satisfaction can have a long-term impact on employee retention rates and innovation, which can ultimately lead to economic benefits.

Answer accuracy and practicality: The percentage of generated answers that can be used in business as is (no correction rate)

By measuring the extent to which the AI's answers and suggestions can be used as is, you can objectively understand the AI's practical level. The fewer corrections required, the greater the reliability in the field, which directly leads to improved ROI.

On the other hand, if we only focus on the rate of uncorrected content, we risk placing excessive expectations on AI. It is important to analyze not only the number of corrections, but also which parts needed correction and improve accuracy by improving the root cause.

Employee Experience (EX): Reduced time spent searching allows employees to focus on their core tasks

If employees can reduce the time they spend gathering information and organizing documents in their daily work, they can devote more time to core, value-added tasks such as planning and face-to-face communication.

Although these positive effects are qualitative, in the long term they will increase employee motivation and retention rates, leading to stronger organizational capabilities, which will ultimately result in a higher ROI.

Democratization of knowledge: To what extent has the tacit knowledge of veterans been utilized by junior employees via AI (search hit rate, etc.)

By digitizing the experience and know-how of veteran employees and making it accessible to anyone through AI, it will be possible to create a system that is not dependent on individuals. By using search hit rates and answer accuracy as indicators, a space will be created where people can mutually complement each other's knowledge.

This democratization of knowledge will strengthen the overall organizational strength and contribute to accelerating the training of new and junior employees. As a result, generative AI will become easier to use in a variety of tasks, and the return on investment can be expected to increase.

The "hidden variables" that affect ROI: The data quality trap

AI learns based on the input data, so if the data is not properly prepared, no matter how advanced the model applied, it will return incorrect output. No matter how much investment you make, if the fuel you use is poor quality, you will not be able to maximize engine performance. The quality of the data will have a major impact on the results that AI produces. Here we will examine the importance of data quality, which is often overlooked.

The return of "Garbage In, Garbage Out"

There's a long-standing adage in information systems that poor data input will produce poor results, and the same is true for generative AI: even the most cutting-edge models won't deliver a meaningful ROI if the data is incorrect.

RAG and fine tuning refer to existing data to improve the accuracy of answers, but if the data is not organized in the first place, it will be difficult to refer to it properly, increasing the possibility of deriving an incorrect answer.

If the answers returned by AI are repeatedly incorrect, it will be perceived as an "unusable tool" in busy workplaces. As a result, usage rates will decline rapidly, and the investment costs will not be recouped.

Structuring unstructured data is an asset

Huge amounts of PDF files, handwritten meeting minutes, images and audio files stored in disparate folders - organizing this unstructured data into a manageable form can actually be a great asset for a company.

Scattered information increases the risk of repeating the same tasks inefficiently and using the wrong materials. By properly organizing data, you can extract new insights from previously dormant internal information and easily extract materials that will contribute to decision-making.

▼Saison Technology also offers preprocessing applications to improve the accuracy of the generated AI's responses.
⇒ Problem-solving solutions for AI project leaders

Data maintenance is a "maintenance cost"

AI models can only continue to generate value if they are always operated based on the latest and most accurate data. However, because information within a company changes over time, neglecting maintenance increases the risk that the AI will learn and reference incorrect information.

If AI continues to refer to documents that do not reflect the changing facts, it will provide answers that are out of touch with reality. Particularly in cases where industry trends are drastic, using outdated data could cause you to fall behind your competitors.

Therefore, it is necessary to establish operational rules for the process of keeping data up to date, and to establish a system for regular reviews and updates.

Conclusion: AI utilization starts with "data preparation"

The performance of generative AI is heavily dependent on the underlying training data and the environment in which it operates. No matter how excellent the algorithm is, if the data quality is poor, it will make incorrect inferences, and users will quickly become inconvenienced and abandon the system.

It is often thought that introducing the latest models or large-scale cloud services will produce results, but in reality, even if you use expensive models, the effects will be limited if data management is poor.

Furthermore, many companies have a mix of different systems and formats across different departments, and integrating and updating these systems requires a long-term perspective.However, this process itself also serves to visualize intellectual property, so it is necessary to have an ongoing approach.

Ultimately, by aiming to extend the benefits of generative AI beyond the use of individual tools to management-level decision-making, we can expect to establish a competitive advantage and continuously increase ROI.

The person who wrote the article

Affiliation: Data Integration Consulting Department, Data & AI Evangelist

Shinnosuke Yamamoto

After joining the company, he worked as a data engineer, designing and developing data infrastructure, primarily for major manufacturing clients. He then became involved in business planning for the standardization of data integration and the introduction of generative AI environments. From April 2023, he will be working as a pre-sales representative, proposing and planning services related to data infrastructure, while also giving lectures at seminars and acting as an evangelist in the "data x generative AI" field. His hobbies are traveling to remote islands and visiting open-air baths.
(Affiliations are as of the time of publication)

Recommended Content

Related Content