Key Points for Promoting DX! The Value of Data Integration

Step 1. Why is data integration necessary?

First of all, why do companies data integration? Let's consider a situation where data integration is necessary in corporate activities, that is, a case where the same data is needed across multiple systems.

Case 1: Master data integration

Integration to share the same data across multiple systems

Master data, which is basic information about a company, such as data about employees and departments, needs to be shared across multiple systems. For example, employee master data is generally managed in a human resources information system, but if separate systems for expense reimbursement, payroll, and labor are used, data integration with the human resources information system is required.

Case 2: Collaboration in business flow

Integration to pass data to subsequent processes

For example, in the case of sales management in the manufacturing industry, various business systems are involved, such as order management, shipping management, billing management, and inventory management. There is a sequential relationship between these systems, and the order data generated in order management is required in other systems as the sales management business process progresses.

Case 3: Collaboration in data analysis

Integration to collect and return data for analysis

Generally, the system where data is generated and the system where the data is analyzed are different. To analyze data, the data must be collected from the system where it originated. It is also possible to reflect the results of data analysis back to the system where it originated (reverse ETL).

In this way, it is common for the same data to be shared and used across multiple systems in corporate activities.

Step 2. Why do you need data integration tool?

In STEP 1, we introduced three cases where data integration is necessary.
So, what impact does manually integrating these data integration have on actual business operations? There are thought to be two main impacts.

"Occurrence of workload"

For example, in the case of employee master data, the data is updated every time a new employee joins or leaves the company, and each time this update must be manually performed in all systems that use the employee master data.

"Poor data quality"

If data updates such as the former are not reflected in other systems, errors may occur in business operations, and accurate analysis results may not be obtained.

To avoid such challenges, data federation requires data integration tool what is generally called EAI (Enterprise Application Integration)  or ETL (Extract, Transfer, and Load). In response to the need for data integration and the adverse effects of human data integration in corporate activities, data integration tool provides support from the perspective of "automation" and "quality improvement".

From an automation perspective, the mechanical mechanism of a data integration tool eliminates the need for tasks such as moving data or converting it to suit the target system. From a quality improvement perspective, it can prevent human-caused data corruption, such as missed connections or conversion errors. Integration tools with ETL functionality can also create a system that improves and ensures data quality by developing logic to remove missing data.

Step 3. Why do we need data integration platform?

We found that by introducing data integration system, it is possible to improve business efficiency and ensure data quality.
However, it is not a good idea to just blindly introduce data integration system.data integration platformFirst, let's consider the case of data integration that is not based on an infrastructure.

Case 1: Introducing data integration tool for each SaaS implementation

Large companies may employ dozens, or even more than a hundred, SaaS solutions. Furthermore, if a SaaS solution is specialized for a specific business process, its implementation will be carried out by the department responsible for that specific business, and the data integration method will also be selected by each department. As a result, even within a single company, there may be a wide variety of data integration methods, which can lead to inefficiencies in development and operation due to dependency on individuals and vendors, as well as data silos, where systems are closed off within departments.

Case 2: data integration depends on the specifications of mission-critical business systems (ERP)

If the ERP consultant supporting the implementation of mission-critical business systems is not familiar with data integration area, there is a possibility that the mission-critical business systems mission-critical business systems will be configured with an interface that is heavily dependent on the specifications of the core business system.If a data integration mechanism that is heavily dependent on a specific business system is adopted, there is a risk that changes to the specifications of that core business system will have a widespread impact on modifications, causing costs to increase, and that it will be difficult to expand to new data integration between systems.

Preventing these problems with a "data integration platform"

In cases like the one above, our "data integration platform" concept, which unifies data integration mechanisms within a company into a single system, avoids the following drawbacks by utilizing the characteristics of a "hub-and-spoke system" and "loose coupling."

Feature 1: Hub and spoke system

As the name suggests, the hub and spoke approach is an architecture in which data integration platform is at the center (= hub) and the business systems that are the target of data integration are positioned on the periphery (= spokes). Because all data integration is carried out via data integration platform, it eliminates the dependency on individuals, who require different skill sets depending on data integration. In addition, because data integration platform is not limited to a specific business department but is a company-wide platform, it transcends departmental boundaries and eliminates data silos (a situation in which data is distributed and closed off to each business department).

Characteristic 2: Loose coupling

Loose coupling refers to a state in which a high degree of independence is maintained between linked business systems by having data integration platform intervene between them. Specifically, if there is any change in one business system, data integration platform absorbs the impact so that it does not affect other business systems. Because the integration between business systems is loosely coupled, for example, even if you want to introduce a new SaaS, you can do so without worrying about the impact on other business systems. In other words, scalability is ensured.

We collectively refer to the consolidation of data and interfaces through the introduction of data integration platform as Data Integration.

summary

This time, from the perspective of data integration, we talked about the need for data integration, the need for data integration tool, and the need for data integration platform.

• Although data needs to be linked for master integration, business flow implementation, and data analysis, manually linking data results in a huge and complicated workload, and problems such as a decline in data quality arise.
• By introducing data integration tool, we will automate data integration and improve data quality.
• In order to prevent data integration from becoming dependent on individual skills and the creation of data silos, a hub-and-spoke data integration platform will be introduced to create a loosely coupled architecture, eliminating individual skills, data silos, and ensuring scalability.

Do you understand data integration?

Next time, I would like to talk about "standardization of data integration," which is an important concept when it comes to data integration.

The person who wrote the article

Affiliation: Data Integration Consulting Department, Data & AI Evangelist

Shinnosuke Yamamoto

After joining the company, he worked as a data engineer, designing and developing data infrastructure, primarily for major manufacturing clients. He then became involved in business planning for the standardization of data integration and the introduction of generative AI environments. From April 2023, he will be working as a pre-sales representative, proposing and planning services related to data infrastructure, while also giving lectures at seminars and acting as an evangelist in the "data x generative AI" field. His hobbies are traveling to remote islands and visiting open-air baths.
(Affiliations are as of the time of publication)

Recommended Content

Related Content