Project Planning Guide for Introducing data integration platform- How to Eliminate Obstacles to Planning and Avoid Risks

Why data integration projects get stuck in the planning stage

Even if the "what you want to do" is clear, why can't you put it into an action plan? The main reason is that there is a lack of information to determine "what should be done, in what order, and how much effort it will take," due to the complexity of the existing systems that need to be integrated and the wide range of technical uncertainties.

In particular, organizations that have traditionally operated systems with individual optimization within each department have no accumulated know-how for cross-company collaboration. As a result, even if they try to force a schedule without accurately understanding the current situation, they are inevitably left with an unfounded plan. As a result, they are unable to create RFPs (requests for proposals) necessary for budgeting and vendor selection, and projects reach a stalemate.

▼I want to know more about data integration
⇒ data integration / data integration platform | Glossary

Back to Top

The risk of proceeding with a project under uncertainty

If the information necessary for planning is lacking and tool implementation and development is pushed ahead, the project will most likely face the following three problems, which will be fatal to the project's success.

Schedule deviations and breakdowns

The biggest risk is a discrepancy between the expected schedule and the actual work. Due to insufficient prior research, there is a tendency to underestimate the difficulty of connecting to existing systems. Even if it is determined that "connection is possible because an API exists," in reality, there are many cases where specification constraints make it difficult to obtain data, or unexpected data processing is required. When these technical issues are discovered during the implementation phase, additional work is required to resolve them, and the original schedule falls apart. As a result, delivery dates are significantly delayed.

▼I want to know more about the API
⇒ API｜Glossary

Undefined requirements and rework

Inter-system integration goes beyond simple data transfer; it impacts the business process itself. However, if the current business flow involves manual work or subjective judgment by personnel, proceeding with automation without taking these factors into consideration can lead to situations where business operations do not run smoothly once the system goes live. Furthermore, if requirements such as the frequency of data updates and security standards between related departments remain unclear, problems will arise in the testing phase where users are repeatedly told that "the behavior is different from what was intended." This can lead to large-scale rework, such as redesigning and modifying programs.

Budget overruns due to hidden costs

Estimates based on inaccurate plans do not cover all necessary costs. For example, hidden costs such as additional costs for network equipment to meet security requirements, costs for introducing cleansing tools to maintain data quality, and costs for modifying legacy systems can become apparent midway through the project. If these costs are not taken into account in the initial budget, it can be difficult to cover the additional costs, which can force you to cut features or suspend the project.

Back to Top

The role of assessment in enabling planning

In order to resolve the aforementioned "stagnation due to an inability to make plans" and "risks due to uncertain progress," it is effective to conduct an "assessment" that focuses on investigating the current situation and organizing requirements before the full-scale development phase begins.

"Assessment" is a process that visualizes the current system assets and business processes that have been left as black boxes, and clarifies the project scope and risks. By clarifying opaque elements based on facts, it becomes possible to formulate a project plan that is based on evidence, not speculation.

If you don't have a dedicated engineer in-house, you can utilize external support with specialized knowledge of data integration to create objective and highly accurate plans. The specific process is described in detail in the next section.

Back to Top

Specific assessment process

To ensure a solid project plan, the assessment involves two phases: "Current Status Survey" and "Analysis," which turn a vague concept into a concrete implementation plan.

Current situation investigation phase

First, we conduct an investigation to accurately understand the actual state of the company's internal systems and operations, which have become black boxes.

Analysis of the current situation and sharing of goals

Before starting the investigation, we clearly communicate the project's objectives and goals to the relevant departments. We explain that the objectives are not simply a system investigation, but are to "build a company-wide data utilization platform" and "improve business efficiency," and we provide direction on how each department should cooperate (by providing materials and responding to interviews). This prevents oversights in the investigation and stagnation due to inadequate cooperation between departments.

Interviews with each department

We will conduct interviews with the information systems department and each business department to understand the actual situation.

• Information Systems Department: Checks technical requirements such as management system specifications, infrastructure environment, and security policies.
• businessAdministration department: We ask about the actual operational status that is not recorded in the system log, such as the timing of data input in actual work, how it is used, whether manual processing is performed, etc. In particular, we will also identify work outside of the regular flow, such as "exception processing" and "monthly specific processing."

Interface Visualization

Based on the results of the interviews, we will document the connection specifications of the target systems as an "IF list."

• Comprehensive technical specifications: Protocols (REST, SOAP, FTP, JDBC, etc.), data types (structured data, unstructured data), character codes (Unicode, UTF-8), file formats (CSV, PDF, PNG), etc. are described in detail.
• List the current status of the connected systems: List the factors that affect connectivity, such as the operating environment (OS, middleware), cloud/ on-premise classification, and version information.

Data flow visualization

It visualizes not only individual IF information but also the data flow and structure of the entire system.

• Logical data flow: Depicts the overall flow of data, including where it originates and which systems it passes through before being used.
• ER diagram (data structure and relationships): We organize the data structures and relationships held by each system and create basic documentation to confirm how data items are related between systems (mapping consistency).

Analysis Phase

Based on the facts obtained from the investigation, we assess risks and develop specific solutions and implementation plans (concepts).

Risk Analysis & Assessment

From the survey results, we identify factors that hinder feasibility and evaluate priorities.

• Identifying knockout conditions: We identify risks that could be fatal to the project (knockout conditions), such as "no connection method available due to legacy protocols" or "security requirements cannot be met."
• Evaluating difficulty and priority: We determine the priority of the integration by combining the technical difficulty of the integration with its business importance.

Concept creation (grand design formulation)

Based on the analysis, we will develop a comprehensive implementation blueprint that includes the following elements:

• Current issues: We will sort out the bottlenecks that have come to light through the interviews (double entry, time lags, data inconsistencies, etc.) and clarify the issues that need to be resolved.
• Measures and policies: Define specific approaches to solving problems.
• Grand Design:
  • data integration method: We design the optimal architecture, whether it be real-time integration or batch integration, API integration or file integration, etc.
  • Master data policy: Establish a management policy for determining which data from multiple systems will be the "true (golden record)."
  • Data utilization policy: Define how the integrated data will be utilized, whether it will be analyzed with a BI tool or written back to SaaS for business use.
• Risk assessment: Presents the remaining risks in the project and countermeasures.
• Migration Plan: Develop a phased migration plan (roadmap) to minimize business impact.
• Assumed structure and roles: Define the division of roles between your company, vendors, and support partners, as well as the necessary resource structure.

Back to Top

Assessment ROI

The cost and time required for planning may raise concerns about the return on investment, but the benefits of an assessment are clear compared to the risk of losses that would result from starting a project without sufficient certainty.

Eliminate project stalls and improve initial momentum

This eliminates the "I don't know where to start" situation and clarifies specific tasks and procedures, allowing the project team to get started without hesitation and get things up and running in the shortest possible time.

Optimizing costs and improving the accuracy of budget-to-actual management

Estimates made when requirements are unclear tend to include contingency costs (buffers) to avoid risks. By clarifying requirements through upstream processes, unnecessary buffers can be eliminated, enabling orders to be placed at appropriate costs in line with actual conditions. Furthermore, hidden costs can be identified in advance, reducing the risk of budget overruns.

Establishing a company-led system

Through the investigation process, you will gain a deeper understanding of your company's system environment and business issues. This will enable you to build a project structure that allows your company to take the lead in decision-making, rather than relying on the vendor. This is an important factor in stable operation and maintenance after implementation.

Back to Top

Toward reliable project execution

The introduction of data integration platform is an important infrastructure investment that supports a company's data utilization platform. Pushing ahead with a plan without sufficient internal knowledge and in a situation where it is difficult to formulate a plan carries significant risks of schedule disruption and increased costs.

First, conducting an assessment in the early stages of a project to accurately understand the current situation and then going through the process of formulating a feasible plan will ultimately be the quickest and most reliable way to lead the project to success.

Back to Top

summary

As explained in this column, in order to make a data integration project a success, it is essential to accurately understand the current situation before starting and to develop a feasible plan based on that understanding.

Saison Technology offers an assessment (pre-evaluation) support service to eliminate uncertainty in project planning and ensure a smooth launch. Our data integration professionals, with extensive implementation experience, will visualize your company's system environment and business flow and provide practical support in formulating the optimal implementation plan.

If you have issues such as "it's difficult to formulate a plan on your own" or "you would like to hear an expert's opinion on how to proceed specifically," please feel free to contact our "Online Consultation" service.

Back to Top

The person who wrote the article

Affiliation: Data Integration Consulting Department, Solution Architect

K. F

In his previous job, he worked in sales and in-house system engineering at a financial institution. After joining Saison Technology, he worked as a pre-sales representative, supporting proposals and planning services related to data integration platform, while also promoting data utilization methods in the financial field. His hobbies are watching baseball games, visiting hot springs, and watching movies.
(Affiliations are as of the time of publication)

Recommended Content

Related Content