Free online consultation available. Please feel free to contact us.

DWH

  • Glossary

「DWH」

This glossary explains various keywords that will help you understand the mindset necessary for data utilization and successful DX.
This time, we will explain about "DWH," which serves as the foundation for data utilization.

What is a DWH?

DWH (Data Warehouse) is an abbreviation of Data WareHouse. Warehouse is an English word meaning "warehouse," and DWH refers to a "warehouse (database) that stores data" for data analysis and other data utilization.

Traditionally, products such as those from Teradata have been used as packaged products that companies purchase and operate in-house. Recently, DWHs have also begun to be offered as cloud services, with well-known services such as BigQuery from Google Cloud, Redshift from Amazon Web Services (AWS), and Snowflake from Snowflake.

[Kumo Radio] ~What? Databases alone aren't enough for data utilization? ~What is DWH? (4 minutes 38 seconds)

Vtubers Kumoko and Haruka deliver the latest IT information in an easy-to-understand way. In this video, Kumoko gets hooked on data analysis, but the amount of data is so large that her database is slowing down... Why? What should she do?
We'll explain the differences between a regular database and a data warehouse in just 5 minutes!

How is it different from a normal database (RDB)?

The first thing that is difficult to understand about DWH is how it differs from the databases (RDBs) that we normally use. For example, if you are already using a database such as MySQL or PostgreSQL, it can be a little confusing as to why you would need to introduce a separate DWH when you can store your data there.

Common RDB products such as MySQL and PostgreSQL are designed with the "performance to carry out business on the database," allowing data to be rewritten, added, and deleted quickly without causing data inconsistencies. For example, they are designed with functions that prevent duplicate orders from being placed even if multiple orders are placed for a product that only has one of its kind.

Data analysis often requires past data, which means the amount of data to be processed is large, and it is desirable for the type of data required and the way the data is organized to be suitable for analysis. In contrast, DWH is a database product that stores data for analytical purposes and is designed with an emphasis on "analysis performance" so that analytical processing can be performed reliably using analytical queries even on large amounts of data. However, it is not good at rewriting data.

It would be simple if one product could combine both the "performance for carrying out business" and the "performance for analysis," but unfortunately, for technical reasons, this is difficult to do, so a separate product was created that excels at analytical processing, such as aggregating large amounts of data for analysis; this is what led to the creation of DWH.

What is a DWH?
A database that emphasizes "analysis performance" so that analytical processing can be performed reliably using analytical queries even with large amounts of data.

If you are using a regular database (RDB) for data analysis and processing becomes very slow as the amount of data increases, a DWH may be able to solve the problem.

Relationship with "BI tools"

DWH stores data and enables analytical work using analytical queries such as SQL. BI tools are products that focus on visualizing analytical results using graphs, etc. You can also use them in combination to analyze data stored in a DWH using a BI tool.

Relationship with "Data Lake"

While a DWH requires that the data schema (the format of the data to be accepted) be defined in advance before data can be input, a "data lake" can accept a wide variety of large amounts of data (even images, videos, and other data) without having to worry about the data format. As we live in an age where a wide variety of data is generated in large amounts, there is a demand for platforms that can flexibly accept any data, without the need to prepare a schema, and these have become popular.

This can be achieved using Hadoop, or by using object storage services provided by each cloud service (such as Amazon S3).

While DWH can be considered a competing technology at the methodological level, it is also common to combine data lakes and DWHs, taking advantage of their respective characteristics.

"Data preparation" is harder than analysis (DWH and ETL)

Once a data infrastructure was built using a DWH and data analysis became possible, an unexpected fact became clear: preparing the data needed for analysis often took more time than the data analysis itself.

Data is usually scattered across various locations within a company and is in a wide variety of formats. To analyze the data, it must be brought into the DWH, but preprocessing is required to read the data from various locations and convert the data format to align it.

I think it's quite common that people start out trying to do data analysis, but every time they want to analyze something, they end up doing endless, tedious work like retrieving data from various sources in CSV files and preprocessing it in Excel.

This led to the development of "data integration tool" that could connect to a wide variety of data sources, retrieve data, and efficiently perform data conversion, etc. This product category, which was born as a means of solving the problems of DWH, came to be known as "ETL."​

Unexpected pitfalls of data utilization:
Preparing the data needed for analysis often takes more time than the data analysis itself.

data integration tool are necessary to utilize data in a DWH

Many organizations are now trying to utilize data. Data utilization can be promoted using various methods and technologies, but a common approach is to introduce a DWH and BI tool as a company-wide data infrastructure.

When using a DWH, various data integration needs may arise, and the ability to effectively utilize "data integration tool" (known as ETL, EAI, iPaaS, etc.) can make the difference between success and failure in data utilization. For example,

  • How to bring data into DWH:
    Data exists in a variety of formats across various systems, both internally and in the cloud.
  • Methods for processing data on DWH:
    It may be necessary to process the data before analysis, such as by formatting the data.
  • How to extract data stored in a DWH and use it externally:
    Once a DWH is established as a data infrastructure, there will be a need to utilize the data on the DWH in various external systems.
  • Ways to combine with other technologies such as data lakes:
    For example, if you want to combine the strengths of each, such as receiving various data in a data lake, processing the data from the data lake, and then inputting it into a DWH, you will need a way to link the data lake and DWH (or you will also need this when migrating your data infrastructure).

Doing all of these tasks manually every time something happens would be time-consuming and troublesome. Also, it's not possible to know in advance what data is needed and how to analyze it; in many cases, it becomes clear only after actually analyzing and utilizing the data.

Therefore, in order to utilize DWH data integration tool that achieves the following characteristics is required.

Able to connect to a wide variety of systems and data

You will need the ability to connect to a variety of internal systems, Excel files in various formats used throughout the company, various cloud services that have been introduced, and a wide variety of data and systems.

Sufficiently high processing performance

The data used for analysis is often large, such as data accumulated from the past. It is necessary to be able to process large amounts of data quickly. Simple, convenient tools are not enough; a method that can be used for full-scale analysis is required.

Automation of various processes

There is a need for a method that can be used as an automation tool, such as automating the regular process of acquiring data and inputting it into a DWH, organizing data on the DWH, and, for example, inputting data accumulated on the DWH into machine learning for processing and automatically reflecting the prediction results in business operations.

Ease of use for worksites

When utilizing data, it can be difficult to analyze in advance what is required of an IT system, as this involves repeatedly collecting and analyzing data to gain insights.

Let's say you've discovered something through analysis. You may find that you want to perform additional data analysis, but in order to do so, you need additional new data, or that you need to implement an automated linkage process to turn your findings into business results. This is likely to happen often. If you have to request and wait for IT modifications every time you notice something like this, data utilization will not progress efficiently. However, if you try to do these things manually, you will end up drowning in data chores and will not be able to make progress.

Furthermore, this kind of awareness is more easily achieved by people familiar with the business than by IT specialists, so it is desirable for them to take the initiative in this regard.

Therefore, to successfully utilize data, it is desirable to be able to quickly redesign how data integration on-site, and for the initiative to be led by someone who has a good understanding of the current state and needs of the business and operations.

No-code or low-code tools that allow for flexible development of data integration using only a GUI would enable on-site personnel to quickly resolve such needs and efficiently utilize data.

Related keywords (for further understanding)

  • ETL
    • In the recent trend of actively working on data utilization, the majority of the work is not the data analysis itself, but rather the collection and preprocessing of data scattered around, from on-premise to cloud. This is a means to carry out such processing efficiently.
  • BI tools
    • It is a tool that can aggregate and analyze data along various analytical axes. It is a means of analyzing data and obtaining analytical results, and has a function that allows you to report the analytical results in easy-to-read formats such as graphs.
  • Data Lake
  • Data Mesh
  • Object Storage
  • iPaaS
    • A cloud service that "connects" various clouds with external systems and data simply by operating on a GUI is called iPaaS.
  • No-code/Low-code

DataSpider trial version and free online seminar

"DataSpider," data integration tool developed and sold by our company, also has ETL functions and is data integration tool with a proven track record of being widely used as a means of supporting the utilization of DWH.

Unlike regular programming, development can be done using only the GUI (no-code), without writing any code, and it offers "high development productivity," "full-fledged performance that can serve as the foundation for business (professional use)," and "ease of use that can be used by those in the field (even non-programmers can use it)."
It can smoothly solve the problem of "connecting disparate systems and data," which is hindering not only data utilization but also the successful utilization of various IT technologies such as cloud computing.

We offer a free trial version and hold online seminars where you can try out the software for free, so we hope you will give it a try.

Glossary Column List

Alphanumeric characters and symbols

A row

Ka row

Sa row

Ta row

Na row

Ha row

Ma row

Ya row

Ra row

Wa row

»Data Utilization Column List

Recommended Content

Related Content

Return to column list