Transfer learning

Glossary

"Transfer learning"

This glossary explains various keywords that will help you understand the mindset necessary for data utilization and successful DX.
This time, let's think about how learning is done in machine learning, which is currently attracting a lot of attention in the use of IT.

What is transfer learning?

Transfer learning is an attempt to use existing machine learning capabilities to apply them to other applications.
In deep learning (neural networks), transfer learning is the process of adapting an existing trained model to a different task by removing the output layer, adding a new output layer, and training only the output layer.

A method for effectively reusing existing learning results

(At the time of writing) when people talk about AI, they are usually referring to machine learning. However, the problem with utilizing machine learning is how to prepare the trained model needed for your purpose.

Please also see here for more information on machine learning.
⇒Machine Learning | Glossary

You might think that if necessary, it's fine to just let it learn and create it, but since it's often a lot of work to prepare a huge amount of training data and train it from scratch every time you want to do something new, there are situations where other, more efficient methods are desired.

Therefore, instead of learning from scratch, researchers have begun to reuse existing trained models to solve other tasks that are needed by the researchers themselves. The well-known "fine-tuning" is one example of such an approach.

⇒ Fine Tuning | Glossary

If you're unfamiliar with the more widely known terms "machine learning" and "fine tuning," see above.

The difference between "fine tuning" and "transfer learning"

The difference between this and fine tuning is somewhat unclear, and fine tuning itself is sometimes considered to be a type of transfer learning in a broad sense. Here, I will introduce transfer learning as an approach that takes a different approach from fine tuning, or as a way of thinking about it.

Fine-tuning involves preparing new training data for an existing trained model, conducting additional training, and "making changes to the existing trained model itself" to adjust its capabilities and adapt it to a new application (see the article above).

In contrast, transfer learning takes a different approach from fine-tuning and aims to reuse existing trained models.

Do not modify the existing trained model itself

Fine-tuning can have a negative impact on what has already been learned when teaching something new. For example, if a model that has already been trained to distinguish between "apples" and "mandarins" is given the ability to distinguish between "green apples," the model's already acquired ability to distinguish between "apples" may deteriorate.

To avoid such problems, transfer learning tries to reuse existing learning results by not making (too many) changes to existing trained models. What it does is to change only the "outside" of the existing trained model.

Fine-tuning: Adjusting the existing neural network itself through retraining
Transfer learning: leaving the existing neural network untouched and adjusting it externally

Neural network example

Neural networks (NNs), which are widely used today, generally have a layered structure. There is an input layer where data is input from outside, and each layer is connected to the next layer, where the data is processed and passed on to the next layer, and finally the data is output from the output layer.

This is a rough illustration, but deep learning has a multi-layer structure like this (the "deep" in deep learning means multi-layered).

↑Output layer where the processing results are
xxxxxxxxx
xxxxxxxxx
XXXXXXXXX ↑Data is processed layer by layer
xxxxxxxxx
xxxxxxxxx
↑Input layer where data is input

Data flows from the input layer to the output layer. The neurons (circles) in the diagram above are connected to each other like a spider's web between the previous and next layers.

Although we won't go into detail in this article, "learning" means that each neuron adjusts the weights (parameters) of the connections between neurons so that it can return an appropriate output for the input. When using a trained model, data is input from the input layer, processed at each layer, and then input to the next layer, with the processed results being output from the final output layer.

Apologies for the rough illustration, but the following shows the difference between "fine tuning" and "transfer learning" in terms of the extent to which the model is retrained using additional training data. The "●" indicates the extent to which the model is retrained.

Fine Tuning:

The output layer where the results are obtained
●●●●●●
●●●●●●
●●●●●●
●●●●●●
●●●●●●
The data input layer

Although it may suppress learning in neurons close to the input layer, it forces re-learning over a wide range of neurons.

Transfer learning:

The output layer where the results are obtained
●●●●●●
xxxxxxxxx
xxxxxxxxx
xxxxxxxxx
xxxxxxxxx
The data input layer

Only the output layer is removed (or not removed), a new output layer is added, and that part is trained, but the other layers are not changed.

Fine-tuning basically targets the entire neural network that has already been trained, and modifies the existing trained parameters with new data. By continuing the training that has been done so far with additional data, the existing acquired capabilities themselves are modified.

In contrast, transfer learning aims to "repurpose already acquired capabilities" without touching existing learning results. By "re-training only the output layer" or "adding a new layer outside the existing neural network and learning only the new layer," the goal is to avoid touching the already trained neural network and to "make it possible to do something different from before" by adjusting the new layer added outside.

However, even with fine tuning, there are cases where relearning on the output layer side is the focus and relearning on the input layer side is suppressed, making it difficult to clearly distinguish between the two.

The output side of a neural network is the part where adjustments are made to a specific output, and it is thought that the essence of learning is not often present. Therefore, it is easier to imagine it as the intention to remove only that part, and "reactivate its capabilities" for a different purpose while maintaining the existing "essence of the learning results."

By avoiding altering the essence of abilities, it is possible to avoid unintended negative effects caused by additional learning. This is the idea of "repurposing" existing abilities for other uses without changing their essence.

The "just reuse" approach

The idea of "repurposing something that has already been completed without making any changes and making use of its good points" is an approach that can be seen elsewhere as well.

For example, if you want to use machine learning in your work to achieve results, in addition to the approach of modifying and improving the performance of the existing "trained model" itself, you can also try to achieve results by thinking about what capabilities the machine learning you have actually created has and how it might be useful, and then "devising ways to use it in business."

This approach can also be considered a kind of transfer learning approach in that it attempts to effectively utilize existing models as they are. (When utilizing machine learning, this type of trial and error on the business side is generally an important point - see the article "Machine Learning").

Thinking beyond machine learning

This idea of "making good use of what already exists" is also widely seen in the use of IT. It is not about "creating something new" or "modifying and changing" but rather about "making use of what already exists, adapting it to suit your own purposes and using it as is."

For example, in IT system development, rather than developing a new system or modifying (customizing, etc.) existing software, one approach is to make good use of packaged software or cloud services that are provided as they are.

Developing a system from scratch in-house
Customize package software to suit your company's business needs and change its functions
By using package software and cloud services as they are and linking them with external IT using "connecting" technologies such as "EAI" and "iPaaS," you can tailor them to your needs.

Whether it's IT system development or the use of machine learning, it's important to develop things from scratch when the time comes, but making good use of what already exists is an approach that is likely to become increasingly important in the future.

In the cloud era, it will no longer be possible to change the functions of AWS or kintone to suit your company's needs. The key to success will be to simply repurpose what is already there and use it effectively.

"Connecting" technology that utilizes existing assets and services

If the service itself cannot be changed, or if machine learning is used as is, it is necessary to create a situation where existing technologies can be used effectively. From now on, it will be important to be able to assemble IT that meets our needs through ingenuity outside the service. To do this, we need to be able to freely link various IT assets such as existing systems, data, and cloud services, and combine them to suit our needs.

Please utilize "connecting" technology

There are ways to efficiently develop these various "integration processes" using only a GUI. These are "connecting" technologies known as "EAI," "ETL," and "iPaaS," such as "DataSpider" and "HULFT Square." By utilizing these, new and old systems can be integrated smoothly and efficiently.

Can be used with GUI only

Unlike regular programming, there is no need to write code. By placing and configuring icons on the GUI, you can achieve integration with a wide variety of systems, data, and cloud services.

Being able to develop using a GUI is also an advantage

No-code development using only a GUI may seem like a simple compromise compared to full-scale programming. However, being able to develop using only a GUI allows on-site personnel to proactively work on cloud integration themselves. On-site personnel are the ones who know the business best.

Full-scale processing can be implemented

There are many products that claim to allow development using only a GUI, but some people may have a negative impression of such products as being too simple.

It is true that things like "it's easy to make, but it can only do simple things," "when I tried to execute a full-scale process it couldn't process and crashed," or "it didn't have the high reliability or stable operating capacity to support business operations, which caused problems" tend to occur.

"DataSpider" and "HULFT Square" are easy to use, but also allow you to create processes at the same level as full-scale programming. They have the same high processing power as full-scale programming, as they are internally converted to Java and executed, and have a long history of supporting corporate IT. They combine the benefits of "GUI only" with full-scale capabilities.

No need to operate in-house as it is iPaaS

DataSpider can be operated securely on a system under your own management. With HULFT Square, a cloud service (iPaaS), this "connecting" technology itself can be used as a cloud service without the need for in-house operation, eliminating the hassle of in-house implementation and system operation.

Related keywords (for further understanding)

Machine learning related keywords

Keywords related to Generative AI/ChatGPT

Keywords related to data integration and system integration

EAI
- It is a concept of "connecting" systems by data integration, and is a means of freely connecting various data and systems. It is a concept that has been used since long before the cloud era as a way to effectively utilize IT.
ETL
- In the recent trend of actively working on data utilization, the majority of the work is not the data analysis itself, but rather the collection and preprocessing of data scattered around, from on-premise to cloud. This is a means to carry out such processing efficiently.
iPaaS
- A cloud service that "connects" various clouds with external systems and data simply by operating on a GUI is called iPaaS.

Are you interested in "iPaaS" and "connecting" technologies?

Try out our products that allow you to freely connect various data and systems, from on-premise IT systems to cloud services, and make successful use of IT.

The ultimate "connecting" tool: data integration software "DataSpider" and data integration platform "HULFT Square"

"DataSpider," data integration tool developed and sold by our company, is a "connecting" tool with a long history of success. "HULFT Square," a data integration platform, is a "connecting" cloud service developed using DataSpider technology.

Another feature is that development can be done using only the GUI (no code) without writing code like in regular programming, so business staff who have a good understanding of their company's business can take the initiative to use it.

Try out DataSpider/ HULFT Square 's "connecting" technology:

There are many simple collaboration tools on the market, but this tool can be used with just a GUI, is easy enough for even non-programmers to use, and has "high development productivity" and "full-fledged performance that can serve as the foundation for business (professional use)."

It can smoothly solve the problem of "connecting disparate systems and data" that hinders successful IT utilization. We regularly hold free trial versions and hands-on sessions where you can try it out for free, so we hope you will give it a try.

Free product introduction seminar

Why not try a PoC to see if "HULFT Square" can transform your business?

Why not try verifying how "connecting" can be utilized in your business, the feasibility of solving problems using data integration, and the benefits that can be obtained?

I want to automate data integration with SaaS, but I want to confirm the feasibility of doing so.
We want to move forward with data utilization, but we have issues with system integration
I want to consider data integration platform to achieve DX.

PoC Program | HULFT Square

Transfer learning