Symmetric key cryptography / DES / AES (Advanced Encryption Standard)

Glossary

"Symmetric key cryptography / DES / AES (Advanced Encryption Standard)"

This glossary explains various keywords that will help you understand the mindset necessary for data utilization and successful DX.
This time, we will explain about symmetric key cryptography, a technology that supports "safety and security" in IT, and AES.

What is symmetric key cryptography / DES / AES (Advanced Encryption Standard)?

Symmetric key cryptography is a type of cryptography that uses a common cryptographic key for encryption and decryption. It is a technology that has been used since ancient times to safely send and receive confidential information by sharing common secret information, and it is also a traditional cryptographic technology that is distinguished from public key cryptography, which has emerged in recent years.
AES (Advanced Encryption Standard) is a symmetric key encryption algorithm that is widely used around the world and was standardized in 2001 by the National Institute of Standards and Technology (NIST) after a public call for proposals.

Examples of encryption and decryption

You may not have heard of the term "symmetric key cryptography," but what people generally refer to simply as "cryptography" is almost always symmetric key cryptography. It is not simply called "cryptography" because it is a more traditional form of cryptography that is distinguished from the recently introduced "public key cryptography." So what exactly is "cryptography" and what does it involve?

Example of an ancient code: Caesar cipher

Encryption technology has been used since ancient times, long before the IT era. It was used in military and diplomatic affairs to prevent the leakage of confidential information, ensuring that messages would be transmitted in a form that would make it impossible to know what was written on them, even if orders or other documents fell into the hands of the enemy.

One of the most famous codes used in ancient times is the Caesar cipher. For example, suppose you have the following cipher text:

ZWPWOLEZAN EO CKKZ

It's hard to understand what's written here, but let's try to "shift the alphabetical order by four letters" on this mysterious alphabet string. For example, "If it's A, it becomes E," and if it's Z, it becomes D.

DATASPIDER IS GOOD

What at first glance seemed to be written incomprehensible words became instantly readable. There is some talk that the real Caesar (of ancient Rome) may have used a more complicated code, but this type of code is generally called the "Caesar cipher."

Substitution cipher

Now, the Caesar cipher can be summarized as follows.

Encryption algorithm: Shift the alphabet by a specified number of characters
Encryption key: Shift by 4 characters

In order to decrypt ciphertext and return it to the original plaintext, you need to know "what algorithm was used" and also "what parameters (encryption key) were used to encrypt it" (in the case of the Caesar cipher, "how many characters were shifted").

The Caesar cipher can also be generalized as an encryption algorithm. Specifically, it can be considered a very simple example of a method of "replacing one character with another," and encryption algorithms that use character substitution are sometimes called "substitution ciphers."

For example, the following is an example of a substitution cipher that is a more generalized version of the Caesar cipher.

Encryption algorithm: Plain text characters are replaced with other characters in a specific procedure to create cipher text.
Encryption key: For an alphabetic document, a substitution table for "26 alphabetic characters" to "26 alphabetic characters"

Substitution ciphers can be even stronger because they do not require every single character to be substituted.

Encryption algorithm: Replace any "two-letter alphabet sequence" with a "two-letter alphabet sequence" using a specified procedure
Encryption key: Substitution table for "26 alphabet characters x 26 alphabet characters" to "26 alphabet characters x 26 alphabet characters"

Codebreaking example (frequency analysis)

However, this does not mean that encryption in this way is safe and cannot be deciphered. Simple substitution ciphers can be easily deciphered, and even highly complex substitution ciphers established at the national level (such as Nazi Germany's Enigma cipher or the Japanese Navy's D cipher) can be deciphered.

Next, let me show you how a simple substitution cipher can be easily deciphered.

For example, suppose plain text written in English is encrypted using a substitution cipher, which replaces each letter of the alphabet with a different letter. It is known that there is a bias in the frequency of alphabetic characters in English text. For example, it is known that "Q" appears rarely, "E" appears frequently, and "T", "A", "O", "N", etc. are also common. Then,

Counting the frequency of letters in the ciphertext
The most common letter is probably "E"
The next most common character is probably one of the more common characters, such as "TAON."

Once you find a combination that can be deduced in this way, it is surprisingly easy to decipher. In "The Dancing Men," a Sherlock Holmes story written by Conan Doyle, there is a story in which it is discovered that "graffiti of dancing men left at the crime scene" is actually a coded message, and the code is deciphered in exactly this way. The following is an excerpt from Aozora Bunko.

⇒Translated by Mikami Otokichi, revised by Okubo Yuu, THE ADVENTURE OF THE DANCING MEN by Arthur Conan Doyle (Aozora Bunko)

In other words, this doll with its hands raisedis the letter E. As you know, E is the most commonly used letter in the English alphabet, and it is so common that it can be found in any short sentence. Of the fifteen letters in the first message, four are the same, so it makes sense to call them E.

Once one letter is determined, the rest of the alphabet can be inferred in the same way.

If my reading is correct and this is a message from someone who was once intimate with the lady, then the combination of the E's on both ends and the three letters in the middle may indeed refer to the name ELSIE, "Elsie." Closer inspection reveals that this combination appears three times at the end of the message. If so, this must be a message addressed to Elsie. Thus I found L, S, and I. But what is it trying to convey? Only four letters precede the name Elsie, ending with an E. It must mean COME. I considered other four-letter names ending in an E, but none suited the situation. Then I found C, O, and M, so I tried again with the first message.

This work was written in 1903, during the heyday of the British Empire, and before computers. However, considerable research had already been done into codes and code-breaking, and even writers of mystery novels knew that simple substitution codes could be deciphered based on the frequency of letters.

Cryptanalysis example (brute force)

Ciphers can also be deciphered by "brute force." For example, the Caesar cipher only allows for variations in the number of characters that can be shifted, from "1 to 25 characters." In that case, by "trying all 25 patterns," a readable sentence will emerge from among them, allowing the cipher to be deciphered. In this way, ciphers can be deciphered by brute force by examining all possible patterns of the encryption key.

Especially now that we have computers, this kind of "brute force" cryptanalysis has become easier. If the encryption key is only 16 bits long, all you have to do is try all 65,536 combinations, which means it can be decrypted in most cases. Even with 32 bits, there are 4,294,967,295 (4.2 billion) combinations, so if you can brute force 10,000 combinations per second (if a 5GHz CPU can perform one calculation within 500,000 clock cycles), you can exhaust all of them in four days. What's more, we now live in an age where GPUs, which are capable of massive parallel calculations, can be used for cryptanalysis, making brute force attacks increasingly easier.

Public encryption algorithms

Now, let's recap: Symmetric key cryptography is an encryption algorithm in which the encryption key used to encrypt plaintext is the same as the key (or secret information) used to decrypt ciphertext back to plaintext.

Plaintext → (encryption algorithm + encryption key) → ciphertext
Ciphertext → (encryption algorithm + encryption key) → plaintext

For example, in the Caesar cipher, you would "shift four letters," or in a substitution cipher for each letter of the alphabet, you would "make this doll correspond to E."

A cryptographic algorithm is secure if it means that someone who does not know the encryption key cannot guess the plaintext even if they know only the ciphertext. For example, a simple substitution cipher can be deciphered by analyzing the frequency of character occurrences, so it is considered an insecure encryption algorithm.

In other words, to encrypt data and deliver it to the recipient safely and securely,

Use a "secure encryption algorithm"
- There are no exploitable weaknesses and it cannot be decrypted by brute force.
The process of safely sharing an "encryption key" (secret information) between the sender and receiver to prevent leaks, etc.

It is clear that the following is necessary. Of these, I believe that most of you are already taking care to handle encryption keys (such as passwords) with care. All that remains is to use a "secure encryption algorithm." One option to consider in this situation is the "Advanced Encryption Standard (AES)," which is widely used around the world.

AES is a technology that can be called the "global standard symmetric key encryption." Another feature is that it is a cryptographic algorithm in which "all details about the cryptographic implementation are publicly available." You might think that "keeping the encryption mechanism secret" is safer than "making all cryptographic technology publicly available," but in fact, this is not the case.

Why use a "public encryption algorithm"?

For a long time in history, the mainstream approach to security was to keep the encryption algorithm itself secret. This was especially common when encryption was primarily used in military and diplomatic affairs, but now, with the spread of IT, publicly available encryption algorithms are more commonly used.

Cryptographic technology is no longer primarily used to prevent enemy decryption in diplomatic or military situations, but is now used as a means of safely and securely delivering data in private economic activities.We are now in an era where "safe and secure cryptographic technology that anyone can use" is important as a foundation that supports private economic activities, rather than as a top-secret means of protecting state secrets.

Old encryption technology
- A technique used in war and diplomacy, and in the world of state secrets and espionage.
Current cryptography
- This technology is expected to play an important role as a means of "delivering data safely and securely" in the private sector's IT utilization and as a foundation for social and economic activities.

As research into cryptography advances and computer computing power improves dramatically, it is becoming increasingly difficult to maintain security. By making the technology public, the knowledge of cryptography researchers from around the world can be pooled, increasing the likelihood of achieving advanced technology. Furthermore, with publicly available technology, cryptography researchers from around the world can continue to research weaknesses on a daily basis, and the fact that no fatal weaknesses have been found should itself provide evidence that the technology is secure.

Of course, there is a risk that someone (or some evil country) has noticed a weakness but is keeping it a secret, or that a "secret weakness" has been embedded in the first place, and anyone who knows the secret can decrypt it at will (there is actually a "conspiracy theory" about this), but nowadays public encryption algorithms are often used.

DES (Data Encryption Standard) cipher

The first such "publicly available encryption algorithm for global use" was the Data Encryption Standard (DES), standardized in 1976 by the National Bureau of Standards (NBS), an agency of the U.S. government.

1972: Report on the need for standard cryptography

It all began in 1972, when the use of computers was still in its infancy, when the US government issued a report on the "need for computer security."

Although it was still early days before computers would become widespread and play an active role in society, research results were submitted to the government stating that, as computers become more widely used in society in the future, it would be important to ensure computer security as the foundation for safety and security. Based on this, standard cryptographic technology would be created as the foundation for safety and security in society.

The Apollo moon landing took place in 1969, and the Vietnam War ended on April 30, 1975, during the height of the Cold War between the United States and the Soviet Union. Even back then, the need for safe and secure standard technology to be the foundation of society was already recognized.

1973: The requirements specifications were made public and solicited, but none were adopted.

The National Bureau of Standards (NBS) established the requirements that cryptography must meet in order to be adopted as an official Federal Information Processing Standard (FIPS) in the United States, and then publicly solicited applications for cryptographic technologies that met those requirements. However, no technology that met the requirements was submitted, and all were rejected.

1974: A second public competition was held, and IBM's proposal was selected.

The following year, a public competition was held again, and the "Lucifer cipher" developed by IBM was adopted. DES was then created based on this. (As I will explain later, the fact that IBM's proposal was not adopted as is would later have a lasting impact.)

1976: DES is born

DES is born. DES is an abbreviation for Data Encryption Standard, which translates to "data encryption standard." It is a block cipher with a key length of 56 bits. This advanced encryption technology became available to everyone in the private sector.

It is composed only of operations that are easy to implement, such as XOR operations, bit shifts, and table lookups, and is characterized by its high processing speed and ease of implementation in hardware.

Doubts about DES encryption and its surprising consequences (Important lessons for building a secure foundation)

The DES technology that IBM proposed in response to the public tender was not adopted. DES was a modified version of IBM's proposal, and it was revealed that the NSA (the US intelligence agency, the organization to which the famous Snowden belonged) was involved in the process. Furthermore, there were specific incidents that raised suspicions, such as the following:

IBM wondered if the 56-bit key length was too short, but the NSA convinced IBM that 56 bits was sufficient and would be more convenient for civilians, so they reduced the key length.
The design of the encryption had a "mysterious design element" (S-BOX). S-BOX was not included in IBM's original proposal, and appeared after the NSA's involvement, and the reason for its design was unclear.

This led to widespread suspicion (or conspiracy theory) that DES contained a backdoor that allowed only the NSA to easily decrypt the encryption.

I believe this incident is a significant lesson that makes us think about what we must do and what we must not do in order to create something that the public at large can trust as safe and secure.

Cryptography research advances dramatically based on DES (DES becomes the standard for safety and security)

Before the development of DES, research and use of specific cryptographic techniques was almost exclusively limited to military and intelligence efforts, and was a specialized field.

The emergence of DES, an open technology with all details made public and freely available, led to active development and use of cryptographic technology. Even ordinary universities began to conduct cryptographic research based on DES. By becoming a standard technology, it became the very infrastructure for cryptographic technology research, and people began to think, "If no problems are found when it is tested by cryptography researchers around the world, it is probably reasonably secure."

The true meaning of DES's questionable design was revealed in 1990

In 1990, the true meaning of DES's "mysterious design" was finally revealed.

In 1990, a paper was published on a general attack method known as "differential attack," which could be widely used with symmetric key cryptography. This resulted in a series of incidents in which a number of cryptographic algorithms were severely damaged by the differential attack and became unsafe to use. This was during the bubble era, when Japan was enjoying prosperity, but unfortunately, many of the domestic cryptographic technologies proposed by Japan, which thought it was a world leader in technology (such as FEAL, developed by NTT laboratories), also ended up in disappointment.

While many encryption technologies have been defeated by differential attacks, DES, a long-standing encryption technology, has not suffered fatal damage from differential attacks. In fact, IBM had discovered a differential attack method in 1974, but it had only been kept secret as a national secret.

The "mysterious design of S-BOX" that has been the subject of DES suspicion was that it was designed to be resistant to differential attacks. It was a design that was made with the desire for everyone in the world to use secure encryption technology. However, this ultimately gave rise to conspiracy theories and led to a lack of trust in DES. This conclusion makes us think about what safety and security should be.

What is AES (Advanced Encryption Standard)?

AES (Advanced Encryption Standard) was developed in response to the inadequacy of DES, and is currently the most widely used symmetric key encryption algorithm in the world.

The 56-bit key length of DES was not strong enough against brute force attacks, and there were already cases of successful brute force cryptanalysis of DES in the 1990s. Furthermore, there were also cases where the defensive power was weakened by differential and linear attacks.

Therefore, DES was extended by "Triple DES (3DES)", which increases the encryption strength to an effective 112-bit equivalent by performing the DES encryption process three times, and the encryption algorithm "AES" was adopted as its successor technology after public submissions from around the world.

Eliminates major problems with DES

56-bit encryption is no longer strong enough, and new attacks such as differential and linear attacks have been addressed.
Dispelling suspicions caused by the lack of transparency in the process of creating DES

Everything is transparent and open

The public offering was led by the National Institute of Standards and Technology (NIST), but "the entire process from the public offering to the decision-making process was open and transparent" to avoid any suspicions.

The technology itself has been made public, with the source code for the reference implementation in C++ and Java being made available as open source, and the reasons for implementing it that way and the selection process have all been made public.

European proposal adopted

From among applications from around the world, Rijndael, created by researchers at the Catholic University of Leuven in Belgium, was selected (October 2000), and was officially announced as AES in March 2001. In other words, AES wasnot created in the United States.

Encryption strength

You can choose from 128-bit, 192-bit, or 256-bit key lengths. The block length (unit of encryption) is 128 bits. Rijndael itself allows for variable key and block lengths, allowing for more flexible bit lengths, but the AES public offering standards fixed the above as AES.

Free and open to anyone around the world

There are no issues with copyrights, and the technology itself is open, so anyone in the world can use it freely and free of charge.

Implementation and processing can be done efficiently

It was also chosen because it was highly evaluated for its efficient implementation and processing compared to other methods.

Easy "safety explanation" for users

Because this is the history behind the creation and use of this encryption algorithm, when asked to explain the security of the system, it is sufficient to say, "We use the world standard AES."

Currently (at the time of writing) unscathed by the attack

AES is being checked daily for weaknesses, and new methods of attacking the encryption are being discovered, but so far no problems that would reduce the strength of the encryption have been found (although weaknesses that could serve as a foothold for attacks have been found).

Alternatively, the fact that AES has been in existence for over 20 years and has not found any problems even after being verified by researchers and research institutions around the world is seen as evidence that AES is "safe and secure."

The significance of "proven options" is heavier than you might think

Given this history, and the fact that it is used around the world, including by governments, AES is well-suited to being adopted as a data encryption technology.

Not only is it considered to have a reasonable level of security, but it also has the advantage of having a well-developed ecosystem, with AES-compatible software being widely available in the world. Another advantage is that it is easy to explain the reason for its adoption by saying, "We use AES, the global standard."

Let's consider the difficulty of ensuring "safety and security" with another example.

The fact that something has become a standard technology and that the relevant aspects have already been established greatly reduces the amount of work we have to do ourselves when it comes to dealing with various risks related to the technology, or the many problems that arise when actually operating it as an IT system. This fact carries more weight than we might expect. Let's consider this with another example related to safety and security, which is easier to imagine.

AES is a technology (block cipher) that encrypts "blocks of data" such as files, but it does not alone provide all the elemental technologies needed to create an environment in which data can be used safely and securely. For example, when sending data from a business system at the head office to a business system at a branch office, or from company A to company B, an environment in which data can be transferred safely and securely is also required.

Many people may think, "It should be easy to make, since it's just a matter of transferring files," but when you consider the possibility of line failures and data corruption along the communication path, or the possibility of related systems failing and halting processing midway, if the application is not something that can be resolved by simply "hoping that it will arrive if I send it" (which is probably the case for the majority of IT use in business), it quickly becomes a requirement that cannot be easily achieved.

It is actually a common phenomenon that files (data) can be corrupted due to communication errors during transfer.
- Simply requesting that a file be sent ensures that the file arrives at the destination without being corrupted or distorted.
- It is necessary to understand what problems may occur during transfer and to prepare the necessary recovery and monitoring functions.
Security concerns on the transmission path
- There are cases where it is necessary to achieve a level that can pass security certification systems, such as proper encryption and not leaving unnecessary temporary data.
Adaptable to a wide variety of environments
- It may be necessary to support a variety of environments, including mainframes, Linux, various UNIX systems, Windows, and cloud computing.
- In particular, it is often difficult to implement a transfer process that can be used smoothly between vastly different environments, such as the cloud and mainframe.
- Furthermore, there may be a need to continually respond to OS upgrades and new environments such as new cloud services.
Developing an operating environment and surrounding ecosystem based on actual operations
- In addition to the transfer function, peripheral functions such as logging, user permissions, and their management functions must also be in place (although this is often forgotten).
- Ability to deal with difficult situations that often arise during operations. For example, when line trouble or system failure occurs, you need to understand and implement the logs and functions that are practically necessary in situations such as "which files were delivered properly?" or "disputes about who was responsible for sending or not sending."
Taking these factors into consideration, it takes time and effort to convince customers that the product is made safely and securely.
- It is necessary to explain to the customer that "the system is OK" and assure them, but this is not easy. Just testing and checking every time a system is released takes time and effort.

Even something as simple as "simply transferring a file" can quickly become like this if you consider the need to establish a "file sharing infrastructure" that can be used reliably to the point where it can be entrusted with the task at hand. So what should we do? Similarly, it would be realistic to adopt a product that is "standard technology" and "where related aspects have already been established."

Our company's file integration middleware, HULFT, is an example of an existing, standard product that can eliminate many of the problems mentioned above all at once. It has been used for many years as a means of linking core mission-critical system, core system in fields that require a high level of safety and security, such as mission-critical system, core system, so in Japan it is sometimes enough to explain to customers that "we use HULFT as the platform."

⇒File Linkage | Glossary

Understand and adopt proven products

Some people may have had the impression that cryptography ensures safety and security by keeping things secret and hidden, but the idea is now to adopt "public technologies that are standard in that field" and make good use of the track record and trust that has been built up by making those technologies public.

Of course, it is important to understand that there are risks involved in adopting "technology that everyone uses," such as if AES has an unknown weakness that is being secretly exploited, but it is also important to consider that adopting proprietary technology also carries risks, and that dealing with these is not easy.

Technology Selection
- Adopting the standard technology AES as the foundation for safety and security
It is important to be able to explain why
- Be able to explain why adopting AES is appropriate
To actually achieve safety and security
- The use of passwords used for encryption is the key to ensuring safety and security, so pay close attention to it.

In other words, it's not about "making something" but about "making good use of what already exists." In many cases, this is expected to lead to greater safety and security, reduce the amount of things that need to be made, and reduce the risks and hassles that you have to bear yourself. It will also often be easier to explain why you selected that technology.

This should be extremely beneficial for IT engineers. If you can properly understand why you should adopt AES (and HULFT), I believe you can use it as an effective and advantageous card when proposing and implementing IT systems.

Related keywords (for further understanding)

MFT
- Although file integration can be technically challenging, there are already methods available. By using MFT, a highly sophisticated file integration platform can be smoothly established.
File Linkage
- If you are wondering why we use "file transfer" as a means of collaboration, please see here.
EAI
- This explains how important and meaningful it is to be able to data integration from various systems.
iPaaS
- A cloud service that "connects" various clouds with external systems and data simply by operating on a GUI is called iPaaS.

Interested in HULFT?

Please try out HULFT, the de facto standard and pinnacle of Japanese MFT products. It has an overwhelming track record, meeting the needs of financial institutions, which have strict requirements for IT systems, as a platform product.

HULFT: The de facto standard for MFT

In addition to supporting both new and old operating environments, from mainframes to Linux and Windows, we are also working to enhance functionality that will be useful in the cloud-native era, such as file integration with cloud services (supports transfers with major clouds such as Amazon S3) and support for container technology.

It also has the functionality to efficiently handle the various processes that inevitably occur before and after file transfer is also used as a means to combine new and old IT technologies, such as by freely linking mainframes and clouds without the need for engineers who are knowledgeable about both.

HULFT is a long-standing product with a long track record, but it is also being adapted to current standard technologies and IT environments, such as supporting AES encryption.It also supports transfers with object storage such as Amazon S3, and is compatible with container environments, allowing it to be used naturally with cloud-native architectures based on microservices, and it continues to be updated to keep up with the changing times.

The de facto standard for Managed File Transfer (MFT): Latest version

Learn about file transfer mechanisms through HULFT product introduction and online seminars