The "best" statistician, Mr. Nishiuchi, explains why "desires" and "wit" are necessary to make the most of data analysis

Statistician Nishiuchi Kei is the author of "Statistics is the Strongest Science," which was released in 2013. The series has sold over 500,000 copies in total, and he is currently active on the front lines of business, helping companies utilize data and developing various analytical tools, while continuing to talk about the appeal of statistics in easy-to-understand language. We spoke to Nishiuchi, a specialist in the field of statistics, in detail about his career so far, the appeal of data analysis that can be applied to everyday life such as child-rearing, and the ideal type of talent essential for promoting digital transformation in companies.

The story of statistician Nishiuchi Kei

You have been involved in a variety of activities in the field of statistics. Could you tell us about how you became a statistician?

In high school, I didn't have any particular dreams for the future; I was simply a science type who excelled in math and physics, and rather than wanting to unravel the mysteries of the universe, I was interested in the question, "What is a human being?" So I went on to study biology, but I wasn't able to learn as much about humans as I had hoped. So I changed my focus and became interested in psychology and sociology, and that's when I discovered the appeal of statistics. I had been taking statistics classes since my first year of university, and what I found interesting was learning about its applications. I realized that the field I had thought of as a so-called "liberal arts discipline" was actually based on solid data, and I realized how interesting it was to think about how to make use of the results obtained.

You yourself are going on to medical school, but how does that relate to statistics?

Only recently have departments such as the Department of Data Science emerged where students can specialize in statistics, but at the time there were no such departments. While it is certainly possible to study statistics in the Department of Economics or the Department of Engineering, my interest was in understanding humans through statistics. Therefore, I decided to study biostatistics in the Department of Medicine, a field that studies humans.

What kind of career did you pursue after that?

Within the realm of medicine, I studied statistics in the field of public health. Rather than doctors treating each individual patient, society as a whole identifies the causes and risks of disease and takes measures to address them. Within this field, I became interested in health communications. For example, it is already well known that lack of exercise poses significant health risks, so I researched how to encourage people to develop exercise habits. Fortunately, I became an assistant professor at a university at a young age, but I wanted to use my knowledge in a more practical way to benefit society, so I left my career as a researcher and took on my current job assisting companies with data analysis.

I had the impression that statistics itself has a long history, but it seems that specialized departments were only established recently.

Traditionally, statistics education has been conducted within individual faculties, such as the Faculty of Engineering, Faculty of Economics, and Faculty of Agriculture. Even the same mathematical methods often have subtle differences in how they are thought about and used, and statistical methods are often developed to suit the needs of each field. However, the field I studied, public health, was extremely fascinating, and I learned a lot from using statistics in a "mixed martial arts"-like way. To put it simply, public health is about "doing whatever it takes to make people healthy." Some people have backgrounds in economics, while others specialize in policy science and political science. Experts from a variety of fields, including philosophy, ethics, engineering approaches to medical devices, and even bioscience, are active in the public health field. Being able to learn how to use statistics in a relatively well-rounded way, while immersed in a field like mixed martial arts, was a valuable experience for me.

Is the statistical approach the common language that serves as the rules in mixed martial arts?

Despite differences in background, the idea of evidence-based research is becoming a common language. Randomized controlled experiments are being conducted to randomly divide subjects to find causal relationships, and statistical causal inference is being carried out in areas where randomization is difficult. Ultimately, a trend is emerging in which final judgments are made based on evidence, i.e., scientific evidence including data.

By the way, what kind of child were you as a child? Please tell us about any experiences that you had that have influenced you to this day.

For example, when I played Dragon Quest as a child, there was a rumor going around among my friends that monsters were more likely to appear in the background graphics of forests and rocky mountains than on flat land. So, to test whether this was true, I took a stopwatch and measured how many times enemies appeared. I found that it was indeed the case that monsters were more likely to appear in forests and rocky mountains. In other words, when leveling up, going around rocky mountains efficiently can save time. This may be where my first attempt at data analysis began.

Characteristics of people suited to data analysis

You have now started your own company, Data Vehicle, but what was the background to that?

After leaving my position as a university professor, I personally took on work from companies doing data analysis and training analytical personnel, and gradually accumulated know-how. The most difficult part of data analysis is, above all, processing the data, followed by selecting the method and selecting and discarding variables or features. This is one of the areas where data analysis skills really shine, and I thought that this part could be automated. So, by utilizing my unique know-how to automate these processes, I built a product that would allow anyone to perform data analysis easily and with high productivity, and I founded Data Vehicle Inc. to make this my main business. The product I created at that time is still our company's flagship product, dataDiver.

So you've made a business out of providing tools. In fact, I saw your daily tweets and saw that data analysis is your hobby, and that you see developing tools that lead to automation as your job.

Data analysis is almost like a hobby to me. Writing Python or R code for data analysis feels like breathing, and even when I'm chatting with people, ideas like, "It would be interesting to collect and analyze this data..." pop up instantly. For example, the other day, I was inspired by a casual conversation and decided to analyze which hamburger menu item from a certain chain offers the best value for money. By analyzing the number of ingredients (bun, patty, cheese, etc.) used in each menu item, along with the sales price by time of day, we can create a mathematical model that roughly estimates the cost structure. Looking at the difference between the predicted value and the actual price, we can see whether the chain is intentionally pricing certain items at a lower price than others, or conversely, pricing them at a higher price. In conclusion, it appears that the chain's daytime cheeseburgers were priced to offer the best value for money.

Writing R code is as easy as breathing... it's not an easy thing to do. From your perspective, are there any trends in people who are suited to data analysis and those who are not?

Actually, I've been thinking about this a lot recently, and the conclusion I've come to in one sentence is whether you have "earthly desires" and "wit." Data analysis is essentially a mass of earthly desires, wanting to outsmart your competitors without putting in the effort. People who are simply earnest and talented can't completely shake off the habit of working hard and earnest in all aspects of their work, and don't have many earthly desires, either good or bad, like "taking it easy" or "outsmarting others." If you have such earthly desires, you'll be happy when you realize that focusing your efforts on them will make you more profitable and reduce costs.

It's like, "I've finally found a weakness."

You're right. However, while data analysis can reveal weaknesses, some kind of action must be taken to actually exploit them. By "taking action," I mean trying something new that you haven't done before, or stopping something inefficient that you've been doing up until now. This idea truly lies in the world of "wit," which can be neatly rephrased as creativity. I believe that people who combine these "earthly desires" and "wit" are the ones who can apply data analysis to their work.

The important thing is that these two things are in place.

Ideally, you'd have both, but it's fine to partner with others to fill in any gaps. In that case, however, you need to be mindful of communication loss with the people who can help you fill in your gaps. This isn't just about worldly desires and wit. It's generally said that data science skills require all three elements: statistical (mathematical), computational (IT), and human (business). It may be difficult for one person to master all three, but mastering only one element and ignoring the others won't get the job done. For example, even if someone is good at math or IT, if they can't understand what the "human" business side is saying, the communication loss will be so great that it will become a serious bottleneck for a team trying to create value using data science.

Are there few people in companies who possess both qualities?

Recently, we have been offering DX promotion support programs as part of our service menu. We do not simply provide an analytical environment, but also assist with data inventory, clarifying the vision among management, and establishing the necessary meeting bodies for decision-making, all the way to the establishment of processes leading to decision-making flows. Even large companies are short of personnel who can promote DX and create a competitive advantage from data and digital technology. However, by participating in the training menu we offer, it is possible to discover personnel suited to such activities and develop their strengths. Rather than going to the trouble of acquiring personnel from outside the company who can immediately contribute, it is likely to be significantly more cost-effective to find people within the company who are suited to data analysis.

Specifically, what kind of talent is often left unseen within companies?

In fact, a surprising number of liberal arts students have experience with data analysis in their graduation theses. It's not uncommon for students in psychology, economics, sociology, and education to design their own surveys, collect data, and analyze the results in their graduation theses. However, human resources departments at most Japanese companies are unaware of the research that students have conducted in their graduation theses. In fact, it's not uncommon for companies to outsource this work, spending hundreds of thousands of yen, even though they likely have in-house staff with experience using programming languages like R and statistical analysis software like SPSS. In business, whether in marketing or management, analysis often focuses on people, and having mastered the fundamental theories for understanding humans is a major advantage for such professionals. Even those with data analysis experience through graduate school often end up assigned to completely different tasks once they start work, a common mismatch.

The appeal of statistics and the usefulness of tools

What do you think about the appeal of statistics and data analysis?

Business success is an extremely complex phenomenon, but the fascinating thing about statistics is that it allows us to hack these phenomena. In reality, there is a school of thought that says not to worry about rumors and hypotheses, but just like the Dragon Quest example mentioned above, when you actually analyze data in the real world, you can see that even a slight change in method can clearly improve efficiency. There are actually many hidden mechanisms for hacking the world, giving you an overwhelming advantage over those who are unaware of this. Being able to recognize these mechanisms before your competitors will be a major advantage for your company. Isn't it fascinating to think about how to hack and discover things that only geniuses can find?

So tools like those provided by your company are useful when it comes to efficiently hacking into this world.

While some people who enjoy data analysis may be willing to do it themselves, those implementing the system should carefully consider whether it's more cost-effective to continue paying high labor costs or to have the tool do the work for them. It would be more rational to have the personnel you've hired or trained spend their time on more advanced tasks. Rather than relying on experts for every data analysis you need, such as a simple product or promotional idea, it's often more effective and efficient for those with the relevant business knowledge to use convenient analytical tools themselves. Rather than exhausting yourself with the data analysis process itself, you should focus your efforts on determining what tasks to set and how to use the analytical results to implement creative initiatives. We believe that developing the right environment and tools to reduce the workload associated with data analysis will likely be key to advancing digital transformation. In addition to the aforementioned dataDiver analysis tool, we also offer a data preparation tool called dataFerry, which utilizes DataSpider International, provided by Saison Technology. There are a variety of systems that serve as infrastructure for managing data, and HULFT and DataSpider International will likely become even more important solutions in the future as they bridge the gap between them. One of our goals is to become a global software company. In terms of global software companies used in business, the United States is overwhelmingly strong, and while there are also companies from Germany, Israel, and Scandinavia, Japan is not very successful. In that sense, HULFT has a significant presence as one of the few role models, and I feel that there is much to learn from a managerial perspective.

As digital transformation progresses, we are moving away from closed environments within individual companies and into an era where information is increasingly connected. In order to hack the world, we believe we need a data integration platform that transcends the boundaries of industries, business types, and companies. What are your expectations for HULFT Square as the platform for this?

Even in today's world where the importance of DX management is recognized, collecting and processing data remains a significant burden in data science work. HULFT Square, which can promote seamless data integration from legacy business systems to the latest cloud environments (regardless of business or company boundaries), will be a boon for many companies in their DX efforts.