In this blog post, we’ll explore the meaning and value of data mining—one of the leading methods for analyzing big data—as well as its real-world applications and limitations.
The term “big data” has been circulating everywhere for quite some time. A good example of big data’s popularity is that, at one point, a Google search for “Big data” returned approximately 280 million results. Does that number really give you a sense of its scale? That figure was more than double the number of search results for “Iron Man,” the title of a globally popular movie. To put it somewhat dramatically, we can say we are living in the age of big data. As such, most people have likely encountered big data through various media. However, in many cases, the utility of big data does not feel tangible. This is often because big data has not been analyzed in an appropriate manner. Therefore, in this article, we will explore data mining, a widely used methodology for analyzing big data.
Data mining, also known as analytics, can be defined as “inductive reasoning applied to large-scale data, or big data.” In other words, data mining is the process of analyzing big data using statistical techniques to identify meaningful propositions or patterns. The propositions referred to here can be statements like “if p, then q”—which we learned in high school math—or functions of the form “y = f(x).” So, what exactly is a “meaningful” proposition?
The popular variety show “Infinite Challenge” once featured a scene utilizing big data, but this can be seen as an example of big data being used inappropriately. On “Infinite Challenge,” the show identified “Yoo-god” and “thoughtfulness” as big data keywords for TV personality Yoo Jae-suk; in fact, these are words that anyone would immediately associate with him. The show also cited “annoying” as an unexpected result, but this, too, is a word that anyone interested in Yoo Jae-suk could easily come up with.
From these examples, we can get a sense of what constitutes a “meaningful” proposition. If a proposition derived from data mining is something like “Yoo Jae-suk is very considerate,” it cannot be considered a meaningful result. Conversely, the more a proposition derived through data mining is something that cannot be easily deduced through general reasoning, the more valuable it is.
The value of data mining is primarily manifested in the form of commercial benefits. One example is Paris Baguette’s “Weather Sales Index.” The Weather Sales Index, developed in the past, was created by analyzing nationwide meteorological observation data spanning several years and over 1 billion records of product sales data by store. Through this analysis, Paris Baguette discovered that “when it rains, baked goods such as pizza bread sell well.” By using these findings to forecast sales and manage order volumes at each store, Paris Baguette’s baked goods sales increased by 30% in just one month.
It is said that these predictive results from Paris Baguette were, in fact, something that on-site employees had already vaguely sensed. However, sometimes the results of data mining can be difficult to understand. The online bookstore Amazon created a recommendation system using data that tracked which books individual customers bought—or simply viewed without purchasing. When sales from recommendation lists curated by human editors were compared to those from computer-generated lists, the books on the data-driven lists sold better. Neither humans nor computers could explain why customers who read Ernest Hemingway’s books also purchased works by F. Scott Fitzgerald, but that didn’t matter.
In a way, this can be seen as the true power of data. While using results that cannot be explained may be viewed as taking on a certain amount of risk, causal relationships are not particularly important in the conclusions derived from data mining. After all, those conclusions are statistically true. If we acknowledge this, we can ultimately use data to identify new needs that we hadn’t previously considered. Furthermore, data does not commit the errors that human intelligence is prone to, such as overgeneralization or bias.
By now, readers of this article may feel inspired to try their hand at data mining. However, data mining is not always a panacea. The biggest limitation and stumbling block when first attempting data mining is the collection of data. Not only are there few places where enough data has been accumulated to yield meaningful results, but in today’s world—where everyone recognizes the importance of big data—countless companies are reluctant to freely disclose data that could be their lifeline. Nevertheless, public data provided by the government (data.go.kr) is readily accessible to the general public, so let’s make good use of it.
So far, we’ve explored the definition and value of data mining as a method for utilizing data in the era of big data. Data mining isn’t the only way to leverage big data. Other prominent fields that utilize data include text mining—a technology for extracting meaningful information from text data—as well as artificial intelligence and machine learning, which have become familiar in South Korea thanks to the match between Lee Sedol and AlphaGo. Compared to these, data mining has been studied for a longer period, and its research methods have already been standardized. However, the development of data mining does not end here. As vast amounts of data continue to accumulate, and with advancements in computing technology capable of processing even larger datasets, data mining will continue to reveal countless insights to the world that humans have yet to discover.
