In this blog post, we’ll explore the principles and methods of data mining—which involves extracting meaningful information from vast amounts of data and analyzing past patterns to predict the future—as well as real-world examples of its application.
A Digital Ark Emerging from the Flood of Data
In recent years, there’s a term that has frequently appeared in the IT and science sections of online news: “big data.” For modern people, who are more accustomed to using computers than working out, the word “data” has become so commonplace that it barely piques their interest anymore; yet the term “big data” feels both familiar and, in a way, unfamiliar. Since computers and the internet became widely available to the public, the volume of data has skyrocketed. As this volume grew to a point where it became difficult to manage using traditional methods, it came to be called “big data.” A moment’s reflection makes this easy to understand. As recently as the early 2000s, when searching for something online, there was virtually nothing that could be considered “data” compared to today—aside from structured information like encyclopedias. However, today, the moment you press the Enter key on the Internet, the screen is flooded with countless advertising pages and all kinds of information. In an environment where data has exploded in this way, a pressing need has arisen for technology—like an ark—that can quickly and effectively find the meaningful information users seek. In response, engineers have moved beyond traditional methods, such as simple searches, to develop new technologies—and this is precisely what “data mining” is.
How does data mining extract data?
The Department of Industrial Engineering at Seoul National University currently offers a “Data Mining” course at the undergraduate level and covers “Data Mining Techniques” as a related course at the graduate level. Additionally, through research labs such as the Data Science and Business Analytics Lab, the department conducts research on data-driven decision-making as well as machine learning and artificial intelligence methodologies. Just how important is data mining that universities are offering dedicated courses under that exact name and training experts through related research? If it’s that important, one naturally wonders what it’s all about. To understand data mining, it’s sufficient to examine a three-step process. First, we’ll understand the principles and underlying theories of data mining; next, we’ll examine the methodologies and sub-concepts involved in applying data mining; and finally, we’ll explore the concept through real-world applications of data mining.
Data mining, as the term itself suggests, is a process similar to extracting gold from a mine. In vast amounts of data, the information a user seeks often constitutes only a tiny fraction; data mining filters out unnecessary data and extracts only the most valuable information to provide meaningful results to the user. Generally, data is not limited to information found on the internet but encompasses various forms of data, such as documents accumulated by companies and customer information held by credit card companies. However, since it is impossible for humans to analyze such vast amounts of data manually, the data is digitized, and computers are used to perform data mining. To give a simple example, consider a “data mine” consisting of the transaction records of customers at a supermarket. By analyzing this data, patterns such as “People who buy meat also buy lettuce” or “People who buy diapers also buy beer” can be identified. Statistical theory plays a crucial role in this analysis process. Since the analysis does not occur instantly, the time it takes for the computer to perform calculations is also considered a cost; computer science theory comes into play when determining how to create efficient programs to reduce this cost. In other words, data mining can be described as using a “pickaxe”—forged by combining the core elements of statistics and computer science—to mine the data “mine.”
Two Methods for Identifying Past Patterns and Predicting the Future
Now that we have a general sense of what data mining does, let’s delve a little deeper and examine specific data mining methodologies. Data mining can be broadly applied in two ways. The first involves discovering association rules or performing clustering based on similar characteristics in historical data. In this approach, data of similar types are grouped together to identify their common characteristics and model the situation up to the point when the data was collected. For example, using data from a music streaming service, one might discover association rules such as, “People who listen to Kim Kwang-seok’s music tend to listen to Kim Hyun-sik’s music as well,” or “People who listen to Seo Taiji’s music do not listen to Michael Jackson’s music.” This process of deriving insights about the past and present from data is called “descriptive modeling.” Another approach involves predicting the future based on what the data implies. In the example just discussed, one might predict that “people who listen to old pop songs will purchase the songs they’ve played,” and by analyzing subsequent data, one can verify the actual purchase rate after a song has been played. This process of creating models from data to predict the future is called “predictive modeling.” Today, data mining is used to identify patterns in large-scale data and predict future outcomes, combining not only statistical analysis but also machine learning and artificial intelligence technologies.
How is data mining applied in the real world?
We’re now just one step away. Through the examples examined earlier, you’ve likely come to understand that using data mining allows us to identify patterns in the diverse and complex events of the real world and even predict future occurrences. However, these examples represent only a small fraction of its potential. Major corporations and many nations—which we are already well acquainted with—are using data mining to formulate plans for their survival and growth. The types and scales of these applications are diverse, ranging from marketing strategies that analyze customer purchasing patterns to tailor promotional activities, to financial techniques that detect credit card fraud or predict risks and price fluctuations in financial markets, and even to analyzing phrases on social media to gauge public interest and political leanings. As the diversity and accumulation rate of data continue to grow, the potential applications of data mining in this trend are virtually limitless.
Creating New Knowledge in the Age of Data Overload
Isn’t it simpler than you thought? If this is your first encounter with data mining, I believe you’ve immediately grasped its importance. While the discovery of new knowledge prior to the 21st century was based on an understanding of the laws of nature and human behavior, today the creation of new knowledge also occurs through the process of reanalyzing and reevaluating data—accumulated over time but whose significance had not been fully recognized—to uncover the hidden meanings within it. Data mining—which analyzes the past to assess the present and predict the future—will continue to provide humanity with a steady stream of new information. It will serve as an ark for finding necessary information amid a flood of data and as a pickaxe for mining gold from a mine.
