In this blog post, we’ll explore the concept of big data, its processing technologies, various use cases, and challenges related to personal data protection.
With recent advancements in ICT, social media, and mobile devices, enormous amounts of data are being generated. The technology that goes beyond simply collecting and managing this data to analyze it and predict future outcomes is called “big data.” Today, big data has established itself as a crucial technology, with the term widely used when analyzing data across diverse fields such as broadcasting, business, healthcare, finance, retail, and everyday life. Big data is actively utilized not only in personalized recommendation services and advertising by global platform companies like Amazon, Google, and Facebook but also in recommendation features found in apps for restaurants, travel, and shopping. So, how exactly does big data manage and analyze such vast amounts of data?
In the past, the Hadoop MapReduce framework was widely used to process big data. While MapReduce offers the advantage of distributing large-scale data across multiple computers in a cloud environment for parallel processing, it has limitations when it comes to real-time data retrieval and analysis. For this reason, in-memory distributed processing technologies such as Apache Spark, along with various real-time streaming processing technologies, are now being used in conjunction. The distributed query approach reduces the load on individual nodes by having multiple nodes within a cluster process queries simultaneously, thereby providing fast response times through parallel processing. Representative techniques include partitioning and shuffling. Partitioning is a method of dividing data and storing it across multiple nodes based on a specific key, while shuffling is the process of regrouping and reallocating data when joining data based on a key other than the partition key. In contrast, the streaming processing approach analyzes data as soon as it is generated to organize the necessary information in real time. Techniques such as lineage tracking and state checkpointing are utilized in this process. Data is processed across multiple nodes, and the final results are stored in a database; if an error occurs during processing, recovery procedures are performed. Lineage Tracking is a technology that records the processing path of each data point, enabling only the necessary portions to be recalculated in the event of a failure. Additionally, State Checkpointing is a technology that saves the state of the process at intermediate stages so that, in the event of a failure, the process can be resumed from the saved point rather than restarting from the beginning. These technologies are used to process large volumes of data reliably and efficiently.
At the 2014 World Cup in Brazil, Germany defeated the host nation, Brazil, by an overwhelming score of 7–1 and went on to defeat Argentina in the final to claim the championship. It is widely reported that big data technology played a key role behind the German national team’s outstanding performance at that time. Germany attached sensors to players’ shin guards and uniforms to collect data on activity levels, heart rates, and shooting motions, among other metrics. They then analyzed this data to inform training and tactical planning. Similarly, in the sports sector, big data and AI-based analytical technologies are continuously being utilized to enhance performance, manage athletes, and prevent injuries.
Big data technology is also being used to develop election strategies. In the United States, there are well-documented cases where various public data and voter information are analyzed during the election process to identify voters’ interests and preferences and formulate tailored election strategies. In particular, a comprehensive analysis of diverse data—such as social media activity, donation records, and regional characteristics—is used to enhance the efficiency of election campaigns, and such data-driven election strategies have since established themselves as one of the key election analysis techniques in many countries around the world.
Big data is also being actively utilized in “weather-based management,” which involves applying weather information to business operations. The Korea Meteorological Administration publishes observational data, forecast data, and climate data in real time, and various organizations and companies are using this data to formulate business strategies. In sectors such as electric power companies, airlines, retail, and agriculture, weather information is analyzed to forecast demand and mitigate damage caused by unexpected weather changes, while also improving operational efficiency and increasing revenue.
Furthermore, public institutions and private companies are continuously expanding services that utilize weather information by jointly analyzing various datasets.
Big data technology enables more accurate analysis and forecasting across diverse fields, bringing about significant changes throughout society. However, concerns are consistently being raised that such technological advancements could lead to privacy violations and increased social surveillance. “Big Brother” is a term derived from George Orwell’s novel ‘1984’, referring to a society where an absolute authority monitors everyone’s actions and thoughts. In fact, controversy arose regarding the potential use of big data technology for surveillance purposes following the disclosure of large-scale data collection activities by the U.S. National Security Agency (NSA). Even today, as artificial intelligence and big data technologies continue to advance, striking a balance between personal data protection and data utilization has become an increasingly critical social challenge. While big data is making significant contributions to improving people’s quality of life in various fields—including business management, political activities, weather forecasting, medical technology development, and systematic healthcare—there is also the potential for adverse effects such as privacy violations and social surveillance. Therefore, companies and institutions utilizing big data must strictly adhere to legal and ethical standards regarding privacy protection and data usage, and establish clear principles for leveraging technology based on public trust.
