In this blog post, we’ll explore the core principles of Hadoop—which distributes the storage and processing of big data across multiple computers—as well as HDFS, MapReduce, cluster architecture, and the evolution of the Hadoop ecosystem to this day.
Social media and online services are generating enormous amounts of data even as we speak. Users constantly share where they’ve been, what activities they’ve engaged in, and what they’re interested in, generating various forms of data such as photos, videos, posts, and search histories. During the period when Facebook’s user base was growing rapidly, the accumulation of petabytes (PB) of photo data each month was cited as a prime example of the big data era. Today, vast amounts of data are being generated not only on social media but in nearly every field, including e-commerce, finance, manufacturing, healthcare, scientific research, and artificial intelligence. Amid this deluge of data, companies and institutions go beyond simply storing and managing data; they classify and analyze it in the necessary formats to uncover new value. The technologies and methods for effectively storing and processing such vast amounts of data are generally referred to as big data technologies, and Hadoop has established itself as a leading distributed data processing platform in the course of this development.
Hadoop is an open-source software framework designed to distribute, store, and process large volumes of data across multiple computers. Here, “processing” refers to a series of steps involving storing, reading, and computing vast amounts of data, as well as classifying and analyzing it according to specific purposes. In the past, when discussing large-scale data processing challenges, it was often argued that SQL-based relational databases or analytical systems like SAS were not suitable for handling massive amounts of data because they primarily operated on a single computer. However, today, since relational databases also actively support distributed processing and cloud environments, it is difficult to conclusively state that SQL or relational databases themselves are unsuitable for processing large volumes of data. A key feature of Hadoop is that it was designed from the ground up to store and process large-scale data across multiple servers. It distributes data that would be difficult for a single computer to process among multiple computers and operates them as a single, massive system. Two elements that are particularly important for understanding the core of early Hadoop are the Hadoop Distributed File System (HDFS) and MapReduce. Subsequently, YARN, which handles resource management and job scheduling within the cluster, has established itself as a key core component. Today’s Hadoop continues to be developed as a distributed data processing platform that includes HDFS, MapReduce, YARN, and other components.
HDFS is a distributed file system that stores large files by dividing them across multiple computers. Its basic structure centers on DataNodes, which actually store the data, and the NameNode, which manages the file system’s metadata. As explained in the original text, it is easier to understand the structure by thinking of a node as a single unit of data storage; however, strictly speaking, a node refers not to a simple “storage box” but to a server that makes up a Hadoop cluster or the role performed by that server. HDFS divides large files into blocks of a fixed size and distributes them across multiple DataNodes, while the NameNode manages metadata such as file names, directory structures, and the blocks that make up the files. In the past, 64MB blocks were widely used in Hadoop, but this is no longer considered the standard today. Currently, it is common for HDFS to use larger block sizes, and the actual size may vary depending on system settings and the operating environment. When a user attempts to read or write data, HDFS uses the metadata managed by the NameNode to access the necessary DataNodes.
A critical issue that must be considered in distributed storage systems is the possibility of failures in some servers or storage devices. In large-scale clusters using numerous devices, the failure of a specific device should be treated not as an exceptional event but as a situation that can occur at any time. Therefore, HDFS has traditionally employed a method of replicating data blocks and storing them across multiple data nodes. This ensures that even if a specific data node fails or becomes inaccessible, the data remains available through replicas stored on other data nodes. Recently, HDFS has expanded its capabilities beyond simple replication to include features such as erasure coding, which enhances storage space efficiency. Therefore, it is more accurate to understand Hadoop’s distributed storage approach not as a structure that simply scatters data across multiple computers, but as one designed to maintain data availability and stability while accounting for hardware failures.
In simple terms, the typical data processing method involves retrieving data from storage, performing calculations, and then saving the results back to storage. However, if all data must be moved to a single location for each computation—even after distributing massive amounts of data across multiple servers using HDFS—this places a tremendous burden on the network. As the volume of data increases, so do the time required to move the data and the associated network costs. To address this issue, Hadoop actively leverages the concept of “data locality,” which involves performing computations as close as possible to where the data is stored. In early Hadoop, MapReduce was the representative distributed processing model that implemented this concept. In other words, rather than gathering all the large-scale data in one place for computation, this approach involves multiple computers processing their assigned data in parallel and then aggregating the results.
As its name suggests, MapReduce consists of two main processing stages: Map and Reduce. In the Map phase, each piece of input data is processed to generate intermediate key-value pairs. For example, in a task that counts how many times a specific word appears in multiple documents, the word can be represented as the key and the number of occurrences as the value. When words such as “Breast,” “Liver,” and “Cancer” appear, the Map phase creates a single key-value pair for each word and its occurrence count. Once these intermediate results are generated across multiple computers, the data is aggregated based on the relevant keys, and in the Reduce phase, the values corresponding to the same key are aggregated to produce the final result. For example, if the intermediate results “Cancer-1,” “Cancer-1,” and “Cancer-1” are generated from different data sources, these are ultimately summed to yield a result such as “Cancer-3.” In this way, MapReduce is a method that divides a single large computation into several smaller computations, performs them in parallel, and then combines the results.
If too much intermediate data is generated during the Map phase, network traffic increases and the load on the Reduce phase can become excessive. In such cases, depending on the nature of the task, a Combiner can be used to perform a local aggregation of the Map output first. The Combiner acts as a kind of local aggregator between the Map and Reduce phases, reducing the volume of intermediate results. For example, if a single input data set contains “Breast” once, “Liver” once, and “Cancer” twice, rather than transmitting each item as-is, the Combiner can group them—where possible—into “Breast-1,” “Liver-1,” and “Cancer-2” before passing them to the next phase. However, the Combiner is not mandatory in every MapReduce job; it is used selectively when the nature of the operation allows for intermediate aggregation.
Hadoop, which evolved around HDFS and MapReduce, provided an environment capable of storing and processing large volumes of data by connecting multiple general-purpose servers, rather than entrusting all tasks to a single, expensive supercomputer. A collection of computers connected in this way is called a computer cluster. Companies and organizations can start with a relatively small cluster and expand their processing power and storage capacity by adding servers as their data and workload grow. This horizontal scalability played a key role in Hadoop’s widespread adoption. Today, large-scale distributed data processing environments can be built not only on physical on-premises servers but also on cloud infrastructure, and Hadoop and related technologies are being utilized in line with these environmental changes.
Another important feature of Hadoop is that it is open source. Anyone can examine and use the source code, modify it as needed, and develop related technologies. This openness served as a major catalyst for companies and developers—who had previously found it difficult to attempt large-scale data analysis due to exorbitant costs—to leverage distributed data processing technologies. At the same time, as developers and companies from around the world participated in projects, a foundation was laid for the growth of various tools and projects centered around Hadoop. Currently, the Apache Hadoop project is still under active development, and new versions are continuously being released.
Over time, Hadoop and its surrounding ecosystem have also undergone significant changes. Early Hadoop faced criticism that MapReduce programming was relatively difficult, that its disk-centric batch processing approach made it unsuitable for real-time analysis requiring low latency, and that building and operating the entire system was complex. To address these shortcomings, various projects centered around Hadoop have evolved. Hive has evolved into a data warehouse system that allows users to read, write, and manage data in large-scale distributed storage using SQL, and it continues to be developed as a project that supports data analysis on a petabyte scale. Similarly, rather than simply describing ZooKeeper as a “utility program that makes Hadoop easier to use,” it is more accurate to understand it as a distributed coordination service that provides functions such as configuration management, synchronization, and naming in distributed systems. In particular, Hive and ZooKeeper are Apache projects that are still actively maintained today. In contrast, Apache Pig—once one of the flagship tools in the Hadoop ecosystem—is now archived in Apache’s release catalog rather than being a major project with ongoing official releases. Therefore, while Pig, Hive, and ZooKeeper were often grouped together in the past and described as representative Hadoop tools, today it is necessary to distinguish their respective roles and project statuses.
Hadoop is no longer used synonymously with “big data technology itself.” Today’s data processing environments utilize a combination of cloud-based data warehouses and data lakes, various distributed processing engines such as Apache Spark, streaming platforms, and various types of databases. Nevertheless, Hadoop has had a profound impact on the popularization of distributed processing technology, which involves storing and processing large-scale data across multiple computers. In particular, distributed storage via HDFS, data locality—the principle of performing computations close to where the data resides—and the approach of scaling by grouping multiple servers into a single cluster are key concepts for understanding the evolution of modern big data systems. Hive, too, can now integrate not only with HDFS but also with various distributed and cloud storage solutions such as S3, Azure Data Lake, and Google Cloud Storage, and is evolving to integrate with modern data processing tools including Spark and Presto.
Early Hadoop opened the door to the big data era with the concept of grouping multiple relatively inexpensive computers together to store and analyze massive amounts of data. Since then, the technological landscape has changed significantly, and the Hadoop ecosystem has also undergone constant evolution. The description of Hadoop in the past as an “evolving ecosystem with blurred, ambiguous, and constantly shifting boundaries” aptly captures these characteristics. While the composition and roles of the technologies surrounding Hadoop today differ significantly from its early days, Apache Hadoop itself is still under active development, and components such as HDFS, YARN, and MapReduce remain in use. Ultimately, the history of Hadoop is more than just the success story of a single program. It is a case study that demonstrates how the concept of distributing vast amounts of data across multiple computers—rather than concentrating it on a single machine—and having the necessary computations performed collectively by multiple devices has transformed the way data is processed in the real world. Just as in the IT world, where new technologies constantly emerge and the roles of existing technologies shift, Hadoop and the surrounding distributed data processing ecosystem continue to evolve and advance even today.