noc19-cs33 Lec 04-Hadoop Distributed File System (HDFS)

Introduction

This tutorial provides a comprehensive overview of the Hadoop Distributed File System (HDFS) as presented in the lecture from IIT Kanpur's CS33 course. Understanding HDFS is crucial for managing large datasets across distributed systems, making this tutorial highly relevant for students and professionals working in data science, big data, and cloud computing.

Step 1: Understand the Basics of HDFS

• HDFS is designed to store large files across multiple machines in a distributed environment.
• It operates on a master-slave architecture:
  • NameNode: Master server that manages metadata and namespace of the file system.
  • DataNode: Slave servers that store the actual data blocks.

Key Concepts

• Block Size: Default block size is typically 128 MB or 256 MB. Files are split into blocks for storage across DataNodes.
• Replication: Each block is replicated across multiple DataNodes (default is 3 copies) for fault tolerance and reliability.

Step 2: Explore HDFS Architecture

• The architecture consists of two main components:
  • Client: Interacts with HDFS to read or write data.
  • Cluster: Comprises the NameNode and multiple DataNodes.

Client Operations

• When a client writes a file:

  • The client contacts the NameNode to get the block locations.
  • The client writes data directly to the DataNodes in a pipeline fashion.

• When a client reads a file:

  • The client retrieves block locations from the NameNode and accesses the DataNodes directly.

Step 3: Learn About HDFS Commands

• Familiarize yourself with basic HDFS commands to manage files and directories:
  • To create a directory:

    hdfs dfs -mkdir /directory_name
    

  • To upload a file:

    hdfs dfs -put local_file_path /hdfs_directory
    

  • To list files:

    hdfs dfs -ls /directory_name
    

  • To delete a file:

    hdfs dfs -rm /file_name
    

Practical Tips

• Always check file permissions and ownership when working with HDFS commands.
• Use the -help option with any command to learn more about its usage.

Step 4: Understand Data Reliability and Fault Tolerance

• HDFS provides high reliability and fault tolerance through replication.
• If a DataNode fails, HDFS automatically re-replicates the blocks to ensure the required number of replicas is maintained.
• Regular health checks and heartbeats are sent from DataNodes to the NameNode to monitor system status.

Conclusion

In this tutorial, we covered the foundational aspects of the Hadoop Distributed File System, including its architecture, basic commands, and principles of data reliability. Understanding HDFS is essential for effectively handling large datasets in a distributed computing environment.

Next steps may include setting up a local HDFS cluster for hands-on experience or diving deeper into Hadoop ecosystem tools like MapReduce and Apache Spark.