Apache Hudi - Design/Code Walkthrough Session for Contributors

Title: Apache Hudi - Design/Code Walkthrough Session for Contributors

Channel: Siva Balan

Description: Apache Hudi is an open-source Spark library that manages the storage of large analytical datasets over distributed file systems (DFS) like HDFS or cloud stores. It enables querying of these datasets using tools like Spark, Hive, Presto, etc. Hudi brings stream processing capabilities to big data, delivering fresh data efficiently compared to traditional batch processing methods. It supports features like upserts, incremental pull/querying, read-optimized queries, self-managed file sizes to address small file issues in large datasets, GDPR compliance through deletes, async compaction, and more, ensuring ACID semantics over large datasets.

Tutorial:

1. Introduction to Apache Hudi:

   • Apache Hudi is an open-source library for managing large analytical datasets over distributed file systems.
   • It enables efficient querying of large datasets using tools like Spark, Hive, and Presto.
   • Hudi brings stream processing capabilities to big data, offering fresh data efficiently compared to traditional batch processing.

2. Features of Apache Hudi:

   • Upserts: Apache Hudi supports upserts, allowing you to update existing records in the dataset.
   • Incremental Pull/Querying: You can perform incremental pulls and queries on the dataset.
   • Read-Optimized Query: Hudi provides read-optimized queries for efficient data retrieval.
   • Self-Managed File Sizes: It automatically manages file sizes to address small file problems in large datasets.
   • GDPR Compliance: Apache Hudi supports deletes for GDPR compliance in data management.
   • Async Compaction: It offers asynchronous compaction for better performance.
   • ACID Semantics: Hudi ensures ACID semantics over large datasets for data integrity.

3. Walkthrough Session for Contributors:

   • Watch the video provided in the URL: https://www.youtube.com/watch?v=N2eDfU_rQ_U for a detailed design and code walkthrough session.
   • Learn how to contribute to the Apache Hudi project and understand the implementation details.

By following this tutorial, you will gain insights into Apache Hudi, its features, and how to contribute to the project as discussed in the video by Siva Balan.