AWS Kinesis Firehose to S3 Tutorial | Step by Step Setup Guide

Introduction

This tutorial will guide you through the process of setting up an Amazon Kinesis Firehose delivery stream to send data to an S3 bucket. Kinesis Firehose is a powerful tool for real-time data streaming and processing, allowing you to handle large volumes of data efficiently. By the end of this tutorial, you'll have a functional Kinesis Firehose stream that delivers data to S3.

Step 1: Access the Kinesis Firehose Console

• Log in to your AWS Management Console.
• In the search bar, type "Firehose" and select "Kinesis".
• Click on "Delivery streams".
• Click on "Create delivery stream".

Step 2: Configure the Delivery Stream

• Name Your Delivery Stream: Enter a name for your stream, such as "Stock Tickers".
• Select Source: Choose how you want to send data to your stream:
  • For this tutorial, select the option to use the Firehose endpoint directly.
• Enable Server-Side Encryption (optional): If your data is sensitive, enable encryption. For this tutorial, leave it disabled and click "Next".

Step 3: Data Processing Options

• You can set up a data transformation step using AWS Lambda if needed. This allows you to modify the data as it flows through the stream.
• For this tutorial, skip this step and click "Next".

Step 4: Set Destination to S3

• Choose Destination: Select "Amazon S3" from the options.
• Select S3 Bucket: Choose an existing S3 bucket or create a new one for storing the data.
• Set Prefix for Output Files: You can specify a prefix for your files. The default format includes the date and time, which is suitable for most use cases.
• Error Prefix: Optionally set a prefix for error files. Leave this as default for now and click "Next".

Step 5: Configure Buffering and Compression

• Buffer Size: Specify how large the files should be before they are sent to S3. For example, set it to 1 MB.
• Buffer Interval: Set the maximum interval (in minutes) to wait before writing files to S3. For example, set it to 5 minutes.
• Enable Compression: Optionally enable Gzip or another compression method to reduce storage costs.
• Click "Next".

Step 6: Error Logging and IAM Role

• Error Logging: It's recommended to enable error logging to CloudWatch. You can create a new IAM role or use an existing one that has the necessary permissions to write logs.
• For simplicity, choose the option to create a new role automatically and click "Next".

Step 7: Review and Create the Delivery Stream

• Review all your settings on the confirmation page.
• Click on "Create delivery stream". The creation process may take a few minutes.

Step 8: Test the Delivery Stream

• Once the stream is active, click on the stream name to view details.
• Click on "Test with demo data" to start sending test data to your stream.
• Allow it to run for a few minutes to generate data.

Step 9: Verify Data in S3

• Navigate to your S3 bucket in the AWS Management Console.
• Check the folder structure by year, month, and day to find your data files.
• Open a file to verify the data. Each file will contain batched records from your stream.

Conclusion

You have successfully set up an Amazon Kinesis Firehose delivery stream to send data to an S3 bucket. This setup is ideal for real-time data processing and batch analytics. As next steps, consider exploring data transformation with AWS Lambda or integrating Firehose with other AWS services for more advanced processing.