AWS: Data Processing with Lambda, S3, SNS, and EventBridge | Step-by-Step Tutorial

Introduction

This tutorial provides a step-by-step guide to automating AWS services using a Lambda function for a daily stock email alert application. You will learn to work with AWS services such as Lambda, S3, SNS, and EventBridge, and how to set them up to send email notifications based on data stored in S3.

Step 1: Create an S3 Bucket

• Log in to your AWS console and search for S3.
• Click on "Create bucket."
• Choose a unique name for your bucket (it must be globally unique).
• Keep the default settings for general parameters and click "Create bucket."
• After creating the bucket, create a folder to store stock files:
  • Click "Create folder."
  • Name the folder according to the date or the purpose (e.g., "2023-10-01").
• Upload a dummy text file for testing:
  • Click "Upload" and select a file from your local system.
  • Click "Upload" to complete the process.

Step 2: Set Up SNS for Email Notifications

• Search for Simple Notification Service (SNS) in the AWS console.
• Click on "Create topic."
• Give your topic a name and create it using the basic settings.
• Create a subscription for the topic:
  • Click on "Create subscription."
  • Select "Email" as the protocol.
  • Enter the email address you want to subscribe.
  • Click "Create subscription."
• Check your email for a subscription confirmation and confirm it.

Step 3: Create a Lambda Function

• Search for Lambda in the AWS console and click on "Create function."
• Choose "Author from scratch."
• Name your function and select Python as the runtime.
• Click "Create function."
• Add a trigger for the Lambda function:
  • Choose "EventBridge (CloudWatch Events)" as the source.
  • Create a new rule with a name and set a schedule expression (e.g., cron(0 10 * * ? *) for daily at 10 AM).
• Click "Add."

Step 4: Write Lambda Function Code

• In the Lambda function code editor, replace the default code with the following:

import json
import boto3
import datetime

s3 = boto3.client('s3')
sns = boto3.client('sns')

def lambda_handler(event, context):
    bucket_name = 'YOUR_BUCKET_NAME'
    sns_topic_arn = 'YOUR_SNS_TOPIC_ARN'
    today_date = datetime.datetime.now().strftime('%Y-%m-%d')
    file_key = f"{today_date}/stock_data.txt"  # Adjust this based on your file structure

    try:
        response = s3.get_object(Bucket=bucket_name, Key=file_key)
        file_content = response['Body'].read().decode('utf-8')
        
        sns.publish(
            TopicArn=sns_topic_arn,
            Message=file_content,
            Subject='Daily Stock Update'
        )
        return {
            'statusCode': 200,
            'body': json.dumps('Email sent successfully!')
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }

• Replace YOUR_BUCKET_NAME and YOUR_SNS_TOPIC_ARN with the actual values from your S3 bucket and SNS topic.

Step 5: Set Permissions for the Lambda Function

• Go to the "Configuration" tab of your Lambda function and select "Permissions."
• Click on the role name linked to your Lambda function.
• In the IAM Console, create a new role:
  • Choose "Lambda" as the type of service.
  • Attach policies for full access to S3 and SNS (consider using more restrictive permissions in production).
• Save the new role and return to the Lambda function.
• Edit the permissions to assign the new role to your Lambda function.

Step 6: Test Your Lambda Function

• In the Lambda console, select "Test."
• Create a new test event with no input (leave it blank).
• Click "Test" to execute the function.
• Check your email to verify that you received the stock data notification.

Conclusion

You have successfully created a daily stock email alert application using AWS services. You learned to set up S3 for file storage, configure SNS for email notifications, and automate the process with Lambda and EventBridge. As a next step, consider exploring more complex scheduling and notification scenarios or integrating additional data sources for enhanced functionality.