Serverless Migrations

Lambda Event Filtering with a DynamoDB Stream

Saving Money, Saving Time, Saving Headaches

Lambda Event Filtering with a DynamoDB Stream

By
Ryan Jones
January 14, 2022

Introduction

In this article, we will take a look at a use-case that allows us to pre-filter events by certain conditions before they invoke our AWS Lambda functions. We will be leveraging a DynamoDB stream with multiple AWS Lambda functions attached and multiple Lambda Event Source Mappings. The entire PoC (proof-of-concept) uses IaC written with Serverless Framework.

AWS Architecture


Folder Structure

Above is an overview of what a local PoC folder structure will look like. Further down in this document you will see what each file looks like.

DynamoDB Table

Above we are creating our DynamoDB table with stream enabled.

Two Lambdas + IAM Role for DynamoDB Stream

Above we are creating our two Lambda functions and attaching an IAM policy with permissions to log to CloudWatch and interact with our DynamoDB stream. The index.js is a one line log.

AWS Lambda Event Source Mappings

Above we are attaching two separate Lambda event filters.

One for Lambda A which is looking for the following:

  • eventName of "INSERT"
  • PK prefix of “Message#”
  • SK prefix of “Channel#”
  • messageId exists

If “messageId” doesn’t exist, “PK” doesn’t start with “Message#”, SK doesn’t start with “Channel#”, and eventName is not “INSERT” while in the new DynamoDB record, it will not trigger Lambda A.

This event would trigger Lambda A as it meets our filter requirements

The second Lambda event filter which would trigger Lambda B is looking for the following:

  • eventName of “INSERT”
  • PK prefix of “Feed#”

If “PK” doesn’t start with “Feed#” and eventName is not “INSERT” in the new DynamoDB record, it will not trigger Lambda B.

This event would trigger Lambda B as it meets our filter requirements

If one of our conditions is not met then our Lambda functions with filters will not be invoked. The trickiest part of this setup is getting the Filter Pattern correct. For this, I leveraged the AWS documentation and trial + error.

Conclusion

By leveraging Lambda Event Mapping Filters on a DynamoDB Stream we are able to only trigger the necessary Lambda when our conditions are met. Which is a huge improvement over triggering every time DynamoDB has an event take place then filtering inside the Lambda itself.

By doing this filtering outside the Lambda we are saving money (as we are not billed for the Lambda Event Mapping Filter), saving engineering hours to write the filtering code and test, reducing overall noise from millions of non-relevant Lambda invocations happening, and ensuring downstream maintenance is minimal.

If you want to download a copy of the template, you can find it on our website.