Merge pull request #2032 from astan54321/stnndr-feature-sqs-lambda-dd…

…b-cdk-ts

New serverless pattern - sqs-lambda-ddb-cdk-ts

sqs-lambda-ddb-cdk-ts/.gitignore

@@ -0,0 +1,7 @@
+!jest.config.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

sqs-lambda-ddb-cdk-ts/.npmignore

@@ -0,0 +1,6 @@
+*.ts
+!*.d.ts
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

sqs-lambda-ddb-cdk-ts/README.md

@@ -0,0 +1,133 @@
+# Offloading DynamoDB Writes with SQS, Lambda, and CDK
+
+The purpose of this pattern is to address write throttling issues of Provisioned Capacity DynamoDB Tables by offloading the requests to SQS to handle writes asynchronously. This pattern is deployed through an AWS Cloud Development Kit (CDK) app which provisions DynamoDB tables, SQS queues and DLQs, and Lambda functions. Test scripts have been provided for testing.
+
+Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/sqs-lambda-ddb-cdk-ts](https://serverlessland.com/patterns/sqs-lambda-ddb-cdk-ts)
+
+## Architecture Diagram
+
+![architecture_diagram](./images/architecture_diagram.png)
+
+## How it works
+
+A new Amazon SQS queue will be created and configured to invoke an AWS Lambda function that will write the received messages into an Amazon DynamoDB table. The SQS queue redrive policy will be configured to send undeliverable messages to an Amazon SQS Dead-Letter Queue. Undeliverable messages are the ones that couldn't be written to the DynamoDB table with a maximum of `maxReceiveCount` number of attempts.
+
+The Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.
+
+The Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe message retries on message writes when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.
+
+## Considerations
+
+* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended to enable this feature as in a production environment.
+* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired Write Capacity Units (WCUs).
+* This pattern doesn't guarantee message ordering when writing information to the DynamoDB table
+* This application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Prerequisites
+
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. This account needs to be bootstrapped to use AWS Cloud Development Kit.
+* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+* [Node and NPM installed](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/)
+* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) (AWS CDK) installed and bootstrapped
+
+## Deployment Instructions
+
+1. Create a new directory on your machine, navigate to that directory and clone this repository.
+
+```
+git clone https://github.com/aws-samples/serverless-patterns
+```
+
+2. Navigate to the directory of this repository:
+
+```
+cd sqs-lambda-ddb-cdk-ts
+```
+
+3. Install Node Packages:
+
+```
+npm install
+```
+
+4. From the command line, use AWS CDK to synthesize an AWS CloudFormation Template:
+
+```
+cdk synth
+```
+
+5. From the command line, use AWS CDK to deploy the AWS resources:
+
+```
+cdk deploy
+```
+
+## Testing
+
+Use the provided test script to send messages to the SQS queue. We will simulate a scenario where N messages are sent to the queue per second.
+
+1. Navigate to the test-scripts directory and open up the file named push-sqs-messages.js
+
+```
+cd test-scripts
+```
+
+2. In the script, replace 'YOUR_REGION' and 'YOUR_SQS_QUEUE_URL' with your desired AWS Region and SQS Queue URL respectively
+
+3. Using the command line, run the script and specify an integer for N (the number of requests per time interval - 5 is a sufficient number for this demo) and for T(the frequency in milliseconds with which to send the N events - 1000 is sufficient for this demo). Keep this script running in the background until later in step 10. You should begin to see continuous messages in your terminal saying "Batch of N items pushed to SQS at ...".
+
+```
+node push-sqs-messages.js N T
+```
+
+4. At this point, the Lambda function will start inserting items into the DynamoDB table. After a couple of minutes, you should notice the DLQ filling up with messages since DynamoDB will start throttling the requests.
+
+5. Using Cloudwatch Log Insights, run a query on the Lambda function's Log Group to ensure that the requests are being throttled. Feel free to use the new Query Generator feature or simply run the following query:
+
+```
+filter @message like /ProvisionedThroughputExceededException/
+```
+
+
+
+6. Once you have confirmed that DynamoDB is throttling write requests, go back to the [config](./lib/config/tables.config.ts) file in repo and lower the `sqsMaxConcurrency` property to a value less than or equal to the current writeCapacity value of the table (2 is a sufficient value for this demo). If the writeCapacity of the table is already at 2, the lowest `sqsMaxConcurrency` value possible is 2. Increase the `retryAttempts` property value to a high value (20 should be sufficient for this demo) to give Lambda time to keep up with any subsequent throttling events.
+\
+\
+**IMPORTANT:** Write down the approximate timestamp of when you make this change. It will come in handy when reading CloudWatch Logs.
+\
+\
+**NOTE:** For this demo, it's more convenient to make this change through the Lambda Console as making changes to the config file requires you to run cdk deploy again. Do this by navigating to the function -> Configuration Tab -> Concurrency Tab. In a proper testing or production environment, it's recommended to leverage Infrastructure as Code rather than making changes via the AWS Console.
+
+7. After some time, you should notice the rate at which messages come into the DLQ start to decrease. Navigate back to CloudWatch Log Insights and run the same query from step 5. You can adjust the time window as needed.
+\
+\
+Look through the logs and you should notice that there are none or fewer ProvisionedThroughputExceededException errors after the timestamp you noted in the previous step.
+
+8. For a visual representation, you can also look at CloudWatch metrics for **Write throttled requests(count)** in the monitoring tab of the DynamoDB Console. You should see a spike of write throttled requests and then it should decrease around the time you made the config change. It should look similar to the picture below.
+\
+\
+![graph showing spike in write throttling followed by slight decrease](./images/write_throttling_metrics.png)
+
+\
+\
+**NOTE:** If there are no data points loading on the graph, wait a couple of minutes as CloudWatch takes some time to aggregate metrics.
+
+9. <mark>Terminate the script! - It will otherwise continue to incur request cost until stopped.</mark> 
+
+10. Congrats! You have successfully deployed and tested this serverless pattern. If you wish, feel free to play around with different configurations for the AWS services as well as the payload size of the messages. If not, move onto the Cleanup section to clear your resources and stop incurring costs.
+## Cleanup
+
+1. To delete the stack, run the following command:
+
+```
+cdk destroy
+```
+
+
+
+
+----
+Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+
+SPDX-License-Identifier: MIT-0

sqs-lambda-ddb-cdk-ts/bin/sqs-lambda-dynamo.ts

@@ -0,0 +1,21 @@
+#!/usr/bin/env node
+import 'source-map-support/register';
+import * as cdk from 'aws-cdk-lib';
+import { SqsLambdaDynamoStack } from '../lib/sqs-lambda-dynamo-stack';
+
+const app = new cdk.App();
+new SqsLambdaDynamoStack(app, 'SqsLambdaDynamoStack', {
+  /* If you don't specify 'env', this stack will be environment-agnostic.
+   * Account/Region-dependent features and context lookups will not work,
+   * but a single synthesized template can be deployed anywhere. */
+
+  /* Uncomment the next line to specialize this stack for the AWS Account
+   * and Region that are implied by the current CLI configuration. */
+  // env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
+
+  /* Uncomment the next line if you know exactly what Account and Region you
+   * want to deploy the stack to. */
+  // env: { account: '123456789012', region: 'us-east-1' },
+
+  /* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
+});

sqs-lambda-ddb-cdk-ts/cdk.json

@@ -0,0 +1,64 @@
+{
+  "app": "npx ts-node --prefer-ts-exts bin/sqs-lambda-dynamo.ts",
+  "watch": {
+    "include": [
+      "**"
+    ],
+    "exclude": [
+      "README.md",
+      "cdk*.json",
+      "**/*.d.ts",
+      "**/*.js",
+      "tsconfig.json",
+      "package*.json",
+      "yarn.lock",
+      "node_modules",
+      "test"
+    ]
+  },
+  "context": {
+    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
+    "@aws-cdk/core:checkSecretUsage": true,
+    "@aws-cdk/core:target-partitions": [
+      "aws",
+      "aws-cn"
+    ],
+    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
+    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
+    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
+    "@aws-cdk/aws-iam:minimizePolicies": true,
+    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
+    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
+    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
+    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
+    "@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
+    "@aws-cdk/core:enablePartitionLiterals": true,
+    "@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
+    "@aws-cdk/aws-iam:standardizedServicePrincipals": true,
+    "@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
+    "@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
+    "@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
+    "@aws-cdk/aws-route53-patters:useCertificate": true,
+    "@aws-cdk/customresources:installLatestAwsSdkDefault": false,
+    "@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
+    "@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
+    "@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
+    "@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
+    "@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
+    "@aws-cdk/aws-redshift:columnId": true,
+    "@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
+    "@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
+    "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
+    "@aws-cdk/aws-kms:aliasNameRef": true,
+    "@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
+    "@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
+    "@aws-cdk/aws-efs:denyAnonymousAccess": true,
+    "@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
+    "@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
+    "@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
+    "@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
+    "@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
+    "@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
+    "@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true
+  }
+}

sqs-lambda-ddb-cdk-ts/example-pattern.json

@@ -0,0 +1,56 @@
+{
+  "title": "Amazon SQS to AWS Lambda to Amazon DynamoDB with CloudWatch Alarms",
+  "description": "Preventing Amazon DynamoDB write throttling with an Amazon SQS queue-based buffer using CDK and monitoring with CloudWatch alarms.",
+  "language": "TypeScript",
+  "level": "200",
+  "framework": "CDK",
+  "introBox": {
+    "headline": "How it works",
+    "text": [
+      "This code demonstrates how an Amazon SQS queue can be used to buffer records for an Amazon DynamoDB table and mitigate the risk for a client of being throttled.",
+      "The CDK code deploys Ingress and Dead-Letter SQS queues, AWS Lambda functions, destination DynamoDB tables and CloudWatch alarms.",
+      "AWS Lambda function uses NodeJS 20 runtime."
+    ]
+  },
+  "gitHub": {
+    "template": {
+      "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sqs-lambda-dynamodb-cloudwatch-cdk-ts",
+      "templateURL": "serverless-patterns/sqs-lambda-dynamodb-cloudwatch-cdk-ts",
+      "projectFolder": "sqs-lambda-dynamodb-cloudwatch-cdk-ts",
+      "templateFile": "lib/sqs-lambda-dynamo-stack.ts"
+    }
+  },
+  "resources": {
+    "bullets": [
+      {
+        "text": "Troubleshooting throttling issues in Amazon DynamoDB",
+        "link": "https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TroubleshootingThrottling.html"
+      },
+      {
+        "text": "Configuring reserved concurrency",
+        "link": "https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html"
+      }
+    ]
+  },
+  "deploy": {
+    "text": ["cdk deploy"]
+  },
+  "testing": {
+    "text": ["See the GitHub repo for detailed testing instructions."]
+  },
+  "cleanup": {
+    "text": ["<code>cdk destroy</code>"]
+  },
+  "authors": [
+    {
+      "name": "Andrei Stan",
+      "bio": "Cloud Application Developer, AWS.",
+      "linkedin": "astan54321"
+    },
+    {
+      "name": "Willes Mendoza",
+      "bio": "Cloud Application Developer, AWS.",
+      "linkedin": "willesfmendoza"
+    }
+  ]
+}

sqs-lambda-ddb-cdk-ts/jest.config.js

@@ -0,0 +1,8 @@
+module.exports = {
+  testEnvironment: 'node',
+  roots: ['<rootDir>/test'],
+  testMatch: ['**/*.test.ts'],
+  transform: {
+    '^.+\\.tsx?$': 'ts-jest'
+  }
+};

sqs-lambda-ddb-cdk-ts/lib/config/tables.config.ts

@@ -0,0 +1,31 @@
+/*! Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  SPDX-License-Identifier: MIT-0
+ */
+
+import { TableConfig } from "../sqs-lambda-dynamo-stack";
+
+// This file can be edited to add needed tables and their individual limits.
+export const TABLE_CONFIG: TableConfig[] = [
+    {
+        name: "SampleTable1",
+        writeCapacity: 5, // DynamoDB table WCU allocation.
+        sqsMaxConcurrency: 10, // Maximum concurrency the SQS queue uses for Lambda.
+        retryAttempts: 1, // Gives lambda time to retry before sending message to DLQ.
+        lambdaRetryAttempts: 0, // Maximum number of times to retry when the function returns an error.
+    },
+    // {
+    //     name: "SampleTable2",
+    //     writeCapacity: 5,
+    //     sqsMaxConcurrency: 10,
+    //     retryAttempts: 1,
+    //     lambdaRetryAttempts: 0,
+    // },
+    // {
+    //     name: "SampleTable3",
+    //     writeCapacity: 1,
+    //     lambdaReservedConcurrency: 1,
+    //     retryAttempts: 1,
+    //     lambdaRetryAttempts: 0,
+    // },
+]
+

sqs-lambda-ddb-cdk-ts/lib/lambda/index.js

@@ -0,0 +1,45 @@
+/*! Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *  SPDX-License-Identifier: MIT-0
+ */
+
+const { DynamoDBClient, PutItemCommand } = require("@aws-sdk/client-dynamodb");
+const { marshall } = require("@aws-sdk/util-dynamodb");
+
+// Get destination DynamoDB table name from the environment variables
+const tableName = process.env.DESTINATION_TABLE_NAME;
+
+// Create DynamoDB client outside of handler to optimize performance
+const ddb = new DynamoDBClient({
+    maxAttempts: 0, // disable DynamoDB automated retries (so Lambda fails fast and relies on SQS retry capabilities)
+});
+
+// AWS Lambda handler function
+exports.handler = async function (event, context) {
+    console.log(event);
+
+    // For each message in event
+    for (const record of event.Records) {
+        try {
+            // get the message body (i.e. the record to add to DynamoDB)
+            const item = JSON.parse(record.body);
+
+            // try saving the item to destination DynamoDB table
+            await ddb.send(
+                new PutItemCommand(
+                    {
+                        TableName: `${tableName}`,
+                        Item: marshall(item),
+                    }
+                )
+            );
+        } catch (error) { // reoport any error that may come up
+            console.error("FAILED TO HANDLE RECORD: ", error, record);
+            throw error; // throw to tell to SQS that something went wrong with the record
+        }
+    }
+
+    // exit with 200
+    return {
+        statusCode: 200,
+    };
+};