New serverless pattern - sqs-lambda-ddb-cdk-ts #2032

astan54321 · 2024-01-10T21:20:51Z

Issue #, if available:
#2031

Description of changes:
The purpose of this pattern is to address write throttling issues of Provisioned Capacity DynamoDB Tables by offloading the requests to SQS to handle writes asynchronously. This pattern is deployed through an AWS Cloud Development Kit (CDK) app which provisions DynamoDB tables, SQS queues and DLQs, Lambda functions, and Cloudwatch alarms. Test scripts have been provided for testing.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

bfreiberg · 2024-01-17T09:07:22Z

Hi, it looks like you have a subfolder sqs-lambda-dynamodb-cloudwatch-cdk-ts with another pattern? Is that intentional?

bfreiberg · 2024-01-17T09:10:44Z

I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?

astan54321 · 2024-01-17T18:25:44Z

Hi, it looks like you have a subfolder sqs-lambda-dynamodb-cloudwatch-cdk-ts with another pattern? Is that intentional?

Hi Ben, addressed the outdated pattern folder (this was from a previous submission where the pattern also included a cloudwatch alarm)

astan54321 · 2024-01-17T18:29:42Z

I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?

The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the sqs-lambda-ddb-sam-ts subfolder

bfreiberg · 2024-01-22T07:34:45Z

sqs-lambda-ddb-cdk-ts/README.md

+
+The purpose of this pattern is to address write throttling issues of Provisioned Capacity DynamoDB Tables by offloading the requests to SQS to handle writes asynchronously. This pattern is deployed through an AWS Cloud Development Kit (CDK) app which provisions DynamoDB tables, SQS queues and DLQs, and Lambda functions. Test scripts have been provided for testing.
+
+Learn more about this pattern at Serverless Land Patterns: [INSERT SERVERLESS LAND LINK HERE]


Suggested change

Learn more about this pattern at Serverless Land Patterns: [INSERT SERVERLESS LAND LINK HERE]

Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/sqs-lambda-ddb-cdk-ts](https://serverlessland.com/patterns/sqs-lambda-ddb-cdk-ts)

bfreiberg · 2024-01-22T07:35:52Z

sqs-lambda-ddb-cdk-ts/README.md

+
+A new Amazon SQS queue will be created and configured to invoke an AWS Lambda function that will write the received messages into an Amazon DynamoDB table. The SQS queue redrive policy will be configured to send undeliverable messages to an Amazon SQS Dead-Letter Queue. Undeliverable messages are the ones that couldn't be written to the DynamoDB table with a maximum of `maxReceiveCount` number of attempts.
+
+Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.


Suggested change

Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.

The Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.

bfreiberg · 2024-01-22T07:36:38Z

sqs-lambda-ddb-cdk-ts/README.md

+
+Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.
+
+Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe message write retries when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.


Suggested change

Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe message write retries when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.

The Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe retries on message writes when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.

bfreiberg · 2024-01-22T07:36:57Z

sqs-lambda-ddb-cdk-ts/README.md

+
+## Considerations
+
+* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended enable this feature as in a production environment.


Suggested change

* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended enable this feature as in a production environment.

* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended to enable this feature as in a production environment.

bfreiberg · 2024-01-22T07:37:40Z

sqs-lambda-ddb-cdk-ts/README.md

+## Considerations
+
+* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended enable this feature as in a production environment.
+* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired WCUs.


Suggested change

* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired WCUs.

* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired write capacity units (WCUs).

bfreiberg · 2024-01-22T07:41:14Z

sqs-lambda-ddb-cdk-ts/README.md

+\
+**NOTE:** If there are no data points loading on the graph, wait a couple of minutes as CloudWatch takes some time to aggregate metrics.
+
+9. Terminate the script


Please highlight this because it will continue to incur costs if the script is not stopped

bfreiberg · 2024-01-22T07:41:29Z

sqs-lambda-ddb-cdk-ts/README.md

+
+
+----
+Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.


Suggested change

Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.

bfreiberg · 2024-01-22T07:45:00Z

sqs-lambda-ddb-cdk-ts/README.md

+
+
+
+6. Once you have confirmed that DynamoDB is throttling write requests, go back to the [config](./lib/config/tables.config.ts) file in repo and lower the `lambdaReservedConcurrency` property to a value less than or equal to the current writeCapacity value of the table (1 is a sufficient value for this demo). Increase the `retryAttempts` property value to a high value (20 should be sufficient for this demo) to give Lambda time to keep up with any subsequent throttling events.


The writeCapacity of the table is already at 1, how should the lambdaReservedConcurrency be reduced even more?

bfreiberg · 2024-01-22T07:45:31Z

sqs-lambda-ddb-cdk-ts/example-pattern.json

+    ]
+  },
+  "deploy": {
+    "text": ["<code>cdk deploy</code>"]


Suggested change

"text": ["<code>cdk deploy</code>"]

"text": ["cdk deploy"]

bfreiberg · 2024-01-22T07:45:44Z

sqs-lambda-ddb-cdk-ts/example-pattern.json

+    "text": ["See the GitHub repo for detailed testing instructions."]
+  },
+  "cleanup": {
+    "text": ["Delete the stack: <code>cdk destroy</code>."]


Suggested change

"text": ["Delete the stack: <code>cdk destroy</code>."]

"text": ["<code>cdk destroy</code>"]

bfreiberg · 2024-01-22T07:47:26Z

I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?

The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the sqs-lambda-ddb-sam-ts subfolder

Hi, why would a customer not use Application Auto Scaling to adjust the amount of Provisioned Concurrency instead? https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html#managing-provisioned-concurency

astan54321 · 2024-01-30T22:03:24Z

I'm not sure I understand the use-case that this pattern addresses. Could you please elaborate what the benefit is?

The pattern is meant to show how to offload dynamodb writes using sqs + lambda per table pipelines to avoid ddb table throttling. This is done in collaboration with gulid@ and their pattern for the same architecture but using SAM instead (under the sqs-lambda-ddb-sam-ts subfolder

Hi, why would a customer not use Application Auto Scaling to adjust the amount of Provisioned Concurrency instead? https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html#managing-provisioned-concurency

The Provisioned Concurrency value would only affect the number of available execution environments for the Lambda handler, not limit the maximum amount. With this pattern, we are trying to offload the DynamoDB writes to lambda to allow the supposed application side to write as frequently as possible, rather than having to time requests in line with the DynamoDB WCUs on the application side. For this purpose, the Reserved Concurrency value is more useful since it represents the max number of concurrent requests allowed, which limits how fast the Lambda handler is allowed to handle the requests to keep the table from throttling as much as possible and avoid redriving requests.

bfreiberg · 2024-02-02T08:30:06Z

Hi @astan54321 , thanks for your reply. I understand the use-case better now. In this case, I think it would be best to integrate maximum concurrency on the SQS side as well: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/

astan54321 · 2024-02-02T19:26:28Z

Hi @astan54321 , thanks for your reply. I understand the use-case better now. In this case, I think it would be best to integrate maximum concurrency on the SQS side as well: https://aws.amazon.com/blogs/compute/introducing-maximum-concurrency-of-aws-lambda-functions-when-using-amazon-sqs-as-an-event-source/

Tanks for sharing this @bfreiberg! This helps a lot. Will go ahead and integrate this and update the PR.

bfreiberg · 2024-03-01T08:17:42Z

Hi @astan54321, did you have a chance to work on this?

astan54321 · 2024-03-01T14:42:20Z

Hi Ben, yes, we are still working on making edits to the README and making sure the testing stop makes sense. Should have an updated commit early next week.

singledigit · 2024-04-02T22:25:19Z

Any updates on this?

astan54321 · 2024-04-03T20:38:20Z

Hi Ben and Eric, yes just pushed updates with updated functionality to use SQS max concurrency instead of Lambda's reserved concurrency, which fits the goal of the pattern better. Changes were required to the README by another team member and I during the past few weeks which is what took so long with this (along with changing schedules). Let me now if anhy final changes need to be made if this is to be merged into the base ServerLess land repo. Thanks.

bfreiberg

Looks good, just two minor comments

bfreiberg · 2024-04-09T17:17:44Z

sqs-lambda-ddb-cdk-ts/README.md

+8. For a visual representation, you can also look at CloudWatch metrics for **Write throttled requests(count)** in the monitoring tab of the DynamoDB Console. You should see a spike of write throttled requests and then it should decrease around the time you made the config change. It should look similar to the picture below.
+\
+\
+![alt text](./images/write_throttling_metrics.png)


Please make the alt text a bit more relevant for users with accessibility issues

added alt text description

bfreiberg · 2024-04-09T17:18:37Z

sqs-lambda-ddb-cdk-ts/example-pattern.json

+    "text": [
+      "This code demonstrates how an Amazon SQS queue can be used to buffer records for an Amazon DynamoDB table and mitigate the risk for a client of being throttled.",
+      "The CDK code deploys Ingress and Dead-Letter SQS queues, AWS Lambda functions, destination DynamoDB tables and CloudWatch alarms.",
+      "AWS Lambda function uses NodeJS 18 runtime."


Any reason not to use Node v20 as it is the latest supported runtime?

updated node version in example.json and in CDK template. updated AWS SDK imports for testing script

bfreiberg · 2024-04-16T21:07:36Z

Looks good, thanks for your contribution. A Developer Advocate will merge the PR and publish the changes to serverlessland.com.

Andrei Stan added 5 commits January 10, 2024 14:53

initial commit

056f1f5

format pattern resources

ef4ddee

fix failing checks

58a4520

fix TypeScript language formatting

63a39c9

remove cloudwatch alarms

c68b310

benjasl-stripe assigned bfreiberg and benjasl-stripe Jan 10, 2024

astan54321 mentioned this pull request Jan 10, 2024

New pattern submission: sqs-lambda-ddb-cdk-ts #2031

Closed

remove outdated pattern folder

f98866a

bfreiberg reviewed Jan 22, 2024

View reviewed changes

edit documentation

036ef0e

change lambda reserved concurrency to sqs max concurrency

a203f85

bfreiberg reviewed Apr 9, 2024

View reviewed changes

Andrei Stan and others added 2 commits April 9, 2024 17:06

update lambda node version and add README alt text

75c8a40

Add final pattern file

c3d6df3

bfreiberg added the Patterns Guardian approved label Apr 16, 2024

julianwood merged commit 6b42a1f into aws-samples:main Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New serverless pattern - sqs-lambda-ddb-cdk-ts #2032

New serverless pattern - sqs-lambda-ddb-cdk-ts #2032

astan54321 commented Jan 10, 2024

bfreiberg commented Jan 17, 2024 •

edited

Loading

bfreiberg commented Jan 17, 2024

astan54321 commented Jan 17, 2024 •

edited

Loading

astan54321 commented Jan 17, 2024 •

edited

Loading

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg Jan 22, 2024

bfreiberg commented Jan 22, 2024

astan54321 commented Jan 30, 2024 •

edited

Loading

bfreiberg commented Feb 2, 2024

astan54321 commented Feb 2, 2024

bfreiberg commented Mar 1, 2024

astan54321 commented Mar 1, 2024

singledigit commented Apr 2, 2024

astan54321 commented Apr 3, 2024

bfreiberg left a comment

bfreiberg Apr 9, 2024

astan54321 Apr 9, 2024

bfreiberg Apr 9, 2024

astan54321 Apr 9, 2024

bfreiberg commented Apr 16, 2024


		The purpose of this pattern is to address write throttling issues of Provisioned Capacity DynamoDB Tables by offloading the requests to SQS to handle writes asynchronously. This pattern is deployed through an AWS Cloud Development Kit (CDK) app which provisions DynamoDB tables, SQS queues and DLQs, and Lambda functions. Test scripts have been provided for testing.

		Learn more about this pattern at Serverless Land Patterns: [INSERT SERVERLESS LAND LINK HERE]


		A new Amazon SQS queue will be created and configured to invoke an AWS Lambda function that will write the received messages into an Amazon DynamoDB table. The SQS queue redrive policy will be configured to send undeliverable messages to an Amazon SQS Dead-Letter Queue. Undeliverable messages are the ones that couldn't be written to the DynamoDB table with a maximum of `maxReceiveCount` number of attempts.

		Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.

	Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.
	The Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.


		Lambda function will stop writing messages into the DynamoDB table as soon as it catches an exception and will report the remaining messages to Amazon SQS service as failed.

		Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe message write retries when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.

	Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe message write retries when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.
	The Amazon DynamoDB table will have a minimal `ProvisionedThroughput` configuration for demo purposes, so that the users can observe retries on message writes when testing this stack. Real production throughput must be configured in accordance with the actual business requirements and message sizes.


		## Considerations

		* For simplicity and demo purposes, the DynamoDB Table deployed in this stack does not have Point-In-Time Recovery enabled. This pattern serves as a customizable template for a solution. It is recommended enable this feature as in a production environment.

	* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired WCUs.
	* For simplicity, this project uses a [config](./lib/config/tables.config.ts) file to configure the scaling limits of DynamoDB and Lambda. Customers may wish to automate this process to run in a closed loop based off desired write capacity units (WCUs).



		----
		Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.

	Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
	Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.




		6. Once you have confirmed that DynamoDB is throttling write requests, go back to the [config](./lib/config/tables.config.ts) file in repo and lower the `lambdaReservedConcurrency` property to a value less than or equal to the current writeCapacity value of the table (1 is a sufficient value for this demo). Increase the `retryAttempts` property value to a high value (20 should be sufficient for this demo) to give Lambda time to keep up with any subsequent throttling events.

	"text": ["Delete the stack: <code>cdk destroy</code>."]
	"text": ["<code>cdk destroy</code>"]

New serverless pattern - sqs-lambda-ddb-cdk-ts #2032

New serverless pattern - sqs-lambda-ddb-cdk-ts #2032

Conversation

astan54321 commented Jan 10, 2024

bfreiberg commented Jan 17, 2024 • edited Loading

bfreiberg commented Jan 17, 2024

astan54321 commented Jan 17, 2024 • edited Loading

astan54321 commented Jan 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bfreiberg commented Jan 22, 2024

astan54321 commented Jan 30, 2024 • edited Loading

bfreiberg commented Feb 2, 2024

astan54321 commented Feb 2, 2024

bfreiberg commented Mar 1, 2024

astan54321 commented Mar 1, 2024

singledigit commented Apr 2, 2024

astan54321 commented Apr 3, 2024

bfreiberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bfreiberg commented Apr 16, 2024

bfreiberg commented Jan 17, 2024 •

edited

Loading

astan54321 commented Jan 17, 2024 •

edited

Loading

astan54321 commented Jan 17, 2024 •

edited

Loading

astan54321 commented Jan 30, 2024 •

edited

Loading