Fixing msk-lambda-iam-python-sam and msk-lambda-iam-node-sam projects by adding Cloudformation templates as previous CFT doesn't work anymore due to Cloud9 dependency #2524
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bfreiberg
requested changes
Dec 11, 2024
msk-lambda-iam-node-sam/README.md
Outdated
|
|
||
| This project contains source code and supporting files for a serverless application that you can deploy with the AWS Serverless Application Model (AWS SAM) CLI. It includes the following files and folders. | ||
|
|
||
| - HandlerKafka - Code for the application's Lambda function. | ||
| - events - Invocation events that you can use to invoke the function. | ||
| - template.yaml - An AWS SAM template that defines the application's AWS resources. | ||
| - template_original.yaml - An AWS SAM template that defines the application's AWS resources. | ||
| - MSKAndKafkaClientEC2.yaml - A Cloudformation template file that can be used to deploy an MSK cluster and also deploy an EC2 machine with all pre-requisities already installed, so you can directly build and deploy the lambda function and test it out. |
There was a problem hiding this comment.
Suggested change
| - MSKAndKafkaClientEC2.yaml - A Cloudformation template file that can be used to deploy an MSK cluster and also deploy an EC2 machine with all pre-requisities already installed, so you can directly build and deploy the lambda function and test it out. | |
| - MSKAndKafkaClientEC2.yaml - An AWS CloudFormation template file that can be used to deploy an MSK cluster and also deploy an Amazon EC2 instance with all prerequisites already installed, so you can directly build and deploy the Lambda function and test it out. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
|
|
||
| ## Deploy the sample application | ||
| ## Run the Cloudformation template to create the MSK Cluster and Client EC2 machine |
There was a problem hiding this comment.
Suggested change
| ## Run the Cloudformation template to create the MSK Cluster and Client EC2 machine | |
| ## Deploy the CloudFormation template to create the MSK cluster and client EC2 instance |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
|
|
||
| The AWS SAM CLI is a serverless tool for building and testing Lambda applications. It uses Docker to locally test your functions in an Amazon Linux environment that resembles the Lambda execution environment. It can also emulate your application's build environment and API. | ||
| * [Run the Cloudformation template using the file MSKAndKafkaClientEC2.yaml] - You can go to the AWS Cloudformation console, create a new stack by specifying the template file. You can keep the defaults for input parameters or modify them as necessary. Wait for the Cloudformation stack to be created. This Cloudformation template will create an MSK cluster (Provisioned or Serverless based on your selection). It will also create an EC2 machine that you can use as a client. |
There was a problem hiding this comment.
Suggested change
| * [Run the Cloudformation template using the file MSKAndKafkaClientEC2.yaml] - You can go to the AWS Cloudformation console, create a new stack by specifying the template file. You can keep the defaults for input parameters or modify them as necessary. Wait for the Cloudformation stack to be created. This Cloudformation template will create an MSK cluster (Provisioned or Serverless based on your selection). It will also create an EC2 machine that you can use as a client. | |
| * [Deploy the CloudFormation template using the file MSKAndKafkaClientEC2.yaml] - You can go to the AWS CloudFormation console, create a new stack by specifying the template file. You can keep the defaults for input parameters or modify them as necessary. Wait for the CloudFormation stack to be created. This CloudFormation template will create an MSK cluster (Provisioned or Serverless based on your selection). It will also create an EC2 instance that you can use as a client. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
| The AWS SAM CLI is a serverless tool for building and testing Lambda applications. It uses Docker to locally test your functions in an Amazon Linux environment that resembles the Lambda execution environment. It can also emulate your application's build environment and API. | ||
| * [Run the Cloudformation template using the file MSKAndKafkaClientEC2.yaml] - You can go to the AWS Cloudformation console, create a new stack by specifying the template file. You can keep the defaults for input parameters or modify them as necessary. Wait for the Cloudformation stack to be created. This Cloudformation template will create an MSK cluster (Provisioned or Serverless based on your selection). It will also create an EC2 machine that you can use as a client. | ||
|
|
||
| * [Connect to the EC2 machine] - Once the Cloudformation stack is created, you can go to the EC2 console and log into the machine using either "Connect using EC2 Instance Connect" or "Connect using EC2 Instance Connect Endpoint" option under the "EC2 Instance Connect" tab. |
There was a problem hiding this comment.
Suggested change
| * [Connect to the EC2 machine] - Once the Cloudformation stack is created, you can go to the EC2 console and log into the machine using either "Connect using EC2 Instance Connect" or "Connect using EC2 Instance Connect Endpoint" option under the "EC2 Instance Connect" tab. | |
| * [Connect to the EC2 instance] - Once the CloudFormation stack is created, you can go to the EC2 console and log into the instance using either "Connect using EC2 Instance Connect" or "Connect using EC2 Instance Connect Endpoint" option under the "EC2 Instance Connect" tab. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
| * [Run the Cloudformation template using the file MSKAndKafkaClientEC2.yaml] - You can go to the AWS Cloudformation console, create a new stack by specifying the template file. You can keep the defaults for input parameters or modify them as necessary. Wait for the Cloudformation stack to be created. This Cloudformation template will create an MSK cluster (Provisioned or Serverless based on your selection). It will also create an EC2 machine that you can use as a client. | ||
|
|
||
| * [Connect to the EC2 machine] - Once the Cloudformation stack is created, you can go to the EC2 console and log into the machine using either "Connect using EC2 Instance Connect" or "Connect using EC2 Instance Connect Endpoint" option under the "EC2 Instance Connect" tab. | ||
| Note: You may need to wait for some time after the Cloudformation stack is created, as some UserData scripts continue running after the Cloudformation stack shows Created. |
There was a problem hiding this comment.
Suggested change
| Note: You may need to wait for some time after the Cloudformation stack is created, as some UserData scripts continue running after the Cloudformation stack shows Created. | |
| Note: You may need to wait for some time after the CloudFormation stack is created, as some UserData scripts continue running after the CloudFormation stack shows Created. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
| REPORT RequestId: 2d1041e7-fb49-4181-a8ac-15277f5d2b6c Init Duration: 11.37 msDuration: 2696.60 ms Billed Duration: 2697 ms Memory Size: 128 MB Max Memory Used: 128 MB | ||
| ``` | ||
| confirm by pressing y for both the questions | ||
| You should see the lambda function getting deleted and a final confirmation "Deleted successfully" on the command-line |
There was a problem hiding this comment.
Suggested change
| You should see the lambda function getting deleted and a final confirmation "Deleted successfully" on the command-line | |
| You should see the Lambda function getting deleted and a final confirmation "Deleted successfully" on the command line |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
| ```bash | ||
| sam delete | ||
| ``` | ||
| Next you need to delete the Cloudformation template that created the MSK Server and the EC2 machine by going to the Cloudformation console and selecting the stack and then hitting the "Delete" button. It will run for sometime but eventually you should see the stack getting cleaned up. |
There was a problem hiding this comment.
Suggested change
| Next you need to delete the Cloudformation template that created the MSK Server and the EC2 machine by going to the Cloudformation console and selecting the stack and then hitting the "Delete" button. It will run for sometime but eventually you should see the stack getting cleaned up. | |
| Next, you delete the CloudFormation template that created the MSK cluster and the EC2 instance by going to the CloudFormation console and selecting the stack. Then select the "Delete" button. Please note that it might take a while to complete. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-node-sam/README.md
Outdated
|
|
||
| * **Parameter MSKClusterId**: The unique ID of the MSKCluster, eg. a4e132c8-6ad0-4334-a313-123456789012-s2 | ||
| * **Parameter MSKClusterName**: The name of the MSKCluster | ||
| * **Parameter MSKClusterId**: The unique ID of the MSKCluster | ||
| * **Parameter MSKTopic**: The Kafka topic on which the lambda function will listen on |
There was a problem hiding this comment.
Suggested change
| * **Parameter MSKTopic**: The Kafka topic on which the lambda function will listen on | |
| * **Parameter MSKTopic**: The Kafka topic on which the Lambda function will listen on |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-python-sam/README.md
Outdated
|
|
||
| This project contains source code and supporting files for a serverless application that you can deploy with the AWS Serverless Application Model (AWS SAM) CLI. It includes the following files and folders. | ||
|
|
||
| - HandlerKafka - Code for the application's Lambda function. | ||
| - events - Invocation events that you can use to invoke the function. | ||
| - template.yaml - An AWS SAM template that defines the application's AWS resources. | ||
| - template_original.yaml - An AWS SAM template that defines the application's AWS resources. | ||
| - MSKAndKafkaClientEC2.yaml - A Cloudformation template file that can be used to deploy an MSK cluster and also deploy an EC2 machine with all pre-requisities already installed, so you can directly build and deploy the lambda function and test it out. |
There was a problem hiding this comment.
Suggested change
| - MSKAndKafkaClientEC2.yaml - A Cloudformation template file that can be used to deploy an MSK cluster and also deploy an EC2 machine with all pre-requisities already installed, so you can directly build and deploy the lambda function and test it out. | |
| - MSKAndKafkaClientEC2.yaml - A CloudFormation template file that can be used to deploy an MSK cluster and also deploy an EC2 instance with all prerequisites already installed, so you can directly build and deploy the Lambda function and test it out. |
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
msk-lambda-iam-python-sam/README.md
Outdated
| sam build | ||
| sam deploy --guided | ||
| ``` | ||
| ## Run the Cloudformation template to create the MSK Cluster and Client EC2 machine |
There was a problem hiding this comment.
I'm not going to mark every instance, but please double check the correct spelling:
Cloudformation -> CloudFormation
lambda -> Lambda
EC2 machine -> EC2 instance
There was a problem hiding this comment.
Checked in modified Readme.md files with changes incorporated
…m and msk-lambda-iam-python-sam to include retry mechanism for all yum install commands
…a-iam-python-sam based on review feedback from Ben Freiberg
bfreiberg
reviewed
Jan 6, 2025
There was a problem hiding this comment.
Any particular reason why you renamed the file?
There was a problem hiding this comment.
template_original.yaml file has variables for input parameters. In the Cloudformation template, I copy this file to template.yaml and then replace the variables with correct values for input variables such as MSK brokers, topic etc.
|
Looks good. Ready to merge once the file renaming is sorted out |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Fixing msk-lambda-iam-python-sam and msk-lambda-iam-node-sam projects by adding Cloudformation templates as previous CFT doesn't work anymore due to Cloud9 dependency. Added a Cloudformation template yaml file in each of the projects MSKAndKafkaClientEC2.yaml, that installs MSK and also creates a client EC2 machine that has all software pre-installed so the customer can use the EC2 to run sam build and sam deploy commands to test out the lambda function very easily. Modified README.md accordingly
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.