feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions #80

mmiermans · 2024-02-28T23:50:04Z

Goal

MC-681 Emit ML features when an ML item is scheduled, or fails to be scheduled.

Changes

content-common contains shared code to emit Snowplow events, with tests.
fix: prospect-api and curated-corpus-api failed to retry Snowplow events on error, because Snowplow's request library

References

JIRA ticket:

MC-681

Issue filed with Snowplow library:

Retry is disabled by default for POST method snowplow/snowplow-javascript-tracker#1290

Spec PR:

https://github.com/Pocket/spec/pull/211

…ler-dummy-item

* chore: migrate to msw from nock * chore: add test case for null data * fix: dist import

pocket-ci · 2024-03-01T14:00:32Z

Plan Result (corpus-scheduler-lambda-cdk)

CI link

Plan: 0 to add, 1 to change, 0 to destroy.

Update
- aws_lambda_function.corpus-scheduler-sqs-lambda_F2ECDF9F

Change Result (Click me)

  # aws_lambda_function.corpus-scheduler-sqs-lambda_F2ECDF9F will be updated in-place
  ~ resource "aws_lambda_function" "corpus-scheduler-sqs-lambda_F2ECDF9F" {
        id                             = "CorpusSchedulerLambda-Prod-SQS-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:CorpusSchedulerLambda-Prod-SQS-Function:18" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:CorpusSchedulerLambda-Prod-SQS-Function:18/invocations" -> (known after apply)
        tags                           = {
            "environment" = "Prod"
            "service"     = "CorpusSchedulerLambda"
        }
      ~ version                        = "18" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"     = (sensitive value)
                # (4 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

pocket-ci · 2024-03-01T14:01:13Z

Plan Result (prospect-api-cdk)

CI link

Plan: 0 to add, 5 to change, 0 to destroy.

Update
- aws_dynamodb_table.dynamodb_prospects_dynamodb_table_9854E41E
- aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6
- aws_iam_policy.translation-lambda_translation-sqs-lambda_execution-policy_E70E40B9
- aws_lambda_function.bridge-lambda_bridge-sqs-lambda_343B543A
- aws_lambda_function.translation-lambda_translation-sqs-lambda_B9BDF6BA

Change Result (Click me)

  # data.aws_iam_policy_document.application_ecs_service_ecs-iam_data-ecs-task-role-policy_090CC3AD will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "application_ecs_service_ecs-iam_data-ecs-task-role-policy_090CC3AD" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "dynamodb:BatchGet*",
              + "dynamodb:DescribeTable",
              + "dynamodb:Get*",
              + "dynamodb:Query",
              + "dynamodb:Scan",
              + "dynamodb:UpdateItem",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
            ]
        }
      + statement {
          + actions   = [
              + "s3:*",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:s3:::pocket-prospectapi-prod-images",
              + "arn:aws:s3:::pocket-prospectapi-prod-images/*",
            ]
        }
      + statement {
          + actions   = [
              + "events:PutEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:events:us-east-1:996905175585:event-bus/PocketEventBridge-Prod-Shared-Event-Bus",
            ]
        }
    }

  # data.aws_iam_policy_document.translation-lambda_translation-sqs-lambda_execution-policy-document_94EBBD6E will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_iam_policy_document" "translation-lambda_translation-sqs-lambda_execution-policy-document_94EBBD6E" {
      + id      = (known after apply)
      + json    = (known after apply)
      + version = "2012-10-17"

      + statement {
          + actions   = [
              + "logs:CreateLogGroup",
              + "logs:CreateLogStream",
              + "logs:DescribeLogStreams",
              + "logs:PutLogEvents",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:logs:*:*:*",
            ]
        }
      + statement {
          + actions   = [
              + "dynamodb:BatchWriteItem",
              + "dynamodb:DescribeTable",
              + "dynamodb:PutItem",
              + "dynamodb:Query",
              + "dynamodb:UpdateItem",
            ]
          + effect    = "Allow"
          + resources = [
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
              + "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
            ]
        }
      + statement {
          + actions   = [
              + "ec2:AttachNetworkInterface",
              + "ec2:CreateNetworkInterface",
              + "ec2:DeleteNetworkInterface",
              + "ec2:DescribeInstances",
              + "ec2:DescribeNetworkInterfaces",
            ]
          + effect    = "Allow"
          + resources = [
              + "*",
            ]
        }
    }

  # aws_dynamodb_table.dynamodb_prospects_dynamodb_table_9854E41E will be updated in-place
  ~ resource "aws_dynamodb_table" "dynamodb_prospects_dynamodb_table_9854E41E" {
        id                          = "PROAPI-Prod-Prospects"
        name                        = "PROAPI-Prod-Prospects"
        tags                        = {
            "environment" = "Prod"
            "service"     = "ProspectAPI"
        }
        # (9 unchanged attributes hidden)

      - global_secondary_index {
          - hash_key           = "scheduledSurfaceGuid" -> null
          - name               = "scheduledSurfaceGuid-prospectType" -> null
          - non_key_attributes = [] -> null
          - projection_type    = "ALL" -> null
          - range_key          = "prospectType" -> null
          - read_capacity      = 0 -> null
          - write_capacity     = 0 -> null
        }
      + global_secondary_index {
          + hash_key           = "scheduledSurfaceGuid"
          + name               = "scheduledSurfaceGuid-prospectType"
          + non_key_attributes = []
          + projection_type    = "ALL"
          + range_key          = "prospectType"
          + read_capacity      = 5
          + write_capacity     = 5
        }

        # (5 unchanged blocks hidden)
    }

  # aws_iam_policy.application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6 will be updated in-place
  ~ resource "aws_iam_policy" "application_ecs_service_ecs-iam_ecs-task-role-policy_6FC89FB6" {
        id        = "arn:aws:iam::996905175585:policy/ProspectAPI-Prod-TaskRolePolicy"
        name      = "ProspectAPI-Prod-TaskRolePolicy"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "dynamodb:UpdateItem",
                          - "dynamodb:Scan",
                          - "dynamodb:Query",
                          - "dynamodb:Get*",
                          - "dynamodb:DescribeTable",
                          - "dynamodb:BatchGet*",
                        ]
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
                        ]
                      - Sid      = ""
                    },
                  - {
                      - Action   = "s3:*"
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:s3:::pocket-prospectapi-prod-images/*",
                          - "arn:aws:s3:::pocket-prospectapi-prod-images",
                        ]
                      - Sid      = ""
                    },
                  - {
                      - Action   = "events:PutEvents"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:events:us-east-1:996905175585:event-bus/PocketEventBridge-Prod-Shared-Event-Bus"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags      = {
            "environment" = "Prod"
            "service"     = "ProspectAPI"
        }
        # (4 unchanged attributes hidden)
    }

  # aws_iam_policy.translation-lambda_translation-sqs-lambda_execution-policy_E70E40B9 will be updated in-place
  ~ resource "aws_iam_policy" "translation-lambda_translation-sqs-lambda_execution-policy_E70E40B9" {
        id        = "arn:aws:iam::996905175585:policy/ProspectAPI-Prod-Sqs-Translation-ExecutionRolePolicy"
        name      = "ProspectAPI-Prod-Sqs-Translation-ExecutionRolePolicy"
      ~ policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action   = [
                          - "logs:PutLogEvents",
                          - "logs:DescribeLogStreams",
                          - "logs:CreateLogStream",
                          - "logs:CreateLogGroup",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:*:*:*"
                      - Sid      = ""
                    },
                  - {
                      - Action   = [
                          - "dynamodb:UpdateItem",
                          - "dynamodb:Query",
                          - "dynamodb:PutItem",
                          - "dynamodb:DescribeTable",
                          - "dynamodb:BatchWriteItem",
                        ]
                      - Effect   = "Allow"
                      - Resource = [
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects/*",
                          - "arn:aws:dynamodb:us-east-1:996905175585:table/PROAPI-Prod-Prospects",
                        ]
                      - Sid      = ""
                    },
                  - {
                      - Action   = [
                          - "ec2:DescribeNetworkInterfaces",
                          - "ec2:DescribeInstances",
                          - "ec2:DeleteNetworkInterface",
                          - "ec2:CreateNetworkInterface",
                          - "ec2:AttachNetworkInterface",
                        ]
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        tags      = {
            "environment" = "Prod"
            "service"     = "ProspectAPI"
        }
        # (4 unchanged attributes hidden)
    }

  # aws_lambda_function.bridge-lambda_bridge-sqs-lambda_343B543A will be updated in-place
  ~ resource "aws_lambda_function" "bridge-lambda_bridge-sqs-lambda_343B543A" {
        id                             = "ProspectAPI-Prod-Sqs-Bridge-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Bridge-Function:77" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Bridge-Function:77/invocations" -> (known after apply)
        tags                           = {
            "environment" = "Prod"
            "service"     = "ProspectAPI"
        }
      ~ version                        = "77" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"                  = (sensitive value)
                # (5 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

  # aws_lambda_function.translation-lambda_translation-sqs-lambda_B9BDF6BA will be updated in-place
  ~ resource "aws_lambda_function" "translation-lambda_translation-sqs-lambda_B9BDF6BA" {
        id                             = "ProspectAPI-Prod-Sqs-Translation-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Translation-Function:905" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:ProspectAPI-Prod-Sqs-Translation-Function:905/invocations" -> (known after apply)
        tags                           = {
            "environment" = "Prod"
            "service"     = "ProspectAPI"
        }
      ~ version                        = "905" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"                      = (sensitive value)
                # (3 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

pocket-ci · 2024-03-01T14:01:24Z

Plan Result (curated-corpus-api-cdk)

CI link

⚠️ Resource Deletion will happen

This plan contains resource delete operation. Please check the plan result very carefully!

Plan: 2 to add, 0 to change, 1 to destroy.

Create
- local_file.application_ecs_service_appspec_94386B9C
Replace
- null_resource.application_ecs_service_create-task-definition-file_6BADBAF7

Change Result (Click me)

  # local_file.application_ecs_service_appspec_94386B9C will be created
  + resource "local_file" "application_ecs_service_appspec_94386B9C" {
      + content              = jsonencode(
            {
              + Resources = [
                  + {
                      + TargetService = {
                          + Properties = {
                              + LoadBalancerInfo = {
                                  + ContainerName = "app"
                                  + ContainerPort = 4025
                                }
                              + TaskDefinition   = "arn:aws:ecs:us-east-1:996905175585:task-definition/CuratedCorpusAPI-Prod:768"
                            }
                          + Type       = "AWS::ECS::Service"
                        }
                    },
                ]
              + version   = 1
            }
        )
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "appspec.json"
      + id                   = (known after apply)
    }

  # null_resource.application_ecs_service_create-task-definition-file_6BADBAF7 must be replaced
-/+ resource "null_resource" "application_ecs_service_create-task-definition-file_6BADBAF7" {
      ~ id       = "7274471649080069521" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "alwaysRun" = "2024-03-05T20:31:58Z" -> (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 1 to destroy.

…le-candidates

katerinachinnappan and others added 13 commits February 21, 2024 14:30

init

49385b3

feat: corpus scheduler lambda scheduling items

c7d790c

Merge branch 'main' into katerina/feat-corpus-scheduler-lambda-schedu…

73de627

…ler-dummy-item

clean & merge main

e87063b

fix lint errors

dc45f6d

do not include testHelpers during compilation

85c49a3

update

9c946c6

getting url metadata from parser directly

e24b889

update

f4ac740

cleanup

d0e6716

more cleanup

2c5b94d

addressing feedback

6303bcb

chore: migrate to msw from nock (#81)

277c26e

* chore: migrate to msw from nock * chore: add test case for null data * fix: dist import

mmiermans force-pushed the mc-681-scheduler-snowplow-event branch from cfbff18 to 2b63dd8 Compare March 1, 2024 13:56

mmiermans changed the base branch from main to katerina/feat-corpus-scheduler-lambda-schedule-candidates March 1, 2024 13:56

pocket-ci added the corpus-scheduler-lambda-cdk/add-or-update label Mar 1, 2024

pocket-ci added the prospect-api-cdk/add-or-update label Mar 1, 2024

pocket-ci added the curated-corpus-api-cdk/destroy label Mar 1, 2024

katerinachinnappan added 4 commits March 1, 2024 11:47

remove validateSource

78b9c3a

lint fix

b9b8cc7

sending scheduledSource

812725c

Merge branch 'main' into katerina/feat-corpus-scheduler-lambda-schedu…

6d28666

…le-candidates

mmiermans force-pushed the mc-681-scheduler-snowplow-event branch from 260d095 to 8537436 Compare March 4, 2024 19:57

mmiermans added 4 commits March 4, 2024 14:00

feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions

51dcd0d

chore: move common Snowplow logic to content-common

c1d2071

chore: use shared Snowplow code in prospect-api and curated-corpus-api

1298320

fix: Snowplow retries on error

91585d0

chore: move Snowplow Micro to integration test file

e433537

mmiermans force-pushed the mc-681-scheduler-snowplow-event branch from 71a1bd7 to e433537 Compare March 4, 2024 22:02

fix: lint error snowplowHttpProtocol unused

d212be5

Base automatically changed from katerina/feat-corpus-scheduler-lambda-schedule-candidates to main March 4, 2024 23:18

feat: emit Snowplow events on error

1eb7b40

mmiermans closed this Mar 6, 2024

mmiermans deleted the mc-681-scheduler-snowplow-event branch March 29, 2024 17:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions #80

feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions #80

mmiermans commented Feb 28, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading

feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions #80

feat: [MC-681] Implement scheduled_corpus_candidate Snowplow functions #80

Conversation

mmiermans commented Feb 28, 2024 • edited Loading

Goal

Changes

References

pocket-ci commented Mar 1, 2024 • edited Loading

Plan Result (corpus-scheduler-lambda-cdk)

pocket-ci commented Mar 1, 2024 • edited Loading

Plan Result (prospect-api-cdk)

pocket-ci commented Mar 1, 2024 • edited Loading

Plan Result (curated-corpus-api-cdk)

⚠️ Resource Deletion will happen

mmiermans commented Feb 28, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading

pocket-ci commented Mar 1, 2024 •

edited

Loading