I'm currently working on the implementation of SNS-notifications being the intermediary between S3-bucket-uploads and an upload-handler Lambda Function.

The information flow should look like this:

1. Upload of a file to an S3-bucket
2. This should trigger an SNS-topic "upload-notification"
3. Lambda-function ("upload-handler") are subscribed to this SNS-topic: depending on which bucket received a file upload, a certain lambda-function should be triggered by the SNS-notification

--> How can I obtain information like the "S3 bucket name" etc. from the event that triggered the SNS-notification?

I hope for a possibility like with lambda functions where you can extract information e.g. from a JSON-object produced by SNS. If that doesn't exist, I'd be delighted to learn about other approaches, but somehow I need to extract this information programmatically/automatically from SNS to hand it over to the upload-handler lambda function in Step 3.

Details on the terraform definition blocks:

1. aws_sns_topic_subscription:

resource "aws_sns_topic_subscription" "start_from_upload_topic" {
  topic_arn = var.upload_notification_topic_arn
  protocol  = "lambda"
  endpoint  = module.start_from_upload_handler.arn
}

2. aws_s3_bucket_notification:

resource "aws_s3_bucket_notification" "start_from_upload_handler" {
  for_each = local.input_bucket_id_map

  bucket = each.value
  topic {
    topic_arn = module.upload_notification.topic_setup.topic_arn
    events    = ["s3:ObjectCreated:*"]
  }
}

3. SNS-module "upload_notification"

module "upload_notification" {
  source  = "../../modules/sns_topic"
  name    = "${var.platform_settings.prefix}-upload-notification"
  key_arn = var.platform_settings.logging_settings.logging_key_arn
  allowed_producers = [
    "s3.amazonaws.com",
    "lambda.amazonaws.com",
    "edgelambda.amazonaws.com",
    "events.amazonaws.com",
    "states.amazonaws.com",
  ]
  allowed_consumers = ["lambda.amazonaws.com",
    "edgelambda.amazonaws.com",
    "events.amazonaws.com",
    "states.amazonaws.com",
  ]
  tags = local.tags
}

CodePudding user response：

As per documentation, the S3 bucket name (amongst other data like region, event time, bucket ARN, source IP etc.) will be inside the event message that is passed through to the Lambda from S3 via SNS in your case.

Records[0].s3.bucket.name

{  
   "Records":[  
      {  
         "s3":{  
            "bucket":{  
               "name":"bucket-name",
               ...
            },
            ...
         },
         ...
      }
   ]
}

CodePudding user response：

SNS needs to be set-up in conjunction with the Lambda-function and S3-uploads like so (in terraform, excluding KMS for this example):

resource "aws_lambda_permission" "start_from_upload_sns_topic" {
  statement_id  = "AllowExecutionFromSNStopic"
  action        = "lambda:InvokeFunction"
  function_name = module.start_from_upload_handler.arn
  principal     = "sns.amazonaws.com"
  source_arn    = var.upload_notification_topic_arn
}

resource "aws_s3_bucket_notification" "start_from_upload_handler" {
  for_each = var.input_bucket_name_map

  bucket = each.value
  topic {
    topic_arn = var.upload_notification_topic_arn
    events    = ["s3:ObjectCreated:*"]
  }
}

resource "aws_sns_topic_subscription" "start_from_upload_sns_topic" {
  topic_arn = var.upload_notification_topic_arn
  protocol  = "lambda"
  endpoint  = module.start_from_upload_handler.arn
}

The JSON-object the lambda-function receives from the SNS-topic looks like so:

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:eu-central-1:...",
            "Sns": {
                "Type": "Notification",
                "MessageId": "...",
                "TopicArn": "arn:aws:sns:eu-central-1:....",
                "Subject": "Amazon S3 Notification",
                "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"eu-central-1\",\"eventTime\":\"2021-10-27T15:29:38.959Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:...\"},\"requestParameters\":{\"sourceIPAddress\":\"....\"},\"responseElements\":{\"x-amz-request-id\":\"..\",\"x-amz-id-2\":\"...\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"tf-s3-topic-...\",\"bucket\":{\"name\":\"test-bucket-name\",\"ownerIdentity\":{\"principalId\":\"....\"},\"arn\":\"arn:aws:s3:::test-bucket-name\"},\"object\":{\"key\":\"test_file.json\",\"size\":189,\"eTag\":\"....\",\"versionId\":\"...\",\"sequencer\":\"...\"}}}]}",
                "Timestamp": "2021-10-27T15:29:40.086Z",
                "SignatureVersion": "1",
                "Signature": "...",
                "SigningCertUrl": "https://sns.eu-central-1.amazonaws.com/SimpleNotificationService...",
                "UnsubscribeUrl": "https://sns.eu-central-1.amazonaws.com....",
                "MessageAttributes": {}
            }
        }
    ]
}

We're interested in the "Message" - body of the incoming JSON-object, and this finally looks indeed like @Ermiya Eskandary mentioned in his answer pointing to the S3-notification-JSON-event-structure:

{
    'Records': [
        {
        's3': {
            'bucket': {
                'arn': 'arn:aws:s3:...',
                'name': 'bucket-name',
                },
            'object': {
                'key': 'upload_file_name.json',
                },
            },
        }
    ]
}

The take-away here is that one needs to bear in mind that the incoming JSON emitted by SNS has several top-layer dictionary keywords, which need to be "stripped-off" or "dug-through" in order to get to the actual S3-upload-event in the SNS-Message-body, which comes as a string-JSON-format which needs to be loaded into a proper dictionary-object.

Moreover, it is paramount to subscribe the lambda-function to the SNS-topic and allow the SNS-topic in turn to invoke said lambda-function.