how to upload recently modified files from AWS s3 to dynomodb-CodePudding

I have AWS S3 bucket1 and files are getting uploaded daily in s3 but I want to upload recently added files names only not previously added files names in dynomodb.

Bucket 1 :

abc.txt --filename has been added into dynomodb with the previous date. acd.txt --today's file, I wanted to add this filename into dynomodb.

Take the recently uploaded filename in S3 and update it into dynomodb.

Appreciate it if could provide some insights about it.

CodePudding user response：

You should use the S3 Event Notifications to send new file information to Lambda and write that to DynamoDB.

CodePudding user response：

Using S3 event notifications as pointed out, your Lambda function will receive an event similar to the following:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      <snip>
      "s3": {
      <snip>
        "object": {
          "key": "test/key",
          "size": 1024,
          "eTag": "0123456789abcdef0123456789abcdef",
          "sequencer": "0A1B2C3D4E5F678901"
        }
      }
    }
  ]
}

You should be able to parse the key to get the filename, then use boto3 to write the data to DynamoDB using:

ddb = boto3.client('dynamodb')
ddb.put_item(
    TableName='tablename',
    Item={
        "filename": event["Records"][0]["s3"]["object"]["key"],
        "key": 123
)

Note that it will be necessary to configure the appropriate event source mapping between the S3 bucket and the Lambda function, as well as the permissions to write to DDB within the Lambda's execution role.

https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html