THE ISSUE

I'm trying to load data from S3 Bucket using Lambda (NODEJS). I've looked at various topics here where similar questions were posted and have tried to combine reading the data from many of them (here and here and a few more as well).

The setup is S3(Create Events)>SNS>SQS>LAMBDA. The invocation from S3 all the way to LAMBDA is working perfectly. The code below and the Cloudwatch logs below indicates successful invocation of lambda for every PUT event in S3. I have the IAM roles configured to read from SQS and S3 for LAMBDA I am NOT using SSE on the S3 buckets or objects

But regardless of the approaches I have put (both are in the code below) -> the Lambda is not outputting to console the contents of the CSV File -> You can see the final console logs are not being outputted where the code tries to read the S3 data. Has something else changed over the years that needs updating in the S3 API calls?

CODE

const readline = require('readline');
const AWS = require("aws-sdk");
const s3 = new AWS.S3({apiVersion: '2006-03-01'});

exports.handler = async (event) => {
    console.log('###### Received event:', JSON.stringify(event, null, 2));
    for (const record of event.Records) {
        try {
            const body = JSON.parse((record.body));
            // console.log(`###### Record BODY ${JSON.stringify(body, null, 2)}`);
            const message = JSON.parse(body.Message);
            //console.log(`##### Record BODY MESSAGE ${JSON.stringify(message, null, 2)}`);
            const bucket = message.Records[0].s3.bucket.name;
            const key = message.Records[0].s3.object.key;
            console.log(`###### Record BODY bucket ${bucket}`);
            console.log(`###### Record BODY key ${key}`);
            
            const params = {
                Bucket: bucket, Key: key
            };
            
            console.log(`###### THE PARAMS ${JSON.stringify(params, null, 2)}`);
            s3.getObject(params , function (err, data) {
                if (err) {
                    console.log(`###### ERR ${err}`);
                    throw err;
                } else {
                    console.log(`###### ${data.Body.toString()}`);
                }
            
            })            
            const rl = readline.createInterface({
                input: s3.getObject(params).createReadStream()
            });
            
            rl.on('line', function(line) {
                console.log(`####### RECORD DATA LINE ${line}`);
            })
            .on('close', function() {
                console.log(`####### RECORD DATA CLOSED!!!!!`);
            });

        } catch (e ) {
            console.log(`###### Error ${JSON.stringify(e, null, 2)}`);
        }
    }
    return `Successfully processed ${event.Records.length} messages.`;
};

IAM ROLES

For the sake of brevity I've removed the SQS specific details which work fine. I've given ALL the accesses to S3 but just listing getObject here again for brevity.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                ..
                "s3:GetObject",
                ..
            ],
            "Resource": "arn:aws:s3:::my-REDACTED-bucket"
        }
    ]
}

CLOUDWATCH LOGS

2022-08-31T12:00:07.897Z    0191e48d-f3ac-5af8-9334-bf04bcd815d8    INFO    ###### Record BODY bucket my-REDACTED-bucket
2022-08-31T12:00:07.897Z    0191e48d-f3ac-5af8-9334-bf04bcd815d8    INFO    ###### Record BODY key test.csv
2022-08-31T12:00:07.897Z    0191e48d-f3ac-5af8-9334-bf04bcd815d8    INFO    THE PARAMS 
{
    "Bucket": "my-REDACTED-bucket",
    "Key": "test.csv"
}

END RequestId: 0191e48d-f3ac-5af8-9334-bf04bcd815d8

CodePudding user response：

The basic problem is in your async code. Your code is exiting before anything happens. You need to modify the code to await the asynchronous results.

Use:

const data = await s3.getObject(params).promise();

If you prefer to use the streamed, readline approach then see this answer.