Aws Lambda function to save DynamoDb deleted data to S3-CodePudding

I have set up lambda function, firehose, and S3 bucket to save deleted DynamoDB data to S3.

My lambda function was written in C#.

    var client = new AmazonKinesisFirehoseClient();
    try
    {
        context.Logger.LogInformation($"Write to Kinesis Firehose: {list.Count}");
        var request = new PutRecordBatchRequest
        {
            DeliveryStreamName = _kinesisStream,
            Records = new List<Amazon.KinesisFirehose.Model.Record> ()
        };
        foreach (var item in list)
        {
            var stringWrite = new StringWriter();
            
            string json = JsonConvert.SerializeObject(item, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore });
            byte[] byteArray = UTF8Encoding.UTF8.GetBytes(ToLiteral(json));
            var record = new Amazon.KinesisFirehose.Model.Record
            {
                Data = new MemoryStream(byteArray)
            };
            request.Records.Add(record);
        }
        if (request.Records.Count > 0)
        {
            var response = await client.PutRecordBatchAsync(request);
            Console.WriteLine($"FailedPutCount: {response.FailedPutCount} status: {response.HttpStatusCode}");
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
    }

The "list" is a list of objects

There are some messages in Firehose logs:

 "message": "Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed",
"errorCode": "Lambda.FunctionError"

I also see some error msg S3 bucket:

Error>

AccessDenied

Access Denied

TCG5YV3ZM3EQ4DWE

jRDkHxATNADXilsiy59IYkkechd6nqlyAEe0UDuN7qaNZS3zEIjblZJS9mGMktdCSb8AIFUam5I=

However, when when I downloaded the error file. I saw the following:

attemptsMade":4,"arrivalTimestamp":1661897166462,"errorCode":"Lambda.FunctionError","errorMessage":"Check your function and make sure the output is in required format. In addition to that, make sure the processed records contain valid result status of Dropped, Ok, or ProcessingFailed","attemptEndingTimestamp":1661897241573,"rawData":"XXXXXXXXX"

The "rawData" can be decoded to the original json string writing to firehose using

I found my problem. The lambda function to put records to firehose works. The problem was I enabled the data transformation and add C# lambda there. That causes the data format issue. The data transformation function needs to return a different format data. The solution is to disable the Data transformation.