Converting a python dataframe from AWS lambda to CSV in S3-CodePudding

I am trying to convert a write a python Dataframe and store it in S3. I have added the required layers in Lambda (s3fs & fsspec) and have provided requisite permissions to lambda to write to s3. But not I am getting the below error:

"errorMessage": "module 's3fs' has no attribute 'S3FileSystem'"

Below are the relevant lines of my code:

s3 = boto3.client('s3')
df.to_csv('s3://buckets/<my-bukcet-name>/mydata.csv')

Any pointers on what could be the reason for this?

Regards, Dbeings

CodePudding user response：

Instead of including those layers, I would recommend including the Amazon provided AWS Data Wrangler layer.

Then you would use AWS Data Wrangler to write your dataframe directly to S3, like the following:

import awswrangler as wr

wr.s3.to_csv(df, 's3://buckets/<my-bukcet-name>/mydata.csv', index=False)

CodePudding user response：

You need to install fsspec and s3fs python libraries.

Then you just need to import pandas. I used the following example and it works:

import pandas as pd
data=['a','b','c','d']
df = pd.DataFrame(data)
df.to_csv('s3://mybucket/file.csv')