how to read s3 files from apache beam python?-CodePudding

I am using Apache Beam python SDK to read s3 file data.

code I am using

    ip = (pipe
          | beam.io.ReadFromText("s3://bucket_name/file_path")
          | beam.Map(lambda x: x.split(","))
          | beam.Map(print)

          )

but it gives a error

 S3ClientError('Unable to locate credentials', None)}")}

Anybody have any idea how to give credentials to s3 client thorugh apache beam python file

CodePudding user response：

You need to ensure the credentials are available on your worker machines. If you're using dataflow, you could install them in a custom container.