how to connect an s3 bucket w/ airflow-CodePudding

I have an airflow task where I try and load a file into an s3 bucket. I have airflow running on a Ec2 instance. Im running AF version 2.4.3 I have done

pip install 'apache-airflow[amazon]'

I start up my AF server, log in and go to the Admin section to add a connection. I open a new connection and I dont have an option for s3.

My only Amazon options are: Amazon Elastic MapReduce Amazon Redshift Amazon Web services.

what else am I missing?

CodePudding user response：

You need to define aws connection under "Amazon Web Services Connection"

for more details see here

CodePudding user response：

You should define the connection within your DAG.

You should also use a secure settings.ini file to save your secrets, and then call those variables from your DAG.

See this answer for a complete guide: Airflow s3 connection using UI