I'm working on migrating some ETLs onto airflow, and was planning on using the S3 to snowflake operator to copy into snowflake: https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/_api/airflow/providers/snowflake/transfers/s3_to_snowflake/index.html

Currently we do the standard, PUT file into snowflake internal staging, and then COPY INTO our destination table for the file, however if I migrate it over to S3, and then use this operator in a DAG to COPY INTO the destination table, is there any reason to use an internal staging table? Best Practice reason, or any other?

CodePudding user response：

As per snowflake documentation , S3 acts external Stage and using COPY from there into SNOWFLAKE table. But if your business case needs one more layer of staging lean towards it otherwise you can follow documentation. Working well with one layer of external staging for us.

https://docs.snowflake.com/en/user-guide/data-load-s3-copy.html