Python: How to move files in a structured folder based on year/month/date format?-CodePudding

Currently I have a spark job that reads the file, creates a dataframe, does some transformations and then move those records in "year/month/date" format. I am achieving this by:

df.write.option("delimiter", "\t").option("header", False).mode(
            "append"
        ).partitionBy("year", "month", "day").option("compression", "gzip").csv(
            config["destination"]
        )

I want to achieve the same by pythonic way. So, in the end it should look like:

data/2022/04/14
data/2022/04/15

CodePudding user response：

Based on your question , instead of using partitionBy you can also modify your config['destination'] , as s3 will take care of the necessary folder creations underneath the s3 path

s3_dump_path = config["destination"] ### 's3:/test-path/'
>>> curr_date = datetime.now().date()
>>> year,month,day = curr_date.strftime('%Y'),curr_date.strftime('%m'),curr_date.strftime('%d')
>>> s3_new_path = '/'.join([s3_dump_path,year,month,day])
>>> s3_new_path
's3:/test-path//2022/04/14'
>>> config["destination"] = s3_new_path

df.write.option("delimiter", "\t").option("header", False).mode(
            "append"
        ).option("compression", "gzip").csv(
            config["destination"]
        )