I have some CSV files that I want to copy to a specific folder in ADLS based on the date column within the file.

i.e. CSV file has a column named "date" that reads "2022-02-23" on all rows. I want to copy that file to a folder that has the corresponding year and month, such as "/curated/UK/ProjectABC/2022/02"

I've got a Lookup activity that's pointing to the source CSV file and populating a Set Variable activity with the month using this dynamic content - @substring(string(activity('Lookup1').output.firstrow.date),5,2)

Would this be the right approach, to use a variable?

I cant use variables in the Directory portion of the Sink Dataset, as far as I know.

Have you come across this situation before?

CodePudding user response：

Sounds like you're on the right path. You can use absolutely use Dataset parameters:

Then populate them in your pipeline using a variable (or parameter, or expression):