I'm copying data from on-prem files to blob azure storage. The first part of the pipeline is exactly that, pulling the last modified file from on-prem to blob storage (blob_input). After that I'm having problems. I have one file out of 5 that doesn't have actual data but only shows "no data" which makes it impossible to transport to sql, that's the ultimate goal. So I connected my "copy data" to the filtering pipeline that I created. However, my pipeline doesn't filter out the file. Could someone tell me what my pipeline is missing? I have added pictures below to make things clearer but Stackoverflow wouldn't let me post more so I have created a google docs with all the info Files and Path forEach1 forEach2 forEach3 lookup1 ifcondition1 copydata1 copydata2

CodePudding user response：

I reproduced your scenario the problem causing to filter file where it contains only no data is you are using first row as header then the lookup count for that file is 0 which is not equal to 1 because of it is not filtering out that file.

To resolve this, you can try:

This condition @greater(activity('Lookup1').output.count,0), it will filter file if the lookup count is greater than 0.

Output Copy activity is not coping data from daily4 file because it has sample input like you, and it failed in if condition.