I would like to create a pipeline with Copy Data (mapping JSON source to MySQL columns). At this moment it is working, but the tool is copying all the files from my source container after the 'created' trigger. Is it possible to retrieve only 'new added' files in my container, so I do not have double data in MySQL?

Example current situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 1   file 2

Example expected situation:

first copy -> retrieve file 1
(added new file)
second copy -> retrieve file 2

CodePudding user response：

This is actually a very used scenario , you should use LastModifiedDate for copying most recently added files and to prevent duplication.

Please check this link : https://docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview