I am using Azure Data Factory to create the Data pipelines and store the output as JSON files in the Azure blob storage account.

The issue is I am using the Web activity with rest API for retrieving the data and parsing the data along with authorization token to copy activity.

Now I want to filter the output of web activity by removing or skipping the data from web activity before parsing it to the copy activity. So I can save the data by removing or skipping data in the JSON files as I want.

Is it possible to filter the output of web activity before parsing to the copy activity? If possible please let me know. It is urgent to solve this issue.

CodePudding user response：

Here is a simple demo for your usecase. First , im using a jsonPlaceHolder data check this API:

WebActivity :

Under settings, added the jsonPlaceHolder url and GET method, in headers : Content-Type : application/json.

SetVariable:

Under Variables , Name : data , Value : @json(activity('TestWebActivity').output.Response)

Filter Activity:

Under Settings , items : @variables('data') , condition : @startswith(item().username,'B')

Explanation:

here i set the variable name in set activity to "data", so now in variables you can use data anywhere in your pipeline, in Filter activity , i set the data array in "items" and basically each json in the array is reffered as "item()" , so in order to filter the json array based on a value , in condition you can select the key : items().keyName ..

in order to save your data to Blob storage , add a Copy activity after the filter and link the blob storage as a sink..