I'm pretty new to Acure Data Factory - ADF and have stumbled into somthing I would have solved with a couple lines of code.

Background

Main flow:

1. Lookup Activity fetchin an array of ID's to process
2. ForEach Activity looping over input array and uisng a Copy Activity pulling data from a REST API storing it into a database

Step #1 would result in an array containing ID's

{
    "count": 10000,
    "value": [
        {
            "id": "799128160"
        },
        {
            "id": "817379102"
        },
        {
            "id": "859061172"
        },
        ... many more...

Step #2 When the lookup returns a lot of ID's - individual REST calls takes a lot of time. The REST API supports batching ID's using a comma spearated input.

The question How can I convert the array from the input into a new array with comma separated fields? This will reduce the number of Activities and reduce the time to run.

Expecting something like this;

{
    "count": 1000,
    "value": [
        {
            "ids": "799128160,817379102,859061172,...."
        },
        {
            "ids": "n,n,n,n,n,n,n,n,n,n,n,n,...."
        }
        ... many more...

EDIT 1 - 19th Des 22 Using "Until Activity" and keeping track of posistions, I managed to use plain ADF. Would be nice if this could have been done using some simple array manipulation in a code snippet.

CodePudding user response：

The ideal response might be we have to do manipulation with Dataflow -

My sample input:

1. First, I took a Dataflow In that adding a key Generate (Surrogate key) after the source - Say new key field is 'SrcKey'

Data preview of Surrogate key 1

2. Add an aggregate where you group by mod(SrcKey/3). This will group similar remainders into the same bucket.

3. Add a collect column in the same aggregator to collect into an array with expression trim(toString(collect(id)),'[]').

Data preview of Aggregate 1

Store output in single file in blob storage.

OUTPUT