Home > Blockchain >  How to add columns in GroupBy in Aggregate transformation dynamically in Azure Data Flow
How to add columns in GroupBy in Aggregate transformation dynamically in Azure Data Flow

Time:01-12

I have 200 columns ,out of those 100 columns need to be in GroupBy .So its very painful to add the 100 columns manually ,clicking the dropdown,selecting the column .It takes more time to add columns in this manner.

Is there any way ,to lessen the time to add up 100 cols in groupBy all at once.

CodePudding user response:

  • AFAIK, there is no option within the aggregate transformation to dynamically select all the required columns that are of different types to apply group by operation.
  • One way you can try is to manually enter the names of all the columns to which you have to apply group by on, as an array.
  • Let's say I have the following data (all columns of same type):

enter image description here

  • Considering I want to apply group by on id and test columns, the following is how I have given that. I used collect as my aggregate function on team column.

enter image description here

  • This will give the result as shown in the below image:

enter image description here

  • But this requires us to convert all columns into the same type (string) to perform operations. And also converts the grouped data into an array of values as shown in the above picture (not separate columns but an array).

  • Another manual way that you can try is to alter the dataflow JSON to add the column names. If I select 2 columns using drop down,

enter image description here

  • The JSON would be as shown below:

enter image description here

  • So, if you want to include another column, it will change as shown below. You can build a similar string for required columns and edit the dataflow JSON.

enter image description here

  • Related