I have a situation in my json file I have two columns eventid & sectionname which is dynamic in nature. As mentioned in diagram input

I need output like this which transformation I can perform and since section name is dynamic i.e instead of 301 it will come 501 also in future & I don't want my stream to fail is there any way in pyspark or scala.

CodePudding user response：

df_target = (df_source.set_index(list of static columns)
  .rename_axis([New_Column_Name], axis=1)
  .stack()
  .reset_index())

Where df_source is your dataframe in pandas