How to get the final element of a hierarchical array in scala and apply aggregate functions on it?-CodePudding

I have a hierarchical array in a dataframe as :

customerId  accounts
IND0002 [["IND0002","ACC0155","323"],["IND0002","ACC0262","60"]]
IND0003 [["IND0003","ACC0235","631"],["IND0003","ACC0486","400"],["IND0003","ACC0540","53"]]
IND0004 [["IND0004","ACC0116","965"]]

I need to extract the last element lowest element from each list in the array Ex : from the 1st row I should get 323,60 and 2nd row should get 631,400,53

I tried using explode function, but its extracting just the first elemnts

customerId  accounts    col
IND0002 [["IND0002","ACC0155","323"],["IND0002","ACC0262","60"]]    ["IND0002","ACC0155","323"]
IND0002 [["IND0002","ACC0155","323"],["IND0002","ACC0262","60"]]    ["IND0002","ACC0262","60"]
IND0003 [["IND0003","ACC0235","631"],["IND0003","ACC0486","400"],["IND0003","ACC0540","53"]]    ["IND0003","ACC0235","631"]
IND0003 [["IND0003","ACC0235","631"],["IND0003","ACC0486","400"],["IND0003","ACC0540","53"]]    ["IND0003","ACC0486","400"]
IND0003 [["IND0003","ACC0235","631"],["IND0003","ACC0486","400"],["IND0003","ACC0540","53"]]    ["IND0003","ACC0540","53"]
IND0004 [["IND0004","ACC0116","965"]]   ["IND0004","ACC0116","965"]

val newDF1 = CustomerAccountOutput.withColumn("accounts", $"accounts"(size($"accounts")).minus(1))

CustomerAccountOutput.select($"customerID",explode($"accounts"))

CodePudding user response：

Use