I have a list that i want to parse its element and subelement and give them as variable for dataframe query but i get an error can anybody help here is my code
val ListParser = ("age,15,20","revenue,1,2")
val vars = "category in (1,2,4)"
val resultQuery : Dataset[Row] =
if(ListParser.size == 0){
responses.filter(vars)
}else if(ListParser.size == 1){
responses.filter(vars " AND " responses(ListParser(0)).between(ListParser(1).toInt, ListParser(2).toInt))
}else if(ListParser.size >= 2){
responses.filter(vars " AND " {for(a <- ListParser){
val myInnerList : List[String] = a.split(",").map(_.trim).toList
responses(myInnerList(0)).between(myInnerList(1).toInt,myInnerList(2).toInt)
}})
}else{
responses.filter(vars)
}
and i have another question i want only the value of response.filter() to be in resultQuery val
CodePudding user response:
It seems you are mixing SQL syntax with spark object syntax
try (listParser.map( s => {val l =s.split(","); s"""${l(0)} between (${l(1)},${l(2)})"""}) : vars).mkString(" AND ")
(assuming listParser is indeed a list and not a tuple )