Home > Blockchain >  I'm writing an simple udf but it is showing illegal's start of expression as I am new to s
I'm writing an simple udf but it is showing illegal's start of expression as I am new to s

Time:08-16

def nameFilter= (name:String):String =>{
  if(name.startsWith("A")) name
}

val register= udf(nameFilter)


concatDf.select(register(col("Region"))).show()
})

concatDf.select(ageFilter(col("Region"))).show()

I have concatDf where I have a column called Region and I want The Region whose name starts with "A" should be present.

O I have written the following udf. but the exception says

<console>:6: error: illegal start of declaration
         name.startsWith("A") name

CodePudding user response:

If you say

whose name starts with "A" should be present

this means that you want to use filter and not select. If you want, for example, in this table:

 --- ------- 
|id |name   |
 --- ------- 
|1  |Albania|
|2  |Germany|
 --- ------- 

to only keep Albania (since it starts with A), you might want to change your logic.

To do this without UDFs (which is the best way), you can simply do:

df.filter(col("name").startsWith("A"))

But if you want to use a UDF, you still have to use filter, therefore you have to modify your function nameFilter to return a boolean instead of a string, like:

def nameFilter = (name: String) => name.startsWith("A")

Once you do that, you can use:

val register = udf(nameFilter)

To register the function, then finally:

df.filter(register(col("name")))

To actually filter the rows. Hope this is what you need, good luck!

  • Related