Home > front end >  Is .nonEmpty method an action or transformation in spark scala?
Is .nonEmpty method an action or transformation in spark scala?

Time:02-01

Is .nonEmpty method an action or transformation in spark scala?

For ex:

val don = Seq("Hi", "Hello")
if (don.nonEmpty) {
     println("Not empty")
   }

CodePudding user response:

Here is an example where .nonEmpty is used to filter a sequence, so it is a transformation, while the action is collect:

val rdd: RDD[Seq[String]] = sparkContext.parallelize(
  Seq(Seq("Hi", "Hello"), Seq()))
val NotEmptyRDD = rdd.filter(x => x.nonEmpty)
val notEmptyArray = NotEmptyRDD.collect()
for(element <- notEmptyArray)
  System.out.println(element)
  •  Tags:  
  • Related