Spark partition can be removed?

Is a RDD on RDD transformation, the system will automatically remove the partition? There are still active to remove RDD operator?

I hope you bosses disambiguation,

CodePudding user response:

Partition is automatically inherit, unless you colase or repartion partition again,

CodePudding user response:

Me two sentences, so, in the source code, each RDD inherit a class dependency, the dependency is the RDD since
Now there is a RDDA ancestors RDD, for example, then the Dependency of the RDD is NIL empty
There is now a RDDB is RDDA RDD, so this RDDB Dependency is no longer empty, but has the value, the value is the reference RDDA
So there is no automatic removal problem, because in addition to the ancestors RDD Dependency is empty, so RDD in source level is more than a Dependency class!
Personal understanding, hope the general net NongMen comment

CodePudding user response:

Instead of inheriting the dependency each a RDD has dependency, it is RDD source:
The abstract class RDD [T: ClassTag] (
@ transient private var _sc: SparkContext,
@ transient private var deps: Seq [ the Dependency [_]]
)