I have ever running spark contexts. In my code using localcheckpoint to truncate lineage based on my alogorithm after checkpointed i want to remove RDDs reference please help me....

CodePudding user response：

To clean checkpoints you need this conf:

spark.cleaner.referenceTracking.cleanCheckpoints    true    #Default: false; Controls whether to clean checkpoint files if the reference is out of scope.

See spark configuration;

If you want to uncache\remove the RDD:

rdd.unpersist()