I tried spark.eventlog.dir=false and then the Spark history server is not showing any information related to this.

Is there any similar way Job history server wont show our application related information similar to Spark History Server when spark.eventlog.dir is set to false.

CodePudding user response：

This can be done if we set below highlighted property to any location other than default location which is there in mapred-site.xml

yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi -D mapreduce.map.memory.mb=512 -D mapreduce.reduce.memory.mb=512 -Dmapreduce.jobhistory.intermediate-done-dir=<<"new-location">> 2 10

When this is done then the Job history server can't move logs from intermediate done dir to done dir. Since it reads from the location which is configured in mapred-site.xml