Suppress Spark Java API warnings-CodePudding

I am using Spark Java API in my thesis project. Since I need to print a lot of results in the console, I am trying to suppress all spark related INFO logging. I am working the project both locally (for debugging etc.) and remotely on a cluster. I managed to turn off most of them using the below sample configuration.

Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);

final SparkConf sparkConf = new SparkConf().setAppName("WordCount").setMaster("local[16]");
final JavaSparkContext ctx = new JavaSparkContext(sparkConf);

ctx.setLogLevel("ERROR");

However, I am still getting WARNINGS every time I initialize a spark context, which can be disorienting at times. Below I am giving a sample of the console output:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/C:/Users/SKIKK/.m2/repository/org/apache/spark/spark-unsafe_2.12/3.2.2/spark-unsafe_2.12-3.2.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

I tried different suggestions in the forum; however, I was unable to find a working solution.

CodePudding user response：

Here it is suggested to run with Java 8, to avoid these messeges. However, this states that "Since Java 17, the –illegal-access option is entirely removed".

I switched to Java 17. The below line did the magic: Logger.getRootLogger().setLevel(Level.toLevel("ERROR"));