OptionConverter.convertLevel Error in Hadoop Mapreduce job-CodePudding

I am getting a weird error while executing a mapreduce job in my Hadoop cluster. This error is intermittent. Sometimes, it fails the mapper and sometimes it succeeds even if the error is thrown.

2022-12-28 01:20:53,882 ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org/apache/log4j/helpers/OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level; (loaded from file:/opt/hadoop-3.3.0/share/hadoop/common/lib/log4j-1.2.17.jar by sun.misc.Launcher$AppClassLoader@350ed555) called from class org.apache.log4j.config.PropertiesConfiguration (loaded from file:/data/hadoop/yarn/usercache/hdfs-user/appcache/application_1671477750397_6197/filecache/10/job.jar/job.jar by sun.misc.Launcher$AppClassLoader@350ed555).

Gradle File

plugins {
    id 'java'
    id "com.github.davidmc24.gradle.plugin.avro" version "1.3.0"
}
dependencies {
    compileOnly group: 'org.apache.hadoop', name: 'hadoop-client', version: '3.3.0'
    implementation group: 'org.apache.avro', name: 'avro', version: '1.8.1'
    implementation group: 'org.apache.avro', name: 'avro-mapred', version: '1.10.2'
    implementation group: 'io.netty', name: 'netty-buffer', version: '4.1.51.Final'
    implementation  group: 'joda-time', name: 'joda-time', version:'2.8.1'
    implementation group: 'org.javatuples', name: 'javatuples', version: '1.2'
    implementation group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: '2.12.4'
    testImplementation group: 'junit', name: 'junit', version: "$junit4Version"
    implementation project(':libraries:java-dcl')
    implementation project(':libraries:sdp')

    testImplementation group: 'org.assertj', name: 'assertj-core', version: '3.6.1'
}

generateAvroJava {
    source("${projectDir}/src/main/avro")//sourcepath avrofile
}

// The alternative to below is to put all java that is dependent on scala (or on the java dependent on the scala) in the src/main/scala dir
// https://stackoverflow.com/questions/23261075/compiling-scala-before-alongside-java-with-gradle
sourceSets {
    main {
        java {
            srcDirs = ['src/main/java']
        }
    }
}

CodePudding user response：

I had to use the org.apache.log4j.Logger package in all of the internal libraries and got rid of other logger classes. There were many files where the change was required.