Younger brother just learning the spark, with CDH build a cluster server, wrote the spark program running on the yarn, can not find the program debug logs, said in an online modification spark - the log4j under the conf. Properties configuration, increase the:
Log4j. Appender. The FILE=org. Apache. Log4j. DailyRollingFileAppender
. Log4j appenders. FILE. Threshold=DEBUG
. Log4j appenders. FILE. The FILE=/opt/app/sparkServicesLog/spark log
Log4j. Appender. FILE. The DatePattern='. '- dd yyyy - MM
Log4j. Appender. FILE. Layout=org.. Apache log4j. PatternLayout
Log4j. Appender. FILE. Layout. ConversionPattern=[% - 5 p] [% d {MM - dd yyyy - HH: MM: ss}] [% C {1} : % M: % L] % M % n: MM: ss}] [% C {1} : % M: % L] % M % n
Log had, but only the output of the spark logs, no program debug log output, consult everybody a great god, and how to solve??? Thank you very much,,

CodePudding user response:

Used Yarn model, StandAlone mode, we need to inherit Logging, and then use logInfo log, will generate log records on the Executor