How to print stacktrace in pyspark while using py4j logger?-CodePudding

I would like to print stacktrace of the exception raised during the execution of pyspark code. I am using native logger of the pyspark (py4j) but while executing logger.exception() it is failing. I have also tried logger.error()

    log4jLogger = sc._jvm.org.apache.log4j
    logger = log4jLogger.LogManager.getLogger(__name__)
    logger.info('first statement.')
    try:
        raise Exception('Dummy exception')        
    except Exception as e:
        logger.error('Something awful happened') # Does not print stacktrace
        logger.exception('Something awful happened') # print stacktrace but crashes
        
    logger.info('Importing module.')

Output looks like:

21/12/08 11:50:56 INFO __main__: First statement
21/12/08 11:50:56 ERROR __main__: Something awful happened
Traceback (most recent call last):
  File "/home/gaurav.gupta/projects/PoCs/brandMention/pyspark-scripts/dist/main.py", line 94, in <module>
    raise Exception('Dummy exception')        
Exception: Dummy exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gaurav.gupta/projects/PoCs/brandMention/pyspark-scripts/dist/main.py", line 97, in <module>
    logger.exception('Something awful happened')
  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/home/gaurav.gupta/miniconda3/envs/venv_pyspark/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 332, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o37.exception. Trace:
py4j.Py4JException: Method exception([class java.lang.String]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
    at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
    at py4j.Gateway.invoke(Gateway.java:274)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)


21/12/08 11:50:56 INFO SparkUI: Stopped Spark web UI at http://192.168.1.13:4040
21/12/08 11:50:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

CodePudding user response：

I have been able to find a workaround for the problem. I am getting the stack trace in the stringified format and then printing it as usual.

import traceback
strace = ''.join(traceback.format_exception(etype=type(ex), value=ex, tb=ex.__traceback__))
logger.error(strace)