AWS EMR Upgrade from 5.30 to 6.6.0-CodePudding

I am trying to upgrade my current EMR from 5.30.0 to 6.6.0 or even 5.35.0 to run my python batch script. Whenever I run my python file, even if it is a plain print statement I get the following error on any version of EMR from 5.35 to 6.6.0 . Does anyone have any tips or idea on this issue ?

   Traceback (most recent call last):
  File "/mnt1/yarn/usercache/hadoop/appcache/application_11111111_0001/container_16111111110_0001_02_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/mnt1/yarn/usercache/hadoop/appcache/application_111111110_0001/container_16111111630_0001_02_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o92.sql.
: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:127)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:237)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:116)

CodePudding user response：

Looks like the issue was in the applications being installed in the EMR. Hadoop and Hive were not installed on EMR and as a result the above error.