spark-cassandra-connector on EMR serverless (PySpark)-CodePudding

I'm working on getting an application running on EMR Serverless and am having trouble pulling in the spark-cassandra-connector. I have no issue pulling it on my local, but all of my attempts at using the library on EMR Serverless have failed.

When I include the library using --jars s3://XXX/XXXX/spark-cassandra-connector-driver_2.12-3.2.0.jar, I error out on the following line

d = spark \
    .read \
    .format("org.apache.spark.sql.cassandra") \
    .options(table="YYYY", keyspace="YYY") \
    .load()

with the error

py4j.protocol.Py4JJavaError: An error occurred while calling o121.load.
: java.lang.ClassNotFoundException: 
Failed to find data source: org.apache.spark.sql.cassandra. Please find packages at
http://spark.apache.org/third-party-projects.html

When I try to add the package using --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0, the application times out with

com.datastax.spark#spark-cassandra-connector_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-5ee06249-c545-4b92-804f-ecedd322158a;1.0
    confs: [default]
:: resolution report :: resolve 524554ms :: artifacts dl 0ms
    :: modules in use:
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   1   |   0   |   0   |   0   ||   0   |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        module not found: com.datastax.spark#spark-cassandra-connector_2.12;3.2.0

    ==== local-m2-cache: tried

      file:/home/hadoop/.m2/repository/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      file:/home/hadoop/.m2/repository/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

    ==== local-ivy-cache: tried

      /home/hadoop/.ivy2/local/com.datastax.spark/spark-cassandra-connector_2.12/3.2.0/ivys/ivy.xml

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      /home/hadoop/.ivy2/local/com.datastax.spark/spark-cassandra-connector_2.12/3.2.0/jars/spark-cassandra-connector_2.12.jar

    ==== central: tried

      https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

    ==== spark-packages: tried

      https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom

      -- artifact com.datastax.spark#spark-cassandra-connector_2.12;3.2.0!spark-cassandra-connector_2.12.jar:

      https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: com.datastax.spark#spark-cassandra-connector_2.12;3.2.0: not found

        ::::::::::::::::::::::::::::::::::::::::::::::


:::: ERRORS
    Server access error at url https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.pom (java.net.ConnectException: Connection timed out (Connection timed out))

    Server access error at url https://repos.spark-packages.org/com/datastax/spark/spark-cassandra-connector_2.12/3.2.0/spark-cassandra-connector_2.12-3.2.0.jar (java.net.ConnectException: Connection timed out (Connection timed out))

I'm betting the --package issue comes from a firewall configuration problem, but I'm not seeing any way to open up access. As for the --jars issue, I'm not sure why pulling in the .jar isn't enough for Spark to recognize the org.apache.spark.sql.cassandra format.

Any help on either issue would be appreciated, thanks!

CodePudding user response：

Yes, the issue with --packages most probably due your egress settings that prevent accessing the Maven central.

To use --jars you need to specify all necessary jars, like, driver, connector, Java driver, etc. The easiest way to avoid this is to use so-called assembly build that is available on Maven Central too with com.datastax.spark:spark-cassandra-connector-assembly_2.12:3.2.0 coordinate. Just download referenced jar file.