I have used SciSpark for reading numerous netcdf files in Spark 2.x. However, I am unable to compile it for Spark 3.x (due to various types of errors such as cannot be applied to (Array[Int]), found : Array[Long] with required: Array[Int], cannot be applied to (AnyVal), etc.). Unable to even conclude if it is compatible with Spark 3. Could anyone please suggest if:

• SciSpark is compatible with Spark 3, and

• Any alternative to SciSpark in Spark 3.x?

CodePudding user response：

Looks like https://github.com/SciSpark/SciSpark isn't updated to use Spark 3 because last commit was 4 year ago.

CodePudding user response：

If you already have a Spark cluster you can use Apache Sedona to read netcdf files. Documentation about netcdf files is sparse, but you can ask questions in their mailing list.