I am using this spark image from datamechanics, and am assuming that the image has hadoop installed because the name says so. But I can find it in the usual locations (/usr/local, /opt/, etc). Also the docs are not easy to read to understand how the image was built (mostly because I couldn't find a code like file which I can read). Does anyone know if hadoop is actually installed in the datamechanics images. If not, is there an alternate image that is recommended for spark and hadoop? Thanks!

CodePudding user response：

The name of the image is a reference to the Spark downloadable package that includes Spark 3.1.1 with Hadoop 3.2.0 client libraries, as compared to the "bring your own" version...

It does not "come with" scripts to run HDFS and YARN, therefore does have have either "installed". You can search Dockerhub for datanode/namenode images, but neither are necessary to run Spark from the image you've found.