Notebooks are nice, but REPL is sometimes more useful. Am I somehow able to run spark-shell that executes on Databricks? Like:

spark-shell --master https://adb-5022.2.azuredatabricks.net/

I looked through available tools related to Databricks (databricks connect, dbx, ...), but it seems there's no such functionality.

CodePudding user response：

Databricks connect is the tool that you need if you want to execute code from you local machine in the Databricks cluster. Same as the spark-shell, the driver will be on your local machine, and executors are remove. The databricks-connect package installs the modified distribution of the Apache Spark so you can use spark-shell, pyspark, spark-submit, etc. - just make sure that that directory is in the PATH.

P.S. but I really don't understand why notebooks doesn't work for you - spark-shell doesn't have any superior features compared to them.