Pyspark azuresql-CodePudding

I am using pyspark on mac os and trying to read from AzureSQL and I get the below said error

Spark 2.4.6; Scala 2.11 ; java 1.8.0_251

pyspark --jars spark-mssql-connector_2.11_2.4-1.0.2.jar

dbname = "db-test"
servername = "jdbc:sqlserver://"   "samplesql.database.windows.net:1433"
url = servername   ";"   "database_name="   dbname   ";"

df = spark.read
.format("com.microsoft.sqlserver.jdbc.spark") \
.option("url", url) \
.option("dbtable", table_name) \
.option("authentication", "ActiveDirectoryPassword") \
.option("hostNameInCertificate", "*.database.windows.net") \
.option("user", aduser) \
.option("password", adpwd) \
.option("encrypt", "true").load()

: java.sql.SQLException: No suitable driver
        at java.sql.DriverManager.getDriver(DriverManager.java:315)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:105)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:105)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:104)
        at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:35)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

What am i missing here?

CodePudding user response：

Try with --packages flag.

pyspark --packages com.microsoft.azure:spark-mssql-connector_2.11_2.4:1.0.2