How to stream data from SQL Table with Apache Spark with Databricks-CodePudding

I am attempting to stream from sql table using the following:

my_sales =  spark.read.jdbc(jdbcUrl, dbo.table)

static = spark.read.format("csv").load(my_sales)
dataSchema = static.schema

I am trying to read in the data from the table with the following:

rawdf = (spark.readStream 
      .format("csv") \
      .option("maxFilesPerTrigger", 1) \
      .schema(dataSchema) \
      .csv(dataPath)
           )

I am using the following to write the data to the following location

saveloc = '/mnt/raw/streaminglocation/'


streamingQuery = (
  rawdf
  .writeStream
  .format("csv")
  .outputMode("append")
  .option("checkpointLocation", f"{saveloc}/_checkpoints")
  .option("mergeSchema", "true")
  .start(saveloc)
)

However this failing.

Is it possible to stream from a SQL table?

CodePudding user response：

This is not possible. JDBC sources are not supported for Spark Structured Streaming.

Not convinced of the upfront coding either.

Use CDC with Kafka, or materialized updateable views with CDC with KAFKA, or Debezium.