I have a streaming dataset. It reads from Kafka stream and writes into console.
val outputStream = inputStream.writeStream.format("console")
.option("truncate", value = false)
.option("checkpointLocation", "checkpoint")
outputStream.start()
spark.streams.awaitAnyTermination()
However when there are more than one record, I get a message on console :
only showing top 20 rows
And it shows only 20 records. Is there a way to retrieve all records ??
Spark version : 3.1.2
CodePudding user response:
There is an option numRows
, for example: 10000.
val outputStream = inputStream.writeStream.format("console")
.option("truncate", value = false)
.option("numRows",10000)
.option("checkpointLocation", "checkpoint")