Data from kafka through time and device ID, the current state of the equipment, every 10 seconds within 1 minute before statistics to receive information on the number of devices,
Val df=kafkaDataFrame
WithWatermark (" datatime ", "1 minute")
GroupBy (
Window ($datatime "1", "minute", "10 seconds),
"$" devId
). Agg (Max ($" datatime "))

Df. WriteStream
OutputMode (outputMode. Append)
ForeachBatch ((df: Dataset (Row), batch: Long)=& gt; {
Val FMT=DateTimeFormatter. OfPattern (" MM - dd yyyy - HH: MM: ss ")
Val dayStr=LocalDateTime. Now (). The format (FMT)
Df. Foreach (row=& gt; Println (" s "${dayStr} : ${row. ToSeq}"))
})
See the output data is not the last time window, summary data contains several times in front of the window, I as long as the last time?

CodePudding user response:

This is structedStream API, use the stream (SparkStreamingContext) API, defect is unable to use the event time