I'm building an Spark Streaming program that i want that it read certain messages and return they as "unread" to the topic if the message could not be processed by the program.

My intention is that the program eventually read the message again and try to process it.

Is that possible?

CodePudding user response：

No. That is not possible if you think of Kafka offsetting.

CodePudding user response：

If you "return" a record to the original topic, it'll be appended to the end of the topic, and therefore consumed again on the next poll.

This will cause an infinite consumption loop (until whatever condition that put the record back is no longer true)

What you're asking for seems to be a "dead-letter-queue", which is implemented with different topics, not the original. As for how to process that in Spark, you'd probably have to maintain some Try object (Scala), or other boolean type, that knows if a particular event had been processed successfully; then filter that out before writing to any downstream systems.