I'm trying to build a kafka spark structured streaming stateful application with low latency. By saying low latency I mean a couple of hundred of milliseconds each job.

The spark app read data from a kafka topic with partition number that's 2 times of executor core, then process and output it to another kafka topic. The rate that the data is produced into this topic is 100 records/s with approximate 2 kb record size. The DAG of the job indicate that stage that includes reading from kafka source takes 0.5s . This stage basically transform the data from kafka into a dataset of custom case class, followed by groupByKey and flatMapGroupsWithState function from second stage. The shuffle write time in web UI is 0 ms(which should be small because the shuffled data size is around 10~20kb). So AFAIK the only time-consuming operation should be reading from kafka.

I've read about that kafka can perform much better than this. The end-to-end latency can be smaller than 100 ms.

The kafka broker is not heavily loaded. I don't know if it's related to the question but the whole application runs on a kubernetes cluster. And there's pic of this stage and pic of the whole query attached if they might help.

Sorry I cannot post the code. Is there anything I can try doing?

Best Regards

CodePudding user response：

Today I find that some tasks of the first stage takes 0.5s while others do not, which looks very suspicious to me. And I looked more into the kafka settings. There's one configuration called fetch.max.wait.ms, which prevent the consumer task from stoping waiting for new message for 500 ms by default. After reducing this config everything goes fine. More info here: fetch.max.wait.ms