Turn to

A sparkstreaming program, components including mysql kafka and redis

Local [*] consumption mode test performance is very good, a partition can handle 4 w record

Switch to cool yarn model, can receive data, but not the batch processing, task stuck, regardless of the yarn client or yarn cluater all not line, resources to also is very big, set maxbatchsize 40 can consume, 400 consumption washed-up, nor an error

Don't know where is wrong, please teach

CodePudding user response:

If not work, it is suggested that learn flink + k8s

CodePudding user response:

reference 1st floor RivenDong response:

if not work, suggested that learn flink + k8s

Business needs, window consumption pattern is more suitable

CodePudding user response:

The

refer to the original poster JD9527 response:

for help
A sparkstreaming program, components including mysql kafka and redis

Local [*] consumption mode test performance is very good, a partition can handle 4 w record

Switch to cool yarn model, can receive data, but not the batch processing, task stuck, regardless of the yarn client or yarn cluater all not line, resources to also is very big, set maxbatchsize 40 can consume, 400 consumption washed-up, nor an error

Don't know where is wrong, please teach

Locating the problem, about the driver side and executor and code execution