I get outside web server log data through the flume (test data) as a producer, production 1 per second, unable to consumption, kafka side side to stop to wait for production to get not consumption, I tried to produce a set at least 4 seconds, the consumer end to consumption in time, but I've tested production 5 to 10 seconds, can consume to the consumer end, this is how to return a responsibility? From production to consumption, success is about 2 seconds delay, 3 1 h2g virtual machine cluster, the topic is three partitions

CodePudding user response:

The flume is how configuration? I doubt the flume is you brush data into memory channel, because a buffer quantity is not enough, there is no sink in a timely manner to the kafka
Normal the kafka directly to the configuration of the channel is ok, don't sink configuration

CodePudding user response:

I am to do with 2 machine, such as a and b, a machine is used for acquisition, sink avro type setting, the data collection to the b machine, b the sink of the machine set of kafka, as producers of kafka

CodePudding user response:

Can I reply to earn points?

CodePudding user response:

There are two things, lead to this problem
A flume is set on the number of data of: if you write the number of data of kafka amounted to less than the set, so only if the stop will begin to write,
The second is the flume set the amount of data size (file size)