Is it possible to produce a million records a second via kafka? How many servers should it take? Currently i am sending 7000 messages a second from a kafka producer and struggling to exceed this
I've noticed several sources say kafka can send millions of messages a second
I've created a job that uses @Autowired kafkaTemplate and makes a while loop that sends a short text string "asdf" I've set up linger to 1000ms and seeing messages group in sets of 7000. The producer , consumer broker, zookeeper are on the same machine and the broker and zookeeper and a very simple docker image with default configuration
I am maxing out around 7000 requests a second
application.props
spring.kafka.bootstrap-servers=PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093
host.name=localhost
Job to make calls
@Async
@Scheduled(fixedDelay = 15000)
public void scheduleTaskUsingCronExpression() {
generateCalls();
}
private void generateCalls() {
try{
int i = 0;
System.out.println("start");
long startTime = System.currentTimeMillis();
while(i <= 1000000){
String message = "Test Message sadg sad-";
kafkaTemplate.send(TOPIC, message i);
i ;
}
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime));
System.out.println("done");
}
catch(Exception e){
e.printStackTrace();
}
System.out.println("RUNNING");
}
Kakfa partition config
@Bean
public KafkaAdmin kafkaAdmin() {
//String bootstrapAddress = "localhost:29092";
String bootstrapAddress = "localhost:9092";
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
@Bean
public NewTopic testTopic() {
return new NewTopic("test-topic", 6, (short) 1);
}
Kafka consumer consuming messge
@KafkaListener(topics = "test-topic", groupId = "one", concurrency = "6" )
public void listenGroupFoo(String message) {
if(message.indexOf("-0") != -1){
startTime = new Date().getTime();
System.out.println("Starting Message in group foo: " message);
}
else if(message.indexOf("-100000") != -1){
endTime = new Date().getTime();
System.out.println("Received Message in group foo: " message);
System.out.println(endTime - startTime);
}
}
For hardware I have a 10900k with 64gb ram 5ghz clock speed 970 Evo single nvme disk 10 core 20 thread
All requests are from the same machine to the same machine
Is there a better way to organize / optimize the code to make a massive number of requests? Theories:
- Multiple Threads?
- Changing configurations of servers such as tomcat configs (receiving or sending side)?
- Not use the kafkaTemplate that is autowired or creating multiple?
- Modify Hardware to have multiple disks?
- Not use a job for the producer?
- Anything else anyone can think of to help?
CodePudding user response：
Key properties during KafkaProducers throughput tuning are:
Producer configs:
Topic config:
So after studying the linked documentation you will see two groups of properties coupled:
1. Effective batching and compression on Kafka producer
Kafka producer has built in mechanism of batching and compression. The mechanism gets the best results when the producer groups and then compresses many messages into a single batch, so simply setting a high value for batch size and enabling compression sounds very good, and actually it is, but to see what the last piece of puzzle is we need to go through the internal Kafka producer implementation. Kafka producer spawns a separate thread responsible for getting batches from the internal queue and sends them to the Kafka broker, we can name this Sender thread. In the default configuration the Sender gets batches from the queue whenever it is possible - no matter how large the batch size is, so you can set the batch size to 1024MB and still messages will not be grouped and compressed as much as expected. In order to solve this issue, and group and compress many records into single a batch, Kafka comes with linger.ms property, the property tells Sender thread to wait a while until it sends a batch to the Kafka server. In your specific case, theoretically the most effective way is to send one million messages in a single batch, but of course it depends on the size of the messages.
2. High throughput, but possibly inconsistent replication and acknowledgments.
Kafka producer has 3 different policies stating how long the producer needs to wait until it considers the batch as sent to server correctly. The most consistent policy waits until the message is saved by the leader server and required by the topics configuration number of the replication servers (min.in.sync.replicas). As opposed to, the lowest one, which prefers availability over consistency (as CAP theorem said), it is not waiting until any confirmation comes from the server, it works in push mode, you can compare it to sending data via UDP protocol. So in your case you need to use the second one. In order to get this, you need to disable indempotence feature of the Kafka producer, by setting enable.idempotence=false and ack=0 on the producer. The last part is to set min.insync.replicas=1 during topic creation, this property is coupled with acks property on producer side, and tells producer how many copies of data cluster needs to persist before it sends acknowledgment to the producer, it looks like it is not crucial in your case, because your producer with set acks=0 will not wait for any acknowledgments, but min.insync.replicas=1 is also interpreted by the cluster during leader election process - the process will not start for a single topic, until there are no available Kafka servers in a number higher or equal to min.insync.replicas, so to see the whole picture I had to mention it. The last parameter connected with our education is topic property unclean.leader.election.enabled, but it is more connected with availability than performance, but still imho it is good to know about that.
How to test Kafka Producer's performance?
In order to check, how mentioned properties work together (both producer and topic properties), Kafka distribution comes with a neat script named kafka-producer-perf-test. This script will help you check if the configuration fits your needs. Below you can find gists with very usefull commands .
CodePudding user response：
Spring Boot configures the producer acknowledgements by default to 'all'.
- For the highest throughput set the value to 0.
- For no data loss, set the ack-value to all (or -1).
- For high, but not maximum durability and for high but not maximum throughput - set the ack-value to 1. Ack-value 1 can be seen as an intermediate between both of the above.
I think this can be the major bottleneck for your kind of tests.
You can override it via a Spring property (or in YAML):
spring.kafka.producer.acks=0
OTOH, this could result in 1M records sent, but 99% of the messages get lost, so it depends what you want to achieve :-)