Springboot version 2.0.1,
Kafka - clients version 1.0.1,
Kafka server version 0.11

Only enable producers configuration. Idempotence: true for a particular configuration, to ensure the message does not appear to repeat to send, the rest are default.

The situation is the service at the time of the morning producers blocking would happen a topic, specific exception is:
Failed to send. Nested exception is org.apache.kafka.com mon. Errors. TimeoutException:
Expiring 2 record (s) for myTopic - 3:59349 ms has passed since the last append

And all the other producers are not sent, other producers also can appear abnormal:
Org.apache.kafka.com mon. Errors. TimeoutException: Failed to update metadata after 60000 ms.

But only this one service will appear this kind of situation, more than 20 other services involved in kafka were can normal consumption and production, also can work normally after the service restart, but one to five or six o 'clock in the morning and this would happen and need to restart.

Now look at the server network, memory, I/O, CPU, etc., consumption is small, and in the morning has less amount of data.
This topic one day there will be about 150 w control the amount of data, basic it in the daytime, evening will probably have less than 10 w the amount of data.

Now is don't understand why will appear this problem in the morning.

CodePudding user response:

Look at the log, look at this point in time to send the data content is what? Look at this time point in other services without doing anything?

CodePudding user response:

reference 1st floor ml_dark response:

look at the log, look at this point in time to send the data content is what? Look at this time point in other services without doing anything?

Seen the log, no other service in large-scale operation, less than 5% CPU consumption, memory is less than 40%, the I/O and network is the normal state, and the topic of processing logic is very simple, basic is the passthrough, service will accept messages from upstream topic and then transfer agreement to this topic, the topic of consumer is the spark, the data to calculate.

CodePudding user response:

MyTopic - 3:59349 ms has passed since the last append
Feel this theme has a problem, deleted build a try again

CodePudding user response:

The

reference 3 floor ml_dark response:

myTopic - 3:59349 ms has passed since the last append
Feel the topic has a problem, deleted to try to build a

Reconnection is ok, service a restart can sustain her life a day, and is a small amount of data it's easy to appear problem, very strange

CodePudding user response:

Normal data, and there were small amount of data and disconnect, can consider to whether configuration the largest free time lead to connection is broken,

CodePudding user response:

reference 5 floor kkkkk0lllll reply:

normal data, the data quantity is small, disconnect the condition, can consider to whether configured the maximum idle time lead to disconnects,

Configure the largest free time indeed, but the other topic is the same configuration, like sending an SMS verification code topic sometimes ten days not a message a few days are good. The topic is very strange.

CodePudding user response:

How are you solve it, can you to ask