Celery running for about a week or so of time, has been in the treatment of the task, probably around 3-4 per second, is not very big, ran ran don't perform, queue accumulation is a lot of news, through flower go to see the worker state, has been offline, see the log:
Worker and mq connection has been opened, and went to check the reason, played down the work of the operation information, found that the master has been stuck in this
Went to check the source code, source code is roughly such
Cause the problem of speculation:
1. Whether related to environment, resembles the two times, the problem of memory, but low memory situations, restart the worker will return to normal
2. The master card die, receive MSG has been receiving less than?
Have leaders met the problem, seeking help, or help to give some ideas
Registered task description:
Processing messages passed parameters, do some string concatenation placeholder processing, then mass storage, and finally also send_task asynchronous to another queue processing
CodePudding user response:
Added worker log file picture:
Checked the error, should be the mq active segment out of the link, but why will take the initiative to break, mq queue delete source says, queue deleted after celery should be according to the configuration files to create, I tried several times to do create, but there's a really failed, error the same, but considering the production environment, and testing colleagues feedback, no one should go to delete a queue, continue to screening analysis,,,,
CodePudding user response:
Recently too busy didn't update, the next research direction to understand worker retry mechanism, understand it why not go to retry the link