Home > Mobile >  Find the cause of constant downtimes and high RAM and SWAP usage
Find the cause of constant downtimes and high RAM and SWAP usage

Time:11-27

I have a webserver running Plesk and serving websites via "PHP-FPM served by nginx". I notice several downtimes every day, for some of the domains but not all of them. The domains that get down every day are kind of random, they're not the same every day. At the times that the websites are getting "down", I am noticing high RAM and SWAP usage, mostly from mysqld and php-fpm processes. I am attaching a screenshot below.

enter image description here

What I have tried so far:

  • Processes with their RAM usage

    ps -o pid,user,%mem,command ax | sort -b -k3 -r

  • Concurrent connections by IP to reveal possible http based ddos attacks

    netstat -anp |grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort

  • See GET and POST requests on all domains in real time to find anything suspicious

    tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354'

  • mysqltuner.pl for suggestions

  • Changed FPM settings and played with them for over a week and finally returned them to default values due to no effect

Nothing seems to reveal the cause of the issue. Also, restarting any services (the fpm service, or mysqld) does not fix the issue.

CodePudding user response:

PHP has the ability to tell you what memory usage the current execution is using, so you could add debug error_log calls that and add memory_get_usage() to the output you passing to the log and then you can narrow down where the huge jump in memory usage is coming from. in the code and you will then have to work out what that code is doing.

But stepping out of coding to work out what is causing this, this is why large production systems should not have DB and PHP on a single server. as running both means you will have the memory duplicated for the stuff going into the DB. the memory in PHP for the application to build the Query String, then the MySQL Server reaciving it for processing, 2 seperate servers prevents that duplication of memory usage.

CodePudding user response:

Your Wordpress sites are being targeted by bots that periodically attempt to crack the password to the site one letter at a time and randomly switch the site that they target to avoid being detected. This is very common for wordpress sites, especially those that have payment gateways and do not have the wp-login.php file renamed or protected. There are many different plugins that can protect your wp-login page from these types of attacks. I would be very surprised if the cause was something else.

The bots also run on botnets with all different IPs, so they will be able to avoid detection. Try parsing the nginx log file for "wp-login.php" or "wp-admin" or "admin-ajax" which are common attack vectors.

  • Related