5

An ubuntu VPS I run to host a few basic websites seems to have had apache hacked for bitcoin mining.

In my apache error.log I see the following. 

[Sun Dec 15 06:27:58 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.9 with
Suhosin-Patch configured -- resuming normal operations
[Sun Dec 15 06:27:58 2013] [info] Server built: Jul 12 2013 13:38:21
[Sun Dec 15 06:27:58 2013] [debug] prefork.c(1023): AcceptMutex: sysvsem (default: sysvsem
[Sun Dec 15 09:14:16 2013] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 18 total children
curl: try 'curl --help' or 'curl --manual' for more information
./tmp.sh: 1: ./tmp.sh: ^M: not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M100  165k  100  165k    0     0   402k      0 --:--:-- --:--:-- --:--:--  460k
./tmp.sh: 2: ./tmp.sh: ^M: not found
./tmp.sh: 3: ./tmp.sh: ^M: not found
./tmp.sh: 4: ./tmp.sh: ^M: not found
./tmp.sh: 5: ./tmp.sh: ^M: not found
./tmp.sh: 6: ./tmp.sh: ^M: not found
[2013-12-15 10:06:35] Starting Stratum on stratum+tcp://mine.pool-x.eu
[2013-12-15 10:06:35] 1 miner threads started, using 'scrypt' algorithm.
[2013-12-15 10:06:36] Stratum detected new block
[2013-12-15 10:11:05] Stratum detected new block
[2013-12-15 10:12:46] Stratum detected new block
[2013-12-15 10:13:15] Stratum detected new block

The plan is to re-build my VPS, but first I want to understand exactly how this is happening and I am coming to some dead ends. Here are some things I've noticed / tried:

  1. I can reboot my VPS and the mining doesn't seem to start again until a few hours later. Is this a CRON job? (It seems to restart at about 10am every day).

    • Is this firing up on a CRON job? Can't seem to find anything in crontab, but I are there other places I can check?
    • Do they have an IP address of my VPS and exploit something on my server? How would I detect this, error and access logs are inconclusive.

  2. Doesn't appear to be a wordpress vulnerability

  3. I've setup lastcomm and for the www-data user, I see the following - which is clearly the attack taking place. How can I gain more information about what is going on and being run?

    $sudo lastcomm www-data enter image description here

  4. Whats the difference between apache2 and httpd process?

    enter image description here

  5. I've now noticed the following access log entry, but never at a time that coincides with the mining starting back up:

    177.10.216.85 - - [15/Dec/2013:14:19:01 +0000] "POST /cgi-bin/php4?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C$

  6. I've also ran pwdx against the process, it just returned the following

    4529: /

    Whereas the MySQL process returned the following.

    2298: /var/lib/mysql

Update:

I don't think it's the plesk exploit because apache is returning a 404 (correctly?!) I'm now pretty certain this is just a hidden CRON job or something. Maybe even a hidden process. How they got in originally, I'm really not sure!

46.137.41.30 - - [26/Dec/2013:13:25:26 +0000] "POST /cgi-bin/php4?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%6E HTTP/1.1" 404 493 "-" "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26(KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25"
JonB
  • 151
  • 1
  • 5
  • You said access logs are inconclusive but what requests are processed at 10:06? – Matthew Ife Dec 15 '13 at 14:55
  • I've been through all my vhost access and error logs and also the default error log / access log and there doesn't seem to be anything at 10:06. See edit. I have increased the error logging from just warn to debug now. – JonB Dec 15 '13 at 15:35
  • the `httpd` process is the malicious process which has overwritten $arg0 to make it appear to be a apache process (its called httpd on redhat). Running `pwdx` against those processes *may* indicate which vhost its coming from. – Matthew Ife Dec 15 '13 at 16:36
  • So I ran pwdx against the rogue process and against mysql to comapre...see update. – JonB Dec 15 '13 at 17:29
  • 3
    While you're looking into this, remember to contact the mining pool operator. They can (and will) kill the malicious party's account. – Michael Hampton Dec 15 '13 at 18:29
  • Will do that now - I've noticed that if I reboot my VPS the mining starts up again at 10.06pm or 10.06am exactly. I guess they've installed some kind of CRON job, but my knowledge is limited anything more I can investigate in this area? – JonB Dec 16 '13 at 13:52

3 Answers3

3

Have you tried to decode that strange string in the access log?

Paste this string:

%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C$

In this site:

http://www.url-encode-decode.com

A quick google search on the decoded ascii text suggest that it's a Plesk vulnerability being exploited.

http://about-threats.trendmicro.com/RelatedThreats.aspx?language=au&name=The+Perils+of+the+Plesk+Zero-Day+Exploit

Gabriel Talavera
  • 1,377
  • 1
  • 11
  • 18
  • So i finally got round to pulling the full access logs off apache and I don't think it's this causing the issue (see update). – JonB Dec 28 '13 at 22:03
1

You probably don't have the same problem as I did, but my VPS was exploited using a JBoss vulnerabilty, and they installed a web shell (pwn.jsp) and then used it download some perl backdoor shell.

I just wanted to say to be alert on additional backdoors that the attacker could have left. I found my JBoss management console directory with a oddly named WAR file that when deployed would allow the attacker to deploy any application of his choice in my JBoss instalation

I have some more details in another stackoverflow post and even more in a blog post

Community
  • 1
Pedro Rio
  • 111
  • 2
0

cron job definitions are going to be in any of these possible locations as text files:

  • /etc/cron.d/
  • /etc/cron.daily/
  • /etc/cron.hourly/
  • /etc/cron.monthly/
  • /etc/cron.weekly/
  • /var/spool/cron/crontabs

The first 5 are system based cron folders and generally all run at root level, the last is the directory that contains per user crons (eg when you run crontab -e or crontab -l it will load from that directory each user account with created cron jobs will have a file in this folder

take a look in there, its likely there will be a cron task that starts with the following cron definition

6 10,22 * * * ....

anthonysomerset
  • 4,233
  • 2
  • 21
  • 24