-1

We have a linux server with 4Gb of memory. When we start the system it uses only 435Mb. When we start the apache httpd services it take 1000Mb and subsequently automatically it takes all the memory and the server crashes. If we stop Apache it only releases 200Mb of memory. What could be causing this problem ?

Can any one tell me what these hacker are doing? Below is the log. Please help me out for this.

[root@host ~]#  tail -20 /var/log/httpd/dostizone.com-combined.log
180.76.5.143 - - [14/Nov/2011:02:30:16 +0530] "GET /blogs/10248/209403/nfl-panties-since-the-quality-of HTTP/1.1" 403 2298 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.5.88 - - [14/Nov/2011:02:30:31 +0530] "GET /blogs/815/158725/new-jersey-attorney-search HTTP/1.1" 403 2290 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.186 - - [14/Nov/2011:02:30:32 +0530] "GET / HTTP/1.1" 403 5043 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
crawl-66-249-67-137.googlebot.com - - [14/Nov/2011:02:30:20 +0530] "GET /blogs/805/11279/supra-suprano-high-shoes HTTP/1.1" 200 30642 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:30:37 +0530] "GET /blogs/10514/215084/oakland-raiders-sweatpants-tags HTTP/1.1" 403 2297 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
220.181.94.237 - - [14/Nov/2011:02:30:12 +0530] "GET /profile/8509 HTTP/1.1" 200 236894 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.94.237 - - [14/Nov/2011:02:30:43 +0530] "GET /mode-switch?return_url=%2Fblogs%2F8529%2F160217%2Fclimate-jordan-6 HTTP/1.1" 302 1 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:30:44 +0530] "GET /blogs/390/61573/blackhawk-jerseys-from-the-you HTTP/1.1" 403 2293 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
124.115.0.159 - - [14/Nov/2011:02:30:24 +0530] "GET /blogs/693/46081/application/modules/Hecore/externals/scripts/core.js HTTP/1.1" 200 26869 "http://dostizone.com/blogs/693/46081/thomas-sabo-charms-hot-chilli" "Sosospider+(+http://help.soso.com/webspider.htm)"
124.115.0.159 - - [14/Nov/2011:02:30:24 +0530] "GET /blogs/693/46081/application/modules/Activity/externals/scripts/core.js HTTP/1.1" 200 26873 "http://dostizone.com/blogs/693/46081/thomas-sabo-charms-hot-chilli" "Sosospider+(+http://help.soso.com/webspider.htm)"
124.115.0.159 - - [14/Nov/2011:02:30:24 +0530] "GET /blogs/693/46081/application/modules/Hecore/externals/scripts/imagezoom/core.js HTTP/1.1" 200 26899 "http://dostizone.com/blogs/693/46081/thomas-sabo-charms-hot-chilli" "Sosospider+(+http://help.soso.com/webspider.htm)"
180.76.5.153 - - [14/Nov/2011:02:30:50 +0530] "GET /blogs/10252/212268/cleveland-browns-authentic-jerse HTTP/1.1" 403 2298 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:30:51 +0530] "GET /blogs/741/46260/chocolate-ugg-women-boots-1873 HTTP/1.1" 403 2293 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
124.115.1.7 - - [14/Nov/2011:02:30:40 +0530] "GET /blogs/682/97454/swarovski-jewellry-sale-articles HTTP/1.1" 200 25770 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:30:56 +0530] "GET /blogs/779/60941/players-a-to-z-michael-cuddyer HTTP/1.1" 403 2293 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:31:01 +0530] "GET /blogs/469/58551/chicago-bears-news-there-exist HTTP/1.1" 403 2293 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
220.181.94.237 - - [14/Nov/2011:02:30:54 +0530] "GET /blogs/8529/160217/climate-jordan-6 HTTP/1.1" 200 30750 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
180.76.5.59 - - [14/Nov/2011:02:31:05 +0530] "GET /blogs/815/158197/cheap-calgary-flames-jerseys HTTP/1.1" 403 2292 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
crawl-66-249-68-51.googlebot.com - - [14/Nov/2011:02:31:06 +0530] "GET /mode-switch?return_url=%2Fblogs%2F387%2F45679%2Fhandbag-louis-vuitton-judy-mm-m4 HTTP/1.1" 403 2258 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
crawl-66-249-67-137.googlebot.com - - [14/Nov/2011:02:31:10 +0530] "GET /public/temporary/c83b731ecc556d7fd1a7732d9ac16ed6.png HTTP/1.1" 404 2305 "-" "Googlebot-Image/1
user9517
  • 115,471
  • 20
  • 215
  • 297
bibhudatta
  • 1
  • what is the output of top -c ? – adam Nov 13 '11 at 21:34
  • You've got your MaxClients set too high, lower it significantly (maybe only 20 to start) and see if you can keep it going with the memory usage high. I expect this is more to do with your web application that 'hacking'. – Matthew Ife Nov 13 '11 at 21:40
  • Last login: Mon Nov 14 02:01:19 2011 from 114.143.96.188 [root@host ~]# free -m total used free shared buffers cached Mem: 3922 483 3438 0 68 199 -/+ buffers/cache: 216 3705 Swap: 8189 0 8189 – bibhudatta Nov 13 '11 at 21:54
  • 1
    _Help, I'm being hacked by search engines!_ Umm, so far the ips I've looked up are relevant to Google, Baidu and SOSO. Your web server configuration needs tuning. It can't handle the load from normal operation. – Fiasco Labs Nov 13 '11 at 22:43
  • First rule of detecting hacks: If you notice it, you´re not hacked. Professional Hackers are better than that. – Posipiet Nov 14 '11 at 09:48

2 Answers2

1

Can any one tell me what these hacker are doing.

Nothing much. They're requesting a bunch of things, spaced over time, and getting either 404-not-found, or 403-not-allowed. There is one request that got 200-success, and that's for the page at /blogs/805/11279/supra-suprano-high-shoes. The other 200 responses are for items I'd expect you to have, like that core.js file.

The pace of requests is really low, only a few over the course of a minute. Considering those hits are coming from (what looks to me to be) spiders of various kinds, sites out there on the Internet are linking to pages that either don't exist (404) or are not visible to unauthenticated visitors (403).

As for your memory usage, considering your low hit rate, you will want to turn down your MaxClients and/or ServerLimit values in your httpd.conf. Those will consume memory.

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
  • free -m total used free shared buffers cached Mem: 3922 708 3214 0 50 534 -/+ buffers/cache: 122 3799 Swap: 8189 0 8189 top - 03:35:58 up 1 min, 1 user, load average: 0.75, 0.24, 0.08 Tasks: 91 total, 1 running, 90 sleeping, 0 stopped, 0 zombie Cpu(s): 63.8%us, 9.8%sy, 0.0%ni, 24.0%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4016364k total, 1921204k used, 2095160k free, 53564k buffers Swap: 8385888k total, 0k used, 8385888k free, 1683836k cached – bibhudatta Nov 13 '11 at 22:03
  • PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2260 mysql 15 0 283m 33m 4432 S 145.5 0.9 0:16.51 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld 2455 apache 16 0 327m 30m 13m S 0.7 0.8 0:00.13 /usr/sbin/httpd 2458 apache 16 0 327m 30m 13m S 0.7 0.8 0:00.12 /usr/sbin/httpd 2457 apache 16 0 327m 30m 14m S 0.3 0.8 0:00.11 /usr/sbin/httpd 1 root 18 0 10348 688 576 S 0.0 0.0 0:00.50 init [3] 2 root RT -5 0 0 0 S 0.0 0.0 – bibhudatta Nov 13 '11 at 22:04
  • All the request are not coming from out side because in the firewall there is nor request from these ip. 5months before the hacker hack the system and they put a run some program which will genrate multiple http services but that I found and solve but this thime wht these people are doing I donot understand where is the memory going even we stop apache and mysql the memory donot go down. Once it take the memory it is nuable to rlease the memory. – bibhudatta Nov 13 '11 at 22:05
  • I suspect what you're seeing is Linux caching all of the Apache libraries into the block-cache (the `buffers/cache` line in the free output). When figuring out how much RAM is 'free' on Linux, you take the `total` number and subtract `used` from it. Everything else is actually-free, or freeable-upon-demand and counts as free for any process asking for memory. – sysadmin1138 Nov 13 '11 at 22:07
  • Look how memeory loss in 5 sec root@host ~]# free -m total used free shared buffers cached Mem: 3922 2208 1713 0 82 1710 -/+ buffers/cache: 415 3506 Swap: 8189 0 8189 [root@host ~]# [root@host ~]# [root@host ~]# [root@host ~]# free -m total used free shared buffers cached Mem: 3922 2241 1680 0 82 1710 -/+ buffers/cache: 448 3473 Swap: 8189 0 8189 – bibhudatta Nov 13 '11 at 22:09
  • How can stop the hacker. Please let me known is there any solution or I need to reinstall the linux. – bibhudatta Nov 13 '11 at 22:10
  • StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 – bibhudatta Nov 13 '11 at 22:36
  • What is the solution for this – bibhudatta Nov 13 '11 at 22:45
  • "Once it take the memory it is nuable to rlease the memory." It's not that it's unable, it's that there's no point. Free memory does no good, it might as well not be there. Only memory that is in use helps performance. If you want free memory, take it out of the computer and put it on your desk. If you put it in the computer, the computer will *use* it. The more memory the computer uses, the faster it runs. That's why you put more memory in a computer. – David Schwartz Nov 14 '11 at 08:54
0

outmatically it take all the memory and the server crase

Assuming you mean it crashes....it shouldn't. Even if all the memory is used it won't cause Apache nor the kernel to crash. The system may appear to become unresponsive due to high levels of paging - but that's not the same thing at all.

And the only reason this would happen is if you've not configured it properly.

Can any one tell me what these hacker are doing.

I think you need to focus on getting your own stuff sorted out before you worry about what other people might be doing. First thing is to set up your server correctly - Switch off keepalives or reduce the tieout to 1 second, reduce the maxclients. What are your settings for maxrequestsperchild, Maxclients, Startservers, min/maxspareservers, serverlimit?

Since posting I see you've provided some of this information in a comment (should have been updated on the original question) but you've provided conditional configs depending on pre-fork / threading - which is implemented on your server?

However both configs are fairly modest - and for serving static content should not cause significant memory problems.

even we stop the apache just it release 200mb memory

So what's still hogging all the memory? Did you check all the httpd processes had exited?

symcbean
  • 21,009
  • 1
  • 31
  • 52