0

I recently logged into a vps I have (with vultr, if that is of any concern). To find a large amount of nginx logs and higher than expected load average. This server is doing effectively nothing, and just serves the default nginx page on port 80.

An excerpt of the log content is as follows:

    178.154.200.16 - - [05/May/2017:00:22:11 +0000] "GET /philips-nivea-men-anti-irritation-moisturising-shaving-conditioner HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:22:12 +0000] "GET /reach-across-stream-catherine-stephenson HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
95.108.179.22 - - [05/May/2017:00:22:14 +0000] "GET /bt-orange-wood-dual-roller-plastic-fingers-mabtage-tool-hand-joint-mabtagee HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:22:15 +0000] "GET /timur-dvatz-art-catalogue-book HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:22:17 +0000] "GET /small-nativity-scene-set-christmas-ornament-stable-miniature-birth-jesus HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.183.25 - - [05/May/2017:00:22:18 +0000] "GET /biotique-fruit-whitening-lip-balm-lightens-evens-lip-tones-12-grams HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
95.108.213.13 - - [05/May/2017:00:22:22 +0000] "GET /pro-nail-files-trapetz-grain-100-180-set-nails-nail-salon-nail-set HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.183.19 - - [05/May/2017:00:22:25 +0000] "GET /30-x-100mg-strongest-genuine-herbal-blue-male-sex-aid-men-best-one-market HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
5.255.253.67 - - [05/May/2017:00:22:29 +0000] "GET /ew-natural-loofah-sponge-bath-massage-brush-body-face-cleaning-scrubber-pad-uk06 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
93.158.152.40 - - [05/May/2017:00:22:33 +0000] "GET /ponds-sandal-radiance-talc-sunscreen-powder-50-gm-100-gm-300-gm HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:24:34 +0000] "GET /betelnut-container-mortar-end-papua-new-guinea HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:24:34 +0000] "GET /antique-map-nouvelle-carte-de-lafrique HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.49 - - [05/May/2017:00:24:34 +0000] "GET /dswa-keetmanshoop-deutsche-suedwest-afrika-germany-sw-africa-herero-war-ma-48924 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.31 - - [05/May/2017:00:24:35 +0000] "GET /old-k HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:24:35 +0000] "GET /mauchline-tartan-ware-thimble-case-c-1880-cameron-surname-silver-thimble-ch HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.31 - - [05/May/2017:00:24:35 +0000] "GET /antique-20thc-art-deco-platinum-sapphire-diamond-bracelet-c-1930 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.142.180 - - [05/May/2017:00:24:36 +0000] "GET /ancient-egyptian-carnelian-fish-amulet-18th-dynasty-c-1340-bc-amarna-period HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:24:36 +0000] "GET /papua-new-guinea-grain-bag-holder HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.49 - - [05/May/2017:00:24:36 +0000] "GET /1779-bonne-map-isle-de-cuba-havana-manzanillo-guantanamo-bay-santiago-yago HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.31 - - [05/May/2017:00:24:37 +0000] "GET /carte-dune-partie-de-lamerique-pour-la-navigation-caribbean-buache-1740-map HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.142.180 - - [05/May/2017:00:24:37 +0000] "GET /antique-furniture/dressing-tables?page=8 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
37.9.113.30 - - [05/May/2017:00:24:37 +0000] "GET /17th-18th-century-beautiful-hand-carved-rustic-dug-bowl HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
178.154.200.38 - - [05/May/2017:00:24:41 +0000] "GET /gilbert-ellice-sg57-8-1949-silver-wedding-mtd-mint HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.142.58 - - [05/May/2017:00:24:42 +0000] "GET /leather-case-billiards-quiver-madison-black-2-4 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
5.255.253.47 - - [05/May/2017:00:24:45 +0000] "GET /latvia-1923-early-issue-fine-used-15s-120869 HTTP/1.1" 404 152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

It turns out that this has been going on for months, including googlebot as well. On checking the ip addresses, they seem to resolve to yandex/google name servers, so I find it odd that the bots continue to crawl when my sever returns 404 for all requests.

Is this likely due to the server inheriting an old IP for what looks like some kind of shopping site? Or is there something else going on here?

dukky
  • 1
  • Could be anything. It's not a problem, so long as your server is appropriately secured. If you just have the default Nginx page up you might want to just stop the Nginx service. – Tim May 05 '17 at 00:47
  • I suggest that you configure your logging to show the Host field. That will confirm or reject the idea of an old IP. But Yandex is a bugger, it has once looped trying to download the same files for about a week until I made a workaround. When I tried to negotiate with their tech support, they told me they hadn't found a problem. :(( – Zdenek Apr 05 '18 at 20:02

1 Answers1

0

Yes, you're right.

this likely due to the server inheriting an old IP for what looks like some kind of shopping site

Alexander Tolkachev
  • 4,608
  • 3
  • 14
  • 23