Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [useragent]

63 questions
1
vote
2 answers

Search Engine Bot - Large amount of hits

I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot.. Mozilla/5.0 (compatible; Baiduspider/2.0;…
Justin808
  • 307
  • 3
  • 11
1
vote
2 answers

User agent blacklist recommended

Perishable Press publishes a managed user agent blacklist frequently, which blocks bad bots which htaccess directives. Article to be found here: http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/ Would you recommend using…
Chrisissorry
  • 139
  • 7
1
vote
2 answers

Access log Explanation

I am some what new to access_log and just started monitoring it for one of my new sites. The questions I have, is when viewing via -f tail, this is the average output: 000.000.000.000 - - [22/Jun/2011:08:23:11 -0500] "GET /groups/ HTTP/1.1" 200…
ATLChris
  • 123
  • 6
1
vote
1 answer

Extracting Browser from user agent in SQL

I have a database of user agents that I need extract all of the unique browsers from in SQL. The field is like... Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT…
Ben
  • 3,800
  • 18
  • 65
  • 96
1
vote
1 answer

Browser user agent string with all 0s

I've had 2 mac users hit my site today, and both their browser user agents were: Mozilla/5.0 (000000000; 0; 00000 000 00 0 000000; 00000) DDDDDDDDDDDDDDDDDDDD DDDDDDD DDDD DDDDDD DDDDDDDDDDDDD DDDDDDDDDDDDDDD instead of what I would generally…
psychotik
  • 123
  • 3
1
vote
1 answer

lighttpd 1.4.46+ serve specific file based on $HTTP["useragent"]

I have a lighttpd server version 1.4.46+ on Linux which serves file1.xml and file2.xml from htdocs directory as is. There are clients that request these files, each of those send a User-Agent header to the server, which have format MyApp 12345 where…
FearDC
  • 13
  • 3
1
vote
0 answers

Add some text to curl's user agent string? (i.e. curl with a UA that includes curl's default UA & my text)

I am writing a shell script which uses curl(1) to download files. By default curl will set a HTTP User-Agent (UA) to something like curl/7.74.0. I want to be a good internet citizen and set the UA to say what my script is. Since the software that is…
Amandasaurus
  • 31,471
  • 65
  • 192
  • 253
1
vote
1 answer

Excessive requests to /favicon.ico every few minutes from a single user

A significant fraction of the requests to my webserver over the last few months are from a single user requesting /favicon.ico: # tail -n 3 access_log XXX - - [24/Jan/2021:17:40:18 +0000] "GET /favicon.ico HTTP/1.1" 200 1845 "-"…
anon novice
  • 13
  • 2
0
votes
2 answers

What is the point of a bot spoofing various ancient user agents while "GET / HTTP/1.1"?

I found the following interesting traffic in my apache log: 213.159.213.236 - - [16/Dec/2019:03:02:03 -0500] "GET / HTTP/1.1" 200 3797 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FSL 7.0.7.01001)" 213.159.213.236 - -…
Logg
  • 103
  • 2
0
votes
1 answer

keepalive_disable parameters nginx for particular useragent/browsers

I am using keepalive_disable in nginx conf this requires for Syntax: keepalive_disable none | browser ...; Default: keepalive_disable msie6; Context: http, server, location Now i have two problems other than safari, msie6 i was not able to…
Shoaib
  • 1
0
votes
1 answer

Block website scraper in Haproxy

I am using Haproxy. I want to block scrapers from my website. In the haproxy.cfg , I have created a rule. acl blockedagent hdr_sub(user-agent) -i -f /etc/haproxy/badbots.lst http-request deny if blockedagent The file /etc/haproxy/badbots.lst…
Cyberzinga
  • 35
  • 2
  • 6
0
votes
1 answer

How to whitelist User Agent string?

I'm trying to whitelist a very specific User Agent string in Nginx. The below examples demonstrate how to whitelist general types of browsers (eg mozilla or chrome), but the string I want to whitelist has / and ( which breaks the nginx conf. How to…
user421936
  • 241
  • 1
  • 3
  • 4
0
votes
1 answer

Proxy changing my user agent build string

I'm making 2 connections to a web server. The first one is not using a proxy and the second one is using a proxy. In the web server access.log I can see my user agent reported in two different ways. Not using proxy: Mozilla/5.0 (Windows NT 10.0;…
Ivan Maria Spadacenta
  • 41
  • 4
0
votes
2 answers

Analyzing HTTP traffic

I am trying to solve the problem for a while and i couldn't find a proper answer online. I have a file with mobile traffic (from an Android device) and I am trying to check whether an http request is sent from the browser or from a mobile app (for…
kobibo
  • 101
0
votes
2 answers

nginx IF statement matches with non-alphanumerical characters

I'm trying to stop some pesky bots by blocking ther user agents in nginx. What is the best way to put multiple user-agents/matches in the same if statement when they have non-alphanumerical characters and need to be encaptulated with quotation…
alexwatever
  • 101
  • 2
1 2
3
4 5