using Snort IDS with Webcrawler

Question

I am ,totally, newbie in Snort IDS software and IDS concept, and I need to know if it possible to detect stealth-malicious-web crawlers using Snort!. in other words, can I define snort rules to detect malicious web crawlers ?!!

Answer 1

This is a pretty vague question, but in general the answer is probably yes. Anything that you can see in a packet can be alerted on/dropped with snort. So if you see something and you know it is malicious, you can very likely write a snort rule for it. For example, if you know that a specific user agent is malicious and being used in a web crawler you can block that user agent by creating a rule to match that user agent in the http_header. Here's an example rule from the community ruleset that blocks the known malicious user-agent "Brutus AET":

alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"BLACKLIST User-Agent known Malicious user agent Brutus AET"; flow:to_server,established; content:"Mozilla|2F|3.0 |28|Compatible|29 3B|Brutus|2F|AET"; fast_pattern:only; http_header; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,sectools.org/tool/brutus; classtype:misc-activity; sid:26558; rev:3; )

And a second example rule from the community ruleset for alerting on the known malicious user-agent string core-project:

alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"BLACKLIST User-Agent known malicious user-agent string core-project"; flow:to_server, established; content:"User-Agent|3A 20|core-project"; fast_pattern:only; http_header; metadata:policy balanced-ips drop, policy security-ips drop, ruleset community, service http; classtype:misc-activity; sid:21475; rev:3; )

The hard part would be distinguishing between "normal" web crawlers and malicious ones. But once you find one that you know is malicious it's very likely you can use snort to block it and it would probably not be very difficult, you just have to find a unique pattern in the traffic that can be alerted on.
If you have any specific examples of a malicious web crawler that you are trying to block with snort post the details.