Do anyone knows how to stop Automated Scanners from Scanning the Web Application or website ? Leaving the robots.txt Any other way Configuration can be made? Any server side modification?
Asked
Active
Viewed 434 times
1
-
Add a robots.txt with a Disallow: section – rjdkolb Mar 07 '18 at 12:00
-
not everyone respects robots.txt – gdkrmr Mar 07 '18 at 12:01
-
Leaving robots.txt any other way? – Rohit Sharma Mar 07 '18 at 12:19
-
You may need to check for web application firewall. – Savan Gadhiya Mar 12 '18 at 18:25
1 Answers
0
- You can add a /robots.txt file that can ask scanners nicely not to scan all or parts of your site. Most legitimate search engine robots follow the instructions you put in robots.txt
For example:
User-agent: *
Disallow: /api/
- If you want to be more fancy pants, you can add a http get on /api/ that normal browsers won't access. Your application HTML may access /api/customers and /api/suppliers but never /api/ for example.
When an automated scanner reads /api/, you can reject any further requests from that IP for 10 minutes. It may help a little.
- A Web Application Firewall like OWASP ModSecurity may also help
rjdkolb
- 10,377
- 11
- 69
- 89
-
leaving robots.txt any other thing that can be done on server as a part of security ? – Rohit Sharma Mar 07 '18 at 12:16
-