-3

I have searched whole web for solution but didnt found what i need . heres my question

I have a unique link which is shared by user on social networks like fb , twitter etc . When a unique ip click is triggered on that link i will award user some points , but as soon as user share link i see 7-8 database entries and this entries are from social network crawler . So when user share fb crawls and generates false database entries . Not just first time it crawls and crawls everytime after a fixed delay and creating problem . My link like below

kodeinfo.com/user/token/network

Please let me know how i can stop this networks from crawling the url

Imran Iqbal
  • 163
  • 1
  • 11
  • everytime I read "I have searched so much..." and yet I know the answer without thinking for even 1 minute, I should get a penny and I'd be rich. answer: robots.txt – Michael May 30 '14 at 17:51
  • e.g. if I search for "prevent crawling", the first page explains the robots.txt concept. "prevent crawling" was the first phrase that came to my mind, btw. – Michael May 30 '14 at 17:55

2 Answers2

0

Mabye check if Facebook or twitter crawler use a specific User-Agent and ignore those agents.

GiDo
  • 1,280
  • 12
  • 23
0

You can try using the rel="nofollow" attribute on your links. Robots should, in theory, respect this and not follow the link.

Here's more info from Google.

Good luck!

Alexandre R. Janini
  • 498
  • 9
  • 20