0

i'm trying to make a simple GET request to external URLs from my cloud function using node-fetch. When a user pastes a link, I'm making this request to retrieve social media sharing tags to populate title, description and image. Some websites seem to be responding with robot messages. Is there a way around this?

So far I've made sure the origin of the request is coming from my domain and not some strange cloud server. I've also tried defining a browser based user agent in the header without much luck.

Any ideas on other things I can check for?

SupRob
  • 11
  • 2
  • 1
    You _are_ a robot. Could you respect the website's wishes instead? – Evert Nov 07 '21 at 18:01
  • It's no different than what Facebook or Twitter does when you past a link. Not sure what you mean, not a robot. – SupRob Nov 07 '21 at 18:09
  • A server making a request on your behalf is generally considered a robot in this case. – Evert Nov 07 '21 at 18:10
  • The request is triggered by a user, it's the only time the request happens. That is not a robot. If you're not going to be helpful than optimize your time somewhere else – SupRob Nov 07 '21 at 18:19
  • Definitions aside, this is a server that makes requests on a users' behalf. For lots of reasons people don't like that, and they probably express this in (sorry, it's true.. `robots.txt`). You are fighting a war of attrition because no matter how sophisticated you get with avoiding detection, people will continue to try to get more sophisticated in detecting you. – Evert Nov 07 '21 at 18:22
  • So if you can re-think this issue and avoid it, it might be a better path because there is no single answer that's a universal solution to this. – Evert Nov 07 '21 at 18:24

0 Answers0