we are using the gtrendsR package for a research project and would like to run a few thousand queries within a few hours, as the analysis is very fine-grained. To make this work, we want to use residential proxies of a rotating proxy service (the provider has us whitelisted for university research purposes).
Setting the proxy in R via Sys.setenv works well (it rotates through IPs), but when running gtrendsR we get an HTTP error code 407:
“Error in curl::curl_fetch_memory(url, handle = .pkgenv[["cookie_handler"]]) : Received HTTP code 407 from proxy after CONNECT”
We also tried to insert the proxy parameters into the setHandleParameters() function, which didn't work either.
Here's our code:
library(jsonlite)
library(gtrendsR)
Set rotating proxy
Sys.setenv(http_proxy = http://user-username:npassword@domain:port/,
https_proxy = " http://user-username:npassword@domain:port/")
Test if it works: IP should change every time fromJSON() is called and we find that it works.
fromJSON(https://api.myip.com/) # works because it returns a new IP on every call
But it for the gtrends function it does not work the latter returns HTTP error code 407.
gtrends(keyword= "Merkel",
geo= "DE",
category = 19,
time = "2020-09-10 2020-09-17",
gprop="web",
onlyInterest = TRUE)$interest_over_time
gtrends() also does not when we set the parameters in the following function (maybe we do something wrong):
setHandleParameters(user = "XXXXXXXXXX",
password = "XXXXXXXXXX",
domain = "XXXXXXXX",
proxyhost = "XXXXX")
Any help would be greatly appreciated (already searched through similar issues on GitHub). Thanks!
