0

I'm connecting to a server to make thousands of requests. Based on what I'm seeing, it looks like RCurl is opening a connection to the server on every request. I'm wondering if there is a way to keep the connection open for a set amount of time. Here is what I see when I run two requests within a second of each other

* About to connect() to gt-tradeview port 80 (#0)
*   Trying 192.168.141.136... * connected
* Connected to gt-tradeview (192.168.141.136) port 80 (#0)
> POST /House/TradeView/ajax/varys HTTP/1.1
User-Agent: RCurl
Host: gt-tradeview
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/json
Content-Length: 360

< HTTP/1.1 200 OK
< Content-Length: 1690
< Content-Type: application/json
< Server: Microsoft-HTTPAPI/2.0
< Date: Thu, 09 Jan 2014 14:27:49 GMT
< 
* Connection #0 to host gt-tradeview left intact


* About to connect() to gt-tradeview port 80 (#0)
*   Trying 192.168.141.136... * connected
* Connected to gt-tradeview (192.168.141.136) port 80 (#0)
> POST /House/TradeView/ajax/varys HTTP/1.1
User-Agent: RCurl
Host: gt-tradeview
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/json
Content-Length: 227

< HTTP/1.1 200 OK
< Content-Length: 195
< Content-Type: application/json
< Server: Microsoft-HTTPAPI/2.0
< Date: Thu, 09 Jan 2014 14:27:49 GMT
< 
* Connection #0 to host gt-tradeview left intact

It looks like a pretty big waste of time, but I don't see why it tries since the last line is that the connection is left intact. Nevertheless, if I immediately run 

Browse[2]> showConnections()
     description class mode text isopen can read can write

so there aren't any connections open?

here are my curl.opts

mkURL <- function(x) {
  names <- names(x)

  s <- sprintf("%s=%s",names[1],x[1])
  for (i in 2:length(names)) {
    s <- sprintf("%s&%s=%s",s,names[i],x[i])
  }
  URLencode(s)
}
curl.opts = curlOptions(
  httpheader = c(
    'Content-Type'    = "application/x-www-form-urlencoded; charset=UTF-8",
    'Accept'          = "application/json"
   #'Accept-Encoding' = "gzip,deflate,sdch"
  ),
  verbose = TRUE,  #change to TRUE for server feedback
  header = TRUE,
  buffersize = 100000000000,
  useragent = "RCurl"                                                       ###
)

and the call

curlPerform(url = paste0("http://",serverToHit,"/House/TradeView/ajax/varys"),
      #curlPerform(url = "http://192.168.141.148/House/TradeView/ajax/varys",
                postfields = mkURL(parameters),
                .opts = curl.opts,
                writefunction = r$update,
                post = 1L,
                curl = r$curl())
hedgedandlevered
  • 2,314
  • 2
  • 25
  • 54
  • What's `r$curl()`? Are you in the end calling `getCurlHandle` once and then re-using the returned value? – Martin Morgan Jan 09 '14 at 16:00
  • Maybe you can narrow this down a little (editing your question as necessary) e.g., to a specific (publicly accessible) server / protocol that causes problems. This apparently re-uses connections -- `curlPerform(url="http://stackoverflow.com/questions/21023439/keep-connection-open-with-rcurl", nobody=TRUE, verbose=TRUE, curl=r$curl())` – Martin Morgan Jan 09 '14 at 18:36
  • when you say it re-uses connections, does that imply that subsequent requests would be faster? after running that, does showconnections() still show nothing? if so, what does showconnections() do? – hedgedandlevered Jan 09 '14 at 18:44
  • `showConnections` is about base R connections (`file()`, `url()`, etc, see `?connections`); nothing (unfortunately) to do with RCurl. I was gauging reuse by repeating the `curlPerform()` and looking for output that reports "About to connect..." (creating a new curl connection) or "Re-using existing connection..." (re-using previously established curl connection). – Martin Morgan Jan 09 '14 at 18:51
  • ok so it should be safe to assume that there's something about this particular server that is closing the connections, since your only change was to request from a different server? – hedgedandlevered Jan 09 '14 at 19:10
  • I didn't use post (this was my best guess about possible source of problems), postfields, buffersize, useragent, httpheader, ... – Martin Morgan Jan 09 '14 at 19:53

0 Answers0