R: postForm from the RCurl package and issues API call

Question

Does anyone have experience with the limitations of postForm from the RCurl package?

I am pulling data off an server and almost out of nowhere I got the error message * HTTP 1.0, assume close after body and then a 500 Internal Server Error. I tested the configurations and everything seemed to be fine. I've created a clean database and re-uploaded my database 20/30 cases at the time while repeatedly pulling the data with an API/postForm call from R. It all works as it should until I get to around 150 cases then the error message appears. Regardless of the order I upload the cases in the error appears around 150/160 cases and a total files size around 11 to 12 MB. In other words, the error doesn't seem to be dependent on specific cases as it is not the same cases that breaks it

Any advice would be appreciated.

I've attached a screenshot to spice up this rather boring post a bit and to make up for not having a working example,

enter image description here

Update 2013-08-24 19:33:18Z

Here my curlVersion()$version and sessionInfo() information,

> curlVersion()$version
[1] "7.22.0"
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: i686-pc-linux-gnu (32-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=C                 LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RCurl_1.95-4.1 bitops_1.0-6

Update 2013-08-26 05:39:26Z

As suggested in hadley's comment I've added the verbose RCurl output from the call that works and from the call that fails, see below

The call that works with less then 150 cases in the database

> R.object.API <- postForm(R.object.URL, token=R.object.token, content="record", type="flat", format="csv", rawOrLabel="Label", .opts=curlOptions(ssl.verifypeer=TRUE, cainfo=R.object.crt, verbose=TRUE))
* About to connect() to research.org port 443 (#0)
*   Trying xx.xx.xxx.xxx... * connected
* successfully set certificate verify locations:
*   CAfile: /home/dir/research.cert
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*      subject: C=XX; postalCode=XXXXX-XXXX; ST=XX; L=XXXXXX; street=XXX; street=XX XXXXXX XX; O=XXXX, XXX; OU=XXX; CN=research.org
*      start date: 2013-02-04 00:00:00 GMT
*      expire date: 2016-02-04 23:59:59 GMT
*      subjectAltName: research.org matched
*      issuer: C=US; O=XXXXXX; OU=XXXXXX; CN=XXXXXX Server XX
*      SSL certificate verify ok.
> POST /api/ HTTP/1.1
Host: research.org
Accept: */*
Content-Length: 573
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------XXXXXXXXXXXX

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Mon, 26 Aug 2013 05:16:44 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.3.3
< Expires: 0
< cache-control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< 
* Closing connection #0
>

The call fails with more then 150 cases in the database

> R.object.API <- postForm(R.object.URL, token=R.object.token, content="record", type="flat", format="csv", rawOrLabel="Label", .opts=curlOptions(ssl.verifypeer=TRUE, cainfo=R.object.crt, verbose=TRUE))
* About to connect() to research.org port 443 (#0)
*   Trying xx.xx.xxx.xxx... * connected
* successfully set certificate verify locations:
*   CAfile: /home/dir/research.cert
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*      subject: C=XX; postalCode=XXXXX-XXXX; ST=XX; L=XXXXXX; street=XXX; street=XX XXXXXX XX; O=XXXX, XXX; OU=XXX; CN=research.org
*      start date: 2013-02-04 00:00:00 GMT
*      expire date: 2016-02-04 23:59:59 GMT
*      subjectAltName: research.org matched
*      issuer: C=US; O=XXXXXX; OU=XXXXXX; CN=XXXXXX Server XX
*      SSL certificate verify ok.
> POST /api/ HTTP/1.1
Host: research.org
Accept: */*
Content-Length: 573
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------XXXXXXXXXXXX

< HTTP/1.1 100 Continue
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Date: Mon, 26 Aug 2013 05:15:05 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.3.3
< Expires: 0
< cache-control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-Length: 276
< Connection: close
< Content-Type: text/html; charset=UTF-8
< 
* Closing connection #0
Error: Internal Server Error

Have you tried passing a value for curl's --keepalive-time parameter? — scottyaz, Aug 24 '13 at 02:42
Thank you for responding to my question! No, I haven't tried that yet. I'll read the documentation when I get back to my workstation and try passing a value to Rcurl's 'keepalive-time parameter.' — Eric Fail, Aug 24 '13 at 02:55
@scottyaz, I'm using `Rurl` version `1.95-4.1` and I looked up the names of the options that are understood by the `RCurl` package with `listCurlOptions()`, but `--keepalive-time ` wasn't of of the 174 options listed. `keepalive-time` was mentioned at [libcurl manual page](http://curl.haxx.se/docs/manpage.html), but not in `listCurlOptions()`. Would it be possible for you–or anybody else–to provide an of how to keep the connection open longer then the default 60 seconds with a in the `.opts=curlOptions( ... )`? — Eric Fail, Aug 24 '13 at 05:48
I doubt changing the keepalive parameter will affect this case - even if the connection is not kept alive, it should automatically reconnect. I suspect you're sending data in a way that the server doesn't understand, or you have some other server problem. — hadley, Aug 25 '13 at 14:36
@hadley, thank you for your comments. Do you have any suggestions to how I can start identifying the error? In the call in question I am only fetching data of the server, not sending actual date (assuming I use the terminology correct). Thanks. — Eric Fail, Aug 25 '13 at 16:25
@EricFail you said you're using `postForm` which _is_ sending data to the server. It's basically impossible to help you without a reproducible example - or at least a sequence of calls with the verbose rcurl output. — hadley, Aug 25 '13 at 16:47
@hadley, thank you for taking the time to follow up on my comment. I've updated my question with verbose `RCurl` output from the call that works and from the call that fails. — Eric Fail, Aug 26 '13 at 05:41
Other ideas: create one curl handle and reuse it, try using http 1.0, null out the expect header, ask the server maintainer. — hadley, Aug 26 '13 at 13:55
As a very quick check, how about instead of using R to submit the form, you write a simple (local) webpage with a HTML form on it, and see what happens when you submit it through a browser? If it works you probably do have an Rcurl issue, but if not then it's something on the server. — Joe, Aug 27 '13 at 04:33
@Joe, thank you for responding to my question. Would it be possible for you to provide me with a reference or maybe even an example of how such _a simple (local) webpage with a HTML form on it_ can be written and how it can be submitted through a browser? — Eric Fail, Aug 27 '13 at 15:28

score 1 · Answer 1 · answered Aug 24 '13 at 11:07

Doesnt answer your question but relates to options and keepalive:

RCurl uses the libcurl library. This is different to the CURL command line tool. You need to look at the libcurl options here. CURLOPT_TCP_KEEPALIVE maybe what you want. In RCurl this would be listed as tcp.keepalive if it were present in listCurlOptions().

From the easyopt man page this was added in 7.25.0. You can check what version of libcurl RCurl is using by running

> curlVersion()$version
[1] "7.22.0"

Unfortunately the version of libcurl RCurl is using doesnt handle the keep alives yet.

FYI Rcurl uses system curl, so the version will depend on what version you have installed. — hadley, Aug 25 '13 at 14:34

score 0 · Answer 2 · answered Aug 27 '13 at 16:49

Putting this in an answer as it's easier to format. My suggestion is to save the following in a file, fill in the form address (i.e change WHEREVER_YOU_ARE_TRYING_TO_POST_DATA to the appropriate address), name it something like test.html, and open it in your browser.

I've filled in most of the values based upon your above examples, but I have no idea what should go in the token field - this depends upon your specific problem.

<!DOCTYPE html>
<html>
<head></head>
<body>   
<div class="content">
    <form action="WHEREVER_YOU_ARE_TRYING_TO_POST_DATA" method="post">
        token: <input name="token" type="text" size="100" /><br />
        content: <input name="content" type="text" size="100" value="record" /><br />
        type: <input name="type" type="text" size="100" value="flat"/><br />
        format: <input name="format" type="text" size="100" value="csv"/><br />
        rawOrLabel: <input name="rawOrLabel" type="text" size="100" value="Label"/><br />
        <input name="Submit" type="submit" value="submit" />
    </form>
</div>
</body>
</html>

If submitting this form works, but your postForm code doesn't, then something weird is going on with your R code. If they both fail then you have a much deeper problem that probably has nothing to do with R.

thank you for expanding on your comment. Unfortunately your post does not work for me as I also have to submit a server certificate or maybe I am doing something else wrong? — Eric Fail, Aug 28 '13 at 04:00

R: postForm from the RCurl package and issues API call

Update 2013-08-24 19:33:18Z

Update 2013-08-26 05:39:26Z

The call that works with less then 150 cases in the database

The call fails with more then 150 cases in the database

2 Answers2