5

I'm using the Ruby gem rest-client (1.6.7) to retrieve data using HTTP GET requests. However, sometimes the responses are bigger than I want to handle, so I would like some way to have the RestClient stop reading once it exceeds a size limit I set. The documentation says

For cases not covered by the general API, you can use the RestClient::Request class which provide a lower-level API.

but I do not see how that helps me. I do not see anything that looks like a hook into processing the incoming data stream, only operations I could perform after the whole thing is read. I don't want to waste time and memory reading a huge response into a buffer only to discard it.

How can I set a limit on the amount of data read by RestClient in a GET request? Or is there a different client I can use that makes it easy to set such a limit?

Old Pro
  • 24,624
  • 7
  • 58
  • 106
  • 1
    Can you not just use `Content-Length` header to check the response size, and decide then whether to proceed? – EdvardM Jul 10 '15 at 07:51

1 Answers1

2

rest-client uses ruby's Net::HTTP underneath: https://github.com/rest-client/rest-client/blob/master/lib/restclient/request.rb#L303

Unfortunately, it doesn't seem like Net::HTTP will let you abandon response based on its length as it uses, after all, this method to issue all requests: http://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#method-i-transport_request

As you can see, it uses HTTPResponse to read an HTTP response from server: http://ruby-doc.org/stdlib-2.0.0/libdoc/net/http/rdoc/Net/HTTPResponse.html#method-i-read_body

HTTPResponse seems like the place where you could control whether to read all response and store it into memory, or read and throw away. I you don't want even to read the response, I guess you'll need to close the socket.

I don't know whether there are rest-clients with functionality you need. I guess you'll need to write your own little rest-client if you want to have such a fine-grained control.

Timmy
  • 251
  • 3
  • 12
  • I've got some more time to read the code. You can use `:raw_response => true` and in that case you can process the body of response in chunks. https://github.com/rest-client/rest-client/blob/883903f5e5ae6c894db4e1271b3236301d881ee1/lib/restclient/request.rb#L553 Note that the body will be stored anyway. If you want to stop reading the response, you'd have to hack `fetch_body` method to throw away chunks without writing it anywhere. – Timmy Aug 22 '15 at 05:02