How to limit the the rate HTTPoison, or hackney, streams a response when using the `stream_to` option

Question

I have a client consuming a chunked HTTP response.

HTTPoison.request(method, url, body, headers, stream_to: self(), timeout: 50_000, recv_timeout: 50_000)

However HTTPoison is able send messages to the process quicker than we can process them. The process mailbox is getting up to 30K messages and using all our memory. Is there any way to limit the rate at which HTTPoison fetches and forwards the response body to this process.

Happy to drop to the erlang/hackney interface if that exposes more options.

score 3 · Accepted Answer · answered Jul 16 '18 at 14:43

You can have full control of the streaming request by passing async: :once in the request options and then using HTTPoison.stream_next/1 to request more chunks.

response = HTTPoison.request!(
  method,
  url,
  body,
  headers,
  stream_to: self(),
  async: :once, # <-
  timeout: 50_000,
  recv_timeout: 50_000
)

You'll get one chunk of the response at this point. When you've processed that, you can request more chunks like this:

{:ok, response} = HTTPoison.stream_next(response)
receive do
  ...
end

until you finally get %HTTPoison.AsyncEnd{}.

See documentation of HTTPoison.request and HTTPoison.stream_next for more details.

How to limit the the rate HTTPoison, or hackney, streams a response when using the `stream_to` option

1 Answers1