How to read an HTTP chunked response and send chunked response to client in Elixir?

Question

I'm using Elixir/Phoenix and I have an endpoint that returns a chunked response like say a never ending stream of log lines. The log lines, however, come from another service A which also returns a chunked response. I want my endpoint to read the chunked response from service A and pass them through to the client also in chunks. In essence, it's just a proxy for service A, but I can't let the client connect directly to service A because I need to perform some authentication.

Answer 1

Here's an example of sending chunk data with Phoenix:

  def test_1(conn, _params) do
    conn = conn
    |> put_resp_content_type("text/event-stream")
    |> send_chunked(200)

    conn |> chunk("a")
    conn |> chunk("b")
    conn |> chunk("c")

    # send data five times, once per second, to simulate a log
    for n <- 1..5 do
      conn |> chunk(Integer.to_string(n))
      Process.sleep(1000)
    end

    conn
  end

There are a few http libraries available for Elixir/Erlang; I like HTTPoison.

Here is an example that reads chunks from a url, and sends them off as they come in:

  def test_2(conn, _params) do
    url = "http://localhost:4000/test_1"
    %HTTPoison.AsyncResponse{id: id} = HTTPoison.get!(url, %{}, stream_to: self())

    conn = conn
    |> put_resp_content_type("text/event-stream")
    |> send_chunked(200)

    process_httpoison_chunks(conn, id)
  end

  def process_httpoison_chunks(conn, id) do
    receive do
      %HTTPoison.AsyncStatus{id: ^id} ->
        # TODO handle status
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncHeaders{id: ^id, headers: %{"Connection" => "keep-alive"}} ->
        # TODO handle headers
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncChunk{id: ^id, chunk: chunk_data} ->
        conn |> chunk(chunk_data)
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncEnd{id: ^id} ->
        conn
    end
  end

Some references:

• https://github.com/edgurgel/httpoison#async-requests
• http://big-elephants.com/2016-12/http-streaming-in-elixir/
• https://github.com/faithandbrave/elixir-phoenix-chunked-response-example
• https://github.com/edgurgel/httpoison/issues/108