Haskell simpleHTTP of Network.HTTP.Conduit performing slowly for get requests

Question

In my haskell code I've imported Network.HTTP.Conduit as

import Network.HTTP.Conduit

In my main function I make a simple GET request using simpleHTTP

main = do
     response <- simpleHttp "https://github.com/trending?l=ruby"
     return ()

It took 6 minutes 42 secs to complete 100 api requests

time for NUM in `seq 1 1 100`; do ./Testhttp; done
real    6m42.839s
user    0m12.115s
sys 0m2.652s

whereas the ruby alternative just takes 153 secs for 100 api calls using Net::HTTP.get(URI.parse("https://github.com/trending?l=ruby"))

Am I doing something wrong in my haskell code? What are the performant and efficient alternatives to simpleHTTP?

Have you investigated any non language related reasons why this might be? Like difference in user agent or accept encoding for example? Creating a http request is significantly faster in any language then the >1 sec the request takes. As a data point I can mention that when I run your haskell code the result is about the same as your ruby code. When i run the same thing with curl I get twice that. — monocell, Apr 21 '15 at 20:00
I haven't investigated the non language reasons .The Go and the Ruby version take similar amount of time and I expected the Haskell version to also perform similarly but the difference is rather pretty large and I suspect it to be a library issue. — user2512324, Apr 21 '15 at 21:06
It does sound like a library issue. Are you running this in a cabal sandbox? — TheCriticalImperitive, Apr 21 '15 at 21:25

score 1 · Accepted Answer · answered Apr 21 '15 at 19:19

1

The documentation for simpleHttp says:

Note: This function creates a new Manager. It should be avoided in production code.

Your code creates a new manager for each request. If you change it to reuse a single manager, it could be a lot faster. You can use newManager to create a manager. For example:

import Network.HTTP.Conduit
main = do
    request <- parseUrl "https://github.com/trending?l=ruby"
    manager <- newManager conduitManagerSettings
    _response <- httpLbs request manager
    return ()

answered Apr 21 '15 at 19:19

Taylor Fausak

1,106
8
12

Of course, running the loop in your shell will still create new managers. You can run the loop in Haskell with [`mapM_`](http://hackage.haskell.org/package/base-4.8.0.0/docs/Prelude.html#v:mapM_). Like this: `mapM_ (httpLbs request manager) [1 .. 100]`. – Taylor Fausak Apr 21 '15 at 19:21
I repeated the requests to find a better average of the response time for the request. In production I wish to have only one api call and not multiple. – user2512324 Apr 21 '15 at 20:42
1

I think this is the right answer. Just for completeness, I answered this question on Github as well: https://github.com/snoyberg/http-client/issues/118 – Michael Snoyman Apr 22 '15 at 11:48

score 1 · Answer 2 · answered Apr 21 '15 at 21:47

My guess would be that the Haskell library you are using is doing something like an IPv6 DNS request that times out before falling back to IPv4, and Go and Ruby are doing IPv4 requests directly. A few seconds per request is a likely DNS timeout duration and doesn't have any other possible explanation that I can see.

Haskell simpleHTTP of Network.HTTP.Conduit performing slowly for get requests

2 Answers2