3

I'm trying to write a really trivial "echo" webapp using wai; all I want it to do is reply with the data that is POSTed to it (I really don't care about the method, but I'm using curl and curl is using POST, so that's what I'm going for). My trivial web server is this:

import Network.Wai
import Network.HTTP.Types (status200)
import Network.Wai.Handler.Warp (run)
import Control.Monad.Trans ( liftIO )
import Blaze.ByteString.Builder.ByteString (fromByteString)
import qualified Data.Conduit.List as CondList
import Data.Conduit ( ($$), ($=), Flush(Chunk) )

application req = do
  let src = requestBody req $= CondList.map (Chunk ∘ fromByteString)
  return $ ResponseSource status200 [("Content-Type", "text/plain")] src

main = run 3000 application

What I expected this to do is basically tie the request body to the response body, so that when I run curl --data @/usr/share/dict/words localhost:3000; it would spit my words file back at me. Instead, it gives an empty body. Running curl with "-v" shows that my app is replying with a "200 OK" and no data. I'm not sure what I'm doing wrong here.

If I replace the application function with this:

_ ← requestBody req $$ CondList.mapM_ (liftIO ∘ print)
return $ responseLBS status200 [("Content-Type", "text/plain")] "Hello world\n"

and add an OverloadedStrings pragma to allow the "Hello World" part to work, then I do see my app printing the entire request body to stdout, so I know that curl is submitting the data properly. I also get "Hello World" printed to the curl stdout, so I know that curl works as I expect it to. I must be doing something wrong where I'm tying my requestBody to my ResponseSource, but I don't see it.

unor
  • 92,415
  • 26
  • 211
  • 360
tsuraan
  • 240
  • 1
  • 3

1 Answers1

1

You're correctly using conduit, the problem is that the streaming behavior you're trying to get cannot reliably work in the context of HTTP. Essentially, you want to start sending the response body while the client is sending the request body. This could lead to a deadlock, as both the client and server could be stuck in send mode. To avoid this, Warp flushes the request body before sending the response, which is why the request body appears empty at the time the response body is sent.

In order to get the echo behavior correct, you would need to strictly consume the request body and then send it back. Obviously this could be problematic from a memory usage standpoint if there's a large request body, but this is an inherent aspect of HTTP. If you want constant memory echo, my recommendation would be to stream the request body to a file, and then use ResponseFile for the response body.

Michael Snoyman
  • 31,100
  • 3
  • 48
  • 77
  • That makes no sense to me. I don't see why you've added code whose only purpose is to silently disallow useful behavior. – sclv May 30 '12 at 02:46
  • Because you can't start an HTTP response while the client is still trying to send the HTTP request. Doing so can lead to deadlock. How would you implement a webserver without flushing the request body before sending the response? – Michael Snoyman May 30 '12 at 06:03
  • This is very possible using `epoll`-style IO. Very many web servers are capable of doing this. How could it lead to a deadlock? – dflemstr May 30 '12 at 07:00
  • I don't mean deadlock in the sense of the server will stop being responsive, I mean in the sense that the connection itself will freeze. Imagine the client is sending 5MB of a request in 4096KB chunks. It sends the first chunk, the server accepts it, and tries to send the second. Meanwhile, the server starts sending the response. The client won't send more data until the server accepts the second chunk, and the server won't send more until the client accepts the first response chunk. – Michael Snoyman May 30 '12 at 10:21
  • Is this documented in any of the major servers that one can't provide a streaming echo? – sclv May 30 '12 at 10:56
  • I mean clients can accept while they send, should they so desire. And on the other hand, a server can be written cleverly enough to actually accept while it sends as well. There's no reason that if you are attempting to send then you can necessarily no longer accept or vice versa. – sclv May 30 '12 at 10:58
  • It's a gray area of the spec: after looking through briefly, I see no hard evidence one way or another as to whether a client is required to handle the situation described. However, in practice, there do seem to be a number of client libraries that will fail under this circumstance (a brief Google search indicates this). I think that the current behavior is best as a default, but I'd have no problem providing a setting to allow the alternate behavior. (Such a patch is trivial, just move body flushing to after response sending.) If you're interested, please open a ticket on Github. – Michael Snoyman May 30 '12 at 11:20
  • The behaviour sounds sane to me; I just thought I was misunderstanding something fundamental, which worried me. Thanks for the authoritative response :) – tsuraan May 30 '12 at 17:33