Python Recv() stalling

Question

I'm writing a very basic HTTP client:

import socket
from socket import *

Payload = """GET /test.html HTTP/1.1
Accept: */*
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: example.com
Pragma: no-cache

"""

def SendAndReceive(Host, Payload):
    s = socket(AF_INET, SOCK_STREAM)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    s.connect(Host)    
    s.sendall(Payload)
    tdata=[]
    while True:
        data = s.recv(1024)
        if not data: 
           break
        tdata.append(data)
    print ''.join(tdata)
    return ''.join(tdata)

SendAndReceive(("www.example.com",80),Payload)

For some reasons, the recv() stalls for a while (~10 sec), then returns the data. I'm not sure what's wrong with my code, any help would be greatly appreciated.

Thanks!

Does it happen with all HTTP servers or any particular server? — Mikko Ohtamaa, Jan 24 '14 at 20:42
With all HTTP servers. Something must be wrong in this code.. — n00bz0r, Jan 24 '14 at 20:45

score 2 · Accepted Answer · answered Jan 24 '14 at 20:47

You do an HTTP/1.1 request which implicitly means keep-alive, e.g. the server can keep the connection open after the request is done to get more requests on the same connection. In this case the server decides after 10 seconds that it will not wait for more requests, other servers might wait longer. You also don't check the servers response for content-length or chunked encoded but just assume, that the server will close once the request is done. This is wrong (keep-alive). Also, you have a proxy-connection header which does nothing, because this is not a proxy request (and proxy-connection header in itself is invalid, talking to a proxy requires a connection header).

The easiest thing would be to issue a HTTP/1.0 request and not specifying any connection header or setting it to 'close'. Then the server will close the connection after the response is done. Also, you don't have to deal in this case with chunked encoding.

Changing HTTP/1.1 to HTTP/1.0 and keep-alive to close solved the issue, thanks. Quick question, how should I implement the actual recv() buffering ? Let's say this socket do a GET to an image. — n00bz0r, Jan 24 '14 at 20:54
@n00bz0r: good question, lousy comment, ask a new question for that! — SingleNegationElimination, Jan 24 '14 at 20:56

score 0 · Answer 2 · answered Jan 24 '14 at 20:57

As explained by Steffen, this is due to the keep-alive. For example, if you test it with google.com, you will wait for a very long time. You can modify your code and see the timing out as follows:

# Payload remains the same
import socket as socket_module
from socket import *
def SendAndReceive(Host, Payload):
    s = socket(AF_INET, SOCK_STREAM)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    # or you could do s.setblocking(0) for timeout of 0 seconds
    s.settimeout(1)
    s.connect(Host)
    s.sendall(Payload)
    tdata=[]
    while True:
        try:
            data = s.recv(1024)
        except socket_module.error:
            # TIMEOUT
            break
        if not data:
            break
        tdata.append(data)
    print ''.join(tdata)
    return ''.join(tdata)

In this way you wouldn't get an error.

Python Recv() stalling

2 Answers2

Linked