-1

If I do this:

h = httplib2.Http(timeout=60)

resp, content = h.request(uri, method=method, body=body, headers=headers,
                          redirections=redirections,
                          connection_type=connection_type)

If body is more than 64K the data appears to be truncated.

This is on a 32-bit Python runtime (I don't think it happens with a 64-bit runttime).

How do I accomplish this?

Here's the issue:

https://svn.macports.org/ticket/18376

bossylobster
  • 9,993
  • 1
  • 42
  • 61
phil
  • 11
  • 1

2 Answers2

1

First time I've ever heard of this. Here is a short program to show that this works for me. Note that the full.cgi script I have running just returns the request headers and request body back in the response body. When I run this the full 64K+ of content, including the ' the end' roundtrips.

import httplib2

h = httplib2.Http(timeout=60)
body = "x"* (64*1024) + " the end"
uri="http://bitworking.org/projects/httplib2/test/reflector/full.cgi"
resp, content = h.request(uri, method="POST", body=body)
print content

Are you sure you aren't seeing just the TCP segements in wireshark? Those would be truncated to less than 64K, but don't represent the full HTTP request.

Joe Gregorio
  • 770
  • 4
  • 7
  • Actually I tried your example above and it works fine. But change 64 to 70 and it no longer works and exhibits the behavior I am seeing. – phil Jun 21 '11 at 03:54
  • Have you checked the status code of the response? I am seeing 400 and 408 responses in the logs. BTW, I have tried the same code with 70 and it works for me. – Joe Gregorio Jun 21 '11 at 05:26
  • The resulting code is 408. It appears as if python is not flushing its buffers, so the timeout appears. This is version 2.6.6, 32 bit, on the Mac. Works on my PC. – phil Jun 21 '11 at 10:50
1

The problem related to an unpatched build of 2.6.6. In other words, it relates to a known bug that was later fixed. Apparently I got a build of python that did not include this fix.

Please see this post for more info on this problem: https://svn.macports.org/ticket/18376

The patched version set HAVE_POLL=0, forcing python to use select instead. Make sure you are using a version of python that includes this patch, or pushing larger blocks of data will hang.

Another solution is a rewrite of the httplib.py's send method to catch the '35' exception and resend the data.

Here is some code that illustrates this:

            blen = len(str)
            bleft = len(str)
            bpos = 0
            bsize = 1024*8
            while bleft > 0:
                bend = bpos + bsize
                if bend >= blen:
                    bend = blen
                try:
                    slen = self.sock.send(str[bpos:bend])
                except socket.error, v:
                    if v.args[0] == 35:      # unavailable
                        #print('socket unavailable')
                        slen = 0
                        time.sleep(.5)
                    else:
                        raise
                bleft -= slen
                bpos += slen

in place of self.sock.sendall

phil
  • 11
  • 1