0

I have a BytesIO that I'm adding various bytes to. I want to send this in a urllib2.Request via the request.add_data method. How do I do this? When I try

# create request ....
bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
request.add_data(bytesio.getvalue()) 
bytesio.close()

urllib2.urlopen(request) # error "expected buffer, got bytes"

What am I doing wrong? I'm new to Python and not sure how to create a buffer from a BytesIO. Also, when I just try:

request.add_data(bytesio)  # instead of bytesio.getvalue()

I get a "I/O operation on closed file". If I try to wait until after urlopen to call bytesio.close, then the request just hangs because it's waiting for bytesio to be closed.

What do I need to do?

Answer

request.add_data(str(btyesio.getvalue()))
bytesio.close()

Casting to a string made it happy. I haven't tried to see if it all works with StringIO and I haven't tried the differences between Python 2.x and 3.x.

Chad
  • 3,159
  • 4
  • 33
  • 43

4 Answers4

2

Simplest solution: don't use a BytesIO, you don't need it.

urllib2.Request.add_data expects it's argument to be a string, so just give it one.

the call:

bytesio.write(open("C:\img.jpg", "rb").read())

reads the whole file into memory, then writes it to bytesios memory. That means you already have the string in mory, you don't need it twice. So just try:

request = urllib2.Request('http://www.site.com')
with open("C:\img.jpg", "rb") as inputfile:
    request.add_data(inputfile.read())
urllib2.urlopen(request)
mata
  • 67,110
  • 10
  • 163
  • 162
  • I need to add multiple items (files, plain text)... That's why I'm was using BytesIO to build up the data before setting it on the request. – Chad Jun 14 '12 at 06:35
  • Also: you said it expects a string. Are string and "buffer" the same? – Chad Jun 14 '12 at 06:36
  • strings in python have the same interface as [`buffer`](http://docs.python.org/library/functions.html#buffer), so everything expecting a buffer can also handle a string. In python2 you should probably use `StringIO.StringIO` instead of `BytesIO`. In cPython 2.7 `bytes` is the same as `str`, maybe IronPython is different here. - and dont forget to set the `Content-Type` header if you don't want your data to be sent as `application/x-www-form-urlencoded` – mata Jun 14 '12 at 06:56
  • Thanks for the extra comment info, which will help as I transition this up to 3.x. See my updated question for what ended up working! – Chad Jun 20 '12 at 04:25
1

Just remove this line(add it at the end):

bytesio.close()

The other code seems to work for me:

bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
request = urllib2.Request('http://www.site.com')
request.add_data(bytesio.getvalue()) 

urllib2.urlopen(request) # error "expected buffer, got bytes"
bytesio.close()

>>In [30]: urllib2.urlopen(request)
Out[30]: <addinfourl at 52264040 whose fp = <socket._fileobject object at 0x315a450>>

Or using StringIO:

sio = StringIO.StringIO(open("C:\img.jpg", "rb").read())
request = urllib2.Request('http://www.site.com')
request.add_data(sio.getvalue()) 

urllib2.urlopen(request) # error "expected buffer, got bytes"
sio.close()

>>In [14]: urllib2.urlopen(request)
Out[14]: <addinfourl at 49067360 whose fp = <socket._fileobject object at 0x2dfb3d0>>
Tisho
  • 8,320
  • 6
  • 44
  • 52
  • Sorry, I'm still getting the "expected buffer, got bytes" error message when I use getvalue() and close() after urlopen(). I'm using IronPython 2.7, would that cause any problem/difference if this code is working for you? – Chad Jun 14 '12 at 01:09
  • So, are StringIO and BytesIO the same in Python 2.x, just not in 3.x? – Chad Jun 14 '12 at 17:21
1

There is no BytesIO.getvalue() method because it's not needed. Instead, just keep a reference to the underlying buffer.

This works with lists and arrays, as well as bytes objects, but it's sort of a coincidence, rather than an actual design goal...

dilip kumbham
  • 703
  • 6
  • 15
  • Can you please show an example of how I "keep a reference to the underlying buffer"? I'm new to Python. Thanks. – Chad Jun 14 '12 at 16:53
  • I think you can check here http://docs.python.org/faq/programming.html#how-do-i-write-a-function-with-output-parameters-call-by-reference – dilip kumbham Jun 15 '12 at 06:44
0

You can get the whole data of a BytesIO with:

esio.seek(0, os.SEEK_SET)
esio.read()

(SEEK_SET isn't needed, but I use it to clarify things)

So your example:

# create request ....
bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
bytesio.seek(0, os.SEEK_SET)
request.add_data(bytesio.read()) 
bytesio.close()

urllib2.urlopen(request)
dav1d
  • 5,917
  • 1
  • 33
  • 52