How to send BytesIO using requests post

Question

I am trying to send msgpack encoded values to server using requests post to an insert rest endpoint which takes the serialized byte stream as input. But it seems like the values are not getting there properly as I see no values getting inserted in the table. I have never tried this before, so please pardon my ignorance. Here's what I am doing:

buf = io.BytesIO()
for rows in dataframe:
    buf.write(msgpack.packb(rows))
    response = requests.post(url, data=buf, verify=False)
    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        print('Error fetching response using requests')

if I'm not mistaken, the data needs to be an iterable for it to be streamed by requests.post. You might need to instead make a generator function writting to the buf and pass this generator to requests.post instead. the buf might not even be needed in this case. — DevLounge, May 03 '21 at 21:51
buf.seek(0) is the way to do as the buf always points to the end of the buffer. And I also found out that I have to serialize every value instead of list of dicts which is why the insertion was failing. — ShellZero, May 04 '21 at 16:13

score 3 · Answer 1 · answered May 05 '21 at 16:13

3

extending your answer, using a with block is advised in-order to fix memory leak in case of exception.

for rows in dataframe:
    with io.BytesIO(msgpack.packb(rows)) as buf:
        buf.seek(0)
        response = requests.post(url, data=buf, verify=False)
    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        print('Error fetching response using requests')

answered May 05 '21 at 16:13

Alen Paul Varghese

1,278
14
27

1

Pretty cool :) I like this. But do we still have to do seek in this case? – ShellZero May 05 '21 at 16:23

score 2 · Accepted Answer · answered May 05 '21 at 16:07

Whenever I write to the buffer, the pointer will always point to the end of the buffer and waiting for a new write. So, when I pass data=buf, that is going to make an insertion with no values. The solution is to do buf.seek(0) and then pass data=buf as a parameter or simply data=buf.seek(0)

So, the following works:

for rows in dataframe:
    buf = io.BytesIO()
    buf.write(msgpack.packb(rows))
    buf.seek(0)
    response = requests.post(url, data=buf, verify=False)
    try:
        response.raise_for_status()
    except requests.exceptions.HTTPError as err:
        print('Error fetching response using requests')

score 1 · Answer 3 · answered May 05 '21 at 16:19

1

You can also add initial data when you create the BytesIO object so you don't have to call seek:

>>> import io
>>> buf = io.BytesIO(b"test")
>>> buf.tell()
0
>>> buf.read()
b'test'
>>> buf.tell()
4

answered May 05 '21 at 16:19

HTF

6,632
6
30
49

aah. I see :) So, I can simply do data=buf then :) – ShellZero May 05 '21 at 16:22

How to send BytesIO using requests post

3 Answers3