Encode .parquet into io.Bytes

Question

Goal: Upload a Parquet file to MinIO - this requires converting the file to Bytes.

I've been able to do this for .csv, .json and .txt:

bytes = data.to_csv().encode('utf-8')
bytes = json.dumps(self.data, indent=4, separators=(',', ': ')).encode('utf-8')
bytes = data.encode('utf-8')

MinioConn:

from minio import Minio


class MinioConn:
    def __init__(self,
                 host='foo.com:9000',
                 access_key='CENSORED', secret_key='CENSORED',
                 secure=False):
        self.host = host
        self.access_key = access_key
        self.secret_key = secret_key
        self.secure = secure

    def client(self):
        return Minio(self.host, self.access_key, self.secret_key,
                     secure=self.secure)

My Upload Code:

import pandas as pd
import io
from fastparquet import write

import MinioConn

filename = 'myfile.parquet'
# ---
df = pd.DataFrame(data=[['tom', 10], ['nick', 15], ['juli', 14]],
                  columns=['Name', 'Age'])
df.to_parquet(filename)
# ---

data = pd.read_parquet(filename)

bytes = data.encode('utf-8')
buffer = io.BytesIO(bytes)

bucket = 'synthetic-data-gen'

client = MinioConn().client()
client.put_object(bucket,
                f'foo/bar/{filename}',
                data=buffer,
                length=len(bytes),
                content_type='application/{}'.format(filename.split('.', 1)[1]))

Traceback:

Traceback (most recent call last):
  File "test.py", line 16, in <module>
    bytes = data.encode('utf-8')
  File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/pandas/core/generic.py", line 5139, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'encode'

It looks like your post is mostly code; please add some more details.

Does this answer your question? [Sending dataframe as parquet file directly to dropbox](https://stackoverflow.com/questions/68259769/sending-dataframe-as-parquet-file-directly-to-dropbox) — 0x26res, Mar 10 '22 at 15:04
It doesn't; since I need both the `bytes` and the `buffer` ;( — DanielBell99, Mar 10 '22 at 16:09
Yes but it's just a matter of wrapping the bytes in an `io.BytesIO` — 0x26res, Mar 11 '22 at 09:08
I wouldn't call it `bytes` since that confuses it with the `builtin` — jtlz2, Dec 14 '22 at 09:26

0x26res · Answer 1 · 2022-03-25T13:24:16.193

4

If you don't specify a filename, pandas.to_parquet, it will return bytes.

bytes_data = df.to_parquet()
buffer = io.BytesIO(bytes_data)

For older version of pandas:

buffer = io.BytesIO()
bytes_data = df.to_parquet(buffer)
buffer.seek(0)

edited Mar 25 '22 at 13:24

answered Mar 10 '22 at 15:03

0x26res

11,925
11
54
108

`bytes_data` ... `TypeError: expected str, bytes or os.PathLike object, not _io.BytesIO` – DanielBell99 Mar 10 '22 at 15:45
https://stackoverflow.com/q/71956373/17840900 @0x26res – DanielBell99 Apr 21 '22 at 14:52
When you say older versions..? – jtlz2 Dec 14 '22 at 09:28

Encode .parquet into io.Bytes

1 Answers1

Linked