How to query bytearray data in pandas dataframe using duckdb?

Question

df_image : is a pandas data frame with a column labelled 'bytes', which contains image data in bytearray format.

I display the images as follows:

[display(Image(copy.copy(BytesIO(x)).read(),width=300,height=170)) for x in df_image['bytes']]

Now I try a different method:

c = duckdb.query(
    "select a.bytes \
    from \
    df_image a"
).df()

then,

[display(Image(copy.copy(BytesIO(x)).read(),width=300,height=170)) for x in c['bytes']]

this fails with: TypeError: a bytes-like object is required, not 'str'

I guess in BytesIO(x), x was fed in as str.

I tried codecs.escape_decode(bytes(c.loc[0:0,'bytes'][0], "utf-16"))[0].decode("utf-16") but the image data is corrupted.

score 1 · Answer 1 · answered Nov 12 '22 at 21:35

There's no need to use BytesIO, Pandas Series already contains byte array objects. For example:

import duckdb
from IPython.display import Image, display

with open('image.png', 'rb') as f:
    data = f.read()
    
con = duckdb.connect()
con.execute("create table test (b blob)")
rel = con.table("test")
rel.insert([data])
rel.insert([data])

df = con.query("select b from test").df()

print("Pandas Series object type: %s" % type(df['b'][0]))

[display(Image(x, width=300, height=170)) for x in df['b']]

How to query bytearray data in pandas dataframe using duckdb?

1 Answers1