Save a data frame to S3 in feather format

Question

I have a data frame, let's say:

import pandas as pd
df = pd.DataFrame({'a': [1, 4], 'b': [1, 3]})

I want to save it as a feather file to s3 but I can't find a working way to do it.

I tried to use s3bp and s3fs but they don't do the trick.

Any suggestion?

score 5 · Accepted Answer · answered Feb 09 '18 at 20:05

The solution that worked for me is

import boto3
import pandas as pd

from io import BytesIO
from pyarrow.feather import write_feather

df = pd.DataFrame({'a': [1, 4], 'b': [1, 3]})

s3_resource = boto3.resource('s3')
with BytesIO() as f:
    write_feather(df, f)
    s3_resource.Object('bucket-name', 'file_name').put(Body=f.getvalue())

score 0 · Answer 2 · answered Feb 07 '18 at 14:16

0

You can use storefact / simplekv for this without writing to disk.

import pyarrow as pa
from pyarrow.feather import write_feather
import storefact

df = …
store = storefact.get_store('hs3', host="…", bucket="…", access_key="…", secret_key="…")
buf = pa.BufferOutputStream()
write_feather(df, buf)
storage.put('filename.feather', buf.get_result().to_pybytes())

answered Feb 07 '18 at 14:16

Uwe L. Korn

8,080
1
30
42

Does it works also for Python 3? I am not able to use `get_store` or `get_store_from_url`. – amarchin Feb 07 '18 at 15:06
What do you mean for host here? – amarchin Feb 07 '18 at 15:12
It also works for Python 3, I'm using it happily with it. Please post your error if you get one. – Uwe L. Korn Feb 07 '18 at 19:08
You can probably skip the host here. This is only there if you don't connect to Amazon S3 but another S3-compatible service. – Uwe L. Korn Feb 07 '18 at 19:09
The host argument is required. If I try with `host=''` I get `ConnectionRefusedError: [Errno 61] Connection refused`. But I know that the acces_key and the secret are correct since I am using them in other contexts. – amarchin Feb 08 '18 at 09:50
You should be able to connect to AWS using `s3.amazonaws.com` as the host. – Uwe L. Korn Feb 08 '18 at 12:01

score 0 · Answer 3 · answered Jan 12 '21 at 02:31

Simple solution with just Pyarrow and Pandas

import pandas as pd
import pyarrow as pa

s3 = pa.fs.S3FileSystem(region='us-east-1')

df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})

with s3.open_output_stream('my-bucket/path/to.feather') as f:
   pa.feather.write_feather(df, f)

Save a data frame to S3 in feather format

3 Answers3

Simple solution with just Pyarrow and Pandas