Feather is a file format for storing data frames. It allows fast data exchange between Python and R.
Questions tagged [feather]
147 questions
8
votes
1 answer
Unable to write my dataframe using feather (strided data not supported)
When using the feather package (http://blog.cloudera.com/blog/2016/03/feather-a-fast-on-disk-format-for-data-frames-for-r-and-python-powered-by-apache-arrow/) to try and write a simple 20x20 dataframe, I keep getting an error stating that strided…
Paul S.
- 83
- 6
7
votes
1 answer
Convert Pandas DataFrame to & from In-Memory Feather
Using the IO tools in pandas it is possible to convert a DataFrame to an in-memory feather buffer:
import pandas as pd
from io import BytesIO
df = pd.DataFrame({'a': [1,2], 'b': [3.0,4.0]})
buf = BytesIO()
df.to_feather(buf)
However, using…
Ramón J Romero y Vigil
- 17,373
- 7
- 77
- 125
7
votes
3 answers
Save a data frame to S3 in feather format
I have a data frame, let's say:
import pandas as pd
df = pd.DataFrame({'a': [1, 4], 'b': [1, 3]})
I want to save it as a feather file to s3 but I can't find a working way to do it.
I tried to use s3bp and s3fs but they don't do the trick.
Any…
amarchin
- 2,044
- 1
- 16
- 32
6
votes
2 answers
pd.read_feather problems with decimal / thousands separator and rounding problems for floats
I'd like to use .ftr files to quickly analyze hundreds of tables. Unfortunately I have some problems with decimal and thousands separator, similar to that post, just that read_feather does not allow for decimal=',', thousands='.' options. I've tried…
TiTo
- 833
- 2
- 7
- 28
6
votes
0 answers
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array
I am converting a csv file to feather type using the code as below,
import pandas as pd
import feather
df = pd.read_csv('myfile.csv')
feather.write_dataframe(df, 'myfile.feather')
myfile.csv is over 2G and when I run the code I get the error…
huier
- 165
- 2
- 4
- 12
5
votes
1 answer
Load many feather files in a folder into dask
With a folder with many .feather files, I would like to load all of them into dask in python.
So far, I have tried the following sourced from a similar question on GitHub https://github.com/dask/dask/issues/1277
files = [...]
dfs =…
ZeroStack
- 1,049
- 1
- 13
- 25
5
votes
2 answers
Save a tibble with list columns to disk
I would like to save to disk a tibble that has list-columns (for later use inside R only). Ideally I'd like a fast binary format like feather, however, it doesn't seem to support list cols:
test <- tibble(a= list(c(1,2),…
hdkrgr
- 1,666
- 1
- 12
- 22
4
votes
1 answer
How to send file in buffer from Python to Julia
I have a large Pandas DataFrame in Python that I would like to access in a Julia program (as a Julia DataFrames.DataFrame object). As I would like to avoid writing to disk for each file send from Python to Julia, it seems as though storing the…
Jack N
- 324
- 2
- 14
4
votes
1 answer
Is it possible to append to an existing Feathers format file?
I am working on a very huge dataset with 20 million+ records. I am trying to save all that data into a feathers format for faster access and also append as I proceed with me analysis.
Is there a way to append pandas dataframe to an existing feathers…
Anjana Shivangi
- 397
- 2
- 5
- 19
4
votes
1 answer
Installing feather-format downgrades several packages
I ran df.to_feather method in pandas 0.20.3, and got an error message advising me to run:
conda install feather-format -c conda-forge
Running that warns that it will change many python packages, and some appear to be (small) downgrades. Is this…
techvslife
- 2,273
- 2
- 20
- 26
4
votes
1 answer
python to R compatibility in feather with strings
I am hitting an error when reading a feather object into R that was put out from a python session.
In python:
In [248]: import pandas as pd
In [249]: pd.DataFrame({'col': ['a','b','c']}).to_feather('strings_df.feather')
In R:
>…
andrew
- 2,524
- 2
- 24
- 36
4
votes
1 answer
is it possible to specify column types when saving a pandas DataFrame to feather?
Currently, if a column happens to have only nulls, an exception is thrown with the error:
Invalid: Unable to infer type of object array, were all null
It is possible to specify the type of the column, that will be used instead of inferring the…
Ophir Yoktan
- 8,149
- 7
- 58
- 106
3
votes
0 answers
pd.read_feather() cannot read feather file -- OSError: Verification of flatbuffer-encoded Footer failed
I have some py scripts to read data from db and write it to a feather file (with pandas to_feather() method). But somehow the written file cannot be read and it throws error as below:
Traceback (most recent call last):
File…
Atacan
- 55
- 8
3
votes
1 answer
How can I create a feathered brush with JavaFX?
So I'm trying to create a very basic photo editor program in Java, using JavaFX. I got a brush and eraser working pretty well so far the following way:
package application;
import java.io.File;
import javax.imageio.ImageIO;
import…
Dániel Sáfár
- 31
- 2
3
votes
1 answer
how to run a large feather R database
I am trying to use the code below to import a 4GB database (about 9,000,000 obs and 100 variables) into R using a windows 10 with 8GB RAM
library(feather)
memory.limit(size=99999)
rais_transp = read_feather('rais_transp.feather')
but every time I…
user16019366