Stack Overflow
Stack Overflow

Questions tagged [feather]

Feather is a file format for storing data frames. It allows fast data exchange between Python and R.

147 questions
1
vote
1 answer

Memory mapped file format operating on bigger than memory files. Arrow?

I have a matrix with 50K columns, 500K rows that I would like to subset by column-names/index pretty fast without using much memory (e.g. memory mapped). Most columns are {NA,1,2}, minority (1%) of the columns are quantitative or strings. What file…
tafelplankje
  • 563
  • 1
  • 7
  • 21
1
vote
1 answer

Apache Arrow or feather plugin?

I'd like to use local feather-files as a sources in Intake. Is the plugin for feather/arrow not yet existing or am I missing something?
bowlby
  • 649
  • 1
  • 8
  • 18
1
vote
1 answer

Why tibble is showing local time after reading feather file

I have a pandas dataframe with timestamps, like that: df = pd.DataFrame({'a': [pd.Timestamp('2020-01-01 00:00:00')]}) I write it to a feather file with df.to_feather('a.feather'), and read it back in R with df <-…
psarka
  • 1,562
  • 1
  • 13
  • 25
1
vote
1 answer

Read ftr file into R

I want to open a .ftr file in R. [not a .feather file!]. But neither the feather package nor arrow one do the job e.g. both ex1 <- feather::read_feather("bla.ftr") ex2 <- arrow::read_feather("bla.ftr") do the job. The arrow approach gives the…
BeSeLuFri
  • 623
  • 1
  • 5
  • 21
1
vote
1 answer

Pyarrow error - AttributeError: module 'pyarrow' has no attribute 'feather'

I used conda install pyarrow to install pyarrow in Spyder launched through Anaconda navigator. But I received the following error after I try to save a file in feather format. Traceback (most recent call last): File "", line 1, in …
L.Yang
  • 553
  • 1
  • 6
  • 12
1
vote
1 answer

Serialize Pandas DataFrame to in-memory buffer representation

What is the fastest way to serialize a DataFrame to an in-memory representation? Based on some research, it seems to be widely acknowledged that the Apache Feather format is the fastest available format by most metrics. My goal is to get the…
PK123
  • 43
  • 8
1
vote
1 answer

How read feather file with duplicative columns in R

I have a problem with reading data files in feather format which has 2+ columns with duplicative names. Is it possible to drop duplicative names on meta data level or load data with duplicative names in some way. Let say data set "mtcars" is saved…
Andrii
  • 2,843
  • 27
  • 33
1
vote
1 answer

Feather icon not displayed after using react-bootstrap-carousel

I am working on the react js project. I am using the feather icon in my project. It works fine before I used the react-bootstrap-carousel in my project. Image before using react-bootstrap-carousal on project I import the following import RBCarousel…
Pritesh
  • 1,066
  • 3
  • 15
  • 35
1
vote
2 answers

64-bit integer support in feather

Can the feather package in R support 64-bit integers? When the dataset is passed to feather::write_feather(), the column is converted to a 64-bit float, and loses precision. I'd like to avoid converting it to a character. Here's a simplified…
wibeasley
  • 5,000
  • 3
  • 34
  • 62
1
vote
0 answers

ImportError on "import feather" in Python

I am trying to install feathers on my environment but it keeps crashing on "import feather". the error message is: --------------------------------------------------------------------------- ImportError Traceback (most…
Yazeed Hamdan
  • 31
  • 7
1
vote
1 answer

Pandas' feather format is slow when writing a column of None

I'm testing out feather-format as a way to store pandas DataFrame files. The performance of feather seems to be extremely poor when writing columns consisting entirely of None (info() gives 0 non-null object). The following code well encapsulates…
Tal Fishman
  • 111
  • 5
1
vote
0 answers

Fastest way to remotely store and retrieve 300 short time series as pandas dataframes

I have an array with around 300 Pandas time series, each one with no more than 6 columns and no more than 10.000 rows. I need to store these in a separate persistent machine that I can access from other machines. Other machines will read this data…
Saturnix
  • 10,130
  • 17
  • 64
  • 120
1
vote
0 answers

R feather integer64 preserve large numbers

I have a dataset with some large numbers: require(pacman) p_load(bit64, feather, data.table) sampleDT <- data.table(ID = c(1,2,3), Val = c(11111111111,2222222222,3333)) sampleDT[, Val :=…
LeGeniusII
  • 900
  • 1
  • 10
  • 28
1
vote
2 answers

Is it possible to import a subset of big .rds or .feather files into R?

I've found good tips about fast ways to import files into R, but I'm wondering if it is possible to import only a subset of a given file into a variable. In my case, I have a file with 16 million rows saved as .rds (and also as .feather, as I was…
Thiago
  • 121
  • 1
  • 10
1
vote
0 answers

read and write a dataset description as a pandas dataframe "attribute"?

The R-help for feather_metadata states "Returns the dimensions, field names, and types; and optional dataset description." @hrbrmstr Kindly posted a PR to answer this SO question and make it possible to add a dataset description to a feather file…
Vincent
  • 5,063
  • 3
  • 28
  • 39
1 2 3
…
9 10