Stack Overflow
Stack Overflow

Questions tagged [feather]

Feather is a file format for storing data frames. It allows fast data exchange between Python and R.

147 questions
2
votes
2 answers

Conversion to FEATHER file creates huge file

I am trying to turn an .rds file into a .feather file for reading with Pandas in Python. library(feather) # Set working directory data = readRDS("file.rds") data_year = data[["1986"]] # Try 1 write_feather( data_year, "data_year.feather" …
Stücke
  • 868
  • 3
  • 14
  • 41
2
votes
2 answers

How to turn an *.RDS file into a *.FEATHER file?

I am trying to covert an *.rds file in R into a *.feather file for use in Python. library(feather) data = readRDS("file.rds") write_feather(data,"file.feather") However, I receive the following error: > write_feather(data,"file.feather") Error: `x`…
Stücke
  • 868
  • 3
  • 14
  • 41
2
votes
0 answers

How to put a limit in Pandas to the number of rows read from a 'feather' file?

I have a feather datafile that weights approximately 300 MB, name it df.ftr. I can read it with Pandas using the following command: import pandas as pd df = pd.read_feather('df.ftr') However, this dataset contains over 21 million rows and its size…
Marioanzas
  • 1,663
  • 2
  • 10
  • 33
2
votes
0 answers

pandas 1.3.3 to_feather giving ArrowMemoryError

I have a dataset of size around 270MB and I use the following to write to feather file: df.reset_index().to_feather(feather_path) This gives me an error : File "C:\apps\Python\lib\site-packages\pandas\util\_decorators.py", line 207, in wrapper …
SomeDude
  • 13,876
  • 5
  • 21
  • 44
2
votes
1 answer

Pandas read_feather throw an ArrowInvalid: Casting from timestamp[us] error

Problem >>> import pandas as pd >>> df = pd.read_sql("SELECT * FROM table_name", con) >>> df.to_feather("some_file.feather") >>> pd.read_feather("some_file.feather") ArrowInvalid: Casting from timestamp[us] to timestamp[ns] would result in out of…
Adrien Pacifico
  • 1,649
  • 1
  • 15
  • 33
2
votes
1 answer

Pyarrow: read stream into pandas dataframe high memory consumption

I would like to to first write a stream into an arrow file and then later read it back into a pandas dataframe, with as little memory overhead as posible. Writing data in batches works perfectly fine: import pyarrow as pa import pandas as pd import…
delica
  • 1,647
  • 13
  • 17
2
votes
0 answers

CSV - ERROR: BoundsError: and Feather - ERROR: InexactError: - Julia programming

I'm very new to Julia Programming. I have a folder of CSV files (14) that I join to one big data frame and I'm trying to save the big CSV. (rows - 262673020 x columns - 77) when I use CSV.write - I get this ERROR: BoundsError: attempt to access…
orthoeng2
  • 140
  • 1
  • 6
2
votes
2 answers

Converting excel to feather format with python

I have a (daily growing) list of around 100 big excel files, which I analyse in Python. As I have to run several loops over all the files, my analysis are getting slower and slower. Therefore I'd like to convert all excel files into feather format…
TiTo
  • 833
  • 2
  • 7
  • 28
2
votes
1 answer

How can I install the feather format on Windows 10?

I have been trying to install the feather format for Python, but I keep getting the error message below in my command promopt, when I am typing "pip install feather-format". Does anyone know how to proceed in order to get the feather file installed?…
Nicolai MC
  • 19
  • 1
  • 5
2
votes
1 answer

Exporting dataframe with null-able Int64 from pandas to R

I am trying to export a dataframe that contains among others categorial and nullable integer columns such that it can be easily read by R. I put my bets on apache feather, but unfortunately the Int64 datatype from pandas does not seem to be…
cel
  • 30,017
  • 18
  • 97
  • 117
2
votes
1 answer

Reading feather object in R is slow

I am using the feather packages for data exchange between Python (collecting the data) and R (used for analysis), writing and reading the data in Python is extremely fast. However, reading the same feather object in R is VERY slow, on the order of…
guy
  • 1,021
  • 2
  • 16
  • 40
2
votes
1 answer

ImportError with feather/pyarrow

I am getting an error when running import feather: ---> 28 from pyarrow.lib import cpu_count, set_cpu_count 29 from pyarrow.lib import (null, bool_, 30 int8, int16, int32, int64, ImportError: cannot import name…
Luxo_Jr
  • 379
  • 1
  • 3
  • 12
2
votes
0 answers

iOS im trying to crop the face and apply feather effect on the edge of the Image but getting nil renderer?

here i’m geting the image in Img but showing nil but when i print the size of the image or display image (Img) in imageview i m getting its size and image also in imageview then when i pass the size of the image(Img) in renderer, the issue is…
Ramani Hitesh
  • 214
  • 3
  • 15
1
vote
1 answer

identical(X1, X2) is TRUE, but digest::sha1(X1) != digest::sha1(X2)

I have several large data.table saved to disk in .rds files. I'm looking for ways to reduce the time required to import data. I was looking into the feather package. Part of my pipeline is to check for any changes in the input data set based on…
Peter
  • 7,460
  • 2
  • 47
  • 68
1
vote
0 answers

is there a command line tool to read number of rows and columns of a feather file?

I'm working in a project that generates a lot of data, and every month a process writes a csv file with over 6 million records, and then gets converted into a feather file. There's a need to verify the number of records that the feather file has and…
FDX
  • 11
  • 2
1 2 3
…
9 10