6

I am converting a csv file to feather type using the code as below,

import pandas as pd
import feather 
df = pd.read_csv('myfile.csv')
feather.write_dataframe(df, 'myfile.feather')

myfile.csv is over 2G and when I run the code I get the error message as below:

File "table.pxi", line 705, in pyarrow.lib.RecordBatch.from_pandas
File "table.pxi", line 739, in pyarrow.lib.RecordBatch.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

I've looked at similar questions and have found that feather started to support large file over 2G recently. But my feather version is 0.4 so I think mine one is already able to support large file. Why do I get this error? Any ideas would be appreciated, thanks.

huier
  • 165
  • 2
  • 4
  • 12
  • any update on this? – zyxue Jul 04 '18 at 23:51
  • 4
    Seems to be the same as [this issue on github/feather](https://github.com/wesm/feather/issues/325) and [this on Apache ARROW](https://issues.apache.org/jira/browse/ARROW-1907); if I understand correctly, feather is waiting for R bindings to be implemented in Apache ARROW. Only then, feather will be able to accommodate files >2GB. – 0range Nov 29 '18 at 19:19

0 Answers0