0

We have generated a parquet files, one in Dask (Python) and another with R Drill (using the Sergeant packet ). They use a different implementations of parquet see my other parquet question

We are not able to cross read the files (the python can't read the R file and vice versa).
When reading the Python parquet file in the R environment we receive the following error: system error: Illegalstatexception: UTF8 can only annotate binary filed .
When reading the R/Drill parquet file in Dask we get an FileNotFoundError: [Error 2] no such file or directory ...\_metadata (which is self explanatory).
What are the options to cross read parquet files between R and Python?

Any insights would be appreciated.

skibee
  • 1,279
  • 1
  • 17
  • 37

1 Answers1

1

To read drill-like parquet data-sets with fastparquet/dask, you need to pas a list of the filenames, e.g., 

files = glob.glob('mydata/*/*.parquet')
df = dd.read_parquet(files)

The error from going in the other direction might be a bug, or (gathering from your other question), may indicate that you used fixed-length strings, but drill/R doesn't support them.

mdurant
  • 27,272
  • 5
  • 45
  • 74