For example if you load a pyarrow parqetdataset, you can get at the data but is there an easy way of filtering this before converting to datetime.date ? datetime.date is a python object so would be good to have a fast way of cutting the data down before constructing many objects I think? Or maybe I am missing something that makes this unecessary.
Asked
Active
Viewed 147 times
0
-
What kind of filtering do you want to do? – joris Jul 10 '19 at 21:46
-
filter on date range for example. – mathtick Jul 11 '19 at 11:10
-
There is currently not yet any conditional filtering implemented for pyarrow Tables. That will be added in the future, but for now the only resort is to convert to pandas and do the filtering there. – joris Jul 11 '19 at 13:18