PyTables: Select Rows from Table and apply Numexpr

Question

Is it possible to select rows from a table in PyTables and apply a Numexpr-like expression to the output? For example, if I have the table

import tables as tb

class Event(tb.IsDescription):
    x = tb.Float32Col()
    y = tb.Float32Col()
    z = tb.Float32Col()

I would like the array of "x+y" where "z > 10.0".

score 1 · Answer 1 · answered Feb 20 '14 at 13:53

1

Yes it is very possible. PyTables provided the Expr class to do exactly this. It uses numexpr under the covers.

answered Feb 20 '14 at 13:53

Anthony Scopatz

3,265
2
15
14

How? Normally, I would have to do something like this `values = [x['x'] + x['y'] for x in table.where('(z > 10.0)')]`. How could Numexpr be used instead of `x['x'] + x['y']`? – xvtk Feb 20 '14 at 13:58

score 0 · Answer 2 · edited May 23 '17 at 10:31

I think what you are attempting is closely related to what I was trying to find out here. I think it is not easily possible to combine the numexpr calculation and the in-kernel query. A possibility would be to use the where expression of numepxr on the whole table and set the result of the cases not satisfying your condition to 0, or something which would not occur during your calculation and then use a where query with a condition filtering out the 0s.

However, this will yield a massive overhead if you have a high selectivity criterium(low number of hits). If this is the case though and the data fits into memory, you can use Numexpr on the arrays loaded into memory (using the in-kernel query).

PyTables: Select Rows from Table and apply Numexpr

2 Answers2