Checking for list membership with Pytables where method

Question

I'm trying to select rows based on multiple criterion that cannot be easily expressed with the conditional statements that [pytables allow] (http://pytables.github.io/usersguide/condition_syntax.html).

I also don't want to format a really long string as a conditional argument because that seems hacky.

So I was trying to collect the rows in an array, hoping that I can do additional selection on rows

my_list1 = ['a', 'b', 'c']
my_list2 = ['5', '6', '7']

rows_selected = []
for an_item in my_list1:
    for row in table_all.where("letters == an_item"):
        for another_item in my_list2:
            if row['numbers'] == another_item:
                print row
                rows_selected.append(row)

This type of setting would work (although not really ellegant). But the problem is, all of the rows collected in rows_selected lose their identity and become the last assigned row. So basically I end up with a list of duplicate rows.

Is there an elegant way of using list membership as selection criteria for pytables? Or how can I circumvent the duplicate row thing and make sure the rows collected inside the loop remain unique?

Answer 1

You can't just use an_item in your conditional. It doesnt not change with the loop. I am surprised you even got anything out, i.e.

table_all.where("letters == an_item")

should be

table_all.where("letters == %s" % an_item)

Why don't you try something like

row_selected = [ table_all.readWhere("(letters == %s) & (numbers == %s)" % (l, n) ) for l in my_list1 for n in my_list2 ]

this should be much faster than what you have there and is much more elegant