How to debug large Polars pl.select - narrow down offending row

Question

when I encounter a panic error in a select query, how do I narrow down which is / are the offending expressions

Edit: simply by looking at the logs / setting a Param. Without having to alter the code into a sequential binary search for loop.

pl.select([
  expr1, expr2, expr3 .... 
])

remove half of them - if it still fails, remove another half, else put back half. repeat — ignoring_gravity, Feb 07 '23 at 12:26

score 1 · Answer 1 · answered Feb 07 '23 at 12:27

1

Where you have

pl.select([
expr1, expr2, expr3 .... 
])

isolate the list

exprlist=[expr1, expr2, expr3 .... ]

Then do something like:

for i, expr in enumerate(exprlist):
    try:
        pl.select(expr)
    except:
        print(f"{i} is bad")

answered Feb 07 '23 at 12:27

Dean MacGregor

While a for loop may work, this is not ideal when trying to look into issues in aproductiom environment where I may not have the luxury of altering the code ans rerunning. The input data may also not be easily reproducible. I would have thought it wouldn't be hard for the select function to log the offending expression strings if an error occurs, along with the stack trace – user1441053 Feb 07 '23 at 16:42
1

I guess that's the pitfall of asking vague questions. – Dean MacGregor Feb 07 '23 at 18:02

1 Answers1