using warnings.filterwarnings() to convert a warning into an exception

Question

I want to programatically capture when statsmodels.api.OLS raises its "The smallest eigenvalue is ..." warning

This would enable me to filter a large number of OLS systems by whether or not they raise this warning

Ideally, I would like to pick off just particular warnings instead of a blanket filter for any/all warnings

My attempt (below) attempts a blanket filter using warnings.filterwarnings() , it doesn't work

How do I get warnings.filterwarnings() to work? Or is there some other module I should be looking at instead?

import statsmodels.api as sm
import numpy as np
import pandas as pd
import warnings

np.random.seed(123)

nrows = 100

colA = np.random.uniform(0.0, 1.0, nrows)
colB = np.random.uniform(0.0 ,1.0, nrows)
colC = colA + colB  # multicolinear data to generate ill-conditioned system
y = colA + 2 * colB + np.random.uniform(0.0, 0.1, nrows)
X = pd.DataFrame({'colA': colA, 'colB': colB, 'colC': colC})

warnings.filterwarnings('error')  # achieves nothing

warnings.simplefilter('error')
# from https://stackoverflow.com/questions/59961735/cannot-supress-python-warnings-with-warnings-filterwarningsignore
# also achieves nothing

try:
    model = sm.OLS(y, sm.add_constant(X)).fit()
    print(model.summary())
except:
    print('warning successfully captured in try-except')

You may have to dig around in the source code to see if it is actually using python warnings system. Seeing the actual full warning may also give some clues. — hpaulj, Jan 11 '22 at 02:50
Turns out it wasn't a warning, just a print() with the word "warning" in it — PlanetAtkinson, Jan 12 '22 at 10:58

score 0 · Accepted Answer · answered Jan 11 '22 at 03:20

You can get the smallest eigenvalue using model.eigenvals[-1], just check that it is less than 1e-10 to raise an exception. Here's the source that generates the note

errstr = ("The smallest eigenvalue is %6.3g. This might indicate that there are\n"
          "strong multicollinearity problems or that the design matrix is singular.")
try:
    model = sm.OLS(y, sm.add_constant(X)).fit()
    assert model.eigenvals[-1] >= 1e-10, (errstr % model.eigenvals[-1])
    print(model.summary())
except AssertionError as e:
    print('warning successfully captured in try-except. Message below:')
    print(e)

using warnings.filterwarnings() to convert a warning into an exception

1 Answers1