Abstract base classes and Exceptions

Question

Question

Why do virtual subclasses of an abstract Exception created using the ABCMeta.register not match under the except clause?

Background

I'd like to ensure that exceptions that get thrown by a package that I'm using are converted to MyException, so that code which imports my module can catch any exception my module throws using except MyException: instead of except Exception so that they don't have to depend on an implementation detail (the fact that I'm using a third-party package).

Example

To do this, I've tried registering an OtherException as MyException using an abstract base class:

# Tested with python-3.6
from abc import ABC

class MyException(Exception, ABC):
    pass

class OtherException(Exception):
    """Other exception I can't change"""
    pass

MyException.register(OtherException)

assert issubclass(OtherException, MyException)  # passes

try:
    raise OtherException("Some OtherException")
except MyException:
    print("Caught MyException")
except Exception as e:
    print("Caught Exception: {}".format(e))

The assertion passes (as expected), but the exception falls to the second block:

Caught Exception: Some OtherException

Answer 1

Alright, I looked into this some more. The answer is that it's a long-outstanding open issue in Python3 (there since the very first release) and apparently was first reported in 2011. As Guido said in the comments, "I agree it's a bug and should be fixed." Unfortunately, this bug has lingered due to concerns about the performance of the fix and some corner cases that need to be handled.

The core issue is that the exception matching routine PyErr_GivenExceptionMatches in errors.c uses PyType_IsSubtype and not PyObject_IsSubclass. Since types and objects are supposed to be the same in python3, this amounts to a bug.

I've made a PR to python3 that seems to cover all of the issues discussed in the thread, but given the history I'm not super optimistic it's going to get merged soon. We'll see.

Answer 2

The why is easy:

from abc import ABC

class MyException(Exception, ABC):
    pass

class OtherException(Exception):
    """Other exception I can't change"""
    pass

MyException.register(OtherException)

assert issubclass(OtherException, MyException)  # passes
assert OtherException in MyException.__subclasses__()  # fails

Edit: This assert mimics the outcome of the except clause, but does not represent what actually happens. Look at the accept answer for an explanation.

The workaround also is easy:

class OtherException(Exception):
    pass
class AnotherException(Exception):
    pass

MyException = (OtherException, AnotherException)

Answer 3

It seems that CPython once again takes some shortcuts and doesn't bother calling the metaclass's __instancecheck__ method for the classes listed in except clauses.

We can test this by implementing a custom metaclass with __instancecheck__ and __subclasscheck__ methods:

class OtherException(Exception):
    pass

class Meta(type):
    def __instancecheck__(self, value):
        print('instancecheck called')
        return True

    def __subclasscheck__(self, value):
        print('subclasscheck called')
        return True

class MyException(Exception, metaclass=Meta):
    pass

try:
    raise OtherException("Some OtherException")
except MyException:
    print("Caught MyException")
except Exception as e:
    print("Caught Exception: {}".format(e))

# output:
# Caught Exception: Some OtherException

We can see that the print statements in the metaclass aren't executed.

---

I don't know if this is intended/documented behavior or not. The closest thing to relevant information I could find was from the exception handling tutorial:

A class in an except clause is compatible with an exception if it is the same class or a base class thereof

Does that mean that classes have to be real subclasses (i.e. the parent class must be part of the subclass's MRO)? I don't know.

---

As for a workaround: You can simply make MyException an alias of OtherException.

class OtherException(Exception):
    pass

MyException = OtherException

try:
    raise OtherException("Some OtherException")
except MyException:
    print("Caught MyException")
except Exception as e:
    print("Caught Exception: {}".format(e))

# output:
# Caught MyException

In the case that you have to catch multiple different exceptions that don't have a common base class, you can define MyException as a tuple:

MyException = (OtherException, AnotherException)

Answer 4

Well, this doesn't really answer your question directly, but if you're trying to ensure a block of code calls your exception, you could take a different strategy by intercepting with a context manager.

In [78]: class WithException:
    ...:     
    ...:     def __enter__(self):
    ...:         pass
    ...:     def __exit__(self, exc, msg, traceback):
    ...:         if exc is OtherException:
    ...:             raise MyException(msg)
    ...:         

In [79]: with WithException():
    ...:     raise OtherException('aaaaaaarrrrrrggggh')
    ...: 
---------------------------------------------------------------------------
OtherException                            Traceback (most recent call last)
<ipython-input-79-a0a23168647e> in <module>()
      1 with WithException():
----> 2     raise OtherException('aaaaaaarrrrrrggggh')

OtherException: aaaaaaarrrrrrggggh

During handling of the above exception, another exception occurred:

MyException                               Traceback (most recent call last)
<ipython-input-79-a0a23168647e> in <module>()
      1 with WithException():
----> 2     raise OtherException('aaaaaaarrrrrrggggh')

<ipython-input-78-dba8b409a6fd> in __exit__(self, exc, msg, traceback)
      5     def __exit__(self, exc, msg, traceback):
      6         if exc is OtherException:
----> 7             raise MyException(msg)
      8 

MyException: aaaaaaarrrrrrggggh