How to fix mypy inferring loop variable as "object"?

Question

I made this simple function which I want to check with mypy and pylint. It just parses a string and converts it to the appropriate type.

import re
from typing import Any, Callable
    
def parse_constant(constant: str) -> Any:
    for reg, get_val in [
            (re.compile(r'\'(.*)\''), str),
            (re.compile(r'true', re.IGNORECASE), lambda _: True),
            (re.compile(r'false', re.IGNORECASE), lambda _: False),
            (re.compile(r'([0-9]+)'), int),
            (re.compile(r'([0-9]+\.[0-9]+)'), float)
    ]:
        match = reg.fullmatch(constant)
        if match is not None:
            if len(match.groups()) == 0:
                val = None
            else:
                val = match.groups()[0]
            return get_val(val)
    return None

It works fine but mypy complains: I get error: "object" not callable at line 18 (return get_val(val)).

Now if I replace, str by lambda x: str(x) mypy is happy but pylint complains with Lambda may not be necessary.

What is the proper way to fix that?

The cause of this is that MyPy selects the *base* instead of the *union* when mixing types via inference. The base of ``type`` and ``Callable`` is ``object``. Might be relevant: [Why does mypy infer the common base type instead of the union of all contained types?](https://stackoverflow.com/questions/57452652/why-does-mypy-infer-the-common-base-type-instead-of-the-union-of-all-contained-t) — MisterMiyagi, Mar 26 '21 at 09:37

MisterMiyagi · Accepted Answer · 2021-03-26T09:55:22.077

1

The issue is that MyPy must infer get_val from a mixture of Callable and Type. In this case, MyPy selects the base instead of the union of the types. Explicitly annotate the types to avoid too broad inference.

Inside the for loop, only the loop variables can be annotated. By moving the iterable outside the loop, it can be annotated:

import re
from typing import Any, Callable, Pattern, List, Tuple

cases: List[Tuple[Pattern[str], Callable]] = [
    (re.compile(r'\'(.*)\''), str),
    (re.compile(r'true', re.IGNORECASE), lambda _: True),
    (re.compile(r'false', re.IGNORECASE), lambda _: False),
    (re.compile(r'([0-9]+)'), int),
    (re.compile(r'([0-9]+\.[0-9]+)'), float)
]

def parse_constant(constant: str) -> Any:
    for reg, get_val in cases:
        match = reg.fullmatch(constant)
        if match is not None:
            if len(match.groups()) == 0:
                val = None
            else:
                val = match.groups()[0]
            return get_val(val)
    return None

Moving the cases outside of the function has the added advantage that they are created only once. This is especially of importance for re.compile, which is now compiled once and then stored for repeated use.

edited Mar 26 '21 at 09:55

answered Mar 26 '21 at 09:48

MisterMiyagi

44,374
10
104
119

Many thanks for the clear answer (and for providing a clearer question title)! Nevertheless I'm a bit confused about this: mypy complains because basically`str` is not a `Callable` but if I tell him explicitly that it's a `Callable` then he's ok with that. I guess the lesson is to rely on explicit typing when I face this situation. Thanks again. – qouify Mar 26 '21 at 14:28
@qouify I am afraid that is not obvious indeed. The root problem is that inference has multiple choices, such as the literal ``str`` being of type ``Type[str]`` or ``Callable[..., str]``, and has to pick *one*. Notably, once inference has made a selection this is "locked in" even if it leads to conflicts later on. By providing an annotation at a point before inference has "locked in", you can influence which choice is made. – MisterMiyagi Mar 26 '21 at 14:38
I see. I'm probably not familiar enough with the typing mechanism and I guess I should learn more on that to properly use mypy. At first, my assumption was that there is some inheritance mechanism between types in such a way that `Type[str]` inherits from `Callable` but that's obviously a mistake. Thanks for your help. – qouify Mar 26 '21 at 14:44

How to fix mypy inferring loop variable as "object"?

1 Answers1