Python static type hint/check mismatch between Iterable[AnyStr] vs Iterable[str] | Iterable[bytes]

Question

I'm running into this static type hint mismatch (with Pyright):

from __future__ import annotations
from typing import AnyStr, Iterable


def foo(i: Iterable[AnyStr]):
    return i


def bar(i: Iterable[str] | Iterable[bytes]):
    return i


def baz(i: Iterable[str | bytes]):
    return i


def main():
    s = ['a']

    # makes sense to me
    baz(foo(s))  # allowed
    foo(baz(s))  # not allowed

    # makes sense to me
    baz(bar(s))  # allowed
    bar(baz(s))  # not allowed

    bar(foo(s))  # allowed
    foo(bar(s))  # nope -- why?

What's the difference between Iterable[AnyStr] and Iterable[str] | Iterable[bytes]?

Shouldn't they be "equivalent"? (save for AnyStr referring to a single consistent type within a context)

More concretely: what is the right way to type-hint the following?

import random
from typing import Iterable, AnyStr

def foo(i: Iterable[AnyStr]):
    return i

def exclusive_bytes_or_str():  # type-inferred to be Iterator[bytes] | Iterator[str]
    if random.randrange(2) == 0:
        return iter([b'bytes'])
    else:
        return iter(['str'])

foo(iter([b'bytes']))          # fine
foo(iter(['str']))             # fine
foo(exclusive_bytes_or_str())  # same error

I don't think they are equivalent. Presumably `Iterable[AnyStr]` can emit some mixture of stringy types, but `Iterable[str] | Iterable[bytes]` emits *either* all strings *or* all bytes. — khelwood, Aug 21 '21 at 00:01
khelwood nailed it. You can't return `['foo', b'bar']` as an `Iterable[str] | Iterable[bytes]` because it's neither of those types, but it's perfectly valid as an `Iterable[str | bytes]` or `Iterable[AnyStr]`. — Samwise, Aug 21 '21 at 00:29
@khelwood — I don't think that's correct. `AnyStr` is defined in the `typing` module as `AnyStr = TypeVar('AnyStr', str, bytes)`. Since it's a `TypeVar`, it can only be evaluated as one specific type in any one expression, so `Iterable[AnyStr]` *should* also indicate an iterable that consists either of only `str` objects or of only `bytes` objects. This makes it very different to `Iterable[Union[str, bytes]]` https://mypy.readthedocs.io/en/stable/generics.html — Alex Waygood, Aug 21 '21 at 00:53

Kache · Accepted Answer · 2021-08-24T17:25:24.313

Paraphrased answer from erictraut@github:

This isn't really the intended use for a constrained TypeVar. I recommend using an @overload instead:
@overload
def foo(i: Iterable[str]) -> Iterable[str]: ...
@overload
def foo(i: Iterable[bytes]) -> Iterable[bytes]: ...

def foo(i: Iterable[AnyStr]) -> Iterable[AnyStr]:
    return i

Because:

The type Iterable[str] | Iterable[bytes] is not assignable to type Iterable[AnyStr]. A constrained type variable needs to be matched against one of its contraints, not multiple constraints. When a type variable is "solved", it needs to be replaced by another (typically concrete) type. If foo(bar(s)) were allowed, what type would the AnyType@foo type variable resolve to? If it were resolved to type str | bytes, then the concrete return type of foo would be Iterable[str | bytes]. That's clearly wrong.

Python static type hint/check mismatch between Iterable[AnyStr] vs Iterable[str] | Iterable[bytes]

1 Answers1