11

I have created custom Django model-field subclasses based on CharField but which use to_python() to ensure that the model objects returned have more complex objects (some are lists, some are dicts with a specific format, etc.) -- I'm using MySQL so some of the PostGreSql field types are not available.

All is working great, but Pylint believes that all values in these fields will be strings and thus I get a lot of "unsupported-membership-test" and "unsubscriptable-object" warnings on code that uses these models. I can disable these individually, but I would prefer to let Pylint know that these models return certain object types. Type hints are not helping, e.g.:

class MealPrefs(models.Model):
    user = ...foreign key...
    prefs: dict[str, list[str]] = \
       custom_fields.DictOfListsExtendsCharField(
            default={'breakfast': ['cereal', 'toast'], 
                     'lunch': ['sandwich']},
       )

I know that certain built-in Django fields return correct types for Pylint (CharField, IntegerField) and certain other extensions have figured out ways of specifying their type so Pylint is happy (MultiSelectField) but digging into their code, I can't figure out where the "magic" specifying the type returned would be.

(note: this question is not related to the INPUT:type of Django form fields)

Thanks!

Michael Scott Asato Cuthbert
  • 3,442
  • 2
  • 22
  • 52
  • consulted https://stackoverflow.com/questions/51830993/how-to-specify-typings-for-dynamic-fields-similar-to-django-model-field but not a duplicate or specifying an answer. – Michael Scott Asato Cuthbert Jan 24 '19 at 17:30
  • 1
    Where is the code where the warnings coming from? Are you happy to disable any unwanted pylint warnings at a higher level? Or do you want to inform pylint to expect the return types and then catch when there is something unexpected. I usually just disable at a higher level in cases like this :-) – Scott Skiles Jan 25 '19 at 21:16
  • I can disable at a higher level, but I'm trying to understand what makes pylint understand the correct types for built-in Django fields but not ones I create. – Michael Scott Asato Cuthbert Feb 10 '19 at 17:10
  • You should probably add information that you use a plugin [pylint-django](https://github.com/PyCQA/pylint-django). (or not?) – hynekcer Mar 25 '19 at 23:30
  • I was not aware of the plugin. Will do that and happy to award the bounty @hynekcer if you write it as an answer. But certain code is working properly even without the plugin – Michael Scott Asato Cuthbert Mar 26 '19 at 11:55

2 Answers2

5

I had a look at this out of curiosity, and I think most of the "magic" actually comes for pytest-django.

In the Django source code, e.g. for CharField, there is nothing that could really give a type hinter the notion that this is a string. And since the class inherits only from Field, which is also the parent of other non-string fields, the knowledge needs to be encoded elsewhere.

On the other hand, digging through the source code for pylint-django, though, I found where this most likely happens:

in pylint_django.transforms.fields, several fields are hardcoded in a similar fashion:

_STR_FIELDS = ('CharField', 'SlugField', 'URLField', 'TextField', 'EmailField',
               'CommaSeparatedIntegerField', 'FilePathField', 'GenericIPAddressField',
               'IPAddressField', 'RegexField', 'SlugField')

Further below, a suspiciously named function apply_type_shim, adds information to the class based on the type of field it is (either 'str', 'int', 'dict', 'list', etc.)

This additional information is passed to inference_tip, which according to the astroid docs, is used to add inference info (emphasis mine):

astroid can be used as more than an AST library, it also offers some basic support of inference, it can infer what names might mean in a given context, it can be used to solve attributes in a highly complex class hierarchy, etc. We call this mechanism generally inference throughout the project.

astroid is the underlying library used by Pylint to represent Python code, so I'm pretty sure that's how the information gets passed to Pylint. If you follow what happens when you import the plugin, you'll find this interesting bit in pylint_django/.plugin, where it actually imports the transforms, effectively adding the inference tip to the AST node.

I think if you want to achieve the same with your own classes, you could either:

  1. Directly derive from another Django model class that already has the associated type you're looking for.
  2. Create, and register an equivalent pylint plugin, that would also use Astroid to add information to the class so that Pylint know what to do with it.
Samuel Dion-Girardeau
  • 2,790
  • 1
  • 29
  • 37
4

I thought initially that you use a plugin pylint-django, but maybe you explicitly use prospector that automatically installs pylint-django if it finds Django.

The checker pylint neither its plugin doesn't check the code by use information from Python type annotations (PEP 484). It can parse a code with annotations without understanding them and e.g. not to warn about "unused-import" if a name is used in annotations only. The message unsupported-membership-test is reported in a line with expression something in object_A simply if the class A() doesn't have a method __contains__. Similarly the message unsubscriptable-object is related to method __getitem__.

You can patch pylint-django for your custom fields this way:
Add a function:

def my_apply_type_shim(cls, _context=None):  # noqa
    if cls.name == 'MyListField':
        base_nodes = scoped_nodes.builtin_lookup('list')
    elif cls.name == 'MyDictField':
        base_nodes = scoped_nodes.builtin_lookup('dict')
    else:
        return apply_type_shim(cls, _context)
    base_nodes = [n for n in base_nodes[1] if not isinstance(n, nodes.ImportFrom)]
    return iter([cls] + base_nodes)

into pylint_django/transforms/fields.py

and also replace apply_type_shim by my_apply_type_shim in the same file at this line:

def add_transforms(manager):
    manager.register_transform(nodes.ClassDef, inference_tip(my_apply_type_shim), is_model_or_form_field)

This adds base classes list or dict respectively, with their magic methods explained above, to your custom field classes if they are used in a Model or FormView.

Notes:

I thought also about a plugin stub solution that does the same, but the alternative with "prospector" seems so complicated for SO that I prefer to simply patch the source after installation.

Classes Model or FormView are the only classes created by metaclasses, used in Django. It is a great idea to emulate a metaclass by a plugin code and to control the analysis simple attributes. If I remember, MyPy, referenced in some comment here, has also a plugin mypy-django for Django, but only for FormView, because writing annotations for django.db is more complicated than to work with attributes. - I was trying to work on it for one week.

hynekcer
  • 14,942
  • 6
  • 61
  • 99