41

I'm working with a large existing Python codebase and would like to start adding in type annotations so I can get some level of static checking. I'm imagining something like Erlang, Strongtalk, or Typed Scheme/Racket.

I've seen quick-and-dirty decorators that insert dynamic checks based on function parameter and return type annotations, but I'm looking for something that is more robust and that performs checks at compile-time.

What tools are available right now for this kind of thing? I'm familiar with compilers and type checking and am definitely willing to improve an incomplete tool if it has a good foundation.

(Note: I'm not interested in a discussion of the pros/cons of static typing.)

EDIT: An example:

def put(d, k, v):
   d[k] = v

I'd like to be able to annotate the put function as having type put<K,V>(dict<K,V>, K, V) -> None.

UPDATE: The new PEP 484 (Sep 2014) defines a standard for static typing and type annotations in Python 3.5+. There's a type-checking tool called mypy that is compatible with PEP 484.

Kannan Goundan
  • 4,962
  • 3
  • 24
  • 31
  • I don't think so. I want a tool that will let me sort of treat regular Python like a statically typed language. I do not want to compile extension modules. – Kannan Goundan May 17 '11 at 03:18
  • There's no such compile-time thing on Python... It can't know the types before you execute the code. – JBernardo May 17 '11 at 03:33
  • 4
    You can't always do a perfect job, but there are tools that can infer a lot. For example: [PySonar](http://yinwang0.wordpress.com/2010/09/12/pysonar/) – Kannan Goundan May 17 '11 at 03:36
  • 2
    This is so close to impossible it's not worth it. It's impossible in general as you know. Also, you'd need a very sophisticated version of structural typing to get even close to it - `d[k] = v` word with *every single* object `d` that has `__setitem__`, for instance, not just with instance of `dict`. Depending on the actual type of `d`, there may be various incompatible requirements on `k` (e.g. hashable, has `__index__`) and even on `v`. And *every single object* has e.g. `__hash__`, some of them just "don't work" at runtime. **Static typechecking is not possible in Python.** –  May 17 '11 at 13:37
  • 9
    Doing a perfect job is not possible. **I know this.** But Erlang, Smalltalk, and Scheme have static checking tools that are still useful. The `__setitem__` check seems like a standard structural type check. There are ways to specify a required relationship between the types of `d` and `k`. The selective implementation of `__hash__` isn't ideal, but Java has the same issue with `UnsupportedOperationException`. And even if I can't solve every problem, it doesn't mean I can't have a static checker that is still useful for some things. – Kannan Goundan May 20 '11 at 00:09
  • If you want a compile-time typing system use a language that has one. Python neophytes that come from a strongly typed language often have unrealistic expectations that what they are familiar with has actually been helping them as much as they thought. That C++ programmers consider the fact that array indices are known to be in the class of ints to be heartwarming is the real puzzlement when all but a handful of those ints are disastrously wrong. In practice the incidence of Python type errors approaches zero. – msw Jun 15 '11 at 18:18
  • 5
    All of the above reasons given for why Python can't have static type checking also apply to Scheme and it's descendant Racket. But optional static type checking has successfully been added to Racket while preserving the "dynamic" programming style of the language. Gradual Typing for Python (Jeremy Siek et al, mentioned in one answer, should be released soon) applies similar ideas, and more, to Python, so it can be done. I think some have misinterpreted the question as "Can all dynamic checks be replaced by static type checking?" - clearly no, even in Java static types don't do that. – RD1 Nov 06 '12 at 01:37
  • 1
    I agree that static type checking is not possible for **every** Python program. However, for certain subsets of it, it **is** (i.e. those that don't use dynamic features). And for an even bigger subset, type can be statically checked 99.99% of the time. I'm having much trouble using Python in medium (not to say big) applications. Unless the community comes up with solutions, Python will be relegated to "small stuff"". – Mario Rossi Aug 12 '13 at 05:59
  • @KannanGoundan There is the new 'gradual' package you might want to check out (see answer below). You have to annotate your program in some parts, but other than that it seems very promising. – Paul Nov 13 '13 at 07:30

7 Answers7

17

Edit 2016-11-11: Just use mypy. Type hints can be added gradually. In Python 3 source code, it verifies standard PEP 484 type hints. Types can still be expressed in Python 2 using special comments. Guido likes it.

This post was originally written a long time ago before mypy was a thing. I've preserved the post's original content below, even though it isn't quite accurate.

Original post:

You might want to check out some of the projects mentioned in this related StackOverflow post on static analysis for Python.

In summary:

Since Python uses duck typing extensively, things that might be called "type errors" in other languages might end up being "object X doesn't support method Y" in Python.

Edit 2011-05-17:

I agree with delnan that static typing is not possible for Python [apparently wrong]. But since our skepticism doesn't seem to deter you, I can only give you more information on the subject. I present:

  • A discussion of type inference for Python. (Other links are from here.)
  • Guido van van Rossum's articles on adding optional static typing: part 1 and part 2.
  • RPython, a subset of Python that might stand a chance of being statically analyzed enough to do some form of type checking.
Community
  • 1
  • 1
Karmastan
  • 5,618
  • 18
  • 24
  • 1
    From what I can see, those tools don't let me write type annotations. Also, they checks they perform seem to be relatively superficial (when compared with a typical static type system). – Kannan Goundan May 17 '11 at 03:21
  • @Kannan: I have added more links to my answer. – Karmastan May 17 '11 at 16:14
  • 1
    Thanks for the update, but these are all things I was aware of. They all ended up being dead ends for my use case (gradually adding types to a large untyped codebase). – Kannan Goundan May 20 '11 at 00:15
  • Link to "related StackOverflow post on static analysis for Python." is now broken. – Bruno Rijsman Dec 23 '20 at 12:10
11

You may find mypy interesting. It has been proposed for inclusion in Python 3.5 by Guido.

kirbyfan64sos
  • 10,377
  • 6
  • 54
  • 75
6

Check out this post: PySonar: a Static Analyzer for Python. PySonar is a tool that infers types using abstract interpretation (partially executing) of code. It finds all possible execution paths of your program and finds all possible types of all variables.

There are basically three versions of PySonar:

  • Open-sourced Java (Jython indexer)
  • Closed-sourced Java (Hidden in Google)
  • Open-sourced Python (mini-pysonar)

None of them (except of closed source one) is fully implemented. But the basic idea is that you can use it as a basis for your work.

Vanuan
  • 31,770
  • 10
  • 98
  • 102
  • This is what I've found on GitHub [page](https://github.com/yinwang0/mini-pysonar) of mini-pysonar: "I currently have no motivation of developing PySonar further." Makes me sad. – Tomas Tomecek Sep 02 '13 at 11:33
  • @SummerBreeze Yes. It is a concept (read "example"), not the end user solution. It would be great if you could extend it to produce an html file with analyzed data. Currently it only collects the data, but there is no presentation (end-user tool). – Vanuan Sep 02 '13 at 11:43
3

I don't know if this helps but for what it's worth, Jeremy Siek at U of Colorado did some work on gradual typing, and I found this doing a quick search. http://www.wiki.jvmlangsummit.com/pdf/28_Siek_gradual.pdf

My guess (I might be wrong) is there isn't any promising open source tools you can find at the moment since his research looks relatively new.

Your best bet might be to contact the authors and ask if they can release their code to you.

Russell
  • 3,975
  • 7
  • 37
  • 47
2

There is the 'gradual' package for Python 3; see PIP or the Bitbucket Repo

Apparently this is an implementation by the group around Jeremy Siek who seems to be quite an authority in the field of gradual typing.

Some annotations are apparently necessary, here is an example:

from gradual import *

@typed
def calculate_total(a:int, b:int) -> int:
    return a + b//100

As far as annotations go, this is not so bad. I have not used the package so I cannot speak to its quality, but the syntax (and the people behind it) certainly make it look promising.

Lutz Prechelt
  • 36,608
  • 11
  • 63
  • 88
Paul
  • 7,836
  • 2
  • 41
  • 48
2

I like prospector, backend of landscape.io. It combines output of existing analyzers, such as pylint, pyflakes, pep8, frosted..., into one report. Neat.

Tomas Tomecek
  • 6,226
  • 3
  • 30
  • 26
1

I had a similar need some time ago. All the existing solutions that I've found had some problem or does not have a feature that I'd like to have, so I've made my own.

Here's how you use it:

from requiretype import require

@require(name=str, age=(int, float, long))
def greet_person(name, age):
    print "Hello {0} ({1})".format(name, age)

>>> greet_person("John", 42)
Hello John (42)

>>> greet_person("John", "Doe")
# [...traceback...]
TypeError: Doe is not a valid type.
Valid types: <type 'int'>, <type 'float'>, <type 'long'>

>>> greet_person(42, 43)
# [...traceback...]
TypeError: 42 is not a <type 'str'> type

I hope this is useful for you.

For more details look at:

P.S.: (quoting myself from github repo)

For most cases I'd recommend using tests instead of type checking since it's more natural to do that in Python. But, there are some cases where you want/need to specify a specific type to use and since python does not have type checks for parameters here's where this is useful.

ivanalejandro0
  • 1,689
  • 13
  • 13