Neat way to port nonpolyglot code from Py2 to Py3

Question

In many of my python modules, I am using

from itertools import izip_longest

But now I am moving my codebase to Py3 (compatible with Py2 at the same time). And in Py3 izip_longest is renamed to zip_longest. One answer to fix this is recommended in Failing to import itertools in Python 3.5.2, i.e. to change import statement to following.

try:
    # Python 3
    from itertools import zip_longest as izip_longest
except ImportError:
    # Python 2
    from itertools import izip_longest

However, changing this in 20+ modules looks a bit odd to me. What is the neat way of doing this?

can you use library `six`? It's got `from six.moves import zip_longest` which is `izip_longest` in py2 and `zip_longest` in py3 — Macattack, Jun 22 '20 at 18:52

score 2 · Accepted Answer · answered Jun 22 '20 at 18:52

2

You can wrap that exact code inside a module of your own, say itertools_compat.py, and just write

from itertools_compat import izip_longest

wherever you need this function.

answered Jun 22 '20 at 18:52

Thomas

174,939
50
355
478

You’d still need to change 20 modules to use this wrapper. What does 2to3 make of this? – Jun 22 '20 at 18:56
1

2to3 is extensible, you can write your own patterns and replacements, this would be a trivial case. If someone is interested, i could post an example if you make a separate question and link me to it from here or something – Macattack Jun 22 '20 at 19:27
@macattack can you create an answer here itself? – ThinkGeek Jun 22 '20 at 21:41
@LokeshAgrawal I think it's kinda off topic to the question, plus you've already marked this answered, can't get the sweet answer points. The question that'd be more appropriate is how can I refactor my code automatically from `x` -> `y`... – Macattack Jun 22 '20 at 23:58
@macattack I have created a new question https://stackoverflow.com/q/62525336/7707677. Please share your answer over there – ThinkGeek Jun 23 '20 at 00:05

gelonida · Answer 2 · 2020-06-23T10:36:11.217

Making python2 code work with python3 is not for free. And there is a lot of grunt work involved. I think it is difficult to avoid this and in any case you have to do proper testing to be sure your code is working with both versions.

I don't know your code and your project, it's exact contents, whether this project should stay alive for much longer or if you just want to make it survive for a little longer. Thus I'm not sure what's best in your scenario.

In general I would suggest to change your code such, that it looks like python3 code and still runs with python2 and not to write code that looks like python2 code but also runs with python3. (But all depends on your context)

You might want to try out the package future https://python-future.org/ Which provides helpers for writing code, that's running with both versions.

future also contains tools, that try to automatically change code in a way, that it is more likely to run with both versions. Depending on the complexity of your code you still have to do quite some things manually (especially for unicode issues)

The commands are called futurize (make code look like python3 but run with python2) or pasteurize (make code look like python2 but run also with python3)

make backups of all your files (or use some version control like git) before trying this out.

The simple use case is futurize --stage1 yourfile.py

There is one interesting chapter / cheatsheet with many examples, that is worth reading https://python-future.org/compatible_idioms.html?highlight=zip

If you do not want to use the future package or if you encounter situations, that future doesn't handle well I would write an adapter module and use that in your code.

e.g. py2py3compat.py:

try: 
    # Python 3 
    from itertools import zip_longest
except ImportError: 
    # Python 2 
    from itertools import izip_longest as zip_longest

yourpyfile.py:

from py2py3compat import zip_longest

and perform a global search and replace to replace izip_longest with zip_longest

score 1 · Answer 3 · answered Jun 23 '20 at 16:48

I see your other question got closed so I'll post here. So, there's a couple files, and naming is important!

fixer.py

#!/usr/bin/env python3
# you'll need 2to3 and python-modernize installed
from __future__ import absolute_import
import sys
from lib2to3.main import main
import libmodernize

sys.path.append(".")
sys.exit(main('fixers'))

fixers/fix_iziplongest.py

from lib2to3 import fixer_base
import libmodernize

class FixIziplongest(fixer_base.BaseFix):
    PATTERN = """
    power< 'izip_longest' trailer< '(' any* ')' > >
    | import_from< 'from' 'itertools' 'import' 'izip_longest' >
    """

    # This function is only called on matches to our pattern, which should
    # be usage of izip_longest, and the itertools import
    def transform(self, node, results):
        # add the new import (doesn't matter if we do this multiple times
        libmodernize.touch_import('itertools_compat', 'zip_longest', node)
        # remove the old import
        if node.type == syms.import_from:
            node.parent.remove()
            return node
        
        # rename to izip_longest
        node.children[0].value = 'zip_longest'
        return node

Usage is the same as 2to3 - python ./fixer.py -f iziplongest file_to_fix.py (more flags if you want it to apply changes, that'll just show the diff) So, what this does is it would covert this:

from itertools import izip_longest
for x in izip_longest(a, b):
     print(x)

To this:

from itertools_compat import zip_longest
for x in zip_longest(a, b):
     print(x)

Neat way to port nonpolyglot code from Py2 to Py3

3 Answers3