2to3 - how to keep newline characters from input file?

Question

I'm attempting to run 2to3 on Windows machine where *.py files has Unix-style end-line characters. Running 2to3 modifies newline characters in output file.

MCVE:

print2.py content before

print "Hello, world!"\n

Executed command:

2to3 print2.py -w -n

print2.py content after

print("Hello, world!")\r\n

Expected content:

print("Hello, world!")\n

Is it possible to keep old newline characters when 2to3 conversion is performed?

@RégisB. it's more of hack than a solution, but yes, I have something that works without modifying any files in stdlib. — Łukasz Rogalski, Aug 23 '16 at 06:57

Łukasz Rogalski · Accepted Answer · 2016-08-24T12:23:09.307

Since there seems to be no standard way to change this behavior in command line usage, I've prepared very simple Python script, which runs code and patches unwanted behavior.

Here's an example for python modernize, but any 2to3-based tool will do just fine.

# to access function to patch
import lib2to3.refactor
# actual main
import libmodernize.main
# convert str to list of args, not mandatory
import shlex
# patch problematic function, as suggested by @mfripp
lib2to3.refactor._to_system_newlines = lambda input: input 

args = shlex.split("-w -n src")  # prepare args
libmodernize.main.main(args)  # pass args to main, equivalent of running cmdline tool

score 0 · Answer 2 · answered Aug 18 '16 at 22:46

On Windows, the system line separator is \r\n, as we can see in os.py:

if 'posix' in _names:
    ...
    linesep = '\n'
    ...
elif 'nt' in _names:
    ...
    linesep = '\r\n'
    ...

This line separator is used in lib2to3.refactor:

def _to_system_newlines(input):
    if os.linesep != "\n":
        return input.replace(u"\n", os.linesep)
    else:
        return input

So to preserve line separators with the 2to3 script, it should be enough to replace line return input.replace(u"\n", os.linesep) by return input in the above function.

If you don't want to edit the source code, you could patch lib2to3.refactor for the current session by running `lib2to3.refactor._to_system_newlines = lambda input: input`. — Matthias Fripp, Aug 18 '16 at 22:53

score 0 · Answer 3 · answered Dec 08 '17 at 19:37

This issue seems to have previously been reported as Python 11594, still unresolved. In investigating, I found the recommended workaround, setting lib2to3._to_system_newlines doesn't have any effect when lib2to3 is run from Python 3. What did work for me was to override _open_with_encoding thus:

lib2to3.refactor._open_with_encoding = functools.partial(open, newline='')

I've made this functionality available in jaraco.develop. The module can be invoked just as you would invoke lib2to3, but with the patch applied:

python -m jaraco.develop.lib2to3 .

Or to incorporate it into another library like libmodernize:

import runpy
from jaraco.develop import lib2to3
lib2to3.patch_for_newlines()
runpy.run_module('modernize')

These techinques probably won't work on Python 2, and so I suggest running lib2to3 from Python 3.

2to3 - how to keep newline characters from input file?

3 Answers3