Title: 2to3 does not preserve line endings
Type: behavior Stage: test needed
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: benjamin.peterson Nosy List: benjamin.peterson, bialix, eric.araujo, jason.coombs
Priority: normal Keywords: easy

Created on 2011-03-18 16:27 by bialix, last changed 2018-03-13 09:13 by lukasz.langa.

File name Uploaded Description Edit
2to3-crlf-test.diff eric.araujo, 2011-06-09 15:13
Messages (8)
msg131335 - (view) Author: Alexander Belchenko (bialix) Date: 2011-03-18 16:27
I'm using LF-only line-endings for development of my IntelHex library. I'm working on Windows most of the time.

After 2to3 tool has been ran on my library it has not only changed the Python syntax, but it also saved all files with CRLF line-endings. As result I have all changed files completelly modified and diff shows change in every line. 

2to3 tool should respect my line-endings and must not use simple open(xxx, "wt") mode for writing modified files.
msg131349 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-18 18:19
Thanks for the report.  Can you run “python -m test.test_lib2to3”, if possible with a Python 3.x version?  I’ve seen that the tests use binary mode to compare file contents, so maybe you will get an error message that can get us started.
msg131626 - (view) Author: Alexander Belchenko (bialix) Date: 2011-03-21 09:38
@Éric Araujo: I've ran tests with python 3.2. All tests have passed:

Ran 540 tests in 37.688s

msg131639 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-21 12:29
Thanks.  Would you like to work on a unit test or full patch?
msg131645 - (view) Author: Alexander Belchenko (bialix) Date: 2011-03-21 13:16
Éric, thank you for the proposal, but I'm not familiar enough with the codebase to work on it. 

The short scan over the tests reveals that there is at least one test which tries to test CRLF behavior, in the file, but I don't understand what it doing?

    def test_crlf_newlines(self):
        old_sep = os.linesep
        os.linesep = "\r\n"
            fn = os.path.join(TEST_DATA_DIR, "")
            fixes = refactor.get_fixers_from_package("lib2to3.fixes")
            self.check_file_refactoring(fn, fixes)
            os.linesep = old_sep

So, in theory I can modify that test to check what if the file has LF-only line-endings originally, but os.linesep is CRLF, but then I don't know what the content I should create and how to run fixer over that.
msg131652 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-03-21 13:53
I can fix it. I just need to find time. :)
msg137983 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-09 15:13
I was surprised to see that the file was not using CRLF in the new Mercurial repo.  It is also not in the .hgeol file.  I changed it locally, but it doesn’t change anything, the tests pass before and after the change.
msg307865 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2017-12-08 19:25
This issue affected me today. I'm editing a codebase that has mixed line endings in different files. I'd like to patch for Python 3 support without changing line endings. Even invoking a single fixer (print), the line endings mutate. Since I'm running on macOS, the files with CRLF get LF line endings.

Answers [in this question]( suggest the mutation can be suppressed by altering the _to_system_newlines function, but the proposed fix has no effect on the Python 3.6 version of this routine.

I thought I'd encountered this issue before, but I realized after searching that I was thinking of issue10639, which may present a model for retaining the newlines when refactoring the code.

I found I was able to work around the issue by putting in my current directory:

import lib2to3.refactor
import runpy
import functools

if __name__ == '__main__':
	lib2to3.refactor._open_with_encoding = functools.partial(open, newline='')

And invoking `python -m lib2to3-clean` instead of `-m lib2to3`. The addition of newline='' follows the [guidance in the docs]( on how to avoid mutating newlines.

I've released this functionality in [jaraco.develop 4.0]( so others can readily invoke it with `rwt jaraco.develop -- -m jaraco.develop.lib2to3`.
Date User Action Args
2018-03-13 09:13:53lukasz.langasetkeywords: + easy, - patch
2017-12-08 19:25:24jason.coombssetnosy: + jason.coombs

messages: + msg307865
versions: + Python 3.6, - Python 2.7, Python 3.2, Python 3.3
2011-06-09 15:13:39eric.araujosetfiles: + 2to3-crlf-test.diff
keywords: + patch
messages: + msg137983

versions: - Python 3.1
2011-03-21 13:53:32benjamin.petersonsetnosy: bialix, benjamin.peterson, eric.araujo
messages: + msg131652
2011-03-21 13:16:34bialixsetnosy: bialix, benjamin.peterson, eric.araujo
messages: + msg131645
2011-03-21 12:29:29eric.araujosetnosy: bialix, benjamin.peterson, eric.araujo
title: 2to3 tool does not preserve line-endings -> 2to3 does not preserve line endings
messages: + msg131639
stage: test needed
2011-03-21 09:38:15bialixsetnosy: bialix, benjamin.peterson, eric.araujo
messages: + msg131626
2011-03-18 23:51:15benjamin.petersonsetassignee: benjamin.peterson
nosy: bialix, benjamin.peterson, eric.araujo
2011-03-18 18:19:16eric.araujosetnosy: + eric.araujo

messages: + msg131349
versions: + Python 3.1, Python 3.3
2011-03-18 17:18:59r.david.murraysetnosy: + benjamin.peterson
2011-03-18 16:27:27bialixcreate