classification
Title: Tests fail because of git's newline preferences on Windows
Type: behavior Stage: resolved
Components: Tests, Windows Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Jim.Jewett, abarry, benjamin.peterson, brett.cannon, larry, martin.panter, ned.deily, paul.moore, r.david.murray, steve.dower, terry.reedy, tim.golden, vstinner, zach.ware
Priority: Keywords: patch

Created on 2016-06-30 17:16 by abarry, last changed 2017-07-01 18:02 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
add_gitattributes_1.patch abarry, 2016-08-25 14:38 review
test_random_warning_1.patch abarry, 2016-08-25 14:39 review
add_gitattributes_2.patch abarry, 2016-08-25 15:24 review
Pull Requests
URL Status Linked Edit
PR 840 merged zach.ware, 2017-03-27 05:35
PR 844 merged vstinner, 2017-03-27 09:34
PR 2083 merged zach.ware, 2017-06-10 20:10
PR 2084 merged zach.ware, 2017-06-10 20:19
PR 2086 merged zach.ware, 2017-06-10 20:38
Messages (28)
msg269604 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-06-30 17:16
Specifically, test_random and test_sax both fail because of git's newline preferences. Mine is set to convert \r\n to \n on push and the other way around on pull. Relevant output:

test_random:

....EException ignored in: <_io.FileIO name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck'>
...........................EException ignored in: <_io.FileIO name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name='E:\\GitHub\\cpython\\lib\\test\\randv2_32.pck'>
............................
======================================================================
ERROR: test_bug_1727780 (__main__.MersenneTwister_TestBasicOps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_random.py", line 178, in test_bug_1727780
    r = pickle.load(f)
ImportError: No module named 'random\r'

======================================================================
ERROR: test_bug_1727780 (__main__.SystemRandom_TestBasicOps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_random.py", line 178, in test_bug_1727780
    r = pickle.load(f)
ImportError: No module named 'random\r'

----------------------------------------------------------------------
Ran 61 tests in 1.480s

FAILED (errors=2)

test_sax:

(Passing tests omitted)

======================================================================
FAIL: test_expat_binary_file (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 836, in test_expat_binary_file
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_binary_file_bytes_name (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 874, in test_expat_binary_file_bytes_name
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_binary_file_int_name (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 886, in test_expat_binary_file_int_name
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_binary_file_nonascii (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 862, in test_expat_binary_file_nonascii
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_inpsource_byte_stream (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1053, in test_expat_inpsource_byte_stream
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_inpsource_character_stream (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1066, in test_expat_inpsource_character_stream
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_inpsource_filename (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1015, in test_expat_inpsource_filename
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_inpsource_sysid (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1025, in test_expat_inpsource_sysid
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_inpsource_sysid_nonascii (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1040, in test_expat_inpsou
rce_sysid_nonascii
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

======================================================================
FAIL: test_expat_text_file (__main__.ExpatReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):  File "E:\GitHub\cpython\lib\test\test_sax.py", line 847, in test_expat_text_fi
le
    self.assertEqual(result.getvalue(), xml_test_out)
AssertionError: b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'

----------------------------------------------------------------------
Ran 171 tests in 0.200s

FAILED (failures=10)
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "E:\GitHub\cpython\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1261, in <module>
    test_main()
  File "E:\GitHub\cpython\lib\test\test_sax.py", line 1258, in test_main
    XmlReaderTest)
  File "E:\GitHub\cpython\lib\test\support\__init__.py", line 1837, in run_unittest
    _run_suite(suite)
  File "E:\GitHub\cpython\lib\test\support\__init__.py", line 1812, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: multiple errors occurred
msg269636 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-01 01:05
If it were me, I might try to use a version of Git that doesn’t mess with the files. But if this becomes a common problem (esp with the move from Mercurial to Git), maybe we can add a “.gitattributes” file <https://git-scm.com/docs/gitattributes>. We aleady seem to have “.hgeol”.

I presume this is going to affect earlier versions too.
msg269638 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-07-01 01:36
I agree, but I wouldn't expect my line ending preferences to make tests fail. I think that with the move to GitHub already underway, it's not unreasonable to (at least consider to) add a .gitattributes file. Added Brett to nosy since he's responsible for PEP 512.

(I think this should be changed before the final announcement that the cpython repo is on GitHub, but there's no rush until then; although I'm certainly not going to complain if it's done before ;)
msg269664 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-01 12:38
Yes .gitattributes looks like the correct solution.  The problem is that most of the text files we *want* git to transform per platform, it's just a few that we don't.  Mercurial actually added support for this for us when we switched to it.
msg273652 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-08-25 14:38
Here are three patches for this:

- add_gitattributes_1.patch adds `.gitattributes`, with all the newline preferences in `.hgeol` (with the exception of the "native" rule at the end, which in git is the default newline preference).
- fix_newlines_1.patch fixes test_sax to properly handle CRLF (since *.xml files may have either LF or CRLF), as well as a single batch file which wasn't using CRLF line endings for some reason.
- test_random_warning_1.patch is somewhat unrelated, but it makes sure that failures in the `pickle.load` call no longer show ResourceWarnings on stderr.

The two tests which used to fail no longer do so. Maybe add_gitattributes_1.patch should be backported to 2.7, but probably not the other ones.
msg273657 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-08-25 15:24
Whoops, turns out I was using the wrong approach for binary files. Here come add_gitattributes_2.patch and fix_newlines_2.patch (thanks Zachary for pointing this out).
msg273668 - (view) Author: Jim Jewett (Jim.Jewett) (Python triager) Date: 2016-08-25 16:39
I think if the test is ensuring one line-ending type, then there should be another test ensuring the other... but I think it would be best if there were a single test file that had both types of line-endings, just to ensure that the code doesn't cheat by reading only the first line and assuming that the rest of the data is consistent.
msg273752 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-08-27 02:47
Emanuel: fix_newlines_2.patch seems to change the contents of batch from CRLF to LF newlines, undoing revision 640ccb924b5f. This seems like a step in the wrong direction. Notice $ means LF, and ^M$ means CRLF:

$ curl https://bugs.python.org/file44225/fix_newlines_2.patch | cat -A
. . .
diff --git a/Tools/unicode/genwincodecs.bat b/Tools/unicode/genwincodecs.bat$
. . .
-c:\python30\python genwincodec.py 720 > build/cp720.py^M$
. . .
+c:\python30\python genwincodec.py 720 > build/cp720.py$

However, the test_sax change may be sensible.

test_random_warning_1.patch looks okay.
msg273781 - (view) Author: Anilyka Barry (abarry) * (Python triager) Date: 2016-08-27 14:25
Martin: Indeed, seems like it's backwards for some reason. I'm not sure what happened when I regenerated the index; I removed the patches now anyway. I think the .gitattributes patch would be fine to go on its own.

It's my understanding that line endings don't matter for XML files, so it would probably be a better move to fix xml.sax instead of the test. I'll likely open a new issue if we decide to do that.
msg290085 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-24 11:42
In the master branch, .gitattributes now contains:
---
*.pck binary
Lib/test/cjkencodings/* binary
Lib/test/decimaltestdata/*.decTest binary
Lib/test/sndhdrdata/sndhdr.* binary
Lib/test/test_email/data/msg_26.txt binary
Lib/test/xmltestdata/* binary
Lib/venv/scripts/nt/* binary
Lib/test/coding20731.py binary
---

History of the file:
---
$ git log --follow .gitattributes  
commit 060d2d776a29341c079cce37220324f9775140ba
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Sun Mar 5 08:49:45 2017 +0900

    remove merge=union attribute for Misc/NEWS (GH-460)
    
    Github doesn't support it (ref. isaacs/github#487).  So it can't ease
    conflict on Github.
    
    Additionally, it can make trouble when cherry-pick. (ref. GH-212)

commit 2771304357607aa62801a67acc1e3c7c8ec489ce
Author: Benjamin Peterson <benjamin@python.org>
Date:   Wed Feb 22 22:38:48 2017 -0800

    mark various test data binary (#233)

commit 2c700af5a761f83f303cc7295b660ac31f7d4ed1
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Tue Feb 21 18:39:41 2017 +0900

    .gitattribute -> .gitattributes (GH-213)

commit 3f3d0364a91aa8b89e175a49f93ad04dd4676186
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Tue Feb 21 18:17:06 2017 +0900

    Reduce conflict on Misc/NEWS (GH-212)
    
    use "union" merge strategy for Misc/NEWS.
---

test_random and test_sax now pass on my Windows VM... but still fail on the "AMD64 Windows8 3.x" buildbot :-(
http://buildbot.python.org/all/builders/AMD64%20Windows8%203.x/builds/461/steps/test/logs/stdio

Is it possible that Git on this buildbot ignores .gitattributes? Maybe the Git version is too old?
msg290094 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-03-24 16:17
More likely it needs a fresh clone to fix up those files, unless they've changed since the attributes file was added.
msg290617 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-27 14:11
New changeset cf57fe13b44e3994096ae79f32c11475f333a94f by Victor Stinner in branch '3.5':
bpo-27425: Add .gitattributes, fix Windows tests (#844)
https://github.com/python/cpython/commit/cf57fe13b44e3994096ae79f32c11475f333a94f
msg295571 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-09 20:43
Wait, why is this issue still open? Tests now pass on Windows on all
branches. The issue can be closed, except if I missed something?
msg295663 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 19:58
New changeset 6b6e68776663c0dda04b6a36609297728da2ae9e by Zachary Ware in branch 'master':
bpo-27425: Be more explicit in .gitattributes (GH-840)
https://github.com/python/cpython/commit/6b6e68776663c0dda04b6a36609297728da2ae9e
msg295666 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 20:39
New changeset 964c261dc9a6a901f50d5596d88248bfc4251a55 by Zachary Ware in branch '3.6':
[3.6] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2083)
https://github.com/python/cpython/commit/964c261dc9a6a901f50d5596d88248bfc4251a55
msg295667 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 20:40
New changeset afa6a389606402779c5048a7211edbf2678cecff by Zachary Ware in branch '3.5':
[3.5] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2084)
https://github.com/python/cpython/commit/afa6a389606402779c5048a7211edbf2678cecff
msg295675 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-10 22:00
Zach, do you want to update Python 2.7 for your latest change as well?
msg295684 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-11 03:19
> Zach, do you want to update Python 2.7 for your latest change as well?

PR 2086 is in for that, but it's a huge diff due to line endings changing.  As noted on the PR, I want Benjamin to give his OK before I merge it.
msg295728 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-11 19:19
New changeset 5fe8ac69f99d58232a5234cfca78f05f7223a782 by Zachary Ware in branch '2.7':
[2.7] bpo-27425: Be more explicit in .gitattributes (GH-840) (GH-2086)
https://github.com/python/cpython/commit/5fe8ac69f99d58232a5234cfca78f05f7223a782
msg295729 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-11 19:23
It still might be nice to adjust some of the tests to not care about how line endings are checked in, but I think we're in a good enough place now with .gitattributes that we can close this issue.  Any other changes can be done in a new issue.
msg295762 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-12 09:36
I'm happy to see this "old" issue now fixed :-)
msg296519 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-21 05:21
Installed 3.6.2rc1 is showing the same errors for test_random and test_sax.  #30716.
msg296520 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-21 05:23
Why de we would 'want' /r/n on Windows?  What beside Notepad can't handle /n?
msg296534 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-21 09:27
I reproduce the bug after installing Python 3.6.2rc1 on Windows. I reopen the issue and set the priority to release blocker.
msg297465 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-07-01 01:48
So do we have a resolution or resolutions for this yet?  And is bpo-30716 truly a duplicate?  If so, let's use one or the other.
msg297469 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-07-01 04:40
There's nothing that needs to block the release as far as I'm concerned. The main fix was that I needed a fresh clone on my build machine to pick up the gitattribute changes.

The other bug is still open to enhance some tests and skip one in the release tree, but nothing release blocking. I don't see why this one is now either.
msg297478 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-01 07:15
There is still the problem with test_winreg freezing CommandPrompt, #30715.  For me, this is the worst test bug ever.
msg297490 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-07-01 18:02
OK, based on Steve's response, I am closing this again as it sounds like it should not be an issue going forward with releases and also not for dev builds as long as people do a fresh clone if necessary. Further discussion on hardening the tests can take place over on bpo-30716.
History
Date User Action Args
2017-07-01 18:02:10ned.deilysetstatus: open -> closed
priority: release blocker ->
resolution: fixed
messages: + msg297490
2017-07-01 07:15:06terry.reedysetmessages: + msg297478
2017-07-01 04:40:45steve.dowersetmessages: + msg297469
2017-07-01 01:48:25ned.deilysetmessages: + msg297465
2017-06-28 00:57:41vstinnerunlinkissue29530 dependencies
2017-06-21 09:27:03vstinnersetstatus: closed -> open
priority: normal -> release blocker

nosy: + ned.deily, larry, benjamin.peterson
messages: + msg296534

resolution: fixed -> (no value)
2017-06-21 05:23:21terry.reedysetmessages: + msg296520
2017-06-21 05:21:07terry.reedysetnosy: + terry.reedy
messages: + msg296519
2017-06-12 09:36:19vstinnersetmessages: + msg295762
2017-06-11 19:23:19zach.waresetstatus: open -> closed
resolution: fixed
messages: + msg295729

stage: patch review -> resolved
2017-06-11 19:19:42zach.waresetmessages: + msg295728
2017-06-11 03:19:17zach.waresetmessages: + msg295684
2017-06-10 22:00:10vstinnersetmessages: + msg295675
2017-06-10 20:40:11zach.waresetmessages: + msg295667
2017-06-10 20:39:31zach.waresetmessages: + msg295666
2017-06-10 20:38:24zach.waresetpull_requests: + pull_request2150
2017-06-10 20:19:44zach.waresetpull_requests: + pull_request2148
2017-06-10 20:10:16zach.waresetpull_requests: + pull_request2147
2017-06-10 19:58:45zach.waresetmessages: + msg295663
2017-06-09 20:43:32vstinnersetmessages: + msg295571
2017-06-09 20:23:34Mariattasetversions: + Python 3.7
2017-03-27 14:11:36vstinnersetmessages: + msg290617
2017-03-27 09:34:17vstinnersetpull_requests: + pull_request741
2017-03-27 05:35:24zach.waresetpull_requests: + pull_request737
2017-03-24 16:17:27steve.dowersetmessages: + msg290094
2017-03-24 11:42:13vstinnersetnosy: + vstinner
messages: + msg290085
2017-02-11 16:07:35zach.warelinkissue29530 dependencies
2016-08-27 14:25:07abarrysetmessages: + msg273781
2016-08-27 14:12:57abarrysetfiles: - fix_newlines_2.patch
2016-08-27 14:12:49abarrysetfiles: - fix_newlines_1.patch
2016-08-27 02:47:15martin.pantersetmessages: + msg273752
versions: + Python 3.5
2016-08-25 16:39:19Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg273668
2016-08-25 15:24:22abarrysetfiles: + fix_newlines_2.patch
2016-08-25 15:24:09abarrysetfiles: + add_gitattributes_2.patch

messages: + msg273657
2016-08-25 14:39:01abarrysetfiles: + test_random_warning_1.patch
2016-08-25 14:38:53abarrysetfiles: + fix_newlines_1.patch
2016-08-25 14:38:38abarrysetfiles: + add_gitattributes_1.patch
versions: - Python 3.5
messages: + msg273652

keywords: + patch
stage: patch review
2016-07-01 12:38:57r.david.murraysetnosy: + r.david.murray
messages: + msg269664
2016-07-01 01:36:05abarrysetnosy: + brett.cannon
messages: + msg269638
2016-07-01 01:05:35martin.pantersetversions: + Python 2.7, Python 3.5
nosy: + paul.moore, tim.golden, martin.panter, zach.ware, steve.dower

messages: + msg269636

components: + Windows
2016-06-30 17:16:29abarrycreate