The test_lib2to3.test_parser.test_all_project_files() test fails and produces an annoying output. See the issue #13125

Example of a buildbot:

0:11:09 [398/404] test_lib2to3 passed (39 sec)
--- /root/buildarea/3.x.angelico-debian-amd64/build/Lib/lib2to3/tests/data/bom.py	2017-02-11 12:20:43.532000000 +1100
+++ @	2017-04-20 22:05:37.157911808 +1000
@@ -1,2 +1,2 @@
-# coding: utf-8
+# coding: utf-8
 print "BOM BOOM!"

Example by running: ./python -m test -v test_lib2to3
---
test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency) ... /home/haypo/prog/python/master/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/haypo/prog/python/master/Lib/lib2to3/main.py (bad input: type=22, value='=', context=('', (130, 38)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
/home/haypo/prog/python/master/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/haypo/prog/python/master/Lib/lib2to3/tests/pytree_idempotency.py (bad input: type=22, value='=', context=('', (49, 33)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
--- /home/haypo/prog/python/master/Lib/lib2to3/tests/data/bom.py	2017-02-10 23:10:03.392778645 +0100
+++ @	2017-04-20 14:32:49.921613096 +0200
@@ -1,2 +1,2 @@
-# coding: utf-8
+# coding: utf-8
 print "BOM BOOM!"
expected failure
---


The test fails to parse the following code:
---
from __future__ import print_function
import sys
print("WARNING", file=sys.stderr)
---

whereas lib2to3 is able to parse the code. It seems like the code uses a lib2to3.tests.support.driver object. Maybe this object lacks the fixer which parses the __future__ imports?


The minimum fix would be to make the test quiet: remove the warning, don't log anything.

The best would be fix the test.

I added a PR to fix these two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test with the following changes to the test:

1. Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

2. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.

New changeset 14e976e00e65bf343ba0fca016c3c9132a843daf by Benjamin Peterson (Eric Appelt) in branch 'master':
closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
https://github.com/python/cpython/commit/14e976e00e65bf343ba0fca016c3c9132a843daf

Failure on AMD64 Windows8.1 Non-Debug 3.x:

http://buildbot.python.org/all/#/builders/12/builds/585

======================================================================
ERROR: test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release\build\lib\lib2to3\tests\test_parser.py", line 432, in test_all_project_files
    x = diff(filepath, new, encoding=encoding)
  File "D:\buildarea\3.x.ware-win81-release\build\lib\lib2to3\tests\test_parser.py", line 488, in diff
    return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 304, in call
    with Popen(*popenargs, **kwargs) as p:
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 756, in __init__
    restore_signals, start_new_session)
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 1160, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Different failure on AMD64 Windows10 3.x:

======================================================================
FAIL: test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\lib2to3\tests\test_parser.py", line 434, in test_all_project_files
    self.fail("Idempotency failed: %s" % filepath)
AssertionError: Idempotency failed: D:\buildarea\3.x.bolen-windows10\build\lib\lib2to3\btm_matcher.py

x86 Windows7 3.x:

http://buildbot.python.org/all/#/builders/58/builds/563

======================================================================
FAIL: test_nonexisting_with_pipes (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_subprocess.py", line 1194, in test_nonexisting_with_pipes
    self.assertEqual(stderr, "")
AssertionError: 'Traceback (most recent call last):\n  Fil[1129 chars]le\n' != ''
Diff is 1153 characters long. Set self.maxDiff to None to see it.

1e17d4aaff5c7ca972bab437949d2bb51c5b30f7. I don't think you can pin test_subprocess on this change, though.

New changeset a23a2c555c4187f349276fe2f2ceffa953d0afe9 by Benjamin Peterson in branch '3.6':
[3.6] closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242) (GH-5443)
https://github.com/python/cpython/commit/a23a2c555c4187f349276fe2f2ceffa953d0afe9