classification
Title: Failing tests with installed 3.6.2rc1 on Win 10-64
Type: behavior Stage:
Components: Tests, Windows Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: martin.panter, ned.deily, paul.moore, steve.dower, terry.reedy, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-06-20 18:55 by terry.reedy, last changed 2017-06-21 20:41 by terry.reedy.

Messages (6)
msg296477 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-20 18:55
3.6.2rc1, 64 bit version, installed on my Win10 machine, has 5 failures not present in my 32-bit debug repository builds.  Auto Windows updates is on.  Results same after uninstall and re-install.

F:\dev\3x>py -3 -c "import sys; print(sys.version)"
3.6.2rc1 (heads/3.6:268e1fb, Jun 17 2017, 19:01:44) [MSC v.1900 64 bit (AMD64)]

test_codecencodings_iso2022  multiple failures
test_random  multiple failures
test_sax  multiple failures
test_regrtest  failed in test_pcbuild_rt, FileNotFoundError in subprocess
test_tools  2to3 it seems 

Are the failures unique to my machine?  At least some of the failures below don't seem that they should be.  How is the installer tested?

-----------------------
FAil test_incrementaldecoder test.test_codecencodings_iso2022.Test_ISO2022_KR)
has 4 failures like the following, where b'\n' and b'\r\n' are switched (both ways) between expected and actual.

AssertionError:
b'\xe[334 chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != 
b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'

b"\x1[143 chars]\x0f\r\n\x0eH?@2@{@N\x0f \x0e0m<vAX\x0f \x0e5%[631 chars]\r\n" != 
b"\x1[143 chars]\x0f\n\x0eH?@2@{@N\x0f \x0e0m<vAX\x0f \x0e5%@L[617 chars]f.\n"
---------------------

ERROR: test_bug_1727780 (test.test_random.MersenneTwister_TestBasicOps)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python36\lib\test\test_random.py", line 257, in test_bug_1727780
    r = pickle.load(f)
ModuleNotFoundError: No module named 'random\r'

Any 1 of the 3 test files produces this.
-------------------

test_sax has 10 line-ending switch failures like this
b'<?x[36 chars]1"?>\n<HTML xmlns:pp="http://www.isogen.com/pa[1508 chars]TML>' != 
b'<?x[36 chars]1"?>\r\n<HTML xmlns:pp="http://www.isogen.com/[1736 chars]TML>'
---------------------------

ERROR: test_pcbuild_rt (test.test_regrtest.ProgramsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python36\lib\test\test_regrtest.py", line 582, in test_pcbuild_rt
    self.run_batch(script, *rt_args, *self.regrtest_args, *self.tests)
  File "C:\Programs\Python36\lib\test\test_regrtest.py", line 557, in run_batch
    proc = self.run_command(args)
  File "C:\Programs\Python36\lib\test\test_regrtest.py", line 455, in run_command
    **kw)
  File "C:\Programs\Python36\lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Programs\Python36\lib\subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "C:\Programs\Python36\lib\subprocess.py", line 992, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
--------------------------------

I don't understand the test_tools output.  Some tests ran, but test_tools could not be found ;-).

F:\dev\3x>py -3 -m test -v test_tools
== CPython 3.6.2rc1 (heads/3.6:268e1fb, Jun 17 2017, 19:01:44) [MSC v.1900 64 bit (AMD64)]
== Windows-10-10.0.14393-SP0 little-endian
== hash algorithm: siphash24 64bit
== cwd: C:\Users\Terry\AppData\Local\Temp\test_python_14224
== CPU count: 12
== encodings: locale=cp1252, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_tools
test_alter_comments (test.test_tools.test_fixcid.Test) ... ok
test_directory (test.test_tools.test_fixcid.Test) ... ok
test_parse_strings (test.test_tools.test_fixcid.Test) ... ok
test_gprof (test.test_tools.test_gprof2html.Gprof2htmlTests) ... ok
test_POT_Creation_Date (test.test_tools.test_i18n.Test_pygettext)
Match the date format from xgettext for POT-Creation-Date ... ok
test_header (test.test_tools.test_i18n.Test_pygettext)
Make sure the required fields are in the header, according to: ... ok
test_checksum_fodder (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_dash_l (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_dash_s (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_dash_t (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_multiple_files (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_noargs (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_usage (test.test_tools.test_md5sum.MD5SumTests) ... ok
test_inverse_attribute_error (test.test_tools.test_pdeps.PdepsTests) ... ok
test_process_errors (test.test_tools.test_pdeps.PdepsTests) ... ok
test_empty_line (test.test_tools.test_pindent.PindentTests) ... ok
test_escaped_newline (test.test_tools.test_pindent.PindentTests) ... ok
test_multilevel (test.test_tools.test_pindent.PindentTests) ... ok
test_oneline (test.test_tools.test_pindent.PindentTests) ... ok
test_preserve_indents (test.test_tools.test_pindent.PindentTests) ... ok
test_selftest (test.test_tools.test_pindent.PindentTests) ... ok
test_statements (test.test_tools.test_pindent.PindentTests) ... ok
test_help (test.test_tools.test_reindent.ReindentTests) ... ok
test_noargs (test.test_tools.test_reindent.ReindentTests) ... ok
test_reindent_file_with_bad_encoding (test.test_tools.test_reindent.ReindentTests) ... ok
test_analyze_dxp_import (test.test_tools.test_sundry.TestSundryScripts) ... ok
test_sundry (test.test_tools.test_sundry.TestSundryScripts) ... RefactoringTool: Adding transformation: apply
RefactoringTool: Adding transformation: asserts
RefactoringTool: Adding transformation: basestring
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Adding transformation: dict
RefactoringTool: Adding transformation: except
RefactoringTool: Adding transformation: exec
RefactoringTool: Adding transformation: execfile
RefactoringTool: Adding transformation: exitfunc
RefactoringTool: Adding transformation: filter
RefactoringTool: Adding transformation: funcattrs
RefactoringTool: Adding transformation: future
RefactoringTool: Adding transformation: getcwdu
RefactoringTool: Adding transformation: has_key
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Adding transformation: import
RefactoringTool: Adding transformation: imports
RefactoringTool: Adding transformation: imports2
RefactoringTool: Adding transformation: input
RefactoringTool: Adding transformation: intern
RefactoringTool: Adding transformation: isinstance
RefactoringTool: Adding transformation: itertools
RefactoringTool: Adding transformation: itertools_imports
RefactoringTool: Adding transformation: long
RefactoringTool: Adding transformation: map
RefactoringTool: Adding transformation: metaclass
RefactoringTool: Adding transformation: methodattrs
RefactoringTool: Adding transformation: ne
RefactoringTool: Adding transformation: next
RefactoringTool: Adding transformation: nonzero
RefactoringTool: Adding transformation: numliterals
RefactoringTool: Adding transformation: operator
RefactoringTool: Adding transformation: paren
RefactoringTool: Adding transformation: print
RefactoringTool: Adding transformation: raise
RefactoringTool: Adding transformation: raw_input
RefactoringTool: Adding transformation: reduce
RefactoringTool: Adding transformation: reload
RefactoringTool: Adding transformation: renames
RefactoringTool: Adding transformation: repr
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Adding transformation: standarderror
RefactoringTool: Adding transformation: sys_exc
RefactoringTool: Adding transformation: throw
RefactoringTool: Adding transformation: tuple_params
RefactoringTool: Adding transformation: types
RefactoringTool: Adding transformation: unicode
RefactoringTool: Adding transformation: urllib
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Adding transformation: xrange
RefactoringTool: Adding transformation: xreadlines
RefactoringTool: Adding transformation: zip
RefactoringTool: Can't open test_tools: [Errno 2] No such file or directory: 'test_tools'
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't open test_tools: [Errno 2] No such file or directory: 'test_tools'
ERROR
test_sundry_windows (test.test_tools.test_sundry.TestSundryScripts) ... ok
test_files (test.test_tools.test_unparse.DirectoryTestCase) ... Testing C:\Programs\Python36\Lib\test\test_super.py
Testing C:\Programs\Python36\Lib\subprocess.py
Testing C:\Programs\Python36\Lib\string.py
Testing C:\Programs\Python36\Lib\mimetypes.py
Testing C:\Programs\Python36\Lib\test\test_dictcomps.py
Testing C:\Programs\Python36\Lib\site.py
Testing C:\Programs\Python36\Lib\test\pydoc_mod.py
Testing C:\Programs\Python36\Lib\stringprep.py
Testing C:\Programs\Python36\Lib\test\test_cgi.py
Testing C:\Programs\Python36\Lib\struct.py
ok
test_annotations (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_bytes (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_chained_comparisons (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_class_decorators (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_class_definition (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_del_statement (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_dict_comprehension (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_dict_unpacking_in_dict (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_elifs (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_for_else (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_fstrings (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_function_arguments (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_huge_float (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_imaginary_literals (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_integer_parens (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_lambda_parentheses (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_min_int (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_nonlocal (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_raise_from (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_relative_import (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_set_comprehension (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_set_literal (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_shifts (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_starred_assignment (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_try_except_finally (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_unary_parens (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_while_else (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_with_as (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_with_simple (test.test_tools.test_unparse.UnparseTestCase) ... ok
test_with_two_items (test.test_tools.test_unparse.UnparseTestCase) ... ok

======================================================================
ERROR: test_sundry (test.test_tools.test_sundry.TestSundryScripts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Programs\Python36\lib\test\test_tools\test_sundry.py", line 36, in test_sundry
    import_tool(name)
  File "C:\Programs\Python36\lib\test\test_tools\__init__.py", line 21, in import_tool
    return importlib.import_module(toolname)
  File "C:\Programs\Python36\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "C:\Programs\Python36\Tools\scripts\2to3.py", line 5, in <module>
    sys.exit(main("lib2to3.fixes"))
SystemExit: 1

----------------------------------------------------------------------
Ran 59 tests in 3.481s

FAILED (errors=1)
Warning -- logging._handlerList was modified by test_tools
  Before: (2752582445576, [<weakref at 0x00000280E2BE4F48; to '_StderrHandler' at 0x00000280E2BF11D0>, <weakref at 0x00000280E35A4688; to 'StreamHandler' at 0x00000280E36BEE10>], [<weakref at 0x00000280E2BE4F48; to '_StderrHandler' at 0x00000280E2BF11D0>])
  After:  (2752582445576, [<weakref at 0x00000280E2BE4F48; to '_StderrHandler' at 0x00000280E2BF11D0>, <weakref at 0x00000280E35A4688; to 'StreamHandler' at 0x00000280E36BEE10>], [<weakref at 0x00000280E2BE4F48; to '_StderrHandler' at 0x00000280E2BF11D0>, <weakref at 0x00000280E35A4688; to 'StreamHandler' at 0x00000280E36BEE10>])
test test_tools failed
msg296507 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-06-21 03:21
Sounds similar to Issue 27425. Did rc1 get built with mangled newlines or something?
msg296558 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-06-21 14:08
To clarify (because I never trust test output for this), your install has \n line endings everywhere instead of \r\n?
msg296565 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-06-21 16:38
So the pickle issues were because the .gitattributes file changed multiple times since the last release but the files were not modified, so git did not fix them in my build repo. This should cover random and 2to3.

Most of the stdlib is correct CRLF.

XML should be immune to LF/CRLF differences, so we should harden the test (or the library?) against those.

IMO, we should also harden the multibyte tests against newline differences. When they read the test data with 'rb' they can also .replace(b'\r\n', b'\n') - it's totally external to the tests.

test_pcbuild_rt should fail in an install (or better yet, be skipped - we don't ship rt.bat).
msg296566 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-06-21 16:52
For the next build, test_codecencodings_iso2022, test_random, test_sax and test_tools should all pass, as the .gitattributes file excludes those files from conversion. I'm still in favour of hardening the tests.

test_pcbuild_rc should be skipped. I don't think this test breaking in an install is a release blocking issue.

Up to Ned whether we want to fix these tests before 3.6.2. They may affect users who build from the sdist and do a repo-wide line ending conversion (or fork into another repo that does different line ending correction - I know at least two of those exist).
msg296583 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-21 20:41
Steve, what should I do to answer you line-ending question, or do you still need the info?

Victor Stinner reopened #27425, which was about the exact same line-ending failures.
History
Date User Action Args
2017-06-21 20:41:11terry.reedysetmessages: + msg296583
2017-06-21 16:52:59steve.dowersetnosy: + vstinner
2017-06-21 16:52:32steve.dowersetmessages: + msg296566
2017-06-21 16:42:30steve.dowersetpriority: release blocker -> normal
2017-06-21 16:38:19steve.dowersetmessages: + msg296565
2017-06-21 14:08:35steve.dowersetmessages: + msg296558
2017-06-21 03:21:50martin.pantersetnosy: + paul.moore, tim.golden, martin.panter, zach.ware, steve.dower
messages: + msg296507
components: + Tests, Windows
2017-06-20 18:55:28terry.reedycreate