classification
Title: test_context and test_asyncio crash on Windows 7
Type: Stage: resolved
Components: asyncio Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, db3l, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2018-01-23 22:07 by vstinner, last changed 2018-01-24 09:50 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5292 merged yselivanov, 2018-01-24 02:59
Messages (11)
msg310528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-23 22:07
http://buildbot.python.org/all/#/builders/58/builds/498

0:36:25 [252/414/1] test_context crashed (Exit code 3) -- running: test_largefile (239 sec)
Windows fatal exception: code 0xc000001d

Current thread 0x00000944 (most recent call first):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 321 in ctx1_fun
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 339 in test_context_copy_1
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1861 in _run_suite
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1951 in run_unittest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main
Fatal Python error: Illegal instruction

Current thread 0x00000944 (most recent call first):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 321 in ctx1_fun
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_context.py", line 339 in test_context_copy_1
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1861 in _run_suite
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1951 in run_unittest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main


0:46:50 [306/414/2] test_asyncio crashed (Exit code 3) -- running: test_io (487 sec)
Windows fatal exception: code 0xc000001d

Current thread 0x00000d24 (most recent call first):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2191 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2289 in _step
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\events.py", line 88 in _run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 1529 in _run_once
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 391 in run_forever
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 423 in run_until_complete
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2199 in test_context_1
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1861 in _run_suite
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1951 in run_unittest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main
Fatal Python error: Illegal instruction

Current thread 0x00000d24 (most recent call first):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2191 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2289 in _step
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\events.py", line 88 in _run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 1529 in _run_once
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 391 in run_forever
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\asyncio\base_events.py", line 423 in run_until_complete
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_asyncio\test_tasks.py", line 2199 in test_context_1
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 615 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\case.py", line 663 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 122 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\suite.py", line 84 in __call__
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\unittest\runner.py", line 176 in run
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1861 in _run_suite
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\support\__init__.py", line 1951 in run_unittest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 175 in test_runner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 176 in runtest_inner
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest.py", line 130 in runtest
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\runtest_mp.py", line 67 in run_tests_slave
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 517 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 510 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\libregrtest\main.py", line 585 in main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 46 in _main
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\regrtest.py", line 50 in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 85 in _run_code
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\runpy.py", line 193 in _run_module_as_main

Note: the commit 0bad4d63c654d93e1f32ff35026405a3987db5ca doesn't fix these crashes. (The running build 399 hash this fix, and test_context does still crash.)
msg310529 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-23 22:11
test_asyncio started to crash at build 491:

http://buildbot.python.org/all/#/builders/58/builds/491

This build contains the commit f23746a934177c48eff754411aba54c31d6be2f0: "bpo-32436: Implement PEP 567". I'm not 100% sure that it's the cause of the regression. Since test_context added by this commit also crash, it really smells this the cause of the regression.
msg310530 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-23 22:13
It's not a regression, the newly added code doesn't work on some Windows buildbot... looking at it.
msg310531 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-23 22:18
Some hints. Compiler warnings:

..\Python\hamt.c(625): warning C4018: '<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]
..\Python\hamt.c(733): warning C4018: '<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]
..\Python\hamt.c(939): warning C4018: '<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]
..\Python\hamt.c(1122): warning C4018: '<': signed/unsigned mismatch [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]
msg310532 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-23 22:19
No, these are trivial uint32_t/Py_SIZE: `for (uint32_t i = val_idx + 1; i < Py_SIZE(o); i++)`; it's something else.  I suspect popcount instruction...
msg310538 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-24 00:22
I suspect this is a compiler bug.  I can't reproduce it on my windows 7 virtual machine and on AppVeyor.

It's not related to HAMT, btw, as there is a multitude of HAMT-specific tests that all pass.  The crash is specifically in context.c, and the code there is pretty trivial.
msg310541 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 00:36
> I suspect this is a compiler bug.

It may be interesting to add the compiler version to test.pythoninfo. It would be simpler to compare Windows buildbots to identify a bug.

Currently, the only info is:

"sys.version: 3.7.0a4+ (heads/master:9d411c1, Jan 23 2018, 15:13:42) [MSC v.1900 32 bit (Intel)]"

Some environment variables are also dumped:

os.environ[VS100COMNTOOLS]: C:\Program Files\Microsoft Visual Studio 10.0\Common7\Tools\
os.environ[VS140COMNTOOLS]: D:\Program Files\Microsoft Visual Studio 14.0\Common7\Tools\
os.environ[VS90COMNTOOLS]: D:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools\

In the compile step, I see lines like:

"Microsoft (R) Build Engine version 14.0.24730.2"

I recall also issues with outdated ucrtbase DLLs. Maybe we can also dump the version of such DLL?
msg310552 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-24 03:01
In an off-list conversation with David, it was confirmed that the buildbot in question runs on an old hardware without SSE4.2 support.

I decided to simply stop using native popcount instructions as there's no detectable performance difference when using them vs using the fallback code.

See https://github.com/python/cpython/pull/5292 for details.
msg310554 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-24 03:17
New changeset b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab by Yury Selivanov in branch 'master':
bpo-32436: Don't use native popcount() (also fixes bpo-32641) (#5292)
https://github.com/python/cpython/commit/b7a80d543e1e94475ab9c8214f7a9eab4e63c9ab
msg310555 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-01-24 03:17
I've pushed a commit that should fix the buildbot. Please reopen if it doesn't.
msg310578 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-24 09:50
> I've pushed a commit that should fix the buildbot. Please reopen if it doesn't.

test_context and test_asyncio do'nt crash anymore on this buildbot, thanks Yury.

test_asyncio still fail on this buildbot, I created a new issue: bpo-32645, test_asyncio: TLS tests fail on "x86 Windows7" buildbot.
History
Date User Action Args
2018-01-24 09:50:40vstinnersetmessages: + msg310578
2018-01-24 03:17:48yselivanovsetmessages: + msg310555
2018-01-24 03:17:07yselivanovsetstatus: open -> closed
resolution: fixed
messages: + msg310554

stage: resolved
2018-01-24 03:01:48yselivanovsetmessages: + msg310552
stage: patch review -> (no value)
2018-01-24 02:59:34yselivanovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request5139
2018-01-24 00:36:22vstinnersetmessages: + msg310541
2018-01-24 00:23:36vstinnersetnosy: + db3l
2018-01-24 00:22:11yselivanovsetmessages: + msg310538
2018-01-23 22:19:28yselivanovsetmessages: + msg310532
2018-01-23 22:18:22vstinnersetmessages: + msg310531
2018-01-23 22:13:28yselivanovsetmessages: + msg310530
2018-01-23 22:11:40vstinnersetmessages: + msg310529
2018-01-23 22:07:58vstinnercreate