classification
Title: "AMD64 OpenIndiana 3.x" buildbot: assertion failed in PyObject_Call() in test_subprocess.test_preexec()
Type: Stage:
Components: Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: python-dev, vstinner
Priority: normal Keywords: patch

Created on 2014-08-27 13:11 by vstinner, last changed 2014-10-05 15:31 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
fork_exec.patch vstinner, 2014-09-04 23:11 review
Messages (8)
msg225984 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 13:11
I ran test_subprocess.test_preexec() 2000 times: I'm unable to reproduce the issue on my OpenIndiana VM, nor on Linux. I used this command:
gdb -args ./python -m test -F -m test_preexec test_subprocess 

The command on the buildbot is:
./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600 -j4

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8549/steps/test/logs/stdio

[302/390/1] test_subprocess
Assertion failed: (result != NULL && !PyErr_Occurred()) || (result == NULL && PyErr_Occurred()), file Objects/abstract.c, line 2091
Fatal Python error: Aborted

Current thread 0x0000000000000001 (most recent call first):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/subprocess.py", line 1396 in _execute_child
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/subprocess.py", line 865 in __init__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_subprocess.py", line 1303 in test_preexec
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/case.py", line 577 in run
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/case.py", line 625 in __call__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 125 in run
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 87 in __call__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 125 in run
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/suite.py", line 87 in __call__
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/unittest/runner.py", line 168 in run
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support/__init__.py", line 1750 in _run_suite
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/support/__init__.py", line 1784 in run_unittest
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_subprocess.py", line 2470 in test_main
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1280 in runtest_inner
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 967 in runtest
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 532 in main
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1564 in main_in_temp_cwd
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1589 in <module>
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/runpy.py", line 85 in _run_code
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/runpy.py", line 170 in _run_module_as_main
msg225985 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-27 13:27
The error may be close to the call to the gc module: gc.isenabled(), gc.disable() or gc.enable(). The error is maybe an exception raised before the call to PyObject_Call(), because the called gc functions are very simple.
msg226251 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-09-01 23:28
Oh, a lot of tests fail with MemoryError on this buildbot. It may explain this issue.

For example, test_json crashed with SIGSEGV in this build, probably because of an unhandled MemoryError exception:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8557/steps/test/logs/stdio
msg226388 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-09-04 23:11
I reproduced the crash with pyfailmalloc. The problem is that _posixsubprocess.fork_exec() calls gc.enable() on error when preexec_fn is set and the garbage collector was disabled (by fork_exec). Calling a function with an exception set is forbidden.

Attached patch fixes this issue, but fix also error handling in fork_exec() and add some unit tests.
msg226389 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-09-04 23:16
New changeset 16e3d240456f by Victor Stinner in branch 'default':
Issue #22290: PyObject_Call() now fails with an assertion error when called
http://hg.python.org/cpython/rev/16e3d240456f
msg226391 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-09-04 23:28
> For example, test_json crashed with SIGSEGV in this build, probably because of an unhandled MemoryError exception:

I created an issue for this bug: issue #22338. I proposed a fix.
msg228581 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-10-05 15:30
New changeset 0455cbfd7ae6 by Victor Stinner in branch '3.4':
Issue #22290: Fix error handling in the _posixsubprocess module.
https://hg.python.org/cpython/rev/0455cbfd7ae6

New changeset a74c73477a64 by Victor Stinner in branch 'default':
(Merge 3.4) Issue #22290: Fix error handling in the _posixsubprocess module.
https://hg.python.org/cpython/rev/a74c73477a64
msg228582 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-10-05 15:31
I hope that my fix will be enough to fix the issue. Since it was only seen once, I prefer to close the issue.

By the way, it would be nice to give more memory to this buildbot!
History
Date User Action Args
2014-10-05 15:31:54vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg228582
2014-10-05 15:30:37python-devsetmessages: + msg228581
2014-09-04 23:28:19vstinnersetmessages: + msg226391
2014-09-04 23:16:43python-devsetnosy: + python-dev
messages: + msg226389
2014-09-04 23:11:36vstinnersetfiles: + fork_exec.patch
keywords: + patch
messages: + msg226388
2014-09-01 23:28:57vstinnersetmessages: + msg226251
2014-08-27 13:27:29vstinnersetmessages: + msg225985
2014-08-27 13:11:23vstinnercreate