➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Enhance support.reap_children()
Type: Stage: resolved
Components: Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Felt, vstinner
Priority: normal Keywords:

Created on 2017-08-09 12:50 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3036 merged vstinner, 2017-08-09 12:50
PR 3040 closed vstinner, 2017-08-09 13:46
PR 3043 merged vstinner, 2017-08-09 15:23
PR 3044 merged vstinner, 2017-08-09 15:27
PR 3045 merged vstinner, 2017-08-09 15:41
PR 3046 closed vstinner, 2017-08-09 16:03
PR 3056 merged vstinner, 2017-08-10 10:47
PR 3060 merged vstinner, 2017-08-10 13:41
PR 3063 merged vstinner, 2017-08-10 14:09
PR 19263 merged vstinner, 2020-03-31 20:53
PR 19308 closed Michael.Felt, 2020-04-02 17:43
PR 19312 closed vstinner, 2020-04-02 20:08
Messages (23)
msg299996 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:50 
Attached PR enhances the support.reap_children() function:

* reap_children() now sets environment_altered to True to detect bugs using python3 -m test --fail-env-changed
* Replace bare "except:" with "except OSError:" in reap_children()
* Write an unit test for reap_children() using a timeout of 60 seconds
msg299997 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:51 
The GCC job of Travis CI failed with ENV_CHANGED:

Test wait() behavior when waitpid returns WIFSTOPPED; issue29335. ...
Warning -- reap_children() reaped child process 19839
ok

I tested and... WOW! When run in a loop, this test leaks 100 MB per second. It creates a lot of processes.
msg299998 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:54 
> The GCC job of Travis CI failed with ENV_CHANGED:

Ooops, in fact it was a macOS job:
https://travis-ci.org/python/cpython/jobs/262606830


The GCC job failed with much more errors:
https://travis-ci.org/python/cpython/jobs/262606831

---
0:00:02 load avg: 121.34 [  7/403] test_unittest
Warning -- reap_children() reaped child process 11088
Warning -- reap_children() reaped child process 11089
Warning -- reap_children() reaped child process 11090
Warning -- reap_children() reaped child process 11091

0:14:22 load avg: 136.67 [282/403/1] test_select
Warning -- reap_children() reaped child process 14686

0:16:00 load avg: 110.41 [297/403/1] test_socketserver
Warning -- reap_children() reaped child process 15483
Warning -- reap_children() reaped child process 15492
Warning -- reap_children() reaped child process 15499
Warning -- reap_children() reaped child process 15508

0:18:36 load avg: 105.94 [333/403/1] test_thread
Warning -- reap_children() reaped child process 20670
---

For test_socketserver, see bpo-31151.
msg300000 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 13:20 
A reap_children() warning was fixed in test_thread: bpo-31150. It seems like the commit 88eee44a91c991dbbdf284fa220e2928b5de105c was not enough to fix all warnings.
msg300020 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:44 
New changeset 4baca1b0f7325032598cd38e7ceffc79b616d255 by Victor Stinner in branch 'master':
bpo-31160: Fix test_builtin for zombie process (#3043)
https://github.com/python/cpython/commit/4baca1b0f7325032598cd38e7ceffc79b616d255
msg300021 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:44 
New changeset e3510d74aacc477c30f42f2b941d69689bbc478e by Victor Stinner in branch 'master':
bpo-31160: regrtest now reaps child processes (#3044)
https://github.com/python/cpython/commit/e3510d74aacc477c30f42f2b941d69689bbc478e
msg300023 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:59 
New changeset da5e9301877346942fa4279612750d6a09e05153 by Victor Stinner in branch 'master':
bpo-31160: Fix test_random for zombie process (#3045)
https://github.com/python/cpython/commit/da5e9301877346942fa4279612750d6a09e05153
msg300024 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 16:19 
> New changeset 4baca1b0f7325032598cd38e7ceffc79b616d255 by Victor Stinner in branch 'master':
> bpo-31160: Fix test_builtin for zombie process (#3043)

This change introduced a regression:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1159/steps/test/logs/stdio

======================================================================
FAIL: test_input_no_stdout_fileno (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_builtin.py", line 1624, in test_input_no_stdout_fileno
    lines = self.run_child(child, b"quux\r")
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_builtin.py", line 1573, in run_child
    self.assertEqual(status, 0)
AssertionError: 1 != 0
msg300072 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 10:41 
bpo-31173 fixed a leaked child process in test_subprocess.
msg300073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 11:05 
New changeset 6c8c2943d996b59a48d331f61f22cbe72933910e by Victor Stinner in branch 'master':
bpo-31160: test_tempfile: Fix reap_children() warning (#3056)
https://github.com/python/cpython/commit/6c8c2943d996b59a48d331f61f22cbe72933910e
msg300086 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 13:29 
bpo-31151 fixed test_socketserver.
msg300088 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 14:02 
New changeset 719a15b32587de6c4add4385ee3f84a20711340f by Victor Stinner in branch '3.6':
[3.6] bpo-31160: Backport reap_children() fixes from master to 3.6 (#3060)
https://github.com/python/cpython/commit/719a15b32587de6c4add4385ee3f84a20711340f
msg300093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 14:45 
New changeset 1247e2cda514d7a73187e0b53ec8c35d87a34a84 by Victor Stinner in branch '2.7':
[2.7] bpo-31160: Backport reap_children fixes from master to 2.7 (#3063)
https://github.com/python/cpython/commit/1247e2cda514d7a73187e0b53ec8c35d87a34a84
msg300147 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-11 00:12 
I pushed the most important change: reap_children() now makes tests fail with ENV_CHANGED on warning, so I close the issue.
msg365430 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-31 22:27 
New changeset 16d75675d2ad2454f6dfbf333c94e6237df36018 by Victor Stinner in branch 'master':
bpo-31160: Fix race condition in test_os.PtyTests (GH-19263)
https://github.com/python/cpython/commit/16d75675d2ad2454f6dfbf333c94e6237df36018
msg365474 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 14:31 
With PR19263 The AIX bots are now red.

======================================================================
ERROR: test_input_no_stdout_fileno (test.test_builtin.PtyTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_builtin.py", line 1952, in test_input_no_stdout_fileno
    lines = self.run_child(child, b"quux\r")
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_builtin.py", line 1898, in run_child
    support.wait_process(pid, exitcode=0)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/support/__init__.py", line 3432, in wait_process
    os.kill(pid, signal.SIGKILL)
NameError: name 'signal' is not defined
----------------------------------------------------------------------
Ran 101 tests in 30.348s
FAILED (errors=1, skipped=7)
1 test failed again:
    test_builtin

+++++++++++++++
The Buildbot has detected a failed build on builder PPC64 AIX 3.x while building python/cpython.
Full details are available at:
    https://buildbot.python.org/all/#builders/227/builds/565

Buildbot URL: https://buildbot.python.org/all/

Worker for this Build: edelsohn-aix-ppc64
Worker for this Build: aixtools-aix-power6
msg365475 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 14:41 
> With PR19263 The AIX bots are now red.

I know, I saw and I already pushed fixes.

> NameError: name 'signal' is not defined

Fixed by commit afeaea2d6e346f627b24cc9e84e2986a7266a70e.

> 1 test failed again: test_builtin

Fixed by commit 16d75675d2ad2454f6dfbf333c94e6237df36018.
msg365476 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 14:49 
Ah - great. Sorry for the noise then.
msg365477 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 14:49 
> Ah - great. Sorry for the noise then.

It's not noise, it is useful :-)
msg365478 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 15:00 
I think something is not yet what it needs to be:

the bots both finish test with:

test_zip_pickle (test.test_builtin.BuiltinTest) ... ok
Timeout (0:15:00)!
Thread 0x00000001 (most recent call first):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 3435 in wait_process
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_builtin.py", line 1898 in run_child
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_builtin.py", line 1952 in test_input_no_stdout_fileno
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 616 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 659 in run
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 719 in __call__
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/runner.py", line 176 in run
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 2079 in _run_suite
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 2201 in run_unittest
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 209 in _test_module
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 153 in _runtest
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 318 in rerun_failed_tests
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 691 in _main
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 634 in main
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 712 in main
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/runpy.py", line 87 in _run_code
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/runpy.py", line 197 in _run_module_as_main
make: 1254-004 The error code from the last command is 1.
Stop.
program finished with exit code 2
elapsedTime=3501.292487
test_input_no_stdout_fileno (test.test_builtin.PtyTests) ... 

And the bot status is still FAIL (aka red): failed test (failure) uploading test-results.xml (failure)
msg365479 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 15:03 
> I think something is not yet what it needs to be: (...)

https://buildbot.python.org/all/#/builders/227/builds/571 build failed but it has my commit 16d75675d2ad2454f6dfbf333c94e6237df36018. Ok, something failed.

Please open a new issue. This one is closed.
msg365512 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 21:42 
> Please open a new issue. This one is closed.

Pablo Galindo opened bpo-40140, let's use this one.
msg365517 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 22:03 
> Pablo Galindo opened bpo-40140, let's use this one.

Note: Oops, Batuhan created it, Pablo only commented.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75343
2020-04-02 20:08:02vstinnersetpull_requests: + pull_request18675
2020-04-02 17:43:42Michael.Feltsetpull_requests: + pull_request18670
2020-04-01 22:03:51vstinnersetmessages: + msg365517
2020-04-01 21:42:59vstinnersetmessages: + msg365512
2020-04-01 15:03:24vstinnersetmessages: + msg365479
2020-04-01 15:00:19Michael.Feltsetmessages: + msg365478
2020-04-01 14:49:59vstinnersetmessages: + msg365477
2020-04-01 14:49:15Michael.Feltsetmessages: + msg365476
2020-04-01 14:41:24vstinnersetmessages: + msg365475
2020-04-01 14:31:10Michael.Feltsetnosy: + Michael.Felt
messages: + msg365474
2020-03-31 22:27:23vstinnersetmessages: + msg365430
2020-03-31 20:53:07vstinnersetpull_requests: + pull_request18619
2017-08-11 00:12:50vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg300147

stage: resolved
2017-08-10 14:45:42vstinnersetmessages: + msg300093
2017-08-10 14:09:58vstinnersetpull_requests: + pull_request3097
2017-08-10 14:02:02vstinnersetmessages: + msg300088
2017-08-10 13:41:11vstinnersetpull_requests: + pull_request3095
2017-08-10 13:29:53vstinnersetmessages: + msg300086
2017-08-10 11:05:09vstinnersetmessages: + msg300073
2017-08-10 10:47:03vstinnersetpull_requests: + pull_request3091
2017-08-10 10:41:51vstinnersetmessages: + msg300072
2017-08-09 16:19:41vstinnersetmessages: + msg300024
2017-08-09 16:03:40vstinnersetpull_requests: + pull_request3080
2017-08-09 15:59:10vstinnersetmessages: + msg300023
2017-08-09 15:44:35vstinnersetmessages: + msg300021
2017-08-09 15:44:21vstinnersetmessages: + msg300020
2017-08-09 15:41:57vstinnersetpull_requests: + pull_request3079
2017-08-09 15:27:09vstinnersetpull_requests: + pull_request3078
2017-08-09 15:23:08vstinnersetpull_requests: + pull_request3077
2017-08-09 13:46:22vstinnersetpull_requests: + pull_request3074
2017-08-09 13:20:00vstinnersetmessages: + msg300000
2017-08-09 12:54:50vstinnersetmessages: + msg299998
2017-08-09 12:51:10vstinnersetmessages: + msg299997
2017-08-09 12:50:03vstinnercreate