This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: socketserver.ForkingMixIn.server_close() leaks zombie processes
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, vstinner
Priority: Keywords: patch

Created on 2017-08-09 00:40 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
forkingmixin_sleep.patch vstinner, 2017-08-10 11:30 review
Pull Requests
URL Status Linked Edit
PR 3057 merged vstinner, 2017-08-10 11:30
PR 5417 merged vstinner, 2018-01-29 08:36
Messages (13)
msg299959 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 00:40
It seems like test_socketserver leaks child processes on the "x86 Gentoo Refleaks 3.6" buildbot, but I'm unable to reproduce the bug.

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/49/steps/test/logs/stdio

0:48:11 load avg: 3.91 [154/405/2] test_socketserver failed -- running: test_decimal (630 sec)
Warning -- reap_children() reaped child process 1044
beginning 6 repetitions
123456
Warning -- reap_children() reaped child process 1104
Warning -- reap_children() reaped child process 1115
.Warning -- reap_children() reaped child process 1170
Warning -- reap_children() reaped child process 1175
Warning -- reap_children() reaped child process 1184
.Warning -- reap_children() reaped child process 1249
.Warning -- reap_children() reaped child process 1311
Warning -- reap_children() reaped child process 1316
...
test_socketserver leaked [1, 1, 1] memory blocks, sum=3

(...)

test_ForkingUnixStreamServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = /tmp/unix_socket.hqh5x95a
CLASS = <class 'test.test_socketserver.ForkingUnixStreamServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 17938
ok

test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = /tmp/unix_socket.gry6ulhp
CLASS = <class 'test.test_socketserver.ForkingUnixDatagramServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 18212
ok

test_ForkingUDPServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = ('127.0.0.1', 43415)
CLASS = <class 'socketserver.ForkingUDPServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 18281
ok
test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest)
msg299960 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 00:41
Ah, the master branch is also affected. Example:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/47/steps/test/logs/stdio

1:35:05 load avg: 4.64 [212/406] test_socketserver passed -- running: test_subprocess (370 sec)
beginning 6 repetitions
123456
Warning -- reap_children() reaped child process 4641
.Warning -- reap_children() reaped child process 4708
..Warning -- reap_children() reaped child process 4832
.Warning -- reap_children() reaped child process 4916
Warning -- reap_children() reaped child process 4921
..
msg300074 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 11:30
The problem is that socketserver.ForkinMixin doesn't wait until all children completes. It's only calls os.waitpid() in non-blocking module (using os.WNOHANG) after each loop iteration. If a child process completes after the last call to ForkingMixIn.collect_children(), the server leaks zombie processes.

The server must wait until all children completes. Attached PR implements that.

The bug was be reproduced with the attached forkingmixin_sleep.patch.

haypo@selma$ ./python -m test -v -u all test_socketserver --fail-env-changed -m '*Fork*'
(...)
Warning -- reap_children() reaped child process 17093
Warning -- reap_children() reaped child process 17094
(...)
msg300084 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 13:28
New changeset aa8ec34ad52bb3b274ce91169e1bc4a598655049 by Victor Stinner in branch 'master':
bpo-31151: Add socketserver.ForkingMixIn.server_close() (#3057)
https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049
msg300106 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 15:38
bpo-31010 has been marked as a duplicate of this issue.
msg302013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 00:39
I created a thread on the python-dev mailing list to discuss this issue:
https://mail.python.org/pipermail/python-dev/2017-August/148826.html
msg302015 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 00:43
I tag this issue as release blocker as a remainder that we have to agree how to handle threads/processes before Python 3.7 feature freeze.

See also bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close().
msg303041 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-26 13:50
I proposed the bpo-31593 to fix the issue differently in Python 3.6 (and maybe also Python 2.7).
msg311015 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-28 22:19
What's the status of this?  Feature freeze is here!
msg311022 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-28 22:36
> What's the status of this?  Feature freeze is here!

This issue is very to https://bugs.python.org/issue31233#msg311021 as it's status:
* The commit aa8ec34ad52bb3b274ce91169e1bc4a598655049 is backward incompatible and it's not documented
* There is no option to opt-in for the previous behaviour (exit without waiting for child processes)

For beta1, IMHO it's ok to keep the current status. But I would like to see this issue fixed before 3.7 final. Either revert my change, or do something else.
msg311093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-29 11:10
New changeset db8189bb8db609ca3993dec31fb95fdbe134469a by Victor Stinner in branch 'master':
bpo-31233, bpo-31151: Document socketserver changes (#5417)
https://github.com/python/cpython/commit/db8189bb8db609ca3993dec31fb95fdbe134469a
msg316834 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-16 16:28
I created bpo-33540: "socketserver: Add an opt-in option to get Python 3.6 behaviour on server_close()". I set the release blocker priority on this new issue.
msg318116 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 22:01
The bug has been fixed in Python 3.7.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75334
2018-05-29 22:01:13vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg318116

stage: patch review -> resolved
2018-05-16 16:28:12vstinnersetpriority: deferred blocker ->

messages: + msg316834
2018-01-29 11:10:24vstinnersetmessages: + msg311093
2018-01-29 08:36:21vstinnersetstage: patch review
pull_requests: + pull_request5252
2018-01-28 22:36:32vstinnersetmessages: + msg311022
2018-01-28 22:19:08ned.deilysetmessages: + msg311015
2017-09-26 13:50:26vstinnersetmessages: + msg303041
2017-09-18 19:40:13ned.deilysetpriority: release blocker -> deferred blocker
2017-09-13 00:43:15vstinnersetpriority: normal -> release blocker
nosy: + ned.deily
messages: + msg302015

2017-09-13 00:39:25vstinnersetmessages: + msg302013
2017-08-18 23:25:10vstinnersettitle: test_socketserver: Warning -- reap_children() reaped child process -> socketserver.ForkingMixIn.server_close() leaks zombie processes
2017-08-10 15:38:47vstinnersetmessages: + msg300106
2017-08-10 15:38:19vstinnerlinkissue31010 superseder
2017-08-10 13:28:19vstinnersetmessages: + msg300084
2017-08-10 11:30:30vstinnersetpull_requests: + pull_request3092
2017-08-10 11:30:13vstinnersetfiles: + forkingmixin_sleep.patch
keywords: + patch
messages: + msg300074
2017-08-09 00:41:38vstinnersettitle: [3.6] test_socketserver: Warning -- reap_children() reaped child process -> test_socketserver: Warning -- reap_children() reaped child process
messages: + msg299960
versions: + Python 3.7
2017-08-09 00:40:39vstinnercreate