classification
Title: socketserver.ForkingMixIn.server_close() leaks zombie processes
Type: resource usage Stage:
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: haypo, ned.deily
Priority: deferred blocker Keywords: patch

Created on 2017-08-09 00:40 by haypo, last changed 2017-09-18 19:40 by ned.deily.

Files
File name Uploaded Description Edit
forkingmixin_sleep.patch haypo, 2017-08-10 11:30 review
Pull Requests
URL Status Linked Edit
PR 3057 merged haypo, 2017-08-10 11:30
Messages (7)
msg299959 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-09 00:40
It seems like test_socketserver leaks child processes on the "x86 Gentoo Refleaks 3.6" buildbot, but I'm unable to reproduce the bug.

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/49/steps/test/logs/stdio

0:48:11 load avg: 3.91 [154/405/2] test_socketserver failed -- running: test_decimal (630 sec)
Warning -- reap_children() reaped child process 1044
beginning 6 repetitions
123456
Warning -- reap_children() reaped child process 1104
Warning -- reap_children() reaped child process 1115
.Warning -- reap_children() reaped child process 1170
Warning -- reap_children() reaped child process 1175
Warning -- reap_children() reaped child process 1184
.Warning -- reap_children() reaped child process 1249
.Warning -- reap_children() reaped child process 1311
Warning -- reap_children() reaped child process 1316
...
test_socketserver leaked [1, 1, 1] memory blocks, sum=3

(...)

test_ForkingUnixStreamServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = /tmp/unix_socket.hqh5x95a
CLASS = <class 'test.test_socketserver.ForkingUnixStreamServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 17938
ok

test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = /tmp/unix_socket.gry6ulhp
CLASS = <class 'test.test_socketserver.ForkingUnixDatagramServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 18212
ok

test_ForkingUDPServer (test.test_socketserver.SocketServerTest) ... creating server
ADDR = ('127.0.0.1', 43415)
CLASS = <class 'socketserver.ForkingUDPServer'>
server running
test client 0
test client 1
test client 2
waiting for server
done
Warning -- reap_children() reaped child process 18281
ok
test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest)
msg299960 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-09 00:41
Ah, the master branch is also affected. Example:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/47/steps/test/logs/stdio

1:35:05 load avg: 4.64 [212/406] test_socketserver passed -- running: test_subprocess (370 sec)
beginning 6 repetitions
123456
Warning -- reap_children() reaped child process 4641
.Warning -- reap_children() reaped child process 4708
..Warning -- reap_children() reaped child process 4832
.Warning -- reap_children() reaped child process 4916
Warning -- reap_children() reaped child process 4921
..
msg300074 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 11:30
The problem is that socketserver.ForkinMixin doesn't wait until all children completes. It's only calls os.waitpid() in non-blocking module (using os.WNOHANG) after each loop iteration. If a child process completes after the last call to ForkingMixIn.collect_children(), the server leaks zombie processes.

The server must wait until all children completes. Attached PR implements that.

The bug was be reproduced with the attached forkingmixin_sleep.patch.

haypo@selma$ ./python -m test -v -u all test_socketserver --fail-env-changed -m '*Fork*'
(...)
Warning -- reap_children() reaped child process 17093
Warning -- reap_children() reaped child process 17094
(...)
msg300084 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 13:28
New changeset aa8ec34ad52bb3b274ce91169e1bc4a598655049 by Victor Stinner in branch 'master':
bpo-31151: Add socketserver.ForkingMixIn.server_close() (#3057)
https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049
msg300106 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 15:38
bpo-31010 has been marked as a duplicate of this issue.
msg302013 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-13 00:39
I created a thread on the python-dev mailing list to discuss this issue:
https://mail.python.org/pipermail/python-dev/2017-August/148826.html
msg302015 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-13 00:43
I tag this issue as release blocker as a remainder that we have to agree how to handle threads/processes before Python 3.7 feature freeze.

See also bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close().
History
Date User Action Args
2017-09-18 19:40:13ned.deilysetpriority: release blocker -> deferred blocker
2017-09-13 00:43:15hayposetpriority: normal -> release blocker
nosy: + ned.deily
messages: + msg302015

2017-09-13 00:39:25hayposetmessages: + msg302013
2017-08-18 23:25:10hayposettitle: test_socketserver: Warning -- reap_children() reaped child process -> socketserver.ForkingMixIn.server_close() leaks zombie processes
2017-08-10 15:38:47hayposetmessages: + msg300106
2017-08-10 15:38:19haypolinkissue31010 superseder
2017-08-10 13:28:19hayposetmessages: + msg300084
2017-08-10 11:30:30hayposetpull_requests: + pull_request3092
2017-08-10 11:30:13hayposetfiles: + forkingmixin_sleep.patch
keywords: + patch
messages: + msg300074
2017-08-09 00:41:38hayposettitle: [3.6] test_socketserver: Warning -- reap_children() reaped child process -> test_socketserver: Warning -- reap_children() reaped child process
messages: + msg299960
versions: + Python 3.7
2017-08-09 00:40:39haypocreate