classification
Title: test_multiprocessing_forkserver leaks references on Python 3
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: davin, pitrou, vstinner
Priority: normal Keywords:

Created on 2017-06-26 20:16 by vstinner, last changed 2017-09-15 16:04 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2467 merged pitrou, 2017-06-28 09:04
PR 2468 merged pitrou, 2017-06-28 09:28
PR 2469 merged pitrou, 2017-06-28 09:28
PR 2470 merged pitrou, 2017-06-28 09:52
PR 2471 merged pitrou, 2017-06-28 10:47
PR 2472 merged pitrou, 2017-06-28 10:48
PR 2473 merged pitrou, 2017-06-28 10:51
Messages (14)
msg296942 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 20:16
$ ./python -m test --match=test.test_multiprocessing_forkserver.WithProcessesTestBarrier.test_barrier --match=test.test_multiprocessing_forkserver.WithProcessesTestFinalize.test_finalize -R 3:3 test_multiprocessing_forkserver 

test_multiprocessing_forkserver leaked [222, 222, 225] memory blocks, sum=669
test_multiprocessing_forkserver failed

Total duration: 5 sec
Tests result: FAILURE
msg296943 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 20:27
test_multiprocessing_spawn doesn't leak:

haypo@selma$ ./python -m test --match=test.test_multiprocessing_spawn.WithProcessesTestBarrier.test_barrier --match=test.test_multiprocessing_spawn.WithProcessesTestFinalize.test_finalize -R 3:3 test_multiprocessing_spawn
Run tests sequentially
0:00:00 load avg: 0.11 [1/1] test_multiprocessing_spawn
beginning 6 repetitions
123456
......
1 test OK.

Total duration: 10 sec
Tests result: SUCCESS
msg296944 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 20:28
The first message was on Python 3.6. On master, regrtest sees even more kinds of leaks: references, memory blocks and file descriptors.

haypo@selma$ ./python -m test --match=test.test_multiprocessing_forkserver.WithProcessesTestBarrier.test_barrier --match=test.test_multiprocessing_forkserver.WithProcessesTestFinalize.test_finalize -R 3:3 test_multiprocessing_forkserver 
Run tests sequentially
0:00:00 load avg: 0.25 [1/1] test_multiprocessing_forkserver
beginning 6 repetitions
123456
......
test_multiprocessing_forkserver leaked [492, 492, 492] references, sum=1476
test_multiprocessing_forkserver leaked [221, 222, 225] memory blocks, sum=668
test_multiprocessing_forkserver leaked [4, 4, 4] file descriptors, sum=12
test_multiprocessing_forkserver failed

1 test failed:
    test_multiprocessing_forkserver

Total duration: 5 sec
Tests result: FAILURE
msg296945 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-26 20:38
Is it test_finalize or test_barrier?
msg296946 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 20:51
> Is it test_finalize or test_barrier?

Try yourself: it only leaks when you run both :-)
msg297164 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 09:21
New changeset a79f8faccf5e26f55e8b9496ad49d2071b5e299c by Victor Stinner (Antoine Pitrou) in branch 'master':
bpo-30775: Fix refleaks in test_multiprocessing (#2467)
https://github.com/python/cpython/commit/a79f8faccf5e26f55e8b9496ad49d2071b5e299c
msg297166 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 09:27
Hey, thank you Antoine for fixing the last known reference leak!
msg297168 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 09:49
New changeset e022aad73a4151b5628e2476a8465ce6c0d18b8c by Antoine Pitrou in branch '3.6':
[3.6] bpo-30775: Fix refleaks in test_multiprocessing (GH-2467) (#2468)
https://github.com/python/cpython/commit/e022aad73a4151b5628e2476a8465ce6c0d18b8c
msg297169 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 09:50
New changeset f15bf1f3f3104f6ab2229e4b359984489a74685b by Victor Stinner (Antoine Pitrou) in branch '3.5':
[3.5] bpo-30775: Fix refleaks in test_multiprocessing (GH-2467) (#2469)
https://github.com/python/cpython/commit/f15bf1f3f3104f6ab2229e4b359984489a74685b
msg297170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 10:29
New changeset 79d37ae979a65ada0b2ac820279ccc3b1cd41ba6 by Victor Stinner (Antoine Pitrou) in branch 'master':
Clear potential ref cycle between Process and Process target (#2470)
https://github.com/python/cpython/commit/79d37ae979a65ada0b2ac820279ccc3b1cd41ba6
msg297171 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 11:08
New changeset 38d6a40898940d2c62c637fcc8acce05e75de5c2 by Antoine Pitrou in branch '3.6':
[3.6] Clear potential ref cycle between Process and Process target (GH-2470) (#2471)
https://github.com/python/cpython/commit/38d6a40898940d2c62c637fcc8acce05e75de5c2
msg297172 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 11:16
New changeset a9705b777859f555d50eb5dcd5fc4260c42a0188 by Antoine Pitrou in branch '3.5':
[3.5] Clear potential ref cycle between Process and Process target (GH-2470) (#2472)
https://github.com/python/cpython/commit/a9705b777859f555d50eb5dcd5fc4260c42a0188
msg297174 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 11:48
New changeset 12536bd261ba95cd2748f3d7d47768742a6ffa7a by Victor Stinner (Antoine Pitrou) in branch '2.7':
[2.7] Clear potential ref cycle between Process and Process target (GH-2470) (#2473)
https://github.com/python/cpython/commit/12536bd261ba95cd2748f3d7d47768742a6ffa7a
msg302261 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-15 16:04
The Python 2.7 change caused a regression in Sphinx: bpo-31016.
History
Date User Action Args
2017-09-15 16:04:53vstinnersetmessages: + msg302261
versions: + Python 2.7
2017-06-28 11:48:40vstinnersetmessages: + msg297174
2017-06-28 11:16:00pitrousetmessages: + msg297172
2017-06-28 11:08:23pitrousetmessages: + msg297171
2017-06-28 10:51:38pitrousetpull_requests: + pull_request2528
2017-06-28 10:48:21pitrousetpull_requests: + pull_request2527
2017-06-28 10:47:32pitrousetpull_requests: + pull_request2526
2017-06-28 10:29:10vstinnersetmessages: + msg297170
2017-06-28 09:52:15pitrousetpull_requests: + pull_request2525
2017-06-28 09:51:49pitrousetstatus: open -> closed
stage: commit review -> resolved
2017-06-28 09:50:46vstinnersetmessages: + msg297169
2017-06-28 09:49:41pitrousetstatus: pending -> open

messages: + msg297168
2017-06-28 09:35:03pitrousetstatus: open -> pending
resolution: fixed
stage: commit review
2017-06-28 09:28:54pitrousetpull_requests: + pull_request2524
2017-06-28 09:28:40pitrousetpull_requests: + pull_request2523
2017-06-28 09:27:06vstinnersetmessages: + msg297166
2017-06-28 09:21:55vstinnersetmessages: + msg297164
2017-06-28 09:04:37pitrousetpull_requests: + pull_request2521
2017-06-26 20:51:47vstinnersetmessages: + msg296946
2017-06-26 20:38:08pitrousetmessages: + msg296945
2017-06-26 20:33:19vstinnersetnosy: + pitrou, davin
2017-06-26 20:28:38vstinnersetmessages: + msg296944
2017-06-26 20:27:35vstinnersetmessages: + msg296943
2017-06-26 20:18:18vstinnersettitle: [3.6] test_multiprocessing_forkserver leaks references on Python 3.6 -> test_multiprocessing_forkserver leaks references on Python 3
versions: + Python 3.5, Python 3.7
2017-06-26 20:16:57vstinnercreate