classification
Title: Some tests leak temporary files
Type: Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: miss-islington, pablogsal, serhiy.storchaka, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2019-06-26 22:34 by vstinner, last changed 2019-07-08 22:10 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14415 closed vstinner, 2019-06-26 22:35
PR 14416 merged vstinner, 2019-06-26 22:38
PR 14417 merged miss-islington, 2019-06-26 23:40
PR 14527 merged vstinner, 2019-07-01 17:46
PR 14529 merged vstinner, 2019-07-01 17:55
PR 14554 merged miss-islington, 2019-07-02 11:32
PR 14562 merged vstinner, 2019-07-02 21:22
PR 14563 merged vstinner, 2019-07-02 21:33
PR 14564 merged vstinner, 2019-07-02 22:11
PR 14569 merged miss-islington, 2019-07-03 09:10
PR 14570 merged miss-islington, 2019-07-03 09:10
PR 14571 merged miss-islington, 2019-07-03 09:14
PR 14572 merged vstinner, 2019-07-03 10:49
PR 14585 merged miss-islington, 2019-07-04 10:29
PR 14586 merged miss-islington, 2019-07-04 10:29
PR 14601 merged vstinner, 2019-07-05 13:06
PR 14602 merged miss-islington, 2019-07-05 14:15
PR 14643 merged vstinner, 2019-07-08 08:12
PR 14645 merged miss-islington, 2019-07-08 08:49
Messages (25)
msg346692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 22:34
When running tests, the tempoary directory is not left clean: there are temporary files which are not removed.

I wrote a PoC change to detect such bugs and I found that at least test_shutil and test_urllib leak such temporary files/directories.
msg346693 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 22:39
Test leaking a temporary file in test_urllib:
test.test_urllib.urlretrieve_HttpTests.test_short_content_raises_ContentTooShortError_without_reporthook

I'm not sure of my fix:

diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py
index f6ce9cb6d5..fdddd6e2d8 100644
--- a/Lib/urllib/request.py
+++ b/Lib/urllib/request.py
@@ -283,6 +283,7 @@ def urlretrieve(url, filename=None, reporthook=None, data=None):
                     reporthook(blocknum, bs, size)
 
     if size >= 0 and read < size:
+        urlcleanup()
         raise ContentTooShortError(
             "retrieval incomplete: got only %i out of %i bytes"
             % (read, size), result)
msg346699 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 23:39
Using PR 14415, the following test of test_multiprocessing_spawn emits a false alarm because multiprocessing use "Finalizer" objects which are only finalized "later":
test.test_multiprocessing_spawn.WithManagerTestMyManager.test_mymanager_context_prestarted


Workaround, call explicitly _run_finalizers():

diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index eef262d723..bfecbab9ee 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -5651,6 +5651,7 @@ def install_tests_in_module_dict(remote_globs, start_method):
         if need_sleep:
             time.sleep(0.5)
         multiprocessing.process._cleanup()
+        multiprocessing.util._run_finalizers()
         test.support.gc_collect()
 
     remote_globs['setUpModule'] = setUpModule
msg346700 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-26 23:40
New changeset 4c26abd14f1b7242998eb2f7756aa375e0fe714f by Victor Stinner in branch 'master':
bpo-37421: Fix test_shutil: don't leak temporary files (GH-14416)
https://github.com/python/cpython/commit/4c26abd14f1b7242998eb2f7756aa375e0fe714f
msg346703 - (view) Author: miss-islington (miss-islington) Date: 2019-06-26 23:57
New changeset 7fe81ce47ff6725e2e4d667d499e23dac19834af by Miss Islington (bot) in branch '3.8':
bpo-37421: Fix test_shutil: don't leak temporary files (GH-14416)
https://github.com/python/cpython/commit/7fe81ce47ff6725e2e4d667d499e23dac19834af
msg347127 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-02 10:37
About urllib.request, I created: bpo-37475 "What is urllib.request.urlcleanup() function?".
msg347135 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-02 11:32
New changeset 039fb49c185570ab7b02f13fbdc51c859cfd831e by Victor Stinner in branch 'master':
bpo-37421: multiprocessing tests call _run_finalizers() (GH-14527)
https://github.com/python/cpython/commit/039fb49c185570ab7b02f13fbdc51c859cfd831e
msg347137 - (view) Author: miss-islington (miss-islington) Date: 2019-07-02 11:58
New changeset 632cb36084dc9d13f1cdb31a0e7e3ba80745a51a by Miss Islington (bot) in branch '3.8':
bpo-37421: multiprocessing tests call _run_finalizers() (GH-14527)
https://github.com/python/cpython/commit/632cb36084dc9d13f1cdb31a0e7e3ba80745a51a
msg347140 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-02 12:50
New changeset 7cb9204ee1cf204f6f507d99a60f7c5bb359eebb by Victor Stinner in branch 'master':
bpo-37421: urllib.request tests call urlcleanup() (GH-14529)
https://github.com/python/cpython/commit/7cb9204ee1cf204f6f507d99a60f7c5bb359eebb
msg347191 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-03 09:10
New changeset b71d8d67959f3b5efbdfe00066589ac0d8f98aad by Victor Stinner in branch 'master':
bpo-37421: test_winconsoleio doesn't leak temp file anymore (GH-14562)
https://github.com/python/cpython/commit/b71d8d67959f3b5efbdfe00066589ac0d8f98aad
msg347192 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-03 09:10
New changeset 684cb47fffb7af3ac50cb077f6d2a095c9ce20b4 by Victor Stinner in branch 'master':
bpo-37421: test_concurrent_futures cleans up multiprocessing (GH-14563)
https://github.com/python/cpython/commit/684cb47fffb7af3ac50cb077f6d2a095c9ce20b4
msg347193 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-03 09:12
New changeset 74c9dd57771f4f061ee83b069c8e7b37de41246b by Victor Stinner in branch 'master':
bpo-37421: Fix test_distutils.test_build_ext() (GH-14564)
https://github.com/python/cpython/commit/74c9dd57771f4f061ee83b069c8e7b37de41246b
msg347199 - (view) Author: miss-islington (miss-islington) Date: 2019-07-03 09:31
New changeset a2a807f75dc162dfb45fb297aee4961de8008f84 by Miss Islington (bot) in branch '3.8':
bpo-37421: test_winconsoleio doesn't leak temp file anymore (GH-14562)
https://github.com/python/cpython/commit/a2a807f75dc162dfb45fb297aee4961de8008f84
msg347201 - (view) Author: miss-islington (miss-islington) Date: 2019-07-03 09:36
New changeset 79665c698fb8f97475e03c4231067db1ae47addb by Miss Islington (bot) in branch '3.8':
bpo-37421: test_concurrent_futures cleans up multiprocessing (GH-14563)
https://github.com/python/cpython/commit/79665c698fb8f97475e03c4231067db1ae47addb
msg347204 - (view) Author: miss-islington (miss-islington) Date: 2019-07-03 09:48
New changeset 0aefba7f99f0fd9bcb3328a5919e07f9e03676b0 by Miss Islington (bot) in branch '3.8':
bpo-37421: Fix test_distutils.test_build_ext() (GH-14564)
https://github.com/python/cpython/commit/0aefba7f99f0fd9bcb3328a5919e07f9e03676b0
msg347265 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-04 10:29
New changeset 9d40554e0da09a44a8547f3f3a2b9dedfeaf7928 by Victor Stinner in branch 'master':
bpo-37421: Fix multiprocessing get_temp_dir() finalizer (GH-14572)
https://github.com/python/cpython/commit/9d40554e0da09a44a8547f3f3a2b9dedfeaf7928
msg347270 - (view) Author: miss-islington (miss-islington) Date: 2019-07-04 10:46
New changeset 2d438fc0b748b64d518ea8876af3f6963c6d7d60 by Miss Islington (bot) in branch '3.7':
bpo-37421: Fix multiprocessing get_temp_dir() finalizer (GH-14572)
https://github.com/python/cpython/commit/2d438fc0b748b64d518ea8876af3f6963c6d7d60
msg347273 - (view) Author: miss-islington (miss-islington) Date: 2019-07-04 11:34
New changeset 957656ee1d6fb42664274ef2f440a10d26870e2a by Miss Islington (bot) in branch '3.8':
bpo-37421: Fix multiprocessing get_temp_dir() finalizer (GH-14572)
https://github.com/python/cpython/commit/957656ee1d6fb42664274ef2f440a10d26870e2a
msg347343 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-05 13:14
./python -u -m test test_multiprocessing_spawn -R 3:3 still fail:

ERROR: test_context (test.test_multiprocessing_spawn.TestStartMethod)
ERROR: test_set_get (test.test_multiprocessing_spawn.TestStartMethod)

PR 14601 fix these tests when test_multiprocessing_spawn is run more than once.
msg347348 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-05 14:16
New changeset 8fbeb14312b4c1320d31ad86e69749515879d1c3 by Victor Stinner in branch 'master':
bpo-37421: multiprocessing tests now stop ForkServer (GH-14601)
https://github.com/python/cpython/commit/8fbeb14312b4c1320d31ad86e69749515879d1c3
msg347352 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-05 14:22
I created a follow-up issue: bpo-37507 "multiprocessing: Add a stop() method to ForkServer".
msg347355 - (view) Author: miss-islington (miss-islington) Date: 2019-07-05 14:35
New changeset 229f6e85f8b4d57a2e742e0d3fc361c5bd15f1cb by Miss Islington (bot) in branch '3.8':
bpo-37421: multiprocessing tests now stop ForkServer (GH-14601)
https://github.com/python/cpython/commit/229f6e85f8b4d57a2e742e0d3fc361c5bd15f1cb
msg347492 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-08 08:49
New changeset e676244235895aeb6ec3b81ca3ccf4a70e487919 by Victor Stinner in branch 'master':
bpo-37421: test_concurrent_futures stops ForkServer (GH-14643)
https://github.com/python/cpython/commit/e676244235895aeb6ec3b81ca3ccf4a70e487919
msg347493 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-08 09:52
New changeset cdada40b23b1f7f527797ba7cb14c25820b05981 by Victor Stinner (Miss Islington (bot)) in branch '3.8':
bpo-37421: test_concurrent_futures stops ForkServer (GH-14643) (GH-14645)
https://github.com/python/cpython/commit/cdada40b23b1f7f527797ba7cb14c25820b05981
msg347513 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-08 22:10
It seems like tests don't leak temporary files anymore:

$ mkdir ~/TEMP
$ TMPDIR=~/TEMP TEMPDIR=~/TEMP ./python -m test -v -r -u all,-gui 
$ ls ~/TEMP
# empty directory

I close the issue.
History
Date User Action Args
2019-07-08 22:10:28vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg347513

stage: patch review -> resolved
2019-07-08 09:52:04vstinnersetmessages: + msg347493
2019-07-08 08:49:32miss-islingtonsetpull_requests: + pull_request14456
2019-07-08 08:49:18vstinnersetmessages: + msg347492
2019-07-08 08:12:03vstinnersetpull_requests: + pull_request14455
2019-07-05 14:35:42miss-islingtonsetmessages: + msg347355
2019-07-05 14:22:42vstinnersetmessages: + msg347352
2019-07-05 14:16:02vstinnersetmessages: + msg347348
2019-07-05 14:15:58miss-islingtonsetpull_requests: + pull_request14417
2019-07-05 13:14:30vstinnersetmessages: + msg347343
2019-07-05 13:06:20vstinnersetpull_requests: + pull_request14416
2019-07-04 11:34:39miss-islingtonsetmessages: + msg347273
2019-07-04 10:46:03miss-islingtonsetmessages: + msg347270
2019-07-04 10:29:21vstinnersetmessages: + msg347265
2019-07-04 10:29:19miss-islingtonsetpull_requests: + pull_request14404
2019-07-04 10:29:11miss-islingtonsetpull_requests: + pull_request14403
2019-07-03 10:49:53vstinnersetpull_requests: + pull_request14390
2019-07-03 09:48:20miss-islingtonsetmessages: + msg347204
2019-07-03 09:36:27miss-islingtonsetmessages: + msg347201
2019-07-03 09:31:43miss-islingtonsetmessages: + msg347199
2019-07-03 09:14:16miss-islingtonsetpull_requests: + pull_request14389
2019-07-03 09:12:31vstinnersetmessages: + msg347193
2019-07-03 09:10:42miss-islingtonsetpull_requests: + pull_request14388
2019-07-03 09:10:35vstinnersetmessages: + msg347192
2019-07-03 09:10:22miss-islingtonsetpull_requests: + pull_request14387
2019-07-03 09:10:13vstinnersetmessages: + msg347191
2019-07-02 22:11:28vstinnersetpull_requests: + pull_request14382
2019-07-02 21:33:06vstinnersetpull_requests: + pull_request14381
2019-07-02 21:22:37vstinnersetpull_requests: + pull_request14380
2019-07-02 12:50:22vstinnersetmessages: + msg347140
2019-07-02 11:58:17miss-islingtonsetmessages: + msg347137
2019-07-02 11:32:59miss-islingtonsetpull_requests: + pull_request14372
2019-07-02 11:32:35vstinnersetmessages: + msg347135
2019-07-02 10:37:45vstinnersetmessages: + msg347127
2019-07-01 17:55:09vstinnersetpull_requests: + pull_request14342
2019-07-01 17:46:07vstinnersetpull_requests: + pull_request14340
2019-06-26 23:57:52miss-islingtonsetnosy: + miss-islington
messages: + msg346703
2019-06-26 23:40:09miss-islingtonsetpull_requests: + pull_request14230
2019-06-26 23:40:04vstinnersetmessages: + msg346700
2019-06-26 23:39:26vstinnersetmessages: + msg346699
title: Tests leak temporary files -> Some tests leak temporary files
2019-06-26 22:41:22vstinnersetnosy: + zach.ware, serhiy.storchaka, pablogsal
2019-06-26 22:39:17vstinnersetmessages: + msg346693
2019-06-26 22:38:25vstinnersetpull_requests: + pull_request14229
2019-06-26 22:35:23vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request14228
2019-06-26 22:34:15vstinnercreate