classification
Title: Memory leaks regression caused by: Fix threading._shutdown() race condition
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2019-06-13 08:48 by vstinner, last changed 2019-06-13 10:07 by vstinner. This issue is now closed.

Messages (3)
msg345476 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 08:48
https://buildbot.python.org/all/#/builders/189/builds/60

Examples:

test_ssl leaked [3, 3, 3] references, sum=9
test_ssl leaked [3, 3, 3] memory blocks, sum=9
test_decimal leaked [4, 4, 4] references, sum=12
test_decimal leaked [4, 4, 4] memory blocks, sum=12
test_httplib leaked [1, 1, 1] references, sum=3
test_httplib leaked [1, 1, 1] memory blocks, sum=3
test_multiprocessing_fork leaked [27, 27, 27] references, sum=81
test_multiprocessing_fork leaked [27, 27, 27] memory blocks, sum=81
test_multiprocessing_forkserver leaked [27, 27, 27] references, sum=81
test_multiprocessing_forkserver leaked [27, 27, 27] memory blocks, sum=81
test_socketserver leaked [15, 15, 15] references, sum=45
test_socketserver leaked [15, 15, 15] memory blocks, sum=45
test_capi leaked [17, 17, 17] references, sum=51
test_capi leaked [17, 17, 17] memory blocks, sum=51
test_smtplib leaked [47, 47, 47] references, sum=141
test_smtplib leaked [47, 47, 47] memory blocks, sum=141
test_io leaked [23254, 23250, 23254] references, sum=69758
test_io leaked [7703, 7701, 7703] memory blocks, sum=23107
test_pickle leaked [6, 6, 6] references, sum=18
test_pickle leaked [6, 6, 6] memory blocks, sum=18
test_httpservers leaked [43, 43, 43] references, sum=129
test_httpservers leaked [43, 43, 43] memory blocks, sum=129
test_nntplib leaked [1, 1, 1] references, sum=3
test_nntplib leaked [1, 1, 1] memory blocks, sum=3
test_os leaked [5, 5, 5] references, sum=15
test_os leaked [5, 5, 5] memory blocks, sum=15
test_subprocess leaked [4, 4, 4] references, sum=12
test_subprocess leaked [4, 4, 4] memory blocks, sum=12
test_socket leaked [7, 7, 7] references, sum=21
test_socket leaked [7, 7, 7] memory blocks, sum=21
test_multiprocessing_spawn leaked [27, 27, 27] references, sum=81
test_multiprocessing_spawn leaked [27, 27, 27] memory blocks, sum=81
test_concurrent_futures leaked [3, 3, 3] references, sum=9
test_concurrent_futures leaked [3, 3, 3] memory blocks, sum=9
test_asyncio leaked [62, 62, 62] references, sum=186
test_asyncio leaked [62, 62, 62] memory blocks, sum=186


I used git bisect and I found... my own change :-)

commit 468e5fec8a2f534f1685d59da3ca4fad425c38dd
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Jun 13 01:30:17 2019 +0200

    bpo-36402: Fix threading._shutdown() race condition (GH-13948)
    
    Fix a race condition at Python shutdown when waiting for threads.
    Wait until the Python thread state of all non-daemon threads get
    deleted (join all non-daemon threads), rather than just wait until
    Python threads complete.
    
    * Add threading._shutdown_locks: set of Thread._tstate_lock locks
      of non-daemon threads used by _shutdown() to wait until all Python
      thread states get deleted. See Thread._set_tstate_lock().
    * Add also threading._shutdown_locks_lock to protect access to
      threading._shutdown_locks.
    * Add test_finalization_shutdown() test.
msg345478 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 09:00
Oh. It's a stupid bug: will be fixed by PR 14047.
msg345493 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-13 10:07
commit 6f75c873752a16a7ad8f35855b1e29f59d048e84 (HEAD -> master, upstream/master)
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Jun 13 12:06:24 2019 +0200

    tbpo-36402: Fix threading.Thread._stop() (GH-14047)
    
    Remove the _tstate_lock from _shutdown_locks, don't remove None.
History
Date User Action Args
2019-06-13 10:07:45vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg345493

stage: resolved
2019-06-13 09:00:09vstinnersetmessages: + msg345478
2019-06-13 08:48:58vstinnercreate