classification
Title: test_lzma: test_refleaks_in_decompressor___init__() leaks 100 handles on Windows
Type: resource usage Stage: resolved
Components: Tests, Windows Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, inada.naoki, miss-islington, paul.moore, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2018-06-20 15:59 by vstinner, last changed 2018-06-23 13:07 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7822 ZackerySpytz, 2018-06-20 16:42
PR 7843 merged vstinner, 2018-06-21 09:36
PR 7871 merged miss-islington, 2018-06-23 08:36
PR 7872 merged vstinner, 2018-06-23 08:40
Messages (10)
msg320087 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-20 15:59
Using my PR 7827, I saw that test_refleaks_in_decompressor___init__() of test_lzma leaks 100 handles on Windows.

Extract of the code:

    @support.refcount_test
    def test_refleaks_in_decompressor___init__(self):
        gettotalrefcount = support.get_attribute(sys, 'gettotalrefcount')
        lzd = LZMADecompressor()
        refs_before = gettotalrefcount()
        for i in range(100):
            lzd.__init__()
        self.assertAlmostEqual(gettotalrefcount() - refs_before, 0, delta=10)


The test comes from bpo-31787: commit d019bc8319ea35e93bf4baa38098ff1b57cd3ee5.
msg320091 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-06-20 16:12
In _lzmamodule.c

    self->lock = PyThread_allocate_lock();

This seems leak some resource.  But I'm not sure this lock contains refcnt on Windows.
msg320093 - (view) Author: Zackery Spytz (ZackerySpytz) * (Python triager) Date: 2018-06-20 16:20
Have a look at #23224 and PR 7822.
msg320094 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-20 16:22
> This seems leak some resource.  But I'm not sure this lock contains refcnt on Windows.

PR 7827 checks for leaks of Windows handles, this issue is about Windows handles, not Python reference leaks. But a Windows lock may use a Windows handle internally, I don't know :-)
msg320095 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-20 16:22
It seems like test_bz2 has the same issue, but I'm not sure:
https://github.com/python/cpython/pull/7827#issuecomment-398810849
msg320096 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-06-20 16:26
I four times tried to update this issue, and only the forth one was successful. Other attempts was failed because "Edit Error: someone else has edited this issue (nosy, components, versions)." This is an extreme case of race condition in real life.
msg320300 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-23 08:35
New changeset 9b7cf757213cf4d7ae1d436d86ad53f5ba362d55 by Victor Stinner in branch 'master':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843)
https://github.com/python/cpython/commit/9b7cf757213cf4d7ae1d436d86ad53f5ba362d55
msg320301 - (view) Author: miss-islington (miss-islington) Date: 2018-06-23 08:53
New changeset efc6bf66a58e96c12116d65984ac69b0f36f85de by Miss Islington (bot) in branch '3.7':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843)
https://github.com/python/cpython/commit/efc6bf66a58e96c12116d65984ac69b0f36f85de
msg320306 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-23 13:06
New changeset 1094891b2e3eafef464819acd4cd04cfd2b59661 by Victor Stinner in branch '3.6':
bpo-33916: Fix bz2 and lzma init when called twice (GH-7843) (GH-7872)
https://github.com/python/cpython/commit/1094891b2e3eafef464819acd4cd04cfd2b59661
msg320307 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-23 13:07
It seems like Python 2.7 is inaffected by the bug:

static int
BZ2File_init(BZ2FileObject *self, PyObject *args, PyObject *kwargs)
{
    ...
#ifdef WITH_THREAD
    if (!self->lock) {
        self->lock = PyThread_allocate_lock();
    }

    if (!self->lock) {
        PyErr_SetString(PyExc_MemoryError, "unable to allocate lock");
        goto error;
    }
#endif
    ...
}

I fixed the leak in Python 3.6, 3.7 and master. I close the issue.
History
Date User Action Args
2018-06-23 13:07:46vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg320307

stage: patch review -> resolved
2018-06-23 13:06:16vstinnersetmessages: + msg320306
2018-06-23 08:53:05miss-islingtonsetnosy: + miss-islington
messages: + msg320301
2018-06-23 08:40:36vstinnersetpull_requests: + pull_request7479
2018-06-23 08:36:07miss-islingtonsetpull_requests: + pull_request7478
2018-06-23 08:35:34vstinnersetmessages: + msg320300
2018-06-21 09:36:40vstinnersetpull_requests: + pull_request7453
2018-06-20 16:42:34ZackerySpytzsetkeywords: + patch
stage: patch review
pull_requests: + pull_request7435
2018-06-20 16:26:35serhiy.storchakasetmessages: + msg320096
2018-06-20 16:23:27serhiy.storchakasetnosy: + serhiy.storchaka

components: + Tests
versions: + Python 3.6, Python 3.7
2018-06-20 16:22:36vstinnersetmessages: + msg320095
2018-06-20 16:22:18vstinnersetmessages: + msg320094
2018-06-20 16:20:30ZackerySpytzsetnosy: + ZackerySpytz
messages: + msg320093
2018-06-20 16:12:32inada.naokisetnosy: + inada.naoki
messages: + msg320091
2018-06-20 15:59:40vstinnercreate