classification
Title: test_msilib is potentially leaking references and memory blocks
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lukasz.langa, miss-islington, pablogsal, steve.dower, vstinner
Priority: high Keywords: patch

Created on 2019-06-01 16:15 by pablogsal, last changed 2019-06-24 14:11 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13750 merged pablogsal, 2019-06-02 19:35
PR 14339 merged miss-islington, 2019-06-24 11:31
PR 14340 merged vstinner, 2019-06-24 11:33
Messages (9)
msg344182 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-01 16:15
BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Windows8.1 Refleaks 2.7
Builder url: https://buildbot.python.org/all/#/builders/33/
Build url: https://buildbot.python.org/all/#/builders/33/builds/604

Failed tests
------------



Test leaking resources
----------------------

- test_msilib is leaking references


Build summary
-------------

== Tests result: FAILURE then FAILURE ==

360 tests OK.

10 slowest tests:
- test_bsddb3: 4042.9s
- test_mmap: 434.0s
- test_multiprocessing: 313.8s
- test_shelve: 246.5s
- test_mailbox: 239.3s
- test_ssl: 227.2s
- test_lib2to3: 135.6s
- test_largefile: 131.5s
- test_decimal: 126.0s
- test_urllib2_localnet: 122.6s

1 test failed:
    test_msilib

42 tests skipped:
    test_aepack test_al test_applesingle test_bsddb185 test_cd test_cl
    test_commands test_crypt test_curses test_dbm test_dl test_epoll
    test_fcntl test_fork1 test_gdb test_gdbm test_gl test_grp
    test_imgfile test_ioctl test_kqueue test_linuxaudiodev test_macos
    test_macostools test_mhlib test_nis test_openpty test_ossaudiodev
    test_pipes test_poll test_posix test_pty test_pwd test_readline
    test_resource test_scriptpackages test_spwd test_sunaudiodev
    test_threadsignals test_wait3 test_wait4 test_zipfile64
2 skips unexpected on win32:
    test_gdb test_readline

1 re-run test:
    test_msilib

Total duration: 1 hour 20 min


Tracebacks
----------



Current builder status
----------------------

The builder is failing currently

Commits
-------



Other builds with similar failures
----------------------------------

-  https://buildbot.python.org/all/#/builders/80/builds/608
-  https://buildbot.python.org/all/#/builders/132/builds/505
msg344183 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-01 16:15
This is likely to be caused by: 

https://github.com/python/cpython/pull/13688
msg344304 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-02 19:38
The leak is because creating a 'msilib.Directory' adds an entry on a global variable '_directories', therefore leaking.

My first PR will make sure the _directories is cleared out after the test to stop the leak, but it seems to me that the variable is unused.

@Steve, could you confirm that there is still a reason for the msilib._directories?
msg344311 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-02 20:36
New changeset c0295dba259accc4b247beb22a0b2cc2f31d9850 by Pablo Galindo in branch 'master':
bpo-37124: Fix reference leak in test_msilib (GH-13750)
https://github.com/python/cpython/commit/c0295dba259accc4b247beb22a0b2cc2f31d9850
msg344472 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-03 20:45
I have no idea. I don't ever use msilib, but there are a few people who do that don't have any way to get bugs fixed, so I'll help merge.

If it's just the test leak, probably it's in the test. I don't recall the change having any impact on resource usage. But at worst, removing the test completely is fine by me.
msg344571 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-04 13:36
Python 2.7 is still affected:
https://buildbot.python.org/all/#/builders/33/builds/607
test_msilib leaked [1, 1, 1] references, sum=3
msg346384 - (view) Author: miss-islington (miss-islington) Date: 2019-06-24 11:56
New changeset c9015a7228029bb22fa581b31c4c1ebdc2882814 by Miss Islington (bot) in branch '3.7':
bpo-37124: Fix reference leak in test_msilib (GH-13750)
https://github.com/python/cpython/commit/c9015a7228029bb22fa581b31c4c1ebdc2882814
msg346406 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-24 14:10
New changeset 6cbff564f00fbe17b5443f6986a44ce116d32aee by Victor Stinner in branch '2.7':
bpo-37124: Fix reference leak in test_msilib (GH-13750) (GH-14340)
https://github.com/python/cpython/commit/6cbff564f00fbe17b5443f6986a44ce116d32aee
msg346408 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-24 14:11
I backported Pablo's fix to 2.7 and 3.7 to attempt to fix Windows Refleak 2.7 and Windows Refleak 3.7 buildbots.
History
Date User Action Args
2019-06-24 14:11:16vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg346408

stage: patch review -> resolved
2019-06-24 14:10:05vstinnersetmessages: + msg346406
2019-06-24 11:56:21miss-islingtonsetnosy: + miss-islington
messages: + msg346384
2019-06-24 11:33:38vstinnersetpull_requests: + pull_request14160
2019-06-24 11:31:26miss-islingtonsetstage: patch review
pull_requests: + pull_request14159
2019-06-04 13:36:27vstinnersetnosy: + vstinner
messages: + msg344571
2019-06-03 20:45:13steve.dowersetmessages: + msg344472
2019-06-02 20:36:24pablogsalsetmessages: + msg344311
2019-06-02 19:38:04pablogsalsetmessages: + msg344304
stage: patch review -> (no value)
2019-06-02 19:35:29pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13631
2019-06-02 18:06:03pablogsalsetpriority: normal -> high
nosy: + lukasz.langa
2019-06-01 16:15:53pablogsalsetmessages: + msg344183
2019-06-01 16:15:32pablogsalsetnosy: + steve.dower
2019-06-01 16:15:26pablogsalcreate