This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_zipfile leaks references on Python 3.5 on the wo Refleaks buildbots (Gentoo, Windows)
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: louielu, serhiy.storchaka, vstinner
Priority: normal Keywords: buildbot

Created on 2017-06-15 13:17 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test_x.py vstinner, 2017-06-15 22:08
Pull Requests
URL Status Linked Edit
PR 2220 merged vstinner, 2017-06-15 15:37
PR 2227 merged vstinner, 2017-06-15 22:16
Messages (9)
msg296084 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 13:17
http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.5/builds/27/steps/test/logs/stdio

[336/398/5] test_zipfile
beginning 6 repetitions
123456
......
test_zipfile leaked [3, 9, 8] references, sum=20
test_zipfile leaked [3, 9, 8] memory blocks, sum=20
(...)
Re-running test 'test_zipfile' in verbose mode
(...)
OK (skipped=1)



http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.5/builds/30/steps/test/logs/stdio

[135/398/3] test_zipfile
beginning 6 repetitions
123456
......
test_zipfile leaked [6, 6, 9] references, sum=21
test_zipfile leaked [6, 8, 11] memory blocks, sum=25
(...)
Re-running test 'test_zipfile' in verbose mode
(...)
OK (skipped=1)
msg296085 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 13:18
I tried "./python -m test -R 3:3 test_zipfile" on Linux and my Windows VM, but I'm unable to reproduce the issue. Maybe I passed the wrong options to regrtest (-m test)?
msg296087 - (view) Author: Louie Lu (louielu) * Date: 2017-06-15 13:31
$ git checkout upstream/3.5
$ make -j8
$ ./python.exe -m test -R 3:3 test_zipfile
[1/1] test_zipfile
beginning 6 repetitions
123456
......
1 test OK.


That is a pass on my macOS with upstream/3.5
msg296108 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 15:40
I succeeded to reproduce the bug, but only when running the -jN option, like:

./python -m test -j2 -R 3:3 test_zipfile

I reduced the test to:

./python -m test -j2 -R 3:3 test_zipfile -m test_random_open

This test heavily use random.randint().

In fact, the bug doesn't come from test_zipfile, but it's an issue Python singleton objects. There is a bug in regrtest which doesn't call warmup_caches() in worker processes when using the -jN option (multiprocessing mode).

My PR https://github.com/python/cpython/pull/2220 should fix the bug.

Sadly, I didn't backport test_regrtest functional tests from master to 3.5 yet: see bpo-30383.
msg296109 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 15:43
It seems like Python 3.6 and master (3.7) are not impacted by this bug, thanks to my rework on regrtest (now libregrtest) in Python 3.6.

It's unclear to me if Python 2.7 has the bug: I don't see the warm_caches() function.
msg296128 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 22:08
Ok, using attached test_x.py, I'm able to reproduce this bug on Python 2.7.
msg296129 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-15 22:08
New changeset 33cf0c4cd6e8abe138c3469ca9ec1502410945f0 by Victor Stinner in branch '3.5':
bpo-30675: Fix multiprocessing code in regrtest (#2220)
https://github.com/python/cpython/commit/33cf0c4cd6e8abe138c3469ca9ec1502410945f0
msg296173 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 10:14
New changeset 8bb0863e0282c71d21bc2c1b5184d05d8f3a0988 by Victor Stinner in branch '2.7':
bpo-30675: Fix refleak hunting in regrtest (#2227)
https://github.com/python/cpython/commit/8bb0863e0282c71d21bc2c1b5184d05d8f3a0988
msg296976 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 23:47
I tested: "./python -m test -j2 -R 3:3 -m test_random_open test_zipfile" now pass on 2.7, 3.5, 3.6 and master branches. So this issue is now fixed.

By the way, bpo-30776 should reduce even further false positive when hunting reference leaks.
History
Date User Action Args
2022-04-11 14:58:47adminsetgithub: 74860
2017-06-26 23:47:56vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg296976

stage: resolved
2017-06-16 10:14:11vstinnersetmessages: + msg296173
2017-06-15 22:16:58vstinnersetpull_requests: + pull_request2272
2017-06-15 22:08:42vstinnersetmessages: + msg296129
2017-06-15 22:08:16vstinnersetfiles: + test_x.py

messages: + msg296128
2017-06-15 15:43:43vstinnersetmessages: + msg296109
2017-06-15 15:40:51vstinnersetnosy: + serhiy.storchaka
messages: + msg296108
2017-06-15 15:37:39vstinnersetpull_requests: + pull_request2265
2017-06-15 13:31:17louielusetnosy: + louielu
messages: + msg296087
2017-06-15 13:18:13vstinnersetmessages: + msg296085
2017-06-15 13:17:06vstinnercreate