classification
Title: test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2017-08-10 11:45 by vstinner, last changed 2017-10-27 13:17 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3059 closed vstinner, 2017-08-10 12:12
PR 4102 merged vstinner, 2017-10-24 09:37
PR 4104 merged python-dev, 2017-10-24 10:33
PR 4146 merged serhiy.storchaka, 2017-10-27 10:41
PR 4148 merged python-dev, 2017-10-27 12:35
Messages (11)
msg300075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 11:45
The "x86 Gentoo Refleaks 3.x" buildbot runs tests using -u-cpu to disable the cpu resource. The problem is that DirectoryTestCase.test_files() of Lib/test/test_tools/test_unparse.py uses random:

        # Test limited subset of files unless the 'cpu' resource is specified.
        if not test.support.is_resource_enabled("cpu"):
            names = random.sample(names, 10)

So when we run the same test 7 times, each run uses different data.

I see different options:

* Reseed random using the same seed in dash_R() of regrtest
* Always test all data in test_unparse.py: all files, or select a specific set of interesting files

The random issue is more generic than just test_unparse.py, and so it would be interesting to explore this path. Maybe the random issue explains why some other tests fail randomly.

libregrtest always seeds the random RNG using a seed displayed on the standard output. We should either reuse this seed, or create a new unique seed for each test file, and display it (to be able to reproduce tests).

Reseed random before running each test file can also helps to make tests more reproductible

--

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/52/steps/test/logs/stdio

test_tools leaked [1, 4, 2] memory blocks, sum=7
(...)
Re-running test 'test_tools' in verbose mode
(...)
test_tools leaked [1, 2, 2] memory blocks, sum=5
(...)
1 test failed again:
    test_tools
msg300439 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-17 15:33
I created bpo-31227: "regrtest: reseed random with the same seed before running a test file".
msg302063 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 12:24
test_tools also fails on x86 Gentoo Refleaks 3.6:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/90

test_tools leaked [1, 1, 118] memory blocks, sum=120
(...)
test_tools leaked [7, 1, 6] memory blocks, sum=14
msg304889 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 09:29
"test_tools leaked (...)" still occurs randomly on 3.6 and 3.x Gentoo buildbot.
msg304890 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 09:38
My previous attempt, PR 3059, was a a very generic fix, but my bpo-31227 idea was rejected.

I proposed a new idea, PR 4102, which is restricted to this specific issue:
"test_unparse.DirectoryTestCase now stores the names sample to always
test the same files. It prevents false alarms when hunting reference
leaks."
msg304897 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 10:33
New changeset 8e482bea21cb942804234e36d3c6c896aabd32da by Victor Stinner in branch 'master':
bpo-31174: Fix test_tools.test_unparse (#4102)
https://github.com/python/cpython/commit/8e482bea21cb942804234e36d3c6c896aabd32da
msg304898 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 10:36
Python 2.7 is not affected. Even if Python 2.7 has Demo/parser/test_unparse.py, this test is not part of the Python test suite, and it doesn't use random.sample(names, 10).
msg304900 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 11:18
New changeset d8f78a1fbc0a34224289d436ad67f608fa553f0c by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-31174: Fix test_tools.test_unparse (GH-4102) (#4104)
https://github.com/python/cpython/commit/d8f78a1fbc0a34224289d436ad67f608fa553f0c
msg304901 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-24 11:18
The bug should now be fixed.
msg305117 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-27 12:35
New changeset 7351f9e5a91c403d15c6d556f9989b443f1296f9 by Serhiy Storchaka in branch 'master':
bpo-31174: Improve the code of test_tools.test_unparse. (#4146)
https://github.com/python/cpython/commit/7351f9e5a91c403d15c6d556f9989b443f1296f9
msg305119 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-27 13:17
New changeset bb78898224c99cbf9c6beed8706869f9b66967c2 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-31174: Improve the code of test_tools.test_unparse. (GH-4146) (#4148)
https://github.com/python/cpython/commit/bb78898224c99cbf9c6beed8706869f9b66967c2
History
Date User Action Args
2017-10-27 13:17:38serhiy.storchakasetmessages: + msg305119
2017-10-27 12:35:23python-devsetpull_requests: + pull_request4114
2017-10-27 12:35:13serhiy.storchakasetmessages: + msg305117
2017-10-27 10:41:18serhiy.storchakasetpull_requests: + pull_request4112
2017-10-24 11:18:27vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg304901

stage: patch review -> resolved
2017-10-24 11:18:06vstinnersetmessages: + msg304900
2017-10-24 10:36:49vstinnersetmessages: + msg304898
2017-10-24 10:33:59python-devsetpull_requests: + pull_request4074
2017-10-24 10:33:38vstinnersetmessages: + msg304897
2017-10-24 09:38:50vstinnersetmessages: + msg304890
2017-10-24 09:37:46vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request4072
2017-10-24 09:29:25vstinnersetmessages: + msg304889
2017-09-13 12:26:18vstinnersettitle: test_tools leaks randomly references on x86 Gentoo Refleaks 3.x -> test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x
versions: + Python 3.6
2017-09-13 12:24:14vstinnersetmessages: + msg302063
2017-08-17 15:33:50vstinnersetmessages: + msg300439
2017-08-10 12:12:20vstinnersetpull_requests: + pull_request3093
2017-08-10 11:45:39vstinnercreate