Title: test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x
Type: Stage:
Components: Tests Versions: Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: haypo, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-08-10 11:45 by haypo, last changed 2017-09-13 12:26 by haypo.

Pull Requests
URL Status Linked Edit
PR 3059 open haypo, 2017-08-10 12:12
Messages (3)
msg300075 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 11:45
The "x86 Gentoo Refleaks 3.x" buildbot runs tests using -u-cpu to disable the cpu resource. The problem is that DirectoryTestCase.test_files() of Lib/test/test_tools/ uses random:

        # Test limited subset of files unless the 'cpu' resource is specified.
        if not"cpu"):
            names = random.sample(names, 10)

So when we run the same test 7 times, each run uses different data.

I see different options:

* Reseed random using the same seed in dash_R() of regrtest
* Always test all data in all files, or select a specific set of interesting files

The random issue is more generic than just, and so it would be interesting to explore this path. Maybe the random issue explains why some other tests fail randomly.

libregrtest always seeds the random RNG using a seed displayed on the standard output. We should either reuse this seed, or create a new unique seed for each test file, and display it (to be able to reproduce tests).

Reseed random before running each test file can also helps to make tests more reproductible


test_tools leaked [1, 4, 2] memory blocks, sum=7
Re-running test 'test_tools' in verbose mode
test_tools leaked [1, 2, 2] memory blocks, sum=5
1 test failed again:
msg300439 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-17 15:33
I created bpo-31227: "regrtest: reseed random with the same seed before running a test file".
msg302063 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-13 12:24
test_tools also fails on x86 Gentoo Refleaks 3.6:

test_tools leaked [1, 1, 118] memory blocks, sum=120
test_tools leaked [7, 1, 6] memory blocks, sum=14
Date User Action Args
2017-09-13 12:26:18hayposettitle: test_tools leaks randomly references on x86 Gentoo Refleaks 3.x -> test_tools leaks randomly references on x86 Gentoo Refleaks 3.6 and 3.x
versions: + Python 3.6
2017-09-13 12:24:14hayposetmessages: + msg302063
2017-08-17 15:33:50hayposetmessages: + msg300439
2017-08-10 12:12:20hayposetpull_requests: + pull_request3093
2017-08-10 11:45:39haypocreate