Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_zipfile leaks references on Python 3.5 on the wo Refleaks buildbots (Gentoo, Windows) #74860

Closed
vstinner opened this issue Jun 15, 2017 · 9 comments
Closed

test_zipfile leaks references on Python 3.5 on the wo Refleaks buildbots (Gentoo, Windows) #74860

vstinner opened this issue Jun 15, 2017 · 9 comments
Labels
performance Performance or resource usage tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 30675
Nosy @vstinner, @serhiy-storchaka, @mlouielu
PRs
  • [3.5] bpo-30675: Fix multiprocessing code in regrtest #2220
  • [2.7] bpo-30675: Fix refleak hunting in regrtest #2227
    		•
    Files
  • test_x.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-06-26.23:47:56.331>
created_at = <Date 2017-06-15.13:17:06.946>
labels = ['tests', 'performance']
title = 'test_zipfile leaks references on Python 3.5 on the wo Refleaks buildbots (Gentoo, Windows)'
updated_at = <Date 2017-06-26.23:47:56.329>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-06-26.23:47:56.329>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-26.23:47:56.331>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-06-15.13:17:06.946>
creator = 'vstinner'
dependencies = []
files = ['46955']
hgrepos = []
issue_num = 30675
keywords = ['buildbot']
message_count = 9.0
messages = ['296084', '296085', '296087', '296108', '296109', '296128', '296129', '296173', '296976']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'louielu']
pr_nums = ['2220', '2227']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue30675'
versions = ['Python 3.5']
    @vstinner
    Copy link
    Member Author

    http://buildbot.python.org/all/builders/AMD64%20Windows8.1%20Refleaks%203.5/builds/27/steps/test/logs/stdio

    [336/398/5] test_zipfile
    beginning 6 repetitions
    123456
    ......
    test_zipfile leaked [3, 9, 8] references, sum=20
    test_zipfile leaked [3, 9, 8] memory blocks, sum=20
    (...)
    Re-running test 'test_zipfile' in verbose mode
    (...)
    OK (skipped=1)

    http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.5/builds/30/steps/test/logs/stdio

    [135/398/3] test_zipfile
    beginning 6 repetitions
    123456
    ......
    test_zipfile leaked [6, 6, 9] references, sum=21
    test_zipfile leaked [6, 8, 11] memory blocks, sum=25
    (...)
    Re-running test 'test_zipfile' in verbose mode
    (...)
    OK (skipped=1)

    @vstinner vstinner added tests Tests in the Lib/test dir performance Performance or resource usage labels Jun 15, 2017
    @vstinner
    Copy link
    Member Author

    I tried "./python -m test -R 3:3 test_zipfile" on Linux and my Windows VM, but I'm unable to reproduce the issue. Maybe I passed the wrong options to regrtest (-m test)?

    @mlouielu
    Copy link
    Mannequin

    mlouielu mannequin commented Jun 15, 2017 

    $ git checkout upstream/3.5
$ make -j8
$ ./python.exe -m test -R 3:3 test_zipfile
[1/1] test_zipfile
beginning 6 repetitions
123456
......
1 test OK.

    That is a pass on my macOS with upstream/3.5

    @vstinner
    Copy link
    Member Author

    I succeeded to reproduce the bug, but only when running the -jN option, like:

    ./python -m test -j2 -R 3:3 test_zipfile

    I reduced the test to:

    ./python -m test -j2 -R 3:3 test_zipfile -m test_random_open

    This test heavily use random.randint().

    In fact, the bug doesn't come from test_zipfile, but it's an issue Python singleton objects. There is a bug in regrtest which doesn't call warmup_caches() in worker processes when using the -jN option (multiprocessing mode).

    My PR #2220 should fix the bug.

    Sadly, I didn't backport test_regrtest functional tests from master to 3.5 yet: see bpo-30383.

    @vstinner
    Copy link
    Member Author

    It seems like Python 3.6 and master (3.7) are not impacted by this bug, thanks to my rework on regrtest (now libregrtest) in Python 3.6.

    It's unclear to me if Python 2.7 has the bug: I don't see the warm_caches() function.

    @vstinner
    Copy link
    Member Author

    Ok, using attached test_x.py, I'm able to reproduce this bug on Python 2.7.

    @vstinner
    Copy link
    Member Author

    New changeset 33cf0c4 by Victor Stinner in branch '3.5':
    bpo-30675: Fix multiprocessing code in regrtest (bpo-2220)
    33cf0c4

    @vstinner
    Copy link
    Member Author

    New changeset 8bb0863 by Victor Stinner in branch '2.7':
    bpo-30675: Fix refleak hunting in regrtest (bpo-2227)
    8bb0863

    @vstinner
    Copy link
    Member Author

    I tested: "./python -m test -j2 -R 3:3 -m test_random_open test_zipfile" now pass on 2.7, 3.5, 3.6 and master branches. So this issue is now fixed.

    By the way, bpo-30776 should reduce even further false positive when hunting reference leaks.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    performance Performance or resource usage tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner