Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiprocessing tests log: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown #89372

Closed
vstinner opened this issue Sep 15, 2021 · 7 comments
Closed

multiprocessing tests log: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown #89372

vstinner opened this issue Sep 15, 2021 · 7 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 45209
Nosy @vstinner, @ambv, @miss-islington, @sobolevn
PRs
  • bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing #28377
  • [3.10] bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) #28499
  • [3.9] bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) #28500
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-09-21.18:37:39.102>
created_at = <Date 2021-09-15.13:29:13.571>
labels = ['tests', '3.9', '3.10', '3.11']
title = 'multiprocessing tests log: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown'
updated_at = <Date 2021-09-21.18:37:39.101>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-09-21.18:37:39.101>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-21.18:37:39.102>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-09-15.13:29:13.571>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 45209
keywords = ['patch']
message_count = 7.0
messages = ['401837', '401866', '401902', '402336', '402341', '402342', '402343']
nosy_count = 4.0
nosy_names = ['vstinner', 'lukasz.langa', 'miss-islington', 'sobolevn']
pr_nums = ['28377', '28499', '28500']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45209'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @vstinner
    Copy link
    Member Author

    0:03:25 load avg: 12.33 [250/427] test_multiprocessing_forkserver passed (...)
    /home/vstinner/python/main/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '
    /home/vstinner/python/main/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/psm_33506d9a': [Errno 2] No such file or directory: '/psm_33506d9a'
    warnings.warn('resource_tracker: %r: %s' % (name, e))

    @vstinner vstinner added 3.11 only security fixes tests Tests in the Lib/test dir labels Sep 15, 2021
    @sobolevn
    Copy link
    Member

    I would like to work on this, if no one has started yet.

    @sobolevn
    Copy link
    Member

    The problem was in test_shared_memory_cleaned_after_process_termination test.

    » ./python.exe -m test -v test_multiprocessing_forkserver -m test_shared_memory_cleaned_after_process_termination
== CPython 3.11.0a0 (heads/main:09b4ad11f3, Sep 15 2021, 20:50:50) [Clang 11.0.0 (clang-1100.0.33.16)]
== macOS-10.14.6-x86_64-i386-64bit little-endian
== cwd: /Users/sobolev/Desktop/cpython/build/test_python_14874æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.98 Run tests sequentially
0:00:00 load avg: 1.98 [1/1] test_multiprocessing_forkserver
test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ... ok
/Users/sobolev/Desktop/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/sobolev/Desktop/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/psm_67480e45': [Errno 2] No such file or directory: '/psm_67480e45'
  warnings.warn('resource_tracker: %r: %s' % (name, e))

----------------------------------------------------------------------
Ran 1 test in 0.775s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.6 sec
Tests result: SUCCESS

    My patch (location

    cpython/Lib/test/_test_multiprocessing.py

    Line 4179 in 51056b4

    ):

    resource_tracker.unregister(f"/{name}", "shared_memory")

    With this change no warning is generated:

    » ./python.exe -m test -v test_multiprocessing_forkserver -m test_shared_memory_cleaned_after_process_termination
== CPython 3.11.0a0 (heads/main:09b4ad11f3, Sep 15 2021, 20:50:50) [Clang 11.0.0 (clang-1100.0.33.16)]
== macOS-10.14.6-x86_64-i386-64bit little-endian
== cwd: /Users/sobolev/Desktop/cpython/build/test_python_24281æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.80 Run tests sequentially
0:00:00 load avg: 1.80 [1/1] test_multiprocessing_forkserver
test_shared_memory_cleaned_after_process_termination (test.test_multiprocessing_forkserver.WithProcessesTestSharedMemory) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.732s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 1.5 sec
Tests result: SUCCESS

    My other highly-related PR where I refactor several SharedMemory tests: #28294

    @ambv
    Copy link
    Contributor

    ambv commented Sep 21, 2021

    New changeset f604cf1 by Nikita Sobolev in branch 'main':
    bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377)
    f604cf1

    @miss-islington
    Copy link
    Contributor

    New changeset 09e5016 by Miss Islington (bot) in branch '3.10':
    bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377)
    09e5016

    @vstinner
    Copy link
    Member Author

    New changeset 65268ab by Miss Islington (bot) in branch '3.9':
    bpo-45209: fix UserWarning: resource_tracker in test_multiprocessing (GH-28377) (GH-28500)
    65268ab

    @vstinner
    Copy link
    Member Author

    Thanks Nikita Sobolev for the fix.

    @vstinner vstinner added 3.9 only security fixes 3.10 only security fixes labels Sep 21, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @sharewax sharewax mentioned this issue Aug 18, 2022
    Open
    @KrisThielemans KrisThielemans mentioned this issue Oct 30, 2022
    Closed
    2 tasks
    @cisaacstern cisaacstern mentioned this issue Jul 21, 2023
    Open
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @ambv @vstinner @sobolevn @miss-islington