Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm #82558

Closed
mcepl mannequin opened this issue Oct 5, 2019 · 12 comments
Closed

test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm #82558

mcepl mannequin opened this issue Oct 5, 2019 · 12 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes topic-asyncio

Comments

@mcepl
Copy link
Mannequin

mcepl mannequin commented Oct 5, 2019

BPO 38377
Nosy @vstinner, @ned-deily, @mcepl, @asvetlov, @1st1, @miss-islington
PRs
  • bpo-38377 _multiprocessing.SemLock requires working /dev/shm on Linux. #16593
  • bpo-38377: multiprocessing.SemLock requires working /dev/shm on Linux. #19073
  • bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944
  • [3.9] bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) #20962
  • [3.8] bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962) #20966
  • bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS #20984
  • [3.9] bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984) #20987
  • [3.8] bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984) #20988
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-06-21.21:17:29.495>
created_at = <Date 2019-10-05.13:47:18.114>
labels = ['3.8', '3.9', '3.10', 'expert-asyncio']
title = 'test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm'
updated_at = <Date 2020-06-21.21:17:29.495>
user = 'https://github.com/mcepl'

    bugs.python.org fields:

    activity = <Date 2020-06-21.21:17:29.495>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-21.21:17:29.495>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2019-10-05.13:47:18.114>
creator = 'mcepl'
dependencies = []
files = []
hgrepos = []
issue_num = 38377
keywords = ['patch']
message_count = 12.0
messages = ['354010', '371809', '371819', '371826', '371828', '371833', '371836', '371886', '371891', '371892', '371893', '372021']
nosy_count = 6.0
nosy_names = ['vstinner', 'ned.deily', 'mcepl', 'asvetlov', 'yselivanov', 'miss-islington']
pr_nums = ['16593', '19073', '20944', '20962', '20966', '20984', '20987', '20988']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38377'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
    @mcepl
    Copy link
    Mannequin Author

    mcepl mannequin commented Oct 5, 2019

    ======================================================================
    ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestCGetEventLoop)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

    ======================================================================
    ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestPyGetEventLoop)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

    The problem is that in the OpenBuildService (the build system of SUSE distributions) /dev/shm is limited to:

    abuild@milic:/rpmbuild/BUILD/Python-3.8.0rc1> ls -ld /dev/shm
    drwxr-xr-x 1 root root 0 Oct 5 13:23 /dev/shm
    abuild@milic:    /rpmbuild/BUILD/Python-3.8.0rc1>

    Provided PR will skip test if /dev/shm doesn’t exist or it doesn’t sufficient permissions.

    @mcepl mcepl mannequin added 3.8 only security fixes 3.9 only security fixes topic-asyncio labels Oct 5, 2019
    @vstinner
    Copy link
    Member

    New changeset ddbeb2f by Victor Stinner in branch 'master':
    bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944)
    ddbeb2f

    @vstinner
    Copy link
    Member

    New changeset b1e7361 by Victor Stinner in branch '3.9':
    bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962)
    b1e7361

    @vstinner
    Copy link
    Member

    New changeset e805618 by Victor Stinner in branch '3.8':
    bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962) (GH-20966)
    e805618

    @vstinner
    Copy link
    Member

    Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

    x86-64 macOS 3.9:
    https://buildbot.python.org/all/#builders/696/builds/119

    FAIL: test_resource_tracker_reused (test.test_multiprocessing_forkserver.TestResourceTracker)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5219, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

    @ned-deily
    Copy link
    Member

    Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

    It does appear to be caused by your change. I did a quick check with the 3.8 branch. With that PR, test_multiprocessing_forkserver fails solidly. Backing up to the previous commit, test_multiprocessing_forkserver passes.

    @vstinner
    Copy link
    Member

    test_multiprocessing_forkserver.test_resource_tracker_reused() failed on 3.8, 3.9 and master buildbots. It looks like a real reproducible bug. I'm not sure how my change triggered this bug.

    @vstinner vstinner added the 3.10 only security fixes label Jun 18, 2020
    @vstinner
    Copy link
    Member

    From what I understood, the problem is that creating a SemLock has an impact on the resource tracker.

    The failing unit tests checks something about the resource tracker.

    The test runs a function in a child process. The function is defined in _test_multiprocessing.py. _test_multiprocessing.py calls skip_if_broken_multiprocessing_synchronize() which creates a SemLock(). At the end, the child process modifies the resource tracker because it creates a SemLock().

    There are different options to fix the issue:

    • Modify skip_if_broken_multiprocessing_synchronize() only only attempt to create a SemLock on Linux.
    • Modify skip_if_broken_multiprocessing_synchronize() to only check if the current user is allowed to write into /dev/shm/ directory: that would allow touching the resource tracker.
    • Modify the test to avoid improting _test_multiprocessing in the child process.

    I don't know why the test only fails on macOS.

    Note: I ignored macOS failure when I merged my PR because the macOS job of GitHub Action is failing randomly for at least 3 months and nobody attempted to fix it. So I learnt to ignore this job.

    @vstinner
    Copy link
    Member

    New changeset 3358da4 by Victor Stinner in branch 'master':
    bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
    3358da4

    @miss-islington
    Copy link
    Contributor

    New changeset ec9bc2d by Miss Islington (bot) in branch '3.8':
    bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
    ec9bc2d

    @miss-islington
    Copy link
    Contributor

    New changeset 1529322 by Miss Islington (bot) in branch '3.9':
    bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
    1529322

    @vstinner
    Copy link
    Member

    Ok, macOS buildbots are back to green, I close the issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes topic-asyncio
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @vstinner @ned-deily @miss-islington