Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open() #80523

Closed
xdegaye mannequin opened this issue Mar 18, 2019 · 10 comments
Closed

test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open() #80523

xdegaye mannequin opened this issue Mar 18, 2019 · 10 comments
Labels
3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Mar 18, 2019

BPO 36342
Nosy @brettcannon, @vstinner, @xdegaye, @miss-islington
PRs
  • bpo-36342: Fix test_multiprocessing in test_venv #12513
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-09-24.12:06:09.204>
created_at = <Date 2019-03-18.10:02:39.890>
labels = ['3.8', 'type-bug', 'tests']
title = 'test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()'
updated_at = <Date 2019-09-24.12:06:09.203>
user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-09-24.12:06:09.203>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-24.12:06:09.204>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-03-18.10:02:39.890>
creator = 'xdegaye'
dependencies = []
files = []
hgrepos = []
issue_num = 36342
keywords = ['patch']
message_count = 10.0
messages = ['338188', '338189', '338192', '338240', '338398', '338464', '338495', '338671', '344004', '353078']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'vstinner', 'xdegaye', 'miss-islington']
pr_nums = ['12513']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36342'
versions = ['Python 3.8']
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Mar 18, 2019

    test_multiprocessing fails on platforms lacking a functioning sem_open (Android for example) with:

    ======================================================================
    ERROR: test_multiprocessing (test.test_venv.BasicTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 317, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/data/local/tmp/python/tmp/tmpli0_hkdl/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' returned non-zero exit status 1.

    @xdegaye xdegaye mannequin added 3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Mar 18, 2019
    @vstinner
    Copy link
    Member

    Are you sure that multiprocessing is the issue? On bpo-35978, the test fails because of venv, not because of multiprocessing. Can it be the same issue?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Mar 18, 2019

    You are right, this is a duplicate of bpo-35978.
    Closing as duplicate.

    @xdegaye xdegaye mannequin closed this as completed Mar 18, 2019
    @xdegaye xdegaye mannequin changed the title test_multiprocessing fails on platforms lacking a functioning sem_open test_venv fails on Android with clang Mar 18, 2019
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Mar 18, 2019

    Too hasty again, remembering now why I linked this failure with sem_open :-(

    Re-opening this issue as the test does fail because Android lacks a functioning sem_open implementation. And tagging this issue as a dependency of bpo-35978 as we should wait for the solution on bpo-35978 before fixing this one, if it is still needed.

    The link between the failure and sem_open shows up when running the python statements of test_multiprocessing() in the interpreter on Android:

    generic_x86_64:/data/local/tmp/python $ python
Python 3.8.0a2+ (heads/bpo-36341-dirty:41f0b78cbf, Mar 18 2019, 10:43:05)
[Clang 3.8.275480 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> print(Pool(1).apply_async("Python".lower).get(3))
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python3.8/lib-dynload/_multiprocessing.cpython-38dm.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 118, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 166, in __init__
    self._setup_queues()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 300, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
>>>

    @xdegaye xdegaye mannequin reopened this Mar 18, 2019
    @brettcannon
    Copy link
    Member

    So this is a test_multiprocessing issue? If it is then the title is misleading.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Mar 20, 2019

    It is a test_venv issue related to bpo-32126 and also related to the other issues listed in bpo-32126.

    Changing the title as it is ambiguous indeed.

    check_output() in test_venv.py does not print the full stack trace as it is done by test_asyncio in bpo-32126, is this another issue ?

    @xdegaye xdegaye mannequin changed the title test_venv fails on Android with clang test_venv failure when the platform lacks a functional sem_open() Mar 20, 2019
    @brettcannon
    Copy link
    Member

    I guess my confusion comes from the fact that test_venv in isolation is a totally fine test suite, it just fails when test_multiprocessing runs it due to something multiprocessing-related.

    I've tried tweaking the title to reflect the fact that it's a test_venv failure triggered by test_multiprocessing.

    @brettcannon brettcannon changed the title test_venv failure when the platform lacks a functional sem_open() test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open() Mar 20, 2019
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Mar 23, 2019

    Removing the bpo-35978 dependency as changeset 8bba81f that fixes bpo-35978 only skips test_multiprocessing when test_venv is run from a venv.

    @miss-islington
    Copy link
    Contributor

    New changeset 5437ccc by Miss Islington (bot) (xdegaye) in branch 'master':
    bpo-36342: Fix test_multiprocessing in test_venv (GH-12513)
    5437ccc

    @vstinner
    Copy link
    Member

    It seems like this issue has been fixed in the master branch. Reopen the issue if you consider that the fix should be backported.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @brettcannon @vstinner @miss-islington