classification
Title: test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, miss-islington, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2019-03-18 10:02 by xdegaye, last changed 2019-09-24 12:06 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12513 merged xdegaye, 2019-03-23 13:10
Messages (10)
msg338188 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-18 10:02
test_multiprocessing fails on platforms lacking a functioning sem_open (Android for example) with:

======================================================================
ERROR: test_multiprocessing (test.test_venv.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 317, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/data/local/tmp/python/tmp/tmpli0_hkdl/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' returned non-zero exit status 1.
msg338189 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 10:05
Are you sure that multiprocessing is the issue? On bpo-35978, the test fails because of venv, not because of multiprocessing. Can it be the same issue?
msg338192 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-18 10:30
You are right, this is a duplicate of bpo-35978.
Closing as duplicate.
msg338240 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-18 15:24
Too hasty again, remembering now why I linked this failure with sem_open :-(

Re-opening this issue as the test does fail because Android lacks a functioning sem_open implementation. And tagging this issue as a dependency of bpo-35978 as we should wait for the solution on bpo-35978 before fixing this one, if it is still needed.

The link between the failure and sem_open shows up when running the python statements of test_multiprocessing() in the interpreter on Android:

generic_x86_64:/data/local/tmp/python $ python
Python 3.8.0a2+ (heads/bpo-36341-dirty:41f0b78cbf, Mar 18 2019, 10:43:05)
[Clang 3.8.275480 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> print(Pool(1).apply_async("Python".lower).get(3))
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python3.8/lib-dynload/_multiprocessing.cpython-38dm.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 118, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 166, in __init__
    self._setup_queues()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 300, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
>>>
msg338398 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-03-19 18:57
So this is a test_multiprocessing issue? If it is then the title is misleading.
msg338464 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-20 12:54
It is a test_venv issue related to bpo-32126 and also related to the other issues listed in bpo-32126.

Changing the title as it is ambiguous indeed.

check_output() in test_venv.py does not print the full stack trace as it is done by test_asyncio in bpo-32126, is this another issue ?
msg338495 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-03-20 16:55
I guess my confusion comes from the fact that test_venv in isolation is a totally fine test suite, it just fails when test_multiprocessing runs it due to something multiprocessing-related.

I've tried tweaking the title to reflect the fact that it's a test_venv failure triggered by test_multiprocessing.
msg338671 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-23 11:55
Removing the bpo-35978 dependency as changeset 8bba81fd55873148c65b7d0e6a6effbd63048c76 that fixes bpo-35978 only skips test_multiprocessing when test_venv is run from a venv.
msg344004 - (view) Author: miss-islington (miss-islington) Date: 2019-05-30 21:42
New changeset 5437ccca1424e415a938c583df43d8cc74047d16 by Miss Islington (bot) (xdegaye) in branch 'master':
bpo-36342: Fix test_multiprocessing in test_venv (GH-12513)
https://github.com/python/cpython/commit/5437ccca1424e415a938c583df43d8cc74047d16
msg353078 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-24 12:06
It seems like this issue has been fixed in the master branch. Reopen the issue if you consider that the fix should be backported.
History
Date User Action Args
2019-09-24 12:06:09vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg353078

stage: patch review -> resolved
2019-05-30 21:42:35miss-islingtonsetnosy: + miss-islington
messages: + msg344004
2019-03-23 13:10:15xdegayesetkeywords: + patch
stage: patch review
pull_requests: + pull_request12464
2019-03-23 11:55:05xdegayesetdependencies: - test_venv fails in Travis with GCC
messages: + msg338671
2019-03-20 16:55:28brett.cannonsetmessages: + msg338495
title: test_venv failure when the platform lacks a functional sem_open() -> test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()
2019-03-20 12:54:17xdegayesetmessages: + msg338464
title: test_venv fails on Android with clang -> test_venv failure when the platform lacks a functional sem_open()
2019-03-19 18:57:34brett.cannonsetnosy: + brett.cannon
messages: + msg338398
2019-03-18 15:24:27xdegayesetstatus: closed -> open
messages: + msg338240

dependencies: + test_venv fails in Travis with GCC
resolution: duplicate -> (no value)
stage: resolved -> (no value)
2019-03-18 10:30:37xdegayesetstatus: open -> closed
title: test_multiprocessing fails on platforms lacking a functioning sem_open -> test_venv fails on Android with clang
messages: + msg338192

resolution: duplicate
stage: needs patch -> resolved
2019-03-18 10:05:30vstinnersetnosy: + vstinner
messages: + msg338189
2019-03-18 10:02:39xdegayecreate