This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: compileall fails when the platform lacks a functional sem_open()
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, pitrou
Priority: normal Keywords: patch

Created on 2019-11-19 15:43 by xdegaye, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 17300 closed xdegaye, 2019-11-20 20:48
PR 18047 open ZackerySpytz, 2020-01-18 06:52
Messages (3)
msg356974 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-11-19 15:43
See also the related issues:
#32126: [asyncio] test failure when the platform lacks a functional  sem_open()
#28668: instanciation of multiprocessing.Queue raises ImportError in test_logging

The test failure on android API 24:

generic_x86_64:/data/local/tmp/python $ python -m test -v test_compileall -m test_workers
== CPython 3.9.0a0 (heads/abifa-dirty:cf805c25e6, Nov 18 2019, 16:40:26) [Clang 8.0.2 (https://andro
id.googlesource.com/toolchain/clang 40173bab62ec7462
== Linux-3.10.0+-x86_64-with-libc little-endian
== cwd: /data/local/tmp/python/tmp/test_python_2579
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 Run tests sequentially
0:00:00 [1/1] test_compileall
test_workers (test.test_compileall.CommandLineTestsNoSourceEpoch) ... FAIL
test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch) ... FAIL

======================================================================
FAIL: test_workers (test.test_compileall.CommandLineTestsNoSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_py_compile.py", line 20, in wrapper
    return fxn(*args, **kwargs)
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 707, in test_workers        self.assertRunOK(self.directory, '-j', '0')
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 397, in assertRunOK
    rc, out, err = script_helper.assert_python_ok(
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 157, in assert_pyt
hon_ok
    return _assert_python(True, *args, **env_vars)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 143, in _assert_py
thon
    res.fail(cmd_line)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 70, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-S', '-m', 'compile
all', '/data/local/tmp/python/tmp/tmpc1hy_667', '-j', '0']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python
3.9/lib-dynload/_multiprocessing.cpython-39d.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 425, in <module>
    exit_status = int(not main())
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 403, in main
    if not compile_dir(dest, maxlevels, args.ddir,
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 91, in compile_dir
    with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required sync
hronization primitives needed will not function, see issue 3770.
---

======================================================================
FAIL: test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch)
======================================================================                      [38/374]
FAIL: test_workers (test.test_compileall.CommandLineTestsWithSourceEpoch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 707, in test_workers
    self.assertRunOK(self.directory, '-j', '0')
  File "/data/local/tmp/python/lib/python3.9/test/test_compileall.py", line 397, in assertRunOK
    rc, out, err = script_helper.assert_python_ok(
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 157, in assert_pyt
hon_ok
    return _assert_python(True, *args, **env_vars)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 143, in _assert_py
thon
    res.fail(cmd_line)
  File "/data/local/tmp/python/lib/python3.9/test/support/script_helper.py", line 70, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-S', '-m', 'compile
all', '/data/local/tmp/python/tmp/tmp3llbx9yv', '-j', '0']
stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python
3.9/lib-dynload/_multiprocessing.cpython-39d.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 425, in <module>
    exit_status = int(not main())
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 403, in main
    if not compile_dir(dest, maxlevels, args.ddir,
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 91, in compile_dir
    with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
---

----------------------------------------------------------------------

Ran 2 tests in 0.693s

FAILED (failures=2)
test test_compileall failed
test_compileall failed

== Tests result: FAILURE ==

1 test failed:
    test_compileall

Total duration: 943 ms
Tests result: FAILURE
msg357098 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-11-20 20:07
Changing the title to "compileall fails when the platform lacks a functional  sem_open()" as the problem lies in the compileall module itself.
Nosying Antoine as the author of issue #36786.


compileall fails on android API 24:

generic_x86_64:/data/local/tmp/python $ python -m compileall --workers=2 .
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python3.9/lib-dynload/_multiprocessing.cpython-39d.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/local/tmp/python/lib/python3.9/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 425, in <module>
    exit_status = int(not main())
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 403, in main
    if not compile_dir(dest, maxlevels, args.ddir,
  File "/data/local/tmp/python/lib/python3.9/compileall.py", line 91, in compile_dir
    with ProcessPoolExecutor(max_workers=workers) as executor:
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/data/local/tmp/python/lib/python3.9/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/context.py", line 67, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.9/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
msg358166 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-12-10 07:46
Not interested anymore in android stuff.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 83029
2020-01-18 06:56:03ZackerySpytzsetnosy: + ZackerySpytz
2020-01-18 06:52:47ZackerySpytzsetpull_requests: + pull_request17442
2019-12-10 07:53:59xdegayesetnosy: - xdegaye
2019-12-10 07:46:29xdegayesetnosy: pitrou, xdegaye
messages: + msg358166
2019-11-20 20:48:13xdegayesetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request16792
2019-11-20 20:07:05xdegayesettitle: test_compileall fails when the platform lacks a functional sem_open() -> compileall fails when the platform lacks a functional sem_open()
nosy: + pitrou

messages: + msg357098

components: + Library (Lib), - Tests
2019-11-19 15:43:33xdegayecreate