This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_multiprocessing_pool_circular_import fails in M1 mac
Type: Stage: resolved
Components: macOS, Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, pablogsal, ronaldoussoren
Priority: normal Keywords:

Created on 2021-09-25 21:27 by pablogsal, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (5)
msg402648 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-25 21:27
======================================================================
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pablogsal/github/cpython/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Process return code is 1
command line: ['/Users/pablogsal/github/cpython/python.exe', '-X', 'faulthandler', '-I', '/Users/pablogsal/github/cpython/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/pablogsal/github/cpython/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/context.py", line 119, in Pool
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/pool.py", line 212, in __init__
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/multiprocessing/popen_spawn_posix.py", line 53, in _launch
    parent_r, child_w = os.pipe()
                        ^^^^^^^^^
  File "/Users/pablogsal/github/cpython/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files
/Users/pablogsal/github/cpython/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 65 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/pablogsal/github/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-m8lbung6': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/pablogsal/github/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-w8c_ci83': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/pablogsal/github/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-yzcaa23b': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/pablogsal/github/cpython/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vgij30u5': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---

----------------------------------------------------------------------
Ran 1428 tests in 2.752s

FAILED (failures=1, skipped=6, expected failures=1)
test test_importlib failed
test_importlib failed (1 failure)

== Tests result: FAILURE ==

6 tests OK.
msg402650 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-25 21:49
The same test sometimes hangs forever
msg402651 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-09-25 21:54
As a first step, try adding:

ulimit -n 1000

to the buildbot test step.
msg402653 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-25 22:12
> ulimit -n 1000

If i do this, the test hangs some times instead of crashing
msg402654 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-26 01:57
Seems that the issue went away when configuring the maxfiles with

$ launchctl limit maxfiles

instead of with "ulimit -n"
History
Date User Action Args
2022-04-11 14:59:50adminsetgithub: 89453
2021-09-26 01:57:00pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg402654

stage: resolved
2021-09-25 22:12:24pablogsalsetmessages: + msg402653
2021-09-25 21:54:33ned.deilysetmessages: + msg402651
2021-09-25 21:49:43pablogsalsetmessages: + msg402650
2021-09-25 21:27:51pablogsalcreate