classification
Title: test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()
Type: Stage:
Components: Tests Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: lielfr, lukasz.langa, ned.deily, vstinner
Priority: normal Keywords:

Created on 2021-03-22 10:33 by vstinner, last changed 2021-09-10 11:02 by vstinner.

Messages (9)
msg389301 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-22 10:33
x86-64 macOS 3.x:
https://buildbot.python.org/all/#/builders/366/builds/969

Build triggered by the commit 88d9983b561cd59e5f186d98227de0c1a022b498 which changes PyImport_Import().


The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a limit of 256 file descriptors.


The latest successful build is build 968, whereas the the RLIMIT_NOFILE resource soft limit was also set to 256:
https://buildbot.python.org/all/#/builders/366/builds/968


test.pythoninfo:

* os.uname: posix.uname_result(sysname='Darwin', nodename='mattb-mbp2', release='19.6.0', version='Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64', machine='x86_64')
* sysconfig[HOST_GNU_TYPE]: x86_64-apple-darwin19.6.0
* platform.platform: macOS-10.15.7-x86_64-i386-64bit
* resource.RLIMIT_NOFILE: (256, 9223372036854775807)


FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe
    fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-eik0n2aq': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-g7oeb4aw': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-tiabsvgr': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ykag01b2': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-nl2kdidn': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---
msg389303 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-22 10:51
test_importlib only fails *randomly* on x86-64 macOS 3.x.

For example, the build 970 is a success (but orange): "Tests result: FAILURE then SUCCESS". test_importlib failed with too many open files, and then passed.
msg389334 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-22 18:14
FWIW, I don't recall ever seeing a "too many open files" failure for test_importlib on any of my macOS systems.
msg392245 - (view) Author: Liel Fridman (lielfr) * Date: 2021-04-28 18:54
I have the same error when running the tests on my machine (Same macOS version).
msg400835 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-01 13:49
x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/345

======================================================================
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 53, in _launch
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 120 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-klnfwvgv': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-phgypx9f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---
msg400836 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-01 13:54
> x86-64 macOS 3.10: https://buildbot.python.org/all/#/builders/681/builds/345

On this machine, RLIMIT_NOFILE soft limit is 256.

test.pythoninfo:

resource.RLIMIT_NOFILE: (256, 9223372036854775807)
platform.platform: macOS-11.5.2-x86_64-i386-64bit
socket.hostname: mattb-mbp2

All limits:

resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CORE: (0, 9223372036854775807)
resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_DATA: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_MEMLOCK: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_NOFILE: (256, 9223372036854775807)
resource.RLIMIT_NPROC: (2784, 4176)
resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_STACK: (67104768, 67104768)
resource.pagesize: 4096
msg400837 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-01 13:56
Try setting the following:

$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ sudo ulimit -n 65536
msg400838 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-01 14:10
Before you do it, check the defaults with:

$ launchctl limit maxfiles

I'm curious.
msg401569 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-10 11:02
FYI the test is still failing with the same error on the billenstein-macos builder worker, x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/370

"OSError: [Errno 24] Too many open files"
History
Date User Action Args
2021-09-10 11:02:21vstinnersetmessages: + msg401569
2021-09-01 14:10:58lukasz.langasetmessages: + msg400838
2021-09-01 13:56:08lukasz.langasetnosy: + lukasz.langa
messages: + msg400837
2021-09-01 13:54:52vstinnersetmessages: + msg400836
2021-09-01 13:49:26vstinnersetmessages: + msg400835
2021-04-28 18:54:55lielfrsetnosy: + lielfr
messages: + msg392245
2021-03-22 18:14:29ned.deilysetnosy: + ned.deily
messages: + msg389334
2021-03-22 10:51:31vstinnersetmessages: + msg389303
2021-03-22 10:33:02vstinnercreate