➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()
Type: behavior Stage: commit review
Components: macOS, Tests Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: gregory.p.smith, lielfr, lukasz.langa, miss-islington, ned.deily, ronaldoussoren, vstinner
Priority: normal Keywords: patch

Created on 2021-03-22 10:33 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 29127 merged gregory.p.smith, 2021-10-21 21:20
PR 29144 merged miss-islington, 2021-10-22 01:22
PR 29145 merged miss-islington, 2021-10-22 01:22
Messages (13)
msg389301 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-22 10:33 
x86-64 macOS 3.x:
https://buildbot.python.org/all/#/builders/366/builds/969

Build triggered by the commit 88d9983b561cd59e5f186d98227de0c1a022b498 which changes PyImport_Import().


The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a limit of 256 file descriptors.


The latest successful build is build 968, whereas the the RLIMIT_NOFILE resource soft limit was also set to 256:
https://buildbot.python.org/all/#/builders/366/builds/968


test.pythoninfo:

* os.uname: posix.uname_result(sysname='Darwin', nodename='mattb-mbp2', release='19.6.0', version='Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64', machine='x86_64')
* sysconfig[HOST_GNU_TYPE]: x86_64-apple-darwin19.6.0
* platform.platform: macOS-10.15.7-x86_64-i386-64bit
* resource.RLIMIT_NOFILE: (256, 9223372036854775807)


FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe
    fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-eik0n2aq': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-g7oeb4aw': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-tiabsvgr': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ykag01b2': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-nl2kdidn': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---
msg389303 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-22 10:51 
test_importlib only fails *randomly* on x86-64 macOS 3.x.

For example, the build 970 is a success (but orange): "Tests result: FAILURE then SUCCESS". test_importlib failed with too many open files, and then passed.
msg389334 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-22 18:14 
FWIW, I don't recall ever seeing a "too many open files" failure for test_importlib on any of my macOS systems.
msg392245 - (view) Author: Liel Fridman (lielfr) * Date: 2021-04-28 18:54 
I have the same error when running the tests on my machine (Same macOS version).
msg400835 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-01 13:49 
x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/345

======================================================================
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

stdout:
---

---

stderr:
---
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 53, in _launch
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 120 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-klnfwvgv': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-phgypx9f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---
msg400836 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-01 13:54 
> x86-64 macOS 3.10: https://buildbot.python.org/all/#/builders/681/builds/345

On this machine, RLIMIT_NOFILE soft limit is 256.

test.pythoninfo:

resource.RLIMIT_NOFILE: (256, 9223372036854775807)
platform.platform: macOS-11.5.2-x86_64-i386-64bit
socket.hostname: mattb-mbp2

All limits:

resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_CORE: (0, 9223372036854775807)
resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_DATA: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_MEMLOCK: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_NOFILE: (256, 9223372036854775807)
resource.RLIMIT_NPROC: (2784, 4176)
resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
resource.RLIMIT_STACK: (67104768, 67104768)
resource.pagesize: 4096
msg400837 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-01 13:56 
Try setting the following:

$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ sudo ulimit -n 65536
msg400838 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-01 14:10 
Before you do it, check the defaults with:

$ launchctl limit maxfiles

I'm curious.
msg401569 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-10 11:02 
FYI the test is still failing with the same error on the billenstein-macos builder worker, x86-64 macOS 3.10:
https://buildbot.python.org/all/#/builders/681/builds/370

"OSError: [Errno 24] Too many open files"
msg404665 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-21 21:20 
macOS often has an unreasonably low default max number of file descriptors.  This is easy to raise from user code, it's a soft limit.

https://github.com/python/cpython/pull/29127

is out to try that from regrtest itself.
msg404714 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-22 01:22 
New changeset 843b890334ca30cf6af27dffe29cecd06b49f7d9 by Gregory P. Smith in branch 'main':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/843b890334ca30cf6af27dffe29cecd06b49f7d9
msg404719 - (view) Author: miss-islington (miss-islington) Date: 2021-10-22 04:47 
New changeset 8f6aa48cb2dc827a2cb76e35e91bf02d099875c5 by Miss Islington (bot) in branch '3.10':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/8f6aa48cb2dc827a2cb76e35e91bf02d099875c5
msg404720 - (view) Author: miss-islington (miss-islington) Date: 2021-10-22 04:48 
New changeset 216c040bb1fdf73e78d67ab82a43563d7593f874 by Miss Islington (bot) in branch '3.9':
bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
https://github.com/python/cpython/commit/216c040bb1fdf73e78d67ab82a43563d7593f874
History
Date User Action Args
2022-04-11 14:59:43adminsetgithub: 87758
2021-10-22 04:49:25gregory.p.smithsetstatus: open -> closed
resolution: fixed
stage: patch review -> commit review
2021-10-22 04:48:47miss-islingtonsetmessages: + msg404720
2021-10-22 04:47:18miss-islingtonsetmessages: + msg404719
2021-10-22 01:22:38miss-islingtonsetpull_requests: + pull_request27421
2021-10-22 01:22:34miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request27420
2021-10-22 01:22:34gregory.p.smithsetmessages: + msg404714
2021-10-21 21:20:23gregory.p.smithsetkeywords: + patch
pull_requests: + pull_request27414
2021-10-21 21:20:01gregory.p.smithsetassignee: gregory.p.smith
type: behavior
components: + macOS

nosy: + gregory.p.smith, ronaldoussoren
messages: + msg404665
stage: patch review
2021-09-10 11:02:21vstinnersetmessages: + msg401569
2021-09-01 14:10:58lukasz.langasetmessages: + msg400838
2021-09-01 13:56:08lukasz.langasetnosy: + lukasz.langa
messages: + msg400837
2021-09-01 13:54:52vstinnersetmessages: + msg400836
2021-09-01 13:49:26vstinnersetmessages: + msg400835
2021-04-28 18:54:55lielfrsetnosy: + lielfr
messages: + msg392245
2021-03-22 18:14:29ned.deilysetnosy: + ned.deily
messages: + msg389334
2021-03-22 10:51:31vstinnersetmessages: + msg389303
2021-03-22 10:33:02vstinnercreate