Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe() #87758

Closed
vstinner opened this issue Mar 22, 2021 · 13 comments
Closed

test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe() #87758

vstinner opened this issue Mar 22, 2021 · 13 comments
Assignees
@gpshead
Labels
3.10 only security fixes OS-mac tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

BPO 43592
Nosy @gpshead, @ronaldoussoren, @vstinner, @ned-deily, @ambv, @lielfr, @miss-islington
PRs
  • bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest #29127
  • [3.10] bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) #29144
  • [3.9] bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) #29145
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-10-22.04:49:25.641>
created_at = <Date 2021-03-22.10:33:02.909>
labels = ['OS-mac', 'type-bug', 'tests', '3.10']
title = 'test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe()'
updated_at = <Date 2021-10-22.04:49:25.640>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-10-22.04:49:25.640>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-10-22.04:49:25.641>
closer = 'gregory.p.smith'
components = ['macOS', 'Tests']
creation = <Date 2021-03-22.10:33:02.909>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43592
keywords = ['patch']
message_count = 13.0
messages = ['389301', '389303', '389334', '392245', '400835', '400836', '400837', '400838', '401569', '404665', '404714', '404719', '404720']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'ronaldoussoren', 'vstinner', 'ned.deily', 'lukasz.langa', 'lielfr', 'miss-islington']
pr_nums = ['29127', '29144', '29145']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43592'
versions = ['Python 3.10']
    @vstinner
    Copy link
    Member Author

    x86-64 macOS 3.x:
    https://buildbot.python.org/all/#/builders/366/builds/969

    Build triggered by the commit 88d9983 which changes PyImport_Import().

    The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a limit of 256 file descriptors.

    The latest successful build is build 968, whereas the the RLIMIT_NOFILE resource soft limit was also set to 256:
    https://buildbot.python.org/all/#/builders/366/builds/968

    test.pythoninfo:

    • os.uname: posix.uname_result(sysname='Darwin', nodename='mattb-mbp2', release='19.6.0', version='Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64', machine='x86_64')
    • sysconfig[HOST_GNU_TYPE]: x86_64-apple-darwin19.6.0
    • platform.platform: macOS-10.15.7-x86_64-i386-64bit
    • resource.RLIMIT_NOFILE: (256, 9223372036854775807)

    FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

    stdout:
    ---

    ---

    stderr:
    ---

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe
    fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-eik0n2aq': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-g7oeb4aw': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-tiabsvgr': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ykag01b2': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-nl2kdidn': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

    @vstinner vstinner added 3.10 only security fixes tests Tests in the Lib/test dir labels Mar 22, 2021
    @vstinner
    Copy link
    Member Author

    test_importlib only fails *randomly* on x86-64 macOS 3.x.

    For example, the build 970 is a success (but orange): "Tests result: FAILURE then SUCCESS". test_importlib failed with too many open files, and then passed.

    @ned-deily
    Copy link
    Member

    FWIW, I don't recall ever seeing a "too many open files" failure for test_importlib on any of my macOS systems.

    @lielfr
    Copy link
    Mannequin

    lielfr mannequin commented Apr 28, 2021

    I have the same error when running the tests on my machine (Same macOS version).

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 1, 2021

    x86-64 macOS 3.10:
    https://buildbot.python.org/all/#/builders/681/builds/345

    ======================================================================
    FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py']

    stdout:
    ---

    ---

    stderr:
    ---

    Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 53, in _launch
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 120 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-klnfwvgv': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-phgypx9f': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 1, 2021

    x86-64 macOS 3.10: https://buildbot.python.org/all/#/builders/681/builds/345

    On this machine, RLIMIT_NOFILE soft limit is 256.

    test.pythoninfo:

    resource.RLIMIT_NOFILE: (256, 9223372036854775807)
    platform.platform: macOS-11.5.2-x86_64-i386-64bit
    socket.hostname: mattb-mbp2

    All limits:

    resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_CORE: (0, 9223372036854775807)
    resource.RLIMIT_CPU: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_DATA: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_FSIZE: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_MEMLOCK: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_NOFILE: (256, 9223372036854775807)
    resource.RLIMIT_NPROC: (2784, 4176)
    resource.RLIMIT_RSS: (9223372036854775807, 9223372036854775807)
    resource.RLIMIT_STACK: (67104768, 67104768)
    resource.pagesize: 4096

    @ambv
    Copy link
    Contributor

    ambv commented Sep 1, 2021

    Try setting the following:

    $ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ sudo ulimit -n 65536

    @ambv
    Copy link
    Contributor

    ambv commented Sep 1, 2021

    Before you do it, check the defaults with:

    $ launchctl limit maxfiles

    I'm curious.

    @vstinner
    Copy link
    Member Author

    FYI the test is still failing with the same error on the billenstein-macos builder worker, x86-64 macOS 3.10:
    https://buildbot.python.org/all/#/builders/681/builds/370

    "OSError: [Errno 24] Too many open files"

    @gpshead
    Copy link
    Member

    gpshead commented Oct 21, 2021

    macOS often has an unreasonably low default max number of file descriptors. This is easy to raise from user code, it's a soft limit.

    #29127

    is out to try that from regrtest itself.

    @gpshead gpshead added the OS-mac label Oct 21, 2021
    @gpshead gpshead self-assigned this Oct 21, 2021
    @gpshead gpshead added type-bug An unexpected behavior, bug, or error OS-mac labels Oct 21, 2021
    @gpshead gpshead self-assigned this Oct 21, 2021
    @gpshead gpshead added the type-bug An unexpected behavior, bug, or error label Oct 21, 2021
    @gpshead
    Copy link
    Member

    gpshead commented Oct 22, 2021

    New changeset 843b890 by Gregory P. Smith in branch 'main':
    bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
    843b890

    @miss-islington
    Copy link
    Contributor

    New changeset 8f6aa48 by Miss Islington (bot) in branch '3.10':
    bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
    8f6aa48

    @miss-islington
    Copy link
    Contributor

    New changeset 216c040 by Miss Islington (bot) in branch '3.9':
    bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127)
    216c040

    @gpshead gpshead closed this as completed Oct 22, 2021
    @gpshead gpshead closed this as completed Oct 22, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @gpshead gpshead

    Labels
    3.10 only security fixes OS-mac tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    5 participants
    @ambv @gpshead @vstinner @ned-deily @miss-islington