classification
Title: test_selectors test_above_fd_setsize cases fail on OS X due to infinite hard limit
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, neologix, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2013-09-07 21:32 by ned.deily, last changed 2013-09-08 12:58 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
selectors_osx.diff neologix, 2013-09-08 08:37 review
Messages (8)
msg197189 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-07 21:32
======================================================================
ERROR: test_above_fd_setsize (test.test_selectors.PollSelectorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/test/support/__init__.py", line 485, in wrapper
    return func(*args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/test/test_selectors.py", line 312, in test_above_fd_setsize
    resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))
ValueError: current limit exceeds maximum limit

======================================================================
ERROR: test_above_fd_setsize (test.test_selectors.KqueueSelectorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/test/support/__init__.py", line 485, in wrapper
    return func(*args, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/test/test_selectors.py", line 312, in test_above_fd_setsize
    resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))
ValueError: current limit exceeds maximum limit

----------------------------------------------------------------------
Ran 58 tests in 8.080s

FAILED (errors=2, skipped=12)

Looking at the OS X man page for setrlimit(2), it appears the test's strategy of trying to set the soft RLIMIT_NOFILE to the hard RLIMIT_NOFILE ceiling will fail on OS X (at least) if the hard limit is infinite:

>>> import resource
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(2560, 9223372036854775807)

From the man page:
COMPATIBILITY
     setrlimit() now returns with errno set to EINVAL in places that historically
     succeeded.  It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE.
     Use "rlim_cur = min(OPEN_MAX, rlim_max)".

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/setrlimit.2.html
msg197194 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-09-07 22:32
See also issue issue 17409.

The code isn't setting it to RLIM_INFINITY explicitly, though, so this must mean that OSX is reporting an infinite hard limit when the hard limit is not in fact infinite.  Seems like this is an OSX bug that we will have to work around somehow.  Maybe in getrlimit, by doing the max(OPEN_MAX, rlim_max) dance suggested by the osx man page.
msg197231 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-09-08 08:37
> R. David Murray added the comment:
>
> See also issue issue 17409.
>
> The code isn't setting it to RLIM_INFINITY explicitly, though, so this must mean that OSX is reporting an infinite hard limit when the hard limit is not in fact infinite.  Seems like this is an OSX bug that we will have to work around somehow.

Indeed.
I saw this while testing on "custom" buildbots, and opted for the
simplest solution: I added a @require_mac_version decorator, hoping
this would be solved in recent OSX versions. Apparently not :)

As a simple check, does the following work on OSX ?
>>> limit = resource.getrlimit(resource.RLIMIT_NOFILE)
>>> resource.setrlimit(resource.RLIMIT_NOFILE, limit)

Does the attached patch solve this?
msg197232 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-09-08 08:59
"As a simple check, does the following work on OSX ?
 >>> limit = resource.getrlimit(resource.RLIMIT_NOFILE)
 >>> resource.setrlimit(resource.RLIMIT_NOFILE, limit)"

It doesn't produce an exception.

"Does the attached patch solve this?"

With the patch, test_selectors no longer fails.
msg197234 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-09-08 09:35
New changeset 9ba1432fdc5a by Charles-François Natali in branch 'default':
Issue #18963: Fix test_selectors.test_above_fd_setsize on OS X, where the
http://hg.python.org/cpython/rev/9ba1432fdc5a
msg197244 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-09-08 10:21
I knew this wouldn't be so easy with OS X...

http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6916/steps/test/logs/stdio

======================================================================
ERROR: test_above_fd_setsize (test.test_selectors.PollSelectorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py",
line 56, in testPartExecutor
    yield
  File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/case.py",
line 528, in doCleanups
    function(*args, **kwargs)
ValueError: not allowed to raise maximum limit

----------------------------------------------------------------------

Basically, we can't restore RLIMIT_NOFILE to its original value (from
the cleanup callback): since this works on Ned's machine, I assume
this has been fixed in recent OS X versions.
So I'll restore the requires_mac_vers() decorator.
msg197246 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-09-08 10:36
New changeset fa735675e485 by Charles-François Natali in branch 'default':
Issue #18963: skip test_selectors.test_above_fd_setsize on older OS X versions.
http://hg.python.org/cpython/rev/fa735675e485
msg197258 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-09-08 12:58
Alright, it should be fixed now, thanks for the report.
History
Date User Action Args
2013-09-08 12:58:52neologixsetstatus: open -> closed
type: behavior
messages: + msg197258

resolution: fixed
stage: needs patch -> resolved
2013-09-08 10:36:15python-devsetmessages: + msg197246
2013-09-08 10:21:36neologixsetmessages: + msg197244
2013-09-08 09:35:11python-devsetnosy: + python-dev
messages: + msg197234
2013-09-08 08:59:14ned.deilysetmessages: + msg197232
2013-09-08 08:37:38neologixsetfiles: + selectors_osx.diff
keywords: + patch
messages: + msg197231
2013-09-07 22:32:21r.david.murraysetnosy: + r.david.murray
messages: + msg197194
2013-09-07 21:32:00ned.deilycreate