classification
Title: resource.setrlimit doesn't respect -1
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Paul.Price, docs@python, kushal.das, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2013-03-13 15:18 by Paul.Price, last changed 2013-04-20 18:08 by Paul.Price. This issue is now closed.

Files
File name Uploaded Description Edit
0001-Docs-clarify-use-of-1-for-resource.setrlimit-Issue17.patch Paul.Price, 2013-03-21 15:44
setrlimit_doc.patch r.david.murray, 2013-03-21 20:15
setrlimit_doc.patch r.david.murray, 2013-03-21 20:55
setrlimit_doc.patch r.david.murray, 2013-03-21 21:43
setrlimit_doc.patch r.david.murray, 2013-03-21 23:42
Messages (21)
msg184086 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-13 15:18
The docs for resource.setrlimit (http://docs.python.org/2.7/library/resource.html#resource.setrlimit) state: "The limits argument must be a tuple (soft, hard) of two integers describing the new limits. A value of -1 can be used to specify the maximum possible upper limit."

On Mac OSX (10.7.5) with Python 2.7.3 (built fresh), I get the following behaviour:


$ PATH=$HOME/test/bin:$PATH DYLD_LIBRARY_PATH=$HOME/test/lib:$DYLD_LIBRARY_PATH python
Python 2.7.3 (default, Mar 13 2013, 11:02:56) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.9.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (10, -1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not allowed to raise maximum limit
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (10, 10))
>>> 


I get the same behaviour for the system python (2.7.3 built with GCC 4.6.3) in Ubuntu 12.04.1.
msg184093 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-13 17:27
Since it says "maximum possible limit", I think -1 is the maximum system limit, not the maximum value a particular process is allowed to use.  If that is true the error message is presumably accurate.  And if that's true it needs to be clarified in the docs...and perhaps an enhancement request added.  But my guess might be wrong, too.
msg184263 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-15 21:35
Module 'resource' is unix only.  What happens on 3.x or linux?

Given that the doc invites input of -1, it seems to me that raising an exception is wrong; -1 should be interpreted as a valid value.
msg184411 - (view) Author: Kushal Das (kushal.das) * (Python committer) Date: 2013-03-18 01:04
On Fedora 17, x86_64

./python 
Python 3.4.0a0 (default:fb50eb64e097, Feb 22 2013, 11:43:18) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
[92450 refs, 32257 blocks]
>>> resource.setrlimit(resource.RLIMIT_NOFILE, (10, -1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: not allowed to raise maximum limit
msg184465 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-18 16:19
The OSX manpage for setrlimit includes:

COMPATIBILITY
     setrlimit() now returns with errno set to EINVAL in places that histori-
     cally succeeded.  It no longer accepts "rlim_cur = RLIM_INFINITY" for
     RLIM_NOFILE.  Use "rlim_cur = min(OPEN_MAX, rlim_max)".

It's strange that this is not mentioned in the corresponding manpage on Ubuntu 12.04.

It seems the reason we can't use -1 for RLIMIT_NOFILE is because this requests 'unlimited' (RLIM_INFINITY), while there is always a limit on the number of open files (e.g., OPEN_MAX).

Looking at the code (Modules/resource.c), python is doing the sensible thing.  It seems that resource.setrlimit would have to be special-cased for RLIMIT_NOFILE to work as documented when specifying a "-1", but then python would be diverging from the behaviour of the underlying system call.

I therefore propose that the documentation simply be adjusted to note the observed behaviour (i.e., specifying -1 means 'unlimited', and does not work in general for RLIMIT_NOFILE).  I would be happy to provide a patch if this is desired.
msg184551 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-18 23:20
That would be great, thanks.
msg184877 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-21 15:44
Not sure how you want patches formatted, so I went for 'git format-patch'.

Also, this is my first attempt at writing ReST and my first attempt at writing docs for Python, so you may want to double-check I didn't screw up the syntax or style.  Hopefully this doesn't cause more work than writing it yourself....
msg184878 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-21 15:47
P.S. This is relative to the 'default' branch in the public cpython.
msg184902 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 20:15
Thanks for the patch.  You got the patch mechanics correct.

I did a little experimenting, and at least on my linux box, the exception is raised for any(*) resource whose hard limit is not unlimited.  I'm attaching a revised patch that expresses this.  Do you think this provides sufficient clarification?

One question I have relates to the use of the word 'infinite'.  In the shell, the term 'unlimited' is used, so I'm thinking that's what most people will be familiar with.  On the other hand, the man page for the syscall we are wrapping uses the term 'infinite', so one can argue that it is more correct.  I'm ambivalent about which term to use.

(*) I tested this on NO_PROC, since that's the only other one my shell has a hard limit for, and by setting a hard limit via the shell ulimit command and then using (-1, -1) in the setrlimit call.
msg184903 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-21 20:19
That's good; it doesn't have what I added to the description of RLIMIT_NOFILE, but perhaps you chose to leave that out on purpose.

Since both "unlimited" and "infinite" are both used in different contexts, perhaps we should use both, e.g., "infinite/unlimited"?
msg184904 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 20:24
infinite/unlimited sounds like a good compromise :)

Yes, I left NOFILE out on purpose, because the issue applies to any resource that is not unlimited for whatever reason.
msg184908 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 20:55
Here's another attempt, using infinite/unlimited .  I noticed that the following paragraph talks specifically about the error raised, so I moved the explanation to that paragraph.
msg184910 - (view) Author: Paul Price (Paul.Price) Date: 2013-03-21 21:13
You missed out an "is":

   Raises :exc:`ValueError` if an invalid resource is specified, if the new soft
   limit exceeds the hard limit, or if a process tries to raise its hard limit
   (unless the process has an effective UID of super-user).  Specifying an
   infinite/unlmited limit when the hard or system limit for that resource **is** not
   unlimited will result in a :exc:`ValueError`.  ``setrlimit`` may also raise
    :exc:`error` if the underlying system call fails.
msg184911 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-21 21:17
in-finite literally means un-limited or un-bounded. However, 'infinte' has also taken on the meaning of 'actual infinity'. So in my algorithms book, when 'actual infinities' are not in play. I describe the set of counts as unbounded. I suggest you go with just 'unlimited'. I believe constructions such as and/or (which some style guides discourage. because they are a bit ugly) should suggest some contrast or difference, rather than exact synonyms but I could be wrong.
msg184912 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 21:33
Terry: well, we're trying to deal with the fact that the setrlimit man page says:

       The  value  RLIM_INFINITY  denotes  no limit on a resource (both in the
       structure returned by getrlimit() and in the structure passed to  setr-
       limit()).

Oh.  RLIM_INFINITY is defined in the module.  Let me update the patch again :)
msg184914 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 21:43
Here's a patch that adds RLIMIT_INFINITY and uses it.  This constant exists in 2.7 and Python3, but has not been previously documented.
msg184917 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-21 22:09
Both old and new have this sentence:
"Raises :exc:`ValueError` if an invalid resource is specified, if the new soft limit exceeds the hard limit, or if a process tries to raise its hard limit (unless the process has an effective UID of super-user)."

This strikes me as a bit awkward and ambiguous. As a naive reader that does not already know the facts, it is unclear whether the (unless...) applies to the overall sentence (Raise...) or just the last clause (if a process...). I would try moving the 'unless...' either before the whole sentence or before the clause, depending on which is true.
msg184926 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-21 23:42
How about this.
msg187442 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-04-20 17:40
New changeset 186f6bb3e46a by R David Murray in branch '3.3':
#17409: Document RLIM_INFINITY and use it to clarify the setrlimit docs.
http://hg.python.org/cpython/rev/186f6bb3e46a

New changeset 9c4db76d073e by R David Murray in branch '2.7':
#17409: Document RLIM_INFINITY and use it to clarify the setrlimit docs.
http://hg.python.org/cpython/rev/9c4db76d073e

New changeset f1d95b0ab66e by R David Murray in branch 'default':
Merge #17409: Document RLIM_INFINITY and use it to clarify the setrlimit docs.
http://hg.python.org/cpython/rev/f1d95b0ab66e
msg187443 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-20 17:41
There being no objection :) I've committed the patch.
msg187447 - (view) Author: Paul Price (Paul.Price) Date: 2013-04-20 18:08
Thanks!
History
Date User Action Args
2013-04-20 18:08:06Paul.Pricesetmessages: + msg187447
2013-04-20 17:41:52r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg187443

stage: commit review -> resolved
2013-04-20 17:40:57python-devsetnosy: + python-dev
messages: + msg187442
2013-03-22 01:37:36terry.reedysetstage: commit review
2013-03-21 23:42:15r.david.murraysetfiles: + setrlimit_doc.patch

messages: + msg184926
2013-03-21 22:09:29terry.reedysetmessages: + msg184917
2013-03-21 21:43:22r.david.murraysetfiles: + setrlimit_doc.patch

messages: + msg184914
2013-03-21 21:33:51r.david.murraysetmessages: + msg184912
2013-03-21 21:17:30terry.reedysetmessages: + msg184911
2013-03-21 21:13:18Paul.Pricesetmessages: + msg184910
2013-03-21 20:55:51r.david.murraysetfiles: + setrlimit_doc.patch

messages: + msg184908
2013-03-21 20:24:02r.david.murraysetmessages: + msg184904
2013-03-21 20:19:22Paul.Pricesetmessages: + msg184903
2013-03-21 20:15:09r.david.murraysetfiles: + setrlimit_doc.patch

messages: + msg184902
2013-03-21 15:47:12Paul.Pricesetmessages: + msg184878
2013-03-21 15:44:13Paul.Pricesetfiles: + 0001-Docs-clarify-use-of-1-for-resource.setrlimit-Issue17.patch
keywords: + patch
messages: + msg184877
2013-03-18 23:20:19r.david.murraysetversions: + Python 3.2, Python 3.3, Python 3.4
nosy: + docs@python

messages: + msg184551

assignee: docs@python
components: + Documentation
2013-03-18 16:19:22Paul.Pricesetmessages: + msg184465
2013-03-18 01:04:40kushal.dassetnosy: + kushal.das
messages: + msg184411
2013-03-15 21:35:50terry.reedysetnosy: + terry.reedy
messages: + msg184263
2013-03-13 17:27:11r.david.murraysetnosy: + r.david.murray
messages: + msg184093
2013-03-13 15:18:35Paul.Pricecreate