classification
Title: Add resource.prlimit
Type: enhancement Stage: committed/rejected
Components: Extension Modules Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Arfrever, christian.heimes, giampaolo.rodola, haypo, neologix, pitrou, python-dev, vajrasky
Priority: normal Keywords: patch

Created on 2012-12-02 11:14 by christian.heimes, last changed 2013-12-10 18:40 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
prlimit2.patch christian.heimes, 2013-06-24 13:11 review
Messages (20)
msg176783 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-12-02 11:14
Linux has an additional function to setrlimit and getrlimit which supports to query and/or modify resource limits of another process.

http://www.kernel.org/doc/man-pages/online/pages/man2/getrlimit.2.html

The patch implements resource.prlimit(pid, resource[, limits]) on all platforms that have a prlimit() function. I haven't included the regenerated configure script. You have to run autoreconf in order to test the new function.
msg176785 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-12-02 11:34
prlimit() needs glibc 2.13+ (thanks Arfrever).
msg191759 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-24 13:11
Updated patch, now raises PermissionError on EPERM.
msg192578 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-07 18:33
Does anybody want to review the code before I commit it?
msg193361 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2013-07-19 14:46
From my original post which unfortunately was ignored :( :  
http://mail.python.org/pipermail/python-ideas/2012-June/015323.html
...for completeness perhaps it also makes sense to expose Linux-specific RLIMIT constants:

RLIMIT_MSGQUEUE
RLIMIT_NICE
RLIMIT_RTPRIO
RLIMIT_RTTIME
RLIMIT_SIGPENDING
msg193367 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-07-19 16:30
Fedora 18, this is the result after applying your patch and execute your unit test:

[sky@localhost cpython]$ ./python Lib/test/test_resource.py 
test_args (__main__.ResourceTest) ... ok
test_fsize_enforced (__main__.ResourceTest) ... ok
test_fsize_ismax (__main__.ResourceTest) ... ok
test_fsize_toobig (__main__.ResourceTest) ... ok
test_getrusage (__main__.ResourceTest) ... ok
test_prlimit (__main__.ResourceTest) ... ERROR
test_setrusage_refcount (__main__.ResourceTest) ... ok

======================================================================
ERROR: test_prlimit (__main__.ResourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 131, in test_prlimit
    1, resource.RLIMIT_AS)
  File "/home/sky/Code/python/programming_language/cpython/Lib/unittest/case.py", line 614, in assertRaises
    return context.handle('assertRaises', callableObj, args, kwargs)
  File "/home/sky/Code/python/programming_language/cpython/Lib/unittest/case.py", line 150, in handle
    callable_obj(*args, **kwargs)
PermissionError: [Errno 1] Operation not permitted

----------------------------------------------------------------------
Ran 7 tests in 0.990s

FAILED (errors=1)
Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 143, in <module>
    test_main()
  File "Lib/test/test_resource.py", line 140, in test_main
    support.run_unittest(ResourceTest)
  File "/home/sky/Code/python/programming_language/cpython/Lib/test/support.py", line 1615, in run_unittest
    _run_suite(suite)
  File "/home/sky/Code/python/programming_language/cpython/Lib/test/support.py", line 1590, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 131, in test_prlimit
    1, resource.RLIMIT_AS)
  File "/home/sky/Code/python/programming_language/cpython/Lib/unittest/case.py", line 614, in assertRaises
    return context.handle('assertRaises', callableObj, args, kwargs)
  File "/home/sky/Code/python/programming_language/cpython/Lib/unittest/case.py", line 150, in handle
    callable_obj(*args, **kwargs)
PermissionError: [Errno 1] Operation not permitted

With sudo, I got this:

[sky@localhost cpython]$ sudo ./python Lib/test/test_resource.py 
[sudo] password for sky: 
test_args (__main__.ResourceTest) ... ok
test_fsize_enforced (__main__.ResourceTest) ... ok
test_fsize_ismax (__main__.ResourceTest) ... ok
test_fsize_toobig (__main__.ResourceTest) ... ok
test_getrusage (__main__.ResourceTest) ... ok
test_prlimit (__main__.ResourceTest) ... FAIL
test_setrusage_refcount (__main__.ResourceTest) ... ok

======================================================================
FAIL: test_prlimit (__main__.ResourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 131, in test_prlimit
    1, resource.RLIMIT_AS)
AssertionError: ValueError not raised by prlimit

----------------------------------------------------------------------
Ran 7 tests in 1.002s

FAILED (failures=1)
Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 143, in <module>
    test_main()
  File "Lib/test/test_resource.py", line 140, in test_main
    support.run_unittest(ResourceTest)
  File "/home/sky/Code/python/programming_language/cpython/Lib/test/support.py", line 1615, in run_unittest
    _run_suite(suite)
  File "/home/sky/Code/python/programming_language/cpython/Lib/test/support.py", line 1590, in _run_suite
    raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_resource.py", line 131, in test_prlimit
    1, resource.RLIMIT_AS)
AssertionError: ValueError not raised by prlimit
msg200737 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-21 09:42
Thanks for your tests. I may have to rework my test scenario a bit.
msg200901 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-22 09:22
New changeset 796c21e27a92 by Christian Heimes in branch 'default':
Issue #16595: Add prlimit() to resource module
http://hg.python.org/cpython/rev/796c21e27a92
msg200902 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-22 09:27
Let's see how the buildbots respond. I may have to disable some tests for ancient Kernel versions.
msg200906 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-22 09:39
+        self.assertRaises(PermissionError, resource.prlimit,
+                          1, resource.RLIMIT_AS)

Please skip this when run as root.

+        self.assertEqual(resource.prlimit(0, resource.RLIMIT_AS), (-1, -1))

What if it's not that value by default? Please fix the test to make it
more robust.
msg200910 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-22 09:55
Roundup bot hangs. I have addressed your concerns in http://hg.python.org/cpython/rev/8c77117f41a9
msg201203 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-10-25 04:52
The test is failing with ENOSYS on one of the buildbots:
"""
======================================================================
ERROR: test_prlimit (test.test_resource.ResourceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/test/test_resource.py", line 148, in test_prlimit
    1, resource.RLIMIT_AS)
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/unittest/case.py", line 689, in assertRaises
    return context.handle('assertRaises', callableObj, args, kwargs)
  File "/var/lib/buildslave/3.x.murray-gentoo/build/Lib/unittest/case.py", line 158, in handle
    callable_obj(*args, **kwargs)
OSError: [Errno 38] Function not implemented
"""

I wonder if we could add a @support.ignore_enosys decorator...
msg201206 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-25 06:31
New changeset 87d41a5a9077 by Christian Heimes in branch 'default':
Issue #16595: prlimit() needs Linux kernel 2.6.36+
http://hg.python.org/cpython/rev/87d41a5a9077
msg201207 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-25 06:33
The buildbot is Linux-2.6.35-vs2.3.0.36.32-gentoo-i686-Intel-R-_Core-TM-2_CPU_6600_@_2.40GHz-with-gentoo-2.1 but prlimit() requires 2.6.36+. I didn't expect to see a combination of glibc with prlimit() and Kernel without prlimit(). According to man prlimit it's not suppose to fail with ENOSYS, too.
msg201208 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-25 06:34
Or we should extend with supress(OSerror, errno=errno.ENOSYS): ... :-)

(Just kidding, ignored tests must be marked as skipped.)
msg201236 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-10-25 13:13
> I didn't expect to see a combination of glibc with prlimit() and Kernel without prlimit(). According to man prlimit it's not suppose to fail with ENOSYS, too.

Yeah, we've seen this several times on some buildbots.
Note that actually, any syscall can fail with ENOSYS: the configure
check just checks that the libc exposes the wrapper library, but when
the libc wrapper makes the actual syscall, the kernel can return
-ENOSYS if it doesn't implement the syscall.
msg203505 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-20 16:45
Tests are passing on all buildbots for quite some time now.
msg205841 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2013-12-10 18:27
Just received this report on psutil bug tracker:
https://code.google.com/p/psutil/issues/detail?id=455
It seems os.prlimit() is affected by the same problem:

>>> import resource
>>> resource.RLIM_INFINITY
-1
>>>
msg205842 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2013-12-10 18:28
s/os.prlimit/resource.prlimit
msg205843 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-10 18:40
How is that a problem? In any case, this shouldn't have anything to do with prlimit(), please open another issue.
History
Date User Action Args
2013-12-10 18:40:35pitrousetstatus: open -> closed
resolution: fixed
messages: + msg205843
2013-12-10 18:28:27giampaolo.rodolasetmessages: + msg205842
2013-12-10 18:27:41giampaolo.rodolasetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg205841
2013-11-20 16:45:33christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg203505

stage: commit review -> committed/rejected
2013-10-25 13:13:36neologixsetmessages: + msg201236
2013-10-25 06:34:59hayposetmessages: + msg201208
2013-10-25 06:33:23christian.heimessetmessages: + msg201207
2013-10-25 06:31:29python-devsetmessages: + msg201206
2013-10-25 04:52:07neologixsetnosy: + neologix
messages: + msg201203
2013-10-22 09:55:25christian.heimessetmessages: + msg200910
2013-10-22 09:39:19pitrousetstatus: pending -> open

messages: + msg200906
2013-10-22 09:27:12christian.heimessetstatus: open -> pending

messages: + msg200902
stage: patch review -> commit review
2013-10-22 09:22:30python-devsetnosy: + python-dev
messages: + msg200901
2013-10-21 09:42:25christian.heimessetmessages: + msg200737
2013-07-19 16:31:00vajraskysetnosy: + vajrasky
messages: + msg193367
2013-07-19 14:46:16giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg193361
2013-07-07 18:33:24christian.heimessetnosy: + pitrou, haypo
messages: + msg192578
2013-06-24 13:12:08christian.heimessetfiles: - prlimit.patch
2013-06-24 13:11:59christian.heimessetfiles: + prlimit2.patch

messages: + msg191759
2012-12-03 08:02:43Arfreversetnosy: + Arfrever
2012-12-02 11:34:28christian.heimessetmessages: + msg176785
2012-12-02 11:14:41christian.heimescreate