Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_resource fails when file size is limited #37883

Open
nnorwitz mannequin opened this issue Jan 31, 2003 · 19 comments
Open

test_resource fails when file size is limited #37883

nnorwitz mannequin opened this issue Jan 31, 2003 · 19 comments
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@nnorwitz
Copy link
Mannequin

nnorwitz mannequin commented Jan 31, 2003

BPO 678264
Nosy @loewis, @giampaolo, @devdanzin, @bitdancer, @wiggin15

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2003-01-31.18:06:24.000>
labels = ['type-bug', 'tests', '3.9', '3.10', '3.11']
title = 'test_resource fails when file size is limited'
updated_at = <Date 2021-12-17.00:35:56.774>
user = 'https://bugs.python.org/nnorwitz'

bugs.python.org fields:

activity = <Date 2021-12-17.00:35:56.774>
actor = 'ajaksu2'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2003-01-31.18:06:24.000>
creator = 'nnorwitz'
dependencies = []
files = []
hgrepos = []
issue_num = 678264
keywords = []
message_count = 18.0
messages = ['14342', '14343', '14344', '14345', '14346', '14347', '81850', '114216', '116449', '116479', '116520', '117049', '117051', '117122', '117130', '221696', '227778', '251772']
nosy_count = 8.0
nosy_names = ['loewis', 'nnorwitz', 'mdr0', 'sable', 'giampaolo.rodola', 'ajaksu2', 'r.david.murray', 'wiggin15']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue678264'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

@nnorwitz
Copy link
Mannequin Author

nnorwitz mannequin commented Jan 31, 2003

test_resource does:

print resource.RLIM_INFINITY == max

I'm not sure of the benefit of this line. For machines
which have limited file sizes, it causes the test to fail.
Otherwise the test seems to work properly.

@nnorwitz nnorwitz mannequin self-assigned this Jan 31, 2003
@nnorwitz nnorwitz mannequin added the stdlib Python modules in the Lib dir label Jan 31, 2003
@nnorwitz nnorwitz mannequin self-assigned this Jan 31, 2003
@nnorwitz nnorwitz mannequin added the stdlib Python modules in the Lib dir label Jan 31, 2003
@nnorwitz
Copy link
Mannequin Author

nnorwitz mannequin commented Mar 3, 2003

Logged In: YES
user_id=33168

Martin, welcome back! Now I get to assign some bugs to you
to see if you have any input. :-)

@loewis
Copy link
Mannequin

loewis mannequin commented Mar 30, 2003

Logged In: YES
user_id=21627

The rationale of the line is explained in the comment: The
code tests whether the limit is a really large number, i.e.
whether represenation of large limits works.

Unfortunately, it doesn't (which is a separate issue): On
Linux, RLIM_INFINITY is reported as -1. This comes from
RLIM_INFINITY being the larges unsigned long long number,
i.e. 0xffffffffffffffffuLL. The resource module represents
this incorrectly.

I think there is no way to verify that Python computes the
real limit correctly (unless we invoke the shell's ulimit
for comparison). So you could weaken the test to verify that
the limit is non-negative (and, as a side effect, verify
that it doesn't raise an exception).

If you do so, you'll also have to fix the implementation, so
that the weakened-strengthened test passes on Linux.

@nnorwitz
Copy link
Mannequin Author

nnorwitz mannequin commented Mar 30, 2003

Logged In: YES
user_id=33168

Hmmm, sounds like a lot of work. How about something like this:

    expected_max = resource.RLIM_INFINITY
    if expected_max != max:
        # read the file resource limit
        fsize_ulimit = os.popen('ulimit -f').read()
        try:
            # convert the file size to bytes (from 512 byte
blocks)
            expected_max = int(fsize_ulimit.strip()) * 512
        except ValueError:
            raise TestSkipped, "unable to determine expected
resource value"
    print expected_max == max

This works, but I'm pretty sure this is not portable. On
Linux blocks are 1k. This only works when the block size is
512 bytes. Shall we close this bug as a test environment
problem and won't fix?

@loewis
Copy link
Mannequin

loewis mannequin commented Mar 30, 2003

Logged In: YES
user_id=21627

I'm really not that much concerned about fixing the test,
but more about fixing the code itself.

@mdr0
Copy link
Mannequin

mdr0 mannequin commented Mar 10, 2004

Logged In: YES
user_id=994239

I'm running into this problem under AIX 4.3.3 and 5.1. Is
this going to cause a problem if I put python into
production, or is it "safe" to ignore it?

@devdanzin
Copy link
Mannequin

devdanzin mannequin commented Feb 13, 2009

Is this still a problem?

@devdanzin devdanzin mannequin added tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Feb 13, 2009
@BreamoreBoy
Copy link
Mannequin

BreamoreBoy mannequin commented Aug 18, 2010

Closed as no response to msg81850.

@bitdancer
Copy link
Member

The test does not fail if FSIZE is not max on linux/py3k, but max is still represented as -1 on linux. So the reported bug is no longer valid, but Martin's concern has not been addressed.

@bitdancer
Copy link
Member

As a point of information, on my gentoo linux system without largefile support in the kernel, any value 4294967295 or above results in getrlimit reporting -1. Any smaller value is set and reported as itself. (If a sufficiently large value is passed in to setrlimit, an OverflowError results.)

@loewis
Copy link
Mannequin

loewis mannequin commented Sep 16, 2010

As a point of information, on my gentoo linux system without
largefile support in the kernel, any value 4294967295 or above
results in getrlimit reporting -1. Any smaller value is set and
reported as itself. (If a sufficiently large value is passed in to
setrlimit, an OverflowError results.)

I think we really should create new issues for any remaining problems.

AFAICT, the remaining problems are:

  • resource.RLIM_INFINITY is -1 on Linux, when it is meant to be
    a really large value
  • (as you reported) getrlimit returns -1 to indicate RLIM_INFINITY.

I think the core of the problem is that the resource module considers
rlim_t to be a signed type (or at least representable in "long long").
Using FromUnsignedLongLong in the appropriate place might solve the
issue.

@sable
Copy link
Mannequin

sable mannequin commented Sep 21, 2010

For info:
this test fails on AIX 6.1 with py3k with the following error:

test test_resource failed -- Traceback (most recent call last):
  File "/san_cis/home/cis/buildbot/buildbot-aix6/py3k-aix6-xlc/build/Lib/test/test_resource.py", line 28, in test_fsize_ismax
    self.assertEqual(resource.RLIM_INFINITY, max)
AssertionError: 2147483647 != 1073741312

The same test works on AIX 5.3.

Is it related or should I open a new issue?

@sable
Copy link
Mannequin

sable mannequin commented Sep 21, 2010

I suppose the difference comes from the fact that I have:
on AIX 5.3:
$ ulimit -f
unlimited

on AIX 6.1:
$ ulimit -f
1048575

I think the test should be updated to not require "ulimit -f" is unlimited.

@bitdancer
Copy link
Member

That's what the original report is about, as opposed to the linux repr issue that Martin wants to break out into a new ticket (which I will do). Any ideas how to fix the test? It didn't fail for me on linux, so I don't have a good test platform for trying to fix it.

I'm un-assigning Neal since the original report is no longer the cause of the test error. (The test has obviously changed since we don't check print output any more.)

@bitdancer bitdancer removed the stdlib Python modules in the Lib dir label Sep 22, 2010
@bitdancer bitdancer removed the stdlib Python modules in the Lib dir label Sep 22, 2010
@sable
Copy link
Mannequin

sable mannequin commented Sep 22, 2010

The ideal would be to check that RLIMIT_FSIZE corresponds to the ulimit as it has been suggested by Neal Norwitz in msg14345, but since the value reported by ulimit has a different unit for each platform, that would be quite a lot of trouble.

All I can suggest is the following, which checks that both values are somewhat reasonable.

Index: Lib/test/test_resource.py
===================================================================

--- Lib/test/test_resource.py	(révision 84964)
+++ Lib/test/test_resource.py	(copie de travail)
@@ -20,12 +20,17 @@
         except AttributeError:
             pass
         else:
-            # RLIMIT_FSIZE should be RLIM_INFINITY, which will be a really big
-            # number on a platform with large file support.  On these platforms,
-            # we need to test that the get/setrlimit functions properly convert
-            # the number to a C long long and that the conversion doesn't raise
-            # an error.
-            self.assertEqual(resource.RLIM_INFINITY, max)
+            # RLIMIT_FSIZE should be RLIM_INFINITY if 'ulimit -f' is
+            # set to unlimited
+            # RLIM_INFINITY will be a really big number on a platform
+            # with large file support.  On these platforms, we need to
+            # test that the get/setrlimit functions properly convert
+            # the number to a C long long and that the conversion
+            # doesn't raise an error.            
+            self.assertGreater(resource.RLIM_INFINITY, 0)
+            self.assertLessEqual(cur, max)
+            self.assertGreater(max, 0)
+            self.assertLessEqual(max, resource.RLIM_INFINITY)
             resource.setrlimit(resource.RLIMIT_FSIZE, (cur, max))
 
     def test_fsize_enforced(self):

@BreamoreBoy
Copy link
Mannequin

BreamoreBoy mannequin commented Jun 27, 2014

The inline patch in msg117130 has never been committed from what I can see. Can somebody review it please as I'm assuming that it's still valid.

@BreamoreBoy
Copy link
Mannequin

BreamoreBoy mannequin commented Sep 28, 2014

Can somebody take a look at this please. See also bpo-9917.

@wiggin15
Copy link
Mannequin

wiggin15 mannequin commented Sep 28, 2015

This issue is still relevant and can be reproduced by running:

ulimit -Hf 1000
./python [Lib/test/test_resource.py](https://github.com/python/cpython/blob/main/Lib/test/test_resource.py)

(On AIX, the default hard limit is not RLIM_INFINITY so it reproduces on that operating system without the first line)

The patch is still relevant and fixes the issue.
It looks like it's doing the right thing, but it can't be committed until bpo-9917 will be fixed - otherwise the tests will break on Linux, where RLIM_INFINITY is -1 and not greater than 0.

@devdanzin devdanzin mannequin added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Dec 17, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 9, 2022
@iritkatriel iritkatriel added 3.12 bugs and security fixes and removed 3.9 only security fixes labels Oct 3, 2022
@arhadthedev
Copy link
Member

The test does not fail if FSIZE is not max on linux/py3k, but max is still represented as -1 on linux. So the reported bug is no longer valid, but Martin's concern has not been addressed.

Is the issue relevant (or fixable) on modern systems? It was raised 20 years ago so OS behavior could change drastically.

For context: the reported chunk is:

try:
cur, max = resource.getrlimit(resource.RLIMIT_FSIZE)
except AttributeError:
pass
else:
print resource.RLIM_INFINITY == max
resource.setrlimit(resource.RLIMIT_FSIZE, (cur, max))
In 2007, b213704 replaced it with:
try:
(cur, max) = resource.getrlimit(resource.RLIMIT_FSIZE)
except AttributeError:
pass
else:
# RLIMIT_FSIZE should be RLIM_INFINITY, which will be a really big
# number on a platform with large file support. On these platforms,
# we need to test that the get/setrlimit functions properly convert
# the number to a C long long and that the conversion doesn't raise
# an error.
self.assertEqual(resource.RLIM_INFINITY, max)
resource.setrlimit(resource.RLIMIT_FSIZE, (cur, max))

@iritkatriel iritkatriel added 3.13 bugs and security fixes and removed 3.10 only security fixes labels Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants