This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Test failure in resource module on some ZFS systems
Type: behavior Stage: needs patch
Components: Versions: Python 3.11
process
Status: open Resolution: third party
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: gregory.p.smith, iritkatriel, larry, ned.deily, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-07-23 23:17 by larry, last changed 2022-04-11 14:58 by admin.

Messages (7)
msg298919 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2017-07-23 23:17
I use Linux (Ubuntu GNOME, 17.04, 64-bit) on all my computers, and I use  ZFS for the /home partition.  While releasing 3.4.7rc1 and 3.5.4rc1, I encountered this test failure:

% ./python -m test test_resource
0:00:00 load avg: 0.40 [1/1] test_resource
test test_resource failed -- Traceback (most recent call last):
  File "/home/larry/src/python/releases/test35/Lib/test/test_resource.py", line 55, in test_fsize_enforced
    f.write(b"X" * 1024)
OSError: [Errno 27] File too large

1 test failed:
    test_resource
Tests result: FAILURE

If I run this test on another filesystem (e.g. ext4), with the exact same source tree, the test passes.  I also didn't see this failure when releasing 3.4.6 and 3.5.3 earlier this year.

This is probably caused by new behavior in an recent update to ZFS.  So I'm not sure if this is a ZFS bug, and if it is I don't know whether or not we should work around it.  But it's hard to imagine ZFS is behaving correctly here.

I've confirmed that this happens in all four branches tagged (3.4 through 3.7).  Since it causes test failures, I'm willing to consider accepting a PR for 3.4 and 3.5.

Serhiy, I've tagged you since you were the last person to touch resource.getrlimit and resource.setrlimit.  Are you interested / willing to look at this bug?  Do you have a computer with a ZFS filesystem you can use to run experiments?
msg298927 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-24 04:34
I use 32-bit Linux and when tried to install the ZFS DKMS driver got a warning that ZFS is unstable on 32-bit.
msg310542 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2018-01-24 00:39
I'm using ZFS on 64-bit Linux.

I did see a Github issue / commit that claims to address this.  I'm using a reasonably recent ZoL build, that should have that commit, and I still see the behavior.

I actually experimented with it a little, and, hmm.  The test sets the rlimit to 1024 bytes.  I can write exactly 512 bytes to the file; it fails on the 513th.  We *could* change the test to behave like that (write 512 bytes, then write 513 more) but I don't think we should work around their bug.
msg404604 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-10-21 14:22
Can we/should we do anything about this or shall we close?
msg404650 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-21 20:42
this is probably somewhat of an edge case.  filesystems are presumably free to have their own concepts of granularity on size limit enforcement.

zfs may become more popular in future ubuntu versions as they're offering it as a supported primary whole system filesystem.  but zfs is also highly configurable so it likely isn't accurate to just say "zfs" caused this.

that test appears to be setting RLIMIT_FSIZE to 1024 and creating a 1024 byte file.  this could be an edge case, either in our understanding (the limit can never be reached or the limit can be exactly reached?) or in the filesystem's interpretation of that.  my understanding is that it is supposed to be inclusive.  RLIMIT_FSIZE 1024 should allow 1024 byte files.

but for such small unrealistic real world values i can easily imagine filesystems being creative...  people tend not to expect these limits to be so strict.

that test could use refactoring to rely less on strict limits.  that isn't what it was aiming to test.
msg404656 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2021-10-21 20:58
FWIW the test still fails in exactly the same way.  This was building with main, on Pop!_OS 21.04 64-bit.
msg404696 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-10-21 22:37
It doesn't happen on the zfs ubuntu 20.04 machine i've got.  Figuring out an appropriate workaround when refactoring the test for this is best done by someone with a system it fails on.  It'd be guesswork otherwise.
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75183
2021-10-21 22:37:21gregory.p.smithsettitle: Test failure in resource module on ZFS -> Test failure in resource module on some ZFS systems
messages: + msg404696
versions: + Python 3.11, - Python 3.4, Python 3.5, Python 3.6, Python 3.7
2021-10-21 20:58:26larrysetmessages: + msg404656
2021-10-21 20:42:28gregory.p.smithsetstatus: pending -> open
nosy: + gregory.p.smith
messages: + msg404650

2021-10-21 14:22:46iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg404604

resolution: third party
2018-01-24 00:39:15larrysetmessages: + msg310542
2017-07-24 04:34:51serhiy.storchakasetmessages: + msg298927
2017-07-23 23:17:58larrycreate