classification
Title: Add large-file tests for modules using sendfile(2)
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: cstratak, giampaolo.rodola, petr.viktorin, rosslagerwall, vstinner
Priority: normal Keywords: patch

Created on 2019-05-30 07:18 by giampaolo.rodola, last changed 2019-12-06 20:35 by cstratak.

Files
File name Uploaded Description Edit
disk-space.patch giampaolo.rodola, 2019-10-01 16:42
Pull Requests
URL Status Linked Edit
PR 13676 merged giampaolo.rodola, 2019-05-30 07:19
Messages (12)
msg343951 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-30 07:18
The need for this emerged in:
https://bugs.python.org/issue36610#msg343948.
We currently use sendfile(2) syscall in high-level shutil.copyfile() on Linux and socket.sendfile() on UNIX. In addition this PR also tests shutil.copyfile() implementation on OSX which takes advantage of fcopyfile(2) syscall. The goal is to make sure "fast-copy" syscalls can handle files > 2G. On my Ubuntu 18.04 with SSD disk this introduces a 5 secs slowdown to the test run.

According to:
https://github.com/golang/go/issues/13892
...problems may arise on SunOS, so if a BB will turn red this is expected.
msg353533 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-09-30 04:51
New changeset 5bcc6d89bcb622a6786fff632fabdcaf67dbb4e2 by Giampaolo Rodola in branch 'master':
bpo-37096: Add large-file tests for modules using sendfile(2) (GH-13676)
https://github.com/python/cpython/commit/5bcc6d89bcb622a6786fff632fabdcaf67dbb4e2
msg353552 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-30 07:45
TestCopyfile fails on 32-bit system: bpo-38319.
msg353699 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-10-01 15:35
After this change I get some disk space issues on the Fedora rawhide buildbot for the clang installed build only (and strangely enough not for the other jobs). There are currently around 9GB of free space there:

https://buildbot.python.org/all/#/builders/188/builds/1110

======================================================================
ERROR: test_tofromfile (test.test_array.DoubleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-x86_64.clang-installed/build/target/lib/python3.9/test/test_array.py",
line 374, in test_tofromfile
    f.close()
OSError: [Errno 28] No space left on device
msg353701 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-10-01 16:42
Sorry about that. I'm attaching a patch which skips those tests if there's not enough free disk space. If you confirm it works I'll make a PR.
msg353703 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-01 17:08
Should we decorate the new files with @requires('largefile', '...')? Documentation:

    largefile - It is okay to run some test that may create huge
                files.  These tests can take a long time and may
                consume >2 GiB of disk space temporarily.

It may be interesting to add an option to regrtest to specify how much space can be used by tests, an option similar to -M/--memlimit which is for memory.

I don't recall if largefile is enabled by default on buildbots or not :-p
msg353704 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-10-01 17:19
It seems that the -uall argument is passed to regrtest invocation for the buildbot run [0] which invokes the largefile tests (including all the resource intensive tests). However when configure is run you can see: checking whether to enable large file support... no

[0] https://github.com/python/cpython/blob/master/Lib/test/libregrtest/cmdline.py#L81
msg353709 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-01 20:23
> checking whether to enable large file support... no

That's only at the C level to decide how to pass a file position.

It's unrelated to the regrtest "largefile" resource.
msg356368 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-11-11 16:43
Alright added some more disk space at the buildbots, however it seems that it is not related to that. The current Fedora rawhide buildbot has 19GB of free space and the test is still failing.

I tested on the Fedora stable buildbot for which I reduced intentionally the free disk space to 14GB and it passed: https://buildbot.python.org/all/#/builders/233/builds/398

So something else might be at play here.
msg357859 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-12-05 16:02
It seems that the test actually requires >4GB disk space, because it's copying a 2GB file from one place to another.

Also, it's using a temporary directory. On Fedora /tmp is memory-backed by default, it doesn't use regular disk space.
msg357863 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-12-05 16:26
The test passes with a 8GB /tmp, but not a 6GB one.

That could suggest the test needs 3*2GB disk space for temporary files, plus something extra.
msg357944 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-12-06 20:35
Can we reopen the issue?

Clearly this change modifies the expectations of free disk space for the temp files created by the tests.

Or at least clarify that those tests require more than 6gb of free disk space in /tmp for unix
History
Date User Action Args
2019-12-06 20:35:51cstrataksetmessages: + msg357944
2019-12-05 16:26:30petr.viktorinsetmessages: + msg357863
2019-12-05 16:02:56petr.viktorinsetnosy: + petr.viktorin
messages: + msg357859
2019-11-11 16:43:17cstrataksetmessages: + msg356368
2019-10-01 20:23:56vstinnersetmessages: + msg353709
2019-10-01 17:19:53cstrataksetmessages: + msg353704
2019-10-01 17:08:29vstinnersetmessages: + msg353703
2019-10-01 16:42:10giampaolo.rodolasetfiles: + disk-space.patch

messages: + msg353701
2019-10-01 15:35:07cstrataksetnosy: + cstratak
messages: + msg353699
2019-09-30 07:45:20vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg353552

resolution: fixed ->
2019-09-30 04:52:49giampaolo.rodolasetstatus: open -> closed
versions: + Python 3.9, - Python 3.8
resolution: fixed
components: + Tests, - Library (Lib)
type: enhancement
stage: patch review -> resolved
2019-09-30 04:51:58giampaolo.rodolasetmessages: + msg353533
2019-05-30 07:19:26giampaolo.rodolasetkeywords: + patch
stage: patch review
pull_requests: + pull_request13564
2019-05-30 07:18:19giampaolo.rodolacreate