This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients vstinner
Date 2019-01-18.11:15:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1547810144.05.0.534595171139.issue35772@roundup.psfhosted.org>
In-reply-to
Content
The following test_tarfile tests fail on ppc64 when using tmpfs filesystem (which is the case on RHEL package build server):

* test_sparse_file_00 (test.test_tarfile.GNUReadTest)
* test_sparse_file_01 (test.test_tarfile.GNUReadTest)
* test_sparse_file_10 (test.test_tarfile.GNUReadTest)
* test_sparse_file_old (test.test_tarfile.GNUReadTest)

Example of failure:

======================================================================
FAIL: test_sparse_file_00 (test.test_tarfile.GNUReadTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 964, in test_sparse_file_00
    self._test_sparse_file("gnu/sparse-0.0")
  File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 958, in _test_sparse_file
    self.assertLess(s.st_blocks * 512, s.st_size)
AssertionError: 131072 not less than 86016

Bug first report on RHEL8:
https://bugzilla.redhat.com/show_bug.cgi?id=1639490


test_tarfile has _fs_supports_holes() function to check if the filesystem supports sparse files with holes. The function returns True on:

* ext4 filesystem on x86_64 on my Fedora 29 (kernel 4.19)
* ext4 filesystem on x86_64 on my Fedora 29 (kernel 4.19)
* XFS filesystem on ppc64le (kernel 4.18)
* tmpfs filesystem on ppc64le (kernel 4.18)

In short, it always return True on x86_64 and ppc64le Linux kernels.

Problem: in practice, "tmpfs filesystem on ppc64le (kernel 4.18)" doesn't fully support sparse files.

--

Example from:
https://bugzilla.redhat.com/show_bug.cgi?id=1639490#c5

# ls -lhs ~/sparse 
48K -rw-r--r--. 1 root root 84K Jan 18 05:36 /root/sparse

Copy a sparse file from XFS to tmpfs: cp --sparse=always and fallocate --dig fail to punch holes, the file always take 128K on disk on tmpfs.

# cp sparse /root/mytmp/sparse --sparse=always
# ls -lhs /root/mytmp/sparse 
128K -rw-r--r--. 1 root root 84K Jan 18 06:10 /root/mytmp/sparse
# fallocate --dig /root/mytmp/sparse
# ls -lhs /root/mytmp/sparse 
128K -rw-r--r--. 1 root root 84K Jan 18 06:10 /root/mytmp/sparse

Counter example on XFS, source and destionation files use 48K on disk fo 84K of data:

# cp sparse sparse2 --sparse=always
# ls -lhs sparse*
48K -rw-r--r--. 1 root root 84K Jan 18 05:36 sparse
48K -rw-r--r--. 1 root root 84K Jan 18 06:13 sparse2

--

Attached PR fix the _fs_support_holes() heuristic to return properly False on tmpfs on ppc64le.
History
Date User Action Args
2019-01-18 11:15:48vstinnersetrecipients: + vstinner
2019-01-18 11:15:44vstinnersetmessageid: <1547810144.05.0.534595171139.issue35772@roundup.psfhosted.org>
2019-01-18 11:15:44vstinnerlinkissue35772 messages
2019-01-18 11:15:43vstinnercreate