New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_tarfile fails on ppc64le when using tmpfs filesystem #79953
Comments
The following test_tarfile tests fail on ppc64 when using tmpfs filesystem (which is the case on RHEL package build server):
Example of failure: ====================================================================== Traceback (most recent call last):
File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 964, in test_sparse_file_00
self._test_sparse_file("gnu/sparse-0.0")
File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 958, in _test_sparse_file
self.assertLess(s.st_blocks * 512, s.st_size)
AssertionError: 131072 not less than 86016 Bug first report on RHEL8: test_tarfile has _fs_supports_holes() function to check if the filesystem supports sparse files with holes. The function returns True on:
In short, it always return True on x86_64 and ppc64le Linux kernels. Problem: in practice, "tmpfs filesystem on ppc64le (kernel 4.18)" doesn't fully support sparse files. -- Example from: # ls -lhs ~/sparse Copy a sparse file from XFS to tmpfs: cp --sparse=always and fallocate --dig fail to punch holes, the file always take 128K on disk on tmpfs. # cp sparse /root/mytmp/sparse --sparse=always Counter example on XFS, source and destionation files use 48K on disk fo 84K of data: # cp sparse sparse2 --sparse=always -- Attached PR fix the _fs_support_holes() heuristic to return properly False on tmpfs on ppc64le. |
Oops, I forgot to attached my test script: create_sparse.py |
offsets = ( 4096, 12288, 20480, 28672, 36864, 45056, 53248, 61440, 69632,
77824,) These offsets are less than 64 KiB apart. On systems with a 64 KiB page size, this will not result in a sparse file on tmpfs because the effective block size is the page size (tmpfs lives in the page cache). Red Hat Enterprise Linux uses 64 KiB pages on aarch64, ppc64, ppc64le. Only s390x and x86_64 use 4 KiB pages. |
Ah yes, I confirm that ppc64le uses 64 KiB page size: # python3
Python 3.6.8 (default, Jan 11 2019, 01:44:37)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
>>> import resource
>>> resource.getpagesize()
65536 whereas my x86_64 laptop uses 4 KiB page size: $ python3
Python 3.7.2 (default, Jan 3 2019, 09:14:01)
[GCC 8.2.1 20181215 (Red Hat 8.2.1-6)] on linux
>>> import resource
>>> resource.getpagesize()
4096 |
I updated PR 11606 to mention the link between tmpfs and the page size. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: