classification
Title: test_tarfile fails on ppc64le when using tmpfs filesystem
Type: Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: fweimer, lars.gustaebel, miss-islington, vstinner
Priority: normal Keywords: patch, patch

Created on 2019-01-18 11:15 by vstinner, last changed 2019-01-21 09:45 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
create_sparse.py vstinner, 2019-01-18 14:27
Pull Requests
URL Status Linked Edit
PR 11606 merged vstinner, 2019-01-18 11:22
PR 11606 merged vstinner, 2019-01-18 11:22
PR 11610 closed vstinner, 2019-01-18 14:46
PR 11610 closed vstinner, 2019-01-18 14:46
PR 11610 closed vstinner, 2019-01-18 14:46
PR 11630 merged miss-islington, 2019-01-21 09:24
PR 11630 merged miss-islington, 2019-01-21 09:24
Messages (7)
msg333956 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-18 11:15
The following test_tarfile tests fail on ppc64 when using tmpfs filesystem (which is the case on RHEL package build server):

* test_sparse_file_00 (test.test_tarfile.GNUReadTest)
* test_sparse_file_01 (test.test_tarfile.GNUReadTest)
* test_sparse_file_10 (test.test_tarfile.GNUReadTest)
* test_sparse_file_old (test.test_tarfile.GNUReadTest)

Example of failure:

======================================================================
FAIL: test_sparse_file_00 (test.test_tarfile.GNUReadTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 964, in test_sparse_file_00
    self._test_sparse_file("gnu/sparse-0.0")
  File "/builddir/build/BUILD/Python-3.6.6/Lib/test/test_tarfile.py", line 958, in _test_sparse_file
    self.assertLess(s.st_blocks * 512, s.st_size)
AssertionError: 131072 not less than 86016

Bug first report on RHEL8:
https://bugzilla.redhat.com/show_bug.cgi?id=1639490


test_tarfile has _fs_supports_holes() function to check if the filesystem supports sparse files with holes. The function returns True on:

* ext4 filesystem on x86_64 on my Fedora 29 (kernel 4.19)
* ext4 filesystem on x86_64 on my Fedora 29 (kernel 4.19)
* XFS filesystem on ppc64le (kernel 4.18)
* tmpfs filesystem on ppc64le (kernel 4.18)

In short, it always return True on x86_64 and ppc64le Linux kernels.

Problem: in practice, "tmpfs filesystem on ppc64le (kernel 4.18)" doesn't fully support sparse files.

--

Example from:
https://bugzilla.redhat.com/show_bug.cgi?id=1639490#c5

# ls -lhs ~/sparse 
48K -rw-r--r--. 1 root root 84K Jan 18 05:36 /root/sparse

Copy a sparse file from XFS to tmpfs: cp --sparse=always and fallocate --dig fail to punch holes, the file always take 128K on disk on tmpfs.

# cp sparse /root/mytmp/sparse --sparse=always
# ls -lhs /root/mytmp/sparse 
128K -rw-r--r--. 1 root root 84K Jan 18 06:10 /root/mytmp/sparse
# fallocate --dig /root/mytmp/sparse
# ls -lhs /root/mytmp/sparse 
128K -rw-r--r--. 1 root root 84K Jan 18 06:10 /root/mytmp/sparse

Counter example on XFS, source and destionation files use 48K on disk fo 84K of data:

# cp sparse sparse2 --sparse=always
# ls -lhs sparse*
48K -rw-r--r--. 1 root root 84K Jan 18 05:36 sparse
48K -rw-r--r--. 1 root root 84K Jan 18 06:13 sparse2

--

Attached PR fix the _fs_support_holes() heuristic to return properly False on tmpfs on ppc64le.
msg333970 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-18 14:27
Oops, I forgot to attached my test script: create_sparse.py
msg333971 - (view) Author: Florian Weimer (fweimer) Date: 2019-01-18 14:38
offsets = ( 4096, 12288, 20480, 28672, 36864, 45056, 53248, 61440, 69632,
        77824,)

These offsets are less than 64 KiB apart.  On systems with a 64 KiB page size, this will not result in a sparse file on tmpfs because the effective block size is the page size (tmpfs lives in the page cache).

Red Hat Enterprise Linux uses 64 KiB pages on aarch64, ppc64, ppc64le.  Only s390x and x86_64 use 4 KiB pages.
msg333972 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-18 14:42
Ah yes, I confirm that ppc64le uses 64 KiB page size:

# python3
Python 3.6.8 (default, Jan 11 2019, 01:44:37) 
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
>>> import resource
>>> resource.getpagesize()
65536

whereas my x86_64 laptop uses 4 KiB page size:

$ python3
Python 3.7.2 (default, Jan  3 2019, 09:14:01) 
[GCC 8.2.1 20181215 (Red Hat 8.2.1-6)] on linux
>>> import resource
>>> resource.getpagesize()
4096
msg333973 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-18 14:57
I updated PR 11606 to mention the link between tmpfs and the page size.
msg334115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-21 09:24
New changeset b2385458ceddaf3d0d91456923716259d3915023 by Victor Stinner in branch 'master':
bpo-35772: Fix test_tarfile on ppc64 (GH-11606)
https://github.com/python/cpython/commit/b2385458ceddaf3d0d91456923716259d3915023
msg334118 - (view) Author: miss-islington (miss-islington) Date: 2019-01-21 09:44
New changeset d1dd6be613381b996b9071443ef081de8e5f3aff by Miss Islington (bot) in branch '3.7':
bpo-35772: Fix test_tarfile on ppc64 (GH-11606)
https://github.com/python/cpython/commit/d1dd6be613381b996b9071443ef081de8e5f3aff
History
Date User Action Args
2019-01-21 09:45:45vstinnersetkeywords: patch, patch
status: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-01-21 09:44:33miss-islingtonsetnosy: + miss-islington
messages: + msg334118
2019-01-21 09:24:35miss-islingtonsetpull_requests: + pull_request11399
2019-01-21 09:24:30miss-islingtonsetpull_requests: + pull_request11398
2019-01-21 09:24:23vstinnersetmessages: + msg334115
2019-01-18 14:57:31vstinnersetkeywords: patch, patch

messages: + msg333973
2019-01-18 14:46:43vstinnersetpull_requests: + pull_request11341
2019-01-18 14:46:35vstinnersetpull_requests: + pull_request11340
2019-01-18 14:46:28vstinnersetpull_requests: + pull_request11339
2019-01-18 14:42:07vstinnersetkeywords: patch, patch

messages: + msg333972
2019-01-18 14:38:42fweimersetnosy: + fweimer
messages: + msg333971
2019-01-18 14:27:37vstinnersetkeywords: patch, patch
files: + create_sparse.py
messages: + msg333970
2019-01-18 13:43:14vstinnersetkeywords: patch, patch
nosy: + lars.gustaebel
2019-01-18 11:22:40vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request11333
2019-01-18 11:22:38vstinnersetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11332
2019-01-18 11:15:44vstinnercreate