This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_zlib: test_pair() and test_speech128() fail with s390x hardware accelerator
Type: Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: petr.viktorin, vstinner
Priority: normal Keywords: patch

Created on 2022-02-03 08:34 by vstinner, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31096 merged vstinner, 2022-02-03 08:44
Messages (6)
msg412426 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-03 08:34
test_pair() and test_speech128() tests of test_zlib fail on the s390x architecture if zlib uses the s390x hardware accelerator.

RHEL8 downstream issues (with most details):
https://bugzilla.redhat.com/show_bug.cgi?id=1974658

Fedora downstream issues:
https://bugzilla.redhat.com/show_bug.cgi?id=2038848

The s390x has a hardware accelerator for zlib. Depending if the hardware accelerator is used or not, the output (compress data) is different.

Also, test_zlib compress data in two different ways and then expect the same output. test_zlib pass with the software implementation which creates a single (final) compressed block. test_zlib fails with the hardware implementation which creates multiple compressed blocks (the last one is a final block).

Another reason the output differs is the FHT/DHT heuristic. The zlib deflate algorithm can analyze the data distribution and decide whether it wants to use a fixed-Huffman table (FHT) or a dynamic-Huffman table (DHT) for the next block, but the accelerator can't. Furthermore, looking at data in software would kill the accelerator performance. Therefore the following heuristic is used on s390x: the first 4k are compressed with FHT and the rest of the data with DHT. So, compress() creates a single FHT block. compressobj() creates a FHT block, a DHT block and a trailing block.

It is *not a bug* in zlib: the decompression gives back the original content as expected in all cases. The issue is that Python test_zlib makes too many assumptions on how "streamed" data should be compressed. The test expected that compressed data using different ways to call zlib would return the exact same compressed data. If an accelarator is used, it's not always the case.
msg413866 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-23 23:25
I checked os.uname() value on two buildbots, by looking at their test.pythoninfo:

os.uname: posix.uname_result(sysname='Linux', nodename='ztcpip3.pok.ibm.com', release='3.10.0-1160.53.1.el7.s390x', version='#1 SMP Thu Dec 16 04:33:52 EST 2021', machine='s390x')

os.uname: posix.uname_result(sysname='Linux', nodename='dje', release='5.10.0-11-s390x', version='#1 SMP Debian 5.10.92-1 (2022-01-18)', machine='s390x')

So the machine is 's390x'.
msg413939 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-24 23:32
New changeset 9475dc0b8d2a0db40278bbcb88a89b1265a77ec9 by Victor Stinner in branch 'main':
bpo-46623: Skip two test_zlib tests on s390x (GH-31096)
https://github.com/python/cpython/commit/9475dc0b8d2a0db40278bbcb88a89b1265a77ec9
msg413940 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-24 23:32
Even if Python 3.9 and 3.10 are also affected, I prefer to not backport the change since it's not ideal.
msg413941 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-24 23:34
The Python 3.7 package on Fedora also skips these two tests using --ignore option of regrtest:
	
    %ifarch s390x
    --ignore test_speech128 \
    --ignore test_pair \
    %endif

* https://src.fedoraproject.org/rpms/python3.7/pull-request/38#request_diff
* https://src.fedoraproject.org/rpms/python3.7/blob/rawhide/f/python3.7.spec#_1168
msg414437 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2022-03-03 13:20
The current fix skips all of the tests, rather than just the parts that fail.
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90781
2022-03-03 13:20:48petr.viktorinsetstatus: closed -> open

nosy: + petr.viktorin
messages: + msg414437

resolution: fixed ->
2022-02-24 23:34:55vstinnersetmessages: + msg413941
2022-02-24 23:32:50vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg413940

stage: patch review -> resolved
2022-02-24 23:32:13vstinnersetmessages: + msg413939
2022-02-23 23:25:06vstinnersetmessages: + msg413866
2022-02-03 08:44:59vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request29281
2022-02-03 08:34:40vstinnercreate