This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Maciej Gol
Recipients Maciej Gol
Date 2020-01-23.11:15:38
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1579778138.71.0.872285273373.issue39430@roundup.psfhosted.org>
In-reply-to
Content
Hey guys,

We have a component that archives and unarchives multiple files in separate threads that started to
misbehave recently.

We have noticed a bunch of `AttributeError: module 'lzma' has no attribute 'LZMAFile'` errors, which are
unexpected because our python is not compiled with LZMA support.

What is unfortunate, is that given the traceback:

    Traceback (most recent call last):
      File "test.py", line 18, in <module>
        list(pool.map(test_lzma, range(100)))
      File "/opt/lang/python37/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
        yield fs.pop().result()
      File "/opt/lang/python37/lib/python3.7/concurrent/futures/_base.py", line 428, in result
        return self.__get_result()
      File "/opt/lang/python37/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
        raise self._exception
      File "/opt/lang/python37/lib/python3.7/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "test.py", line 14, in test_lzma
        tarfile.open(fileobj=buf, mode="r")
      File "/opt/lang/python37/lib/python3.7/tarfile.py", line 1573, in open
        return func(name, "r", fileobj, **kwargs)
      File "/opt/lang/python37/lib/python3.7/tarfile.py", line 1699, in xzopen
        fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset)
    AttributeError: module 'lzma' has no attribute 'LZMAFile'


the last line of the traceback is right AFTER this block (tarfile.py:1694):

    try:
        import lzma
    except ImportError:
        raise CompressionError("lzma module is not available")


Importing lzma in ipython fails properly:

    In [2]: import lzma                                                                                                                               
    ---------------------------------------------------------------------------
    ModuleNotFoundError                       Traceback (most recent call last)
    <ipython-input-2-c0255b54beb9> in <module>
    ----> 1 import lzma

    /opt/lang/python37/lib/python3.7/lzma.py in <module>
        25 import io
        26 import os
    ---> 27 from _lzma import *
        28 from _lzma import _encode_filter_properties, _decode_filter_properties
        29 import _compression

    ModuleNotFoundError: No module named '_lzma'

When trying to debug the problem, we have noticed it's not deterministic. In order to reproduce it,
we have created a test python that repeatedly writes an archive to BytesIO and then reads from it.
Using it with 5 threads and 100 calls, gives very good chances of reproducing the issue. For us it
was almost every time.

Race condition occurs both on Python 3.7.3 and 3.7.6.
Test script used to reproduce it attached.

I know that the test script writes uncompressed archives and during opening tries to guess the compression.
But I guess this is a legitimate scenario and should not matter in this case.
History
Date User Action Args
2020-01-23 11:15:38Maciej Golsetrecipients: + Maciej Gol
2020-01-23 11:15:38Maciej Golsetmessageid: <1579778138.71.0.872285273373.issue39430@roundup.psfhosted.org>
2020-01-23 11:15:38Maciej Gollinkissue39430 messages
2020-01-23 11:15:38Maciej Golcreate