Title: Creating RPM on Python 2 works, but Python 3 fails because of sys.implementation.cache_tag
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.4
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: dstufft, eric.araujo, pgacv2, steve.dower
Priority: normal Keywords:

Created on 2017-11-16 22:07 by pgacv2, last changed 2021-02-03 18:19 by steve.dower. This issue is now closed.

Messages (2)
msg306400 - (view) Author: Pedro (pgacv2) Date: 2017-11-16 22:07
I am trying to create an RPM on SLES12 SP2. (I think that corresponds to OpenSUSE 42.2.) This is my file, nothing special:

    import setuptools
                     options={'bdist_rpm': {'post_install': '',
                                            'post_uninstall': ''}})

Running `python bdist_rpm` with Python 2 works. However, running `python3 bdist_rpm` outputs the following:

    running bdist_rpm
    running egg_info
    writing top-level names to MyApp.egg-info/top_level.txt


    byte-compiling /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/ to my_file.cpython-34.pyc


    Processing files: MyApp-1.2.3-1.noarch
    error: File not found: /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/
    RPM build errors:
        File not found: /home/pedro/MyApp/build/bdist.linux-x86_64/rpm/BUILDROOT/MyApp-1.2.3-1.x86_64/usr/lib/python3.4/site-packages/MyApp/
    error: command 'rpmbuild' failed with exit status 1

The problem appears to be that setuptools is generating a file that ends in `.cpython-34.pyc`, and later looks for a file without the `.cpython-34` but can't find it.

The RPM generation process on Python 3 goes through `distutils.util.byte_compile()`, which in turn calls `importlib.util.cache_from_source(path)`, where `path` is the file to be byte-compiled. `cache_from_source()` injects the value of `sys.implementation.cache_tag` (which is equal to 'cpython-34' on SLES12 SP2) into the filename of the compiled file. This attribute of `sys` does not exist in Python 2. So it looks like `setuptools` alters the filename with that tag during byte compilation, but later forgets that it modified the filename and fails because it's looking for the original name.

I ended up working around this by patching `distutils.file_util.write_file` to eliminate the .pyc entries from INSTALLED_FILES:

    orig_bytecode_var = os.environ.get('PYTHONDONTWRITEBYTECODE', '')
    os.environ['PYTHONDONTWRITEBYTECODE'] = '1'
    orig_write_file = distutils.file_util.write_file

    def my_patch(*args, **kwargs):
        new_args = list(args)
        if args[0] == 'INSTALLED_FILES':
            new_args[1] = [fname for fname in args[1] if fname[-4:] not in ('.pyc', '.pyo')]
        orig_write_file(*new_args, **kwargs)
    distutils.file_util.write_file = my_patch

                     options={'bdist_rpm': {'post_install': '',
                                            'post_uninstall': ''}})

    os.environ['PYTHONDONTWRITEBYTECODE'] = orig_bytecode_var
    distutils.file_util.write_file = orig_write_file

But I don't think I should have to do anything special to build an RPM on Python 3 vs. Python 2. Is this expected behavior?
msg386327 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:19
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at
Date User Action Args
2021-02-03 18:19:21steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386327

resolution: out of date
stage: resolved
2017-11-16 22:07:02pgacv2create