Issue47123
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2022-03-25 19:47 by ghost43, last changed 2022-04-11 14:59 by admin.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
zipfile_respect_sourcedate.diff | ghost43, 2022-03-25 19:47 | change the ZipFile.writestr to respect SOURCE_DATE_EPOCH |
Messages (1) | |||
---|---|---|---|
msg416015 - (view) | Author: ghost43 (ghost43) * | Date: 2022-03-25 19:47 | |
Currently `ZipFile.writestr` writes the local time into the ZipFile. (depends on both current time and local timezone) See https://github.com/python/cpython/blob/20e6e5636a06fe5e1472062918d0a302d82a71c3/Lib/zipfile.py#L1816-L1817 This makes pip installing a package generate non-reproducible build artifacts. Specifically, `Scripts/*.exe` files (created for packages that define entry_points/console_scripts) are not reproducible on Windows when installed by pip. This also leaks into the `*.dist-info/RECORD` files. For example, after running `pip install wheel` or `pip install pyinstaller`, in `wheel-0.37.1.dist-info/RECORD`, I have this line: ``` ../../Scripts/wheel.exe,sha256=u9TbPw2XNs_F9uy7y2zwumuzAZDbOSB7BXjLHZ0tTHg,97103 ``` in `pyinstaller-4.10.dist-info/RECORD`, I have these lines: ``` ../../Scripts/pyi-archive_viewer.exe,sha256=nC-9muPlIhUC1qvFkXHpyKJyRQqXISXxbUPXQ1XVOiM,97133 ../../Scripts/pyi-bindepend.exe,sha256=udFHiAdndPpSwaIqmhmLEy36IUs1cNNoNQznSEnLJQQ,97128 ../../Scripts/pyi-grab_version.exe,sha256=3ET9E841tFWujFL99aG4frzgwlP9f9pAkMgE0k2UGK0,97131 ../../Scripts/pyi-makespec.exe,sha256=dJkfmITdLJhyPngmqziqqj5tH9qqfeQc5BTubeoXWUs,97127 ../../Scripts/pyi-set_version.exe,sha256=sWmcOVS93fUY-wbdoz6ixBCvjy1tC4Aaw30DMmrmo-0,97130 ../../Scripts/pyinstaller.exe,sha256=haInbhH0pImJn24cW4v917oUZmzXZj8OE89KFh4MO2Y,97112 ``` Upon comparing multiple `Scripts/wheel.exe` files, I've found that the only difference is due to the above-mentioned timestamp embedded inside the exe (or rather, same timestamp embedded twice). The `exe` files get created by `distlib` (vendored by pip). Here is a traceback with an artificial exception to illustrate the codepath: ``` (env) PS C:\tmp> pip install --no-build-isolation pyinstaller Collecting pyinstaller Using cached pyinstaller-4.10-py3-none-win_amd64.whl (2.0 MB) Requirement already satisfied: setuptools in c:\tmp\env\lib\site-packages (from pyinstaller) (61.0.0) Requirement already satisfied: pyinstaller-hooks-contrib>=2020.6 in c:\tmp\env\lib\site-packages (from pyinstaller) (2022.3) Requirement already satisfied: altgraph in c:\tmp\env\lib\site-packages (from pyinstaller) (0.17.2) Requirement already satisfied: pefile>=2017.8.1 in c:\tmp\env\lib\site-packages (from pyinstaller) (2021.9.3) Requirement already satisfied: pywin32-ctypes>=0.2.0 in c:\tmp\env\lib\site-packages (from pyinstaller) (0.2.0) Requirement already satisfied: future in c:\tmp\env\lib\site-packages (from pefile>=2017.8.1->pyinstaller) (0.18.2) Installing collected packages: pyinstaller ERROR: Exception: Traceback (most recent call last): File "C:\tmp\env\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper status = run_func(*args) File "C:\tmp\env\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper return func(self, options, args) File "C:\tmp\env\lib\site-packages\pip\_internal\commands\install.py", line 405, in run installed = install_given_reqs( File "C:\tmp\env\lib\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs requirement.install( File "C:\tmp\env\lib\site-packages\pip\_internal\req\req_install.py", line 769, in install install_wheel( File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 729, in install_wheel _install_wheel( File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 646, in _install_wheel generated_console_scripts = maker.make_multiple(scripts_to_generate) File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make_multiple filenames.extend(self.make(specification, options)) File "C:\tmp\env\lib\site-packages\pip\_internal\operations\install\wheel.py", line 427, in make return super().make(specification, options) File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 429, in make self._make_script(entry, filenames, options=options) File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 329, in _make_script self._write_script(scriptnames, shebang, script, filenames, ext) File "C:\tmp\env\lib\site-packages\pip\_vendor\distlib\scripts.py", line 263, in _write_script raise Exception(f"heyheyhey2. {sha256(launcher)=}. {sha256(shebang)=}. {sha256(zip_data)=}. " + Exception: heyheyhey2. sha256(launcher)='a00a877acefc'. sha256(shebang)='58628e924f22'. sha256(zip_data)='a423496a0482'. ('SOURCE_DATE_EPOCH' in os.environ)=True ``` The interesting code is here: https://github.com/pypa/distlib/blob/d0e3f49df5d1aeb9daeaaabf0391c9e13e4a6562/distlib/scripts.py#L251-L252 This calls into the cpython standard library, where `time.time()` gets written into the file: https://github.com/python/cpython/blob/20e6e5636a06fe5e1472062918d0a302d82a71c3/Lib/zipfile.py#L1816-L1817 Ideally, either `distlib` or the stdlib `zipfile` module should be changed to respect `SOURCE_DATE_EPOCH`, but it's not entirely clear to me which... The attached patch changes `ZipFile.writestr` to respect `SOURCE_DATE_EPOCH` if set. related https://github.com/pypa/distlib/issues/164 related https://github.com/spesmilo/electrum/issues/7739 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:57 | admin | set | github: 91279 |
2022-03-25 19:47:10 | ghost43 | create |