New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update bundled pip to 20.2.1 and setuptools to 49.2.1 #85662
Comments
I'm doing the PR now, based on the latest versions available today: https://pypi.org/project/pip/20.2.1/ If you're a maintainer and there's a reason to not update to to the latest, please let me know asap. All of our subsequent releases should be RC's, so I assume we won't take any changes bigger than targeted fixes before the next full releases. |
Test failure on Windows. I'll take a look tomorrow. ====================================================================== Traceback (most recent call last):
File "D:\a\cpython\cpython\lib\test\test_venv.py", line 476, in do_test_with_pip
self.run_with_capture(venv.create, self.env_dir,
File "D:\a\cpython\cpython\lib\test\test_venv.py", line 76, in run_with_capture
func(*args, **kwargs)
subprocess.CalledProcessError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\a\cpython\cpython\lib\test\test_venv.py", line 536, in test_with_pip
self.do_test_with_pip(False)
File "D:\a\cpython\cpython\lib\test\test_venv.py", line 484, in do_test_with_pip
self.fail(msg.format(exc, details))
AssertionError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. **Subprocess Output** Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\setuptools-49.2.1-py3-none-any.whl Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\pip-20.2.1-py2.py3-none-any.whl Installing collected packages: setuptools, pip Successfully installed pip-20.2.1 setuptools-49.2.1 Traceback (most recent call last): File "D:\a\cpython\cpython\lib\shutil.py", line 613, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl' During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "D:\a\cpython\cpython\lib\tempfile.py", line 802, in onerror _os.unlink(path) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl' During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "D:\a\cpython\cpython\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\a\cpython\cpython\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "D:\a\cpython\cpython\lib\ensurepip\main.py", line 5, in <module> sys.exit(ensurepip._main()) File "D:\a\cpython\cpython\lib\ensurepip\init.py", line 213, in _main return _bootstrap( File "D:\a\cpython\cpython\lib\ensurepip\init.py", line 132, in _bootstrap
File "D:\a\cpython\cpython\lib\tempfile.py", line 827, in __exit__ self.cleanup() File "D:\a\cpython\cpython\lib\tempfile.py", line 831, in cleanup self._rmtree(self.name) File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree return _rmtree_unsafe(path, onerror) File "D:\a\cpython\cpython\lib\shutil.py", line 615, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File "D:\a\cpython\cpython\lib\tempfile.py", line 805, in onerror cls._rmtree(path) File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree return _rmtree_unsafe(path, onerror) File "D:\a\cpython\cpython\lib\shutil.py", line 596, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File "D:\a\cpython\cpython\lib\shutil.py", line 593, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl' ---------------------------------------------------------------------- |
The issue above doesn't appear to repro on 3.9, so I guess master has started leaking a file handle, presumably in zipimport. I'll see what I can track down, but can't be sure I'll have enough time to get it done for RC, so if anyone else wants to help out feel free. |
Okay, I've tracked it down to the new importlib.readers.ZipReader class keeping the zip file open, presumably until it gets GC'd. This is used by certifi to extract the CA certs from the whl when ensurepip is doing the self-install from the mounted wheel. Jason is already on this bug, which is convenient :) I haven't yet figured out whether there's a convenient way for the reader to not keep the ZIP open for as long as it exists, but I think that's going to be the safest fix. We should definitely fix this one ourselves without forcing users to make changes to accommodate. As I mentioned above, it's only in 3.10 right now, but it's blocking updated pip and setuptools versions downlevel. |
Added some test cases to the PR that directly trigger the issue, specifically this one: def test_entered_path_does_not_keep_open(self):
# This is what certifi does on import to make its bundle
# available for the process duration.
c = resources.path('ziptestdata', 'binary.file').__enter__()
self.zip_path.unlink() All the tests I added pass on 3.9 (with minor tweaks for moved test utils). To unblock the upcoming releases, I'm going to do the backports first and leave this as a release blocker for 3.10. |
GitHub Actions has decided not to run CI today, so you'll have to look at Azure Pipelines for the test failures: https://dev.azure.com/Python/cpython/_build/results?buildId=67152&view=logs&j=c83831cd-3752-5cc7-2f01-8276919eb334&t=5a421c4a-0933-53d5-26b9-04b36ad165eb&l=8012 |
You may be right here. I don't fully understand the repro, but it seems to me like you're trying to delete a zip file while you have resources open in that zip file. I think we need a separate issue to capture the underlying defect. |
I think this reproduces the underlying issue: >>> import zipfile
>>> from pathlib import Path
>>> p = zipfile.Path("tst.zip")
>>> Path("tst.zip").unlink()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\UK03306\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1321, in unlink
self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'tst.zip'
>>> Basically, zipfile.Path objects don't have any way to close the underlying zipfile, so you have to delete the path object to free up the file for deletion. In the context of importlib, maybe permanently holding a reference to the zipfile.Path object can't work, and it needs to be re-opened each time it's needed? |
If you look at the PR into 3.9, it includes a test for this exact case. Start by copying that into 3.10 and then make it pass and everything should be good :) |
pip 20.2.1 contains a regression which breaks |
I think that the backport to 3.8 may have some unintended consequences in the last patch release as venv created with 3.8 now exhibit the pip regression (pypa/pip#8695.). Steve, would you be ok if we bump all branches (master, 3.9 and 3.8) to 20.2.3? |
+1 for bumping to 20.2.3, in case an upstream voice is helpful. :) On Fri, Oct 2, 2020 at 8:12 PM Pablo Galindo Salgado <report@bugs.python.org>
|
This change goes into 3.9.0 with Pablo's fix. |
Can this issue be closed? Are we waiting for manual checks? Or is there any remaining thing to do? |
Yeah, there is some code in PR 21748 that should be merged (the new test). |
I have no objection to bumping it further, provided someone has fixed importlib.resources on 3.10 (i.e. my backported tests pass) |
setuptools is still more recent in 3.9 than in master. Look at PR 21748. Can someone try to update setuptools in master? |
I did some bad things in git and so I had to recreate the PR against master, but the importlib.resources tests still fail. Seems like the ensurepip/venv tests are fine though? Did something change in pip/certifi between 20.2.1 and 20.2.3? I still consider the changes a release blocking regression, but if it's not blocking ensurepip then I guess the updated packages can go in now. I'll split up my PR tomorrow, but I'm not feeling like fighting git more tonight. |
Question: Why do we keep setuptools? According to PEP-453:
Which looks like to be the fact now: $ python3.9 -m venv .venv
$ source .venv/bin/activate
$ pip uninstall setuptools
[...]
Successfully uninstalled setuptools-49.2.1
$ pip install --upgrade pip
[...]
Successfully installed pip-20.3.3 |
I agree; would be good to remove it if possible. There are many packages that fail to build without Setuptools being present or --use-pep517 indicated. It would be nice if pip could make --use-pep517 the default, update that in Python, and then remove Setuptools. I would expect that approach will cause a great deal less turmoil in the ecosystem. |
Friendly reminder that this issue is currently blocking the 3.10a5 release. If you are ok with waiting for the next release to include the fix, please say so here or drop me an email/ |
I'm not sure why, but the PR to update the bundled pip to 21.0.1 (and setuptools to 52.0.0) merged cleanly, so this may be obsolete now. |
Do we need to wait for PR 22779 or can it be closed? |
It can be closed. |
That's specifically the ensurepip PR, which is now outdated since pip 21.0.1 is in master. But for this bug report, Steve said "I still consider the changes a release blocking regression, but if it's not blocking ensurepip then I guess the updated packages can go in now" so I don't know if he considers there to be something else that this issue should still track. |
I closed the PR. Jason's fix deals with it, so this is now resolved. |
path
andcontents
to aggressively close handles #22915Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: