classification
Title: Update bundled pip to 20.2.1 and setuptools to 49.2.1
Type: Stage: patch review
Components: Distutils Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Marcus.Smith, dstufft, eric.araujo, eric.snow, jaraco, lkollar, lukasz.langa, ncoghlan, pablogsal, paul.moore, pradyunsg, steve.dower
Priority: release blocker Keywords: patch

Created on 2020-08-05 21:26 by steve.dower, last changed 2020-10-27 03:43 by vstinner.

Pull Requests
URL Status Linked Edit
PR 21748 closed steve.dower, 2020-08-05 22:17
PR 21774 merged steve.dower, 2020-08-07 21:45
PR 21775 merged steve.dower, 2020-08-07 21:49
PR 22527 merged pablogsal, 2020-10-03 19:09
PR 22544 merged pablogsal, 2020-10-04 16:49
PR 22545 merged pablogsal, 2020-10-04 16:51
PR 22779 open steve.dower, 2020-10-19 19:38
PR 22915 merged jaraco, 2020-10-23 12:55
Messages (26)
msg374901 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-05 21:26
I'm doing the PR now, based on the latest versions available today:

https://pypi.org/project/pip/20.2.1/
https://pypi.org/project/setuptools/49.2.1/

If you're a maintainer and there's a reason to not update to to the latest, please let me know asap. All of our subsequent releases should be RC's, so I assume we won't take any changes bigger than targeted fixes before the next full releases.
msg374909 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-05 23:24
Test failure on Windows. I'll take a look tomorrow.

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 476, in do_test_with_pip
    self.run_with_capture(venv.create, self.env_dir,
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 76, in run_with_capture
    func(*args, **kwargs)
subprocess.CalledProcessError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 536, in test_with_pip
    self.do_test_with_pip(False)
  File "D:\a\cpython\cpython\lib\test\test_venv.py", line 484, in do_test_with_pip
    self.fail(msg.format(exc, details))
AssertionError: Command '['C:\\Users\\runneradmin\\AppData\\Local\\Temp\\tmp3cz40z50\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

**Subprocess Output**
Looking in links: c:\Users\RUNNER~1\AppData\Local\Temp\tmped5jdzqn

Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\setuptools-49.2.1-py3-none-any.whl

Processing c:\users\runneradmin\appdata\local\temp\tmped5jdzqn\pip-20.2.1-py2.py3-none-any.whl

Installing collected packages: setuptools, pip

Successfully installed pip-20.2.1 setuptools-49.2.1

Traceback (most recent call last):

  File "D:\a\cpython\cpython\lib\shutil.py", line 613, in _rmtree_unsafe

    os.unlink(fullname)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "D:\a\cpython\cpython\lib\tempfile.py", line 802, in onerror

    _os.unlink(path)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "D:\a\cpython\cpython\lib\runpy.py", line 197, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "D:\a\cpython\cpython\lib\runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "D:\a\cpython\cpython\lib\ensurepip\__main__.py", line 5, in <module>

    sys.exit(ensurepip._main())

  File "D:\a\cpython\cpython\lib\ensurepip\__init__.py", line 213, in _main

    return _bootstrap(

  File "D:\a\cpython\cpython\lib\ensurepip\__init__.py", line 132, in _bootstrap

    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)

  File "D:\a\cpython\cpython\lib\tempfile.py", line 827, in __exit__

    self.cleanup()

  File "D:\a\cpython\cpython\lib\tempfile.py", line 831, in cleanup

    self._rmtree(self.name)

  File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree

    _shutil.rmtree(name, onerror=onerror)

  File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree

    return _rmtree_unsafe(path, onerror)

  File "D:\a\cpython\cpython\lib\shutil.py", line 615, in _rmtree_unsafe

    onerror(os.unlink, fullname, sys.exc_info())

  File "D:\a\cpython\cpython\lib\tempfile.py", line 805, in onerror

    cls._rmtree(path)

  File "D:\a\cpython\cpython\lib\tempfile.py", line 813, in _rmtree

    _shutil.rmtree(name, onerror=onerror)

  File "D:\a\cpython\cpython\lib\shutil.py", line 737, in rmtree

    return _rmtree_unsafe(path, onerror)

  File "D:\a\cpython\cpython\lib\shutil.py", line 596, in _rmtree_unsafe

    onerror(os.scandir, path, sys.exc_info())

  File "D:\a\cpython\cpython\lib\shutil.py", line 593, in _rmtree_unsafe

    with os.scandir(path) as scandir_it:

NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmped5jdzqn\\pip-20.2.1-py2.py3-none-any.whl'



----------------------------------------------------------------------
msg375013 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 18:07
The issue above doesn't appear to repro on 3.9, so I guess master has started leaking a file handle, presumably in zipimport.

I'll see what I can track down, but can't be sure I'll have enough time to get it done for RC, so if anyone else wants to help out feel free.
msg375015 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 19:04
Okay, I've tracked it down to the new importlib.readers.ZipReader class keeping the zip file open, presumably until it gets GC'd. This is used by certifi to extract the CA certs from the whl when ensurepip is doing the self-install from the mounted wheel.

Jason is already on this bug, which is convenient :)

I haven't yet figured out whether there's a convenient way for the reader to not keep the ZIP open for as long as it exists, but I think that's going to be the safest fix.

We should definitely fix this one ourselves without forcing users to make changes to accommodate. As I mentioned above, it's only in 3.10 right now, but it's blocking updated pip and setuptools versions downlevel.
msg375020 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 21:43
Added some test cases to the PR that directly trigger the issue, specifically this one:

    def test_entered_path_does_not_keep_open(self):
        # This is what certifi does on import to make its bundle
        # available for the process duration.
        c = resources.path('ziptestdata', 'binary.file').__enter__()
        self.zip_path.unlink()

All the tests I added pass on 3.9 (with minor tweaks for moved test utils).

To unblock the upcoming releases, I'm going to do the backports first and leave this as a release blocker for 3.10.
msg375022 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 22:03
GitHub Actions has decided not to run CI today, so you'll have to look at Azure Pipelines for the test failures: https://dev.azure.com/Python/cpython/_build/results?buildId=67152&view=logs&j=c83831cd-3752-5cc7-2f01-8276919eb334&t=5a421c4a-0933-53d5-26b9-04b36ad165eb&l=8012
msg375023 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 22:10
New changeset 135de08128a76f49752ac57c316129500275e828 by Steve Dower in branch '3.8':
bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21775)
https://github.com/python/cpython/commit/135de08128a76f49752ac57c316129500275e828
msg375024 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-08-07 23:47
New changeset 70e9243a55be9c32b41f2149cdfa3957f96f8471 by Steve Dower in branch '3.9':
bpo-41490: Update ensurepip to install pip 20.2.1 and setuptools 49.2.1 (GH-21774)
https://github.com/python/cpython/commit/70e9243a55be9c32b41f2149cdfa3957f96f8471
msg376720 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-11 10:56
The CI failed on the PR on the master branch: https://github.com/python/cpython/pull/21748 First, the Travis CI job didn't start. I closed/opened the issue: Travis CI ran. New issue: bpo-41762. Once bpo-41762 will be fixed, it should be possible to attempt again to merge this PR.
msg376729 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-09-11 13:07
> I haven't yet figured out whether there's a convenient way for the reader to not keep the ZIP open for as long as it exists, but I think that's going to be the safest fix.

You may be right here. I don't fully understand the repro, but it seems to me like you're trying to delete a zip file while you have resources open in that zip file. I think we need a separate issue to capture the underlying defect.
msg376731 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2020-09-11 13:41
I think this reproduces the underlying issue:

>>> import zipfile
>>> from pathlib import Path
>>> p = zipfile.Path("tst.zip")
>>> Path("tst.zip").unlink()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\UK03306\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1321, in unlink
    self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'tst.zip'
>>>

Basically, zipfile.Path objects don't have any way to close the underlying zipfile, so you have to delete the path object to free up the file for deletion.

In the context of importlib, maybe permanently holding a reference to the zipfile.Path object can't work, and it needs to be re-opened each time it's needed?
msg376752 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-09-11 23:42
If you look at the PR into 3.9, it includes a test for this exact case. Start by copying that into 3.10 and then make it pass and everything should be good :)
msg377811 - (view) Author: László Kiss Kollár (lkollar) Date: 2020-10-02 14:34
pip 20.2.1 contains a regression which breaks `--system-site-package` integration: https://github.com/pypa/pip/issues/8695. It would be nice to include the latest patch version (20.2.3 at the moment) which fixes this and a few other issues.
msg377812 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-02 14:42
I think that the backport to 3.8 may have some unintended consequences in the last patch release as venv created with 3.8 now exhibit the pip regression (https://github.com/pypa/pip/issues/8695.).

Steve, would you be ok if we bump all branches (master, 3.9 and 3.8) to 20.2.3?
msg377879 - (view) Author: Pradyun Gedam (pradyunsg) * Date: 2020-10-03 12:31
+1 for bumping to 20.2.3, in case an upstream voice is helpful. :)

On Fri, Oct 2, 2020 at 8:12 PM Pablo Galindo Salgado <report@bugs.python.org>
wrote:

>
> Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:
>
> I think that the backport to 3.8 may have some unintended consequences in
> the last patch release as venv created with 3.8 now exhibit the pip
> regression (https://github.com/pypa/pip/issues/8695.).
>
> Steve, would you be ok if we bump all branches (master, 3.9 and 3.8) to
> 20.2.3?
>
> ----------
> nosy: +pablogsal
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue41490>
> _______________________________________
>
msg377961 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-10-04 16:45
New changeset 2cc6dc9896771ef3615abbb5ba80939a2f644a08 by Pablo Galindo in branch 'master':
bpo-41490: Bump vendored pip to version 20.2.3 (#22527)
https://github.com/python/cpython/commit/2cc6dc9896771ef3615abbb5ba80939a2f644a08
msg377962 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-10-04 17:11
New changeset 4b4d60f9287e0e52c2569876456f40efc11529b4 by Pablo Galindo in branch '3.9':
[3.9] bpo-41490: Bump vendored pip to version 20.2.3 (GH-22527). (GH-22544)
https://github.com/python/cpython/commit/4b4d60f9287e0e52c2569876456f40efc11529b4
msg377963 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-10-04 17:11
New changeset 28cd96f2e5cfb16566a75dca8473b71889746f10 by Pablo Galindo in branch '3.8':
[3.8] bpo-41490: Bump vendored pip to version 20.2.3 (GH-22527). (GH-22545)
https://github.com/python/cpython/commit/28cd96f2e5cfb16566a75dca8473b71889746f10
msg377964 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-10-04 17:13
This change goes into 3.9.0 with Pablo's fix.
msg378008 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-05 09:06
Can this issue be closed? Are we waiting for manual checks? Or is there any remaining thing to do?
msg378013 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-05 09:48
Yeah, there is some code in PR 21748 that should be merged (the new test).
msg378053 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-10-05 16:10
New changeset 168a8383c8358eea1b34df0e832f5d652faa6444 by Łukasz Langa (Pablo Galindo) in branch '3.9':
[3.9] bpo-41490: Bump vendored pip to version 20.2.3 (GH-22527). (GH-22544)
https://github.com/python/cpython/commit/168a8383c8358eea1b34df0e832f5d652faa6444
msg378099 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-06 08:52
I have no objection to bumping it further, provided someone has fixed importlib.resources on 3.10 (i.e. my backported tests pass)
msg378314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-09 11:08
setuptools is still more recent in 3.9 than in master. Look at PR 21748.

Can someone try to update setuptools in master?
msg379017 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-19 21:06
I did some bad things in git and so I had to recreate the PR against master, but the importlib.resources tests still fail.

Seems like the ensurepip/venv tests are fine though? Did something change in pip/certifi between 20.2.1 and 20.2.3?

I still consider the changes a release blocking regression, but if it's not blocking ensurepip then I guess the updated packages can go in now. I'll split up my PR tomorrow, but I'm not feeling like fighting git more tonight.
msg379595 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-10-25 18:21
New changeset df8d4c83a6e1727e766191896aeabde886979587 by Jason R. Coombs in branch 'master':
bpo-41490: ``path`` and ``contents`` to aggressively close handles (#22915)
https://github.com/python/cpython/commit/df8d4c83a6e1727e766191896aeabde886979587
History
Date User Action Args
2020-10-27 03:43:12vstinnersetnosy: - vstinner
2020-10-26 19:13:07brett.cannonsetnosy: - brett.cannon
2020-10-25 18:21:53jaracosetmessages: + msg379595
2020-10-23 12:55:12jaracosetpull_requests: + pull_request21845
2020-10-19 21:06:03steve.dowersetmessages: + msg379017
2020-10-19 19:38:43steve.dowersetpull_requests: + pull_request21738
2020-10-09 11:08:44vstinnersetmessages: + msg378314
2020-10-06 08:52:11steve.dowersetmessages: + msg378099
2020-10-05 16:10:19lukasz.langasetmessages: + msg378053
2020-10-05 09:48:41pablogsalsetmessages: + msg378013
2020-10-05 09:06:41vstinnersetmessages: + msg378008
2020-10-04 17:13:29lukasz.langasetmessages: + msg377964
2020-10-04 17:11:34lukasz.langasetmessages: + msg377963
2020-10-04 17:11:13lukasz.langasetmessages: + msg377962
2020-10-04 16:51:52pablogsalsetpull_requests: + pull_request21546
2020-10-04 16:49:31pablogsalsetpull_requests: + pull_request21545
2020-10-04 16:45:38lukasz.langasetmessages: + msg377961
2020-10-03 19:09:37pablogsalsetpull_requests: + pull_request21532
2020-10-03 12:31:05pradyunsgsetmessages: + msg377879
2020-10-02 14:42:46pablogsalsetnosy: + pablogsal
messages: + msg377812
2020-10-02 14:34:37lkollarsetnosy: + lkollar
messages: + msg377811
2020-09-11 23:42:54steve.dowersetmessages: + msg376752
2020-09-11 13:41:46paul.mooresetmessages: + msg376731
2020-09-11 13:07:17jaracosetmessages: + msg376729
2020-09-11 10:56:49vstinnersetnosy: + vstinner
messages: + msg376720
2020-08-07 23:51:38steve.dowersetversions: - Python 3.9
2020-08-07 23:47:47steve.dowersetmessages: + msg375024
2020-08-07 22:33:10steve.dowersetversions: - Python 3.8
2020-08-07 22:10:40steve.dowersetmessages: + msg375023
2020-08-07 22:03:36steve.dowersetmessages: + msg375022
2020-08-07 21:49:14steve.dowersetpull_requests: + pull_request20920
2020-08-07 21:45:39steve.dowersetpull_requests: + pull_request20919
2020-08-07 21:43:54steve.dowersetpriority: normal -> release blocker
nosy: + brett.cannon, eric.snow
messages: + msg375020

2020-08-07 19:04:43steve.dowersetmessages: + msg375015
2020-08-07 18:07:15steve.dowersetmessages: + msg375013
2020-08-05 23:24:46steve.dowersetmessages: + msg374909
2020-08-05 22:17:51steve.dowersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request20892
2020-08-05 21:26:19steve.dowercreate