This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: 3.11.0a3: under tox, sys._base_executable is wrong
Type: Stage: commit review
Components: Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, nedbat, pablogsal, saaketp, steve.dower, vinay.sajip, vstinner
Priority: normal Keywords: 3.11regression, patch

Created on 2021-12-10 00:24 by nedbat, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 30144 merged steve.dower, 2021-12-16 15:06
Messages (30)
msg408166 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-10 00:24
Under tox, sys._base_executable is not an actual file for 3.11.0a3.  It was fine in 3.11.0a2.

To reproduce:

--- 8< --------------------
# tox.ini
[tox]
envlist = py{310,311a2,311}
skipsdist = True

[testenv]
commands =
    python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"

[testenv:py311a2]
# This is the path to 3.11.0a2 if you have it.
basepython = /usr/local/pyenv/pyenv/versions/3.11.0a2/bin/python3
----------------------------

Create a new Python 3.8 virtualenv, and install latest tox (3.24.4 for me).

Then run "tox".  I see:

--------------------------------------------------------------------------------
py310 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310
py310 run-test-pre: PYTHONHASHSEED='534434199'
py310 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310/bin/python
True
py311a2 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2
py311a2 run-test-pre: PYTHONHASHSEED='534434199'
py311a2 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2/bin/python
True
py311 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311
py311 run-test-pre: PYTHONHASHSEED='534434199'
py311 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/usr/local/pyenv/pyenv/versions/3.11.0a3/python
False
_________________________________________________________ summary _________________________________________________________
  py310: commands succeeded
  py311a2: commands succeeded
  py311: commands succeeded
  congratulations :)
--------------------------------------------------------------------------------

This came to my attention because the coverage.py test suite uses "python -m venv" to create environments. They worked under 3.11.0a2, but failed under a3.  I tracked it down to a difference in sys._base_executable.

I couldn't see a difference in those values without tox, but I'm not sure why it changes the results.
msg408167 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-12-10 01:18
Steve, could this be related to the changes in getpath?
msg408168 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-12-10 01:22
Ned, are you able to bisect this or provide a simpler reproducer that doesn't involve tox?
msg408180 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-10 09:14
Commit 9f2f7e42269db74a89fc8cd74d82a875787f01d7 has correct _base_executable

$ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-115-g9f2f7e42269:9f2f7e42269, Dec 10 2021, 10:09:54) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/venv/bin/python'

_base_executable in commit 99fcf1505218464c489d419d4500f126b6d6dc28 is wrong

$ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-116-g99fcf150521:99fcf150521, Dec 10 2021, 10:12:35) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/python'
msg408195 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-10 11:51
git bisect also identifies that commit as the first bad:


99fcf1505218464c489d419d4500f126b6d6dc28 is the first bad commit
commit 99fcf1505218464c489d419d4500f126b6d6dc28
Author: Steve Dower <steve.dower@python.org>
Date:   Fri Dec 3 00:08:42 2021 +0000

    bpo-45582: Port getpath[p].c to Python (GH-29041)

    The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.

    This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.
msg408206 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-12-10 13:05
Indeed, seems my original hunch is correct. Steve, could you take a look when you have some time?
msg408278 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-11 00:10
I'm going to need a decent amount of time to learn all of these components, because I never use this OS, Tox, nor virtualenv :) I'll try and get to it, but don't hold your breath. Luckily, Modules/getpath.py is much easier to follow and modify than the old systems.

If there's a pyvenv.cfg involved, base_executable should be calculated based on the "home" key in it. Previously, I don't think we calculated it at all on Linux - it was just sys.executable before site.py changes anything.

On Windows, it was always intended to be "the executable that new venvs should be created with" so that venvs created from venvs would be based off the same install, rather than trying to chain. I have no idea what the correct path for that is in this context, so could do with some help.
msg408311 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-11 15:45
Tox isn't needed, just venv from the stdlib:


$ python3.11.0a2 -m venv venv_a2

$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/private/tmp/venv_a2/bin/python
True

$ python3.11.0a3 -m venv venv_a3

$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/usr/local/bin/python
False
msg408312 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-11 15:47
What's the contents of the pyvenv.cfg in these cases?

It looks like the first case is definitely wrong, because the base 
executable should not be in "venv_a2" (that's sys.executable), but I 
don't know where it should be on your system.
msg408313 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-11 16:15
The two venvs seem analogous:


$ cat venv_a2/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0

$ ls -al venv_a2/bin
total 72
drwxr-xr-x  13 nedbatchelder  wheel   416 Dec 11 10:43 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 11 10:43 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 11 10:43 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 11 10:43 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 11 10:43 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 11 10:43 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3.11*
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3.11@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    29 Dec 11 10:43 python3.11.0a2@ -> /usr/local/bin/python3.11.0a2

$ cat venv_a3/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0

$ ls -al venv_a3/bin
total 72
drwxr-xr-x  13 nedbatchelder  wheel   416 Dec 11 10:43 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 11 10:43 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 11 10:43 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 11 10:43 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 11 10:43 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 11 10:43 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3.11*
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3.11@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    29 Dec 11 10:43 python3.11.0a3@ -> /usr/local/bin/python3.11.0a3
msg408318 - (view) Author: Saaket Prakash (saaketp) Date: 2021-12-11 17:04
I tried the same stuff as nedbat on WSL2, and I see similar change in the path of sys._base_executable (though I get a different "base" path on a3, so the path exists even there).

$ ~/.pyenv/versions/3.11.0a2/bin/python -m venv venv_a2
$ ~/.pyenv/versions/3.11.0a3/bin/python -m venv venv_a3
$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/venv_a2/bin/python
True

$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/.pyenv/versions/3.11.0a3/bin/python
True

$ cat venv_a2/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a2/bin
include-system-site-packages = false
version = 3.11.0

$ cat venv_a3/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a3/bin
include-system-site-packages = false
version = 3.11.0
msg408319 - (view) Author: Saaket Prakash (saaketp) Date: 2021-12-11 17:14
But on windows with the python.org installer, the behavior is same for both both a2 and a3.

# With a3 installed

> py -m venv venv_a3
> venv_a3/Scripts/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
C:\Users\ss\AppData\Local\Programs\Python\Python311\python.exe
True
> cat venv_a3/pyvenv.cfg
home = C:\Users\ss\AppData\Local\Programs\Python\Python311
include-system-site-packages = false
version = 3.11.0

# With a2 installed

> py -m venv venv_a2
> venv_a2/Scripts/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
C:\Users\ss\AppData\Local\Programs\Python\Python311\python.exe
True
> cat venv_a2/pyvenv.cfg
home = C:\Users\ss\AppData\Local\Programs\Python\Python311
include-system-site-packages = false
version = 3.11.0
msg408483 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-13 22:24
> $ cat venv_a3/pyvenv.cfg
> home = /home/ss/.pyenv/versions/3.11.0a3/bin

> $ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
> /home/ss/.pyenv/versions/3.11.0a3/bin/python
> True

This is the intended behaviour, and yes it's changed from previous versions (but not on Windows, where it was always correct). The previous value was incorrect, hence it was marked as an internal field.

If your venv doesn't have a "python" binary in the directory set as "home" in pyvenv.cfg (which by definition, according to Lib/venv/__init__.py, is dirname(sys.executable) ), then there's some other issue with venv on your platform.
msg408484 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-13 22:25
I'm downgrading this from release blocker. If Vinay thinks there's a venv-related release blocker here he's welcome to raise the priority again, but I only see an intentional change to an internal value. Tools relying on internal fields will have to adapt for 3.11.
msg408489 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2021-12-13 22:46
> This is the intended behaviour, and yes it's changed from previous versions ... The previous value was incorrect, hence it was marked as an internal field.

But the value as it's calculated now seems to give a file that doesn't exist - how can that be correct?
msg408490 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-13 22:50
This started because a set of tests in the coverage.py test suite fail with 3.11.0a3.  They attempt to create a venv in the test, and the test is already running inside a virtualenv.  The venv creation fails, with a reference to a non-existent file.

I wrote the bug about sys._base_executable because that seemed to be the central difference that was causing the problem.  I was not relying on sys._base_executable, venv was.

I will find steps to reproduce that don't reference sys._base_executable if you need them.
msg408491 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-13 22:54
Is sys._base_executable correct without a venv? It should be the same as sys.executable in that case.

venv calculates 'home' here: Lib/venv/__init__.py#L117

        executable = sys._base_executable
        dirname, exename = os.path.split(os.path.abspath(executable))
        context.executable = executable # not relevant to this bug
        context.python_dir = dirname    # written as "home = ..."

If the path doesn't exist later on, it's because it didn't exist in the first place. *That* could be the real bug.
msg408492 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-13 22:56
> But the value as it's calculated now seems to give a file that doesn't exist - how can that be correct?

Because we never actually use the executable referenced by the 'home' path in a pyvenv.cfg. It's only used as a starting point to locate a couple of entries in sys.path. So if the executable isn't there, it's no big deal.

It shouldn't even matter in the nested-creation case, because we still don't execute it then. It's just how we end up with the same value for 'home' in any venvs created within a venv.
msg408493 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-13 23:00
> Because we never actually use the executable referenced by the 'home' path in a pyvenv.cfg.

This is actually not true on Windows or (I believe) some situations on macOS, where we need to use a redirecting launcher to actually launch the binary specified in this value. But on POSIX, because all the references are hardcoded in the binary, we can just copy the original around and it never loses track of its real home and never has to launch its original copy.
msg408494 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-13 23:05
Demonstration of a problem with only stdlib, and no undocumented features. This is on a Mac:

This works:

$ python3.10 -V
Python 3.10.0

$ python3.10 -m venv v310

$ v310/bin/python -m venv v310-nested

$ v310-nested/bin/python -V
Python 3.10.0

This does not work:

$ python3.11 -V
Python 3.11.0a3

$ python3.11 -m venv v311

$ v311/bin/python -m venv 311-nested
Error: [Errno 2] No such file or directory: '/private/tmp/bpo-46028/311-nested/bin/python'
msg408525 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-14 12:50
> $ v311/bin/python -m venv 311-nested
> Error: [Errno 2] No such file or directory: '/private/tmp/bpo-46028/311-nested/bin/python'

I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though? 
Probably that's the issue here - I don't know how else to get the real 
executable *name* other than copying from argv[0], and it isn't supposed 
to be necessary.

You also have v311/bin/python3.11, right? If you use that one, does it 
work? I'm trying to narrow down where the base executable is actually 
being launched and why.
msg408526 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-14 12:57
Or possibly that error is coming from the attempt to copy it? And since 
both executable and base_executable don't have the 3/3.x suffix, the 
copy is failing because the "real" binary does have the suffix.

This could be corrected in getpath.py with a platform-specific quirk 
that searches for suffixed binaries for base_executable, but for 
performance reasons I think we'd prefer to have that check in venv so 
that it doesn't impact every single launch of CPython.

The actual binary could also be added to pyvenv.cfg as another value - 
parsing that out in getpath.py is now considerably easier for someone to 
add than it used to be.
msg408528 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-14 13:15
> I assume /private/tmp/bpo-46028/311-nested/bin/python3.11 exists though? 

Yes, that file exists, but it's a symlink to a non-existent file:

$ ls -al 311-nested/bin
total 0
drwxr-xr-x  5 nedbatchelder  wheel  160 Dec 13 18:04 ./
drwxr-xr-x  6 nedbatchelder  wheel  192 Dec 13 18:04 ../
lrwxr-xr-x  1 nedbatchelder  wheel   21 Dec 13 18:04 python@ -> /usr/local/bin/python
lrwxr-xr-x  1 nedbatchelder  wheel    6 Dec 13 18:04 python3@ -> python
lrwxr-xr-x  1 nedbatchelder  wheel    6 Dec 13 18:04 python3.11@ -> python
$ ls -al /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory
msg408569 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-14 21:41
Does the first venv's 'python' link to python3[.11]? If so, maybe _base_executable should be based on real_executable's filename rather than executable (that is, *after* resolving symlinks rather than before).

I don't *think* that will cause any issues, and it shouldn't be any more expensive to compute. Only has to change for when a pyvenv.cfg is detected I think.
msg408692 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-16 11:48
Here's the experiment again with 3.10.1 and 3.11.0a3, and more ls's:


$ python3.10 -V
Python 3.10.1

$ python3.10 -m venv v310

$ ls -al v310/bin
total 72
drwxr-xr-x  12 nedbatchelder  wheel   384 Dec 16 06:42 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 16 06:42 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 16 06:42 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 16 06:42 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 16 06:42 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 16 06:42 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:42 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:42 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:42 pip3.10*
lrwxr-xr-x   1 nedbatchelder  wheel    10 Dec 16 06:42 python@ -> python3.10
lrwxr-xr-x   1 nedbatchelder  wheel    10 Dec 16 06:42 python3@ -> python3.10
lrwxr-xr-x   1 nedbatchelder  wheel    25 Dec 16 06:42 python3.10@ -> /usr/local/bin/python3.10

$ ls -al /usr/local/bin/python3.10
lrwxr-xr-x  1 nedbatchelder  admin  53 Dec 16 06:38 /usr/local/bin/python3.10@ -> /usr/local/pyenv/pyenv/versions/3.10.1/bin/python3.10

$ v310/bin/python -m venv v310-nested

$ v310-nested/bin/python -V
Python 3.10.1

$ ls -al v310-nested/bin
total 72
drwxr-xr-x  12 nedbatchelder  wheel   384 Dec 16 06:43 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 16 06:43 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 16 06:43 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  2014 Dec 16 06:43 activate
-rw-r--r--   1 nedbatchelder  wheel   940 Dec 16 06:43 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2082 Dec 16 06:43 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   249 Dec 16 06:43 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   249 Dec 16 06:43 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   249 Dec 16 06:43 pip3.10*
lrwxr-xr-x   1 nedbatchelder  wheel    37 Dec 16 06:43 python@ -> /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x   1 nedbatchelder  wheel     6 Dec 16 06:43 python3@ -> python
lrwxr-xr-x   1 nedbatchelder  wheel     6 Dec 16 06:43 python3.10@ -> python

$ ls -al /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 06:42 /private/tmp/bpo46028/v310/bin/python@ -> python3.10


$ python3.11 -V
Python 3.11.0a3

$ python3.11 -m venv v311

$ ls -al v311/bin
total 72
drwxr-xr-x  12 nedbatchelder  wheel   384 Dec 16 06:45 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 16 06:45 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 16 06:45 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 16 06:45 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 16 06:45 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 16 06:45 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:45 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:45 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   246 Dec 16 06:45 pip3.11*
lrwxr-xr-x   1 nedbatchelder  wheel    10 Dec 16 06:45 python@ -> python3.11
lrwxr-xr-x   1 nedbatchelder  wheel    10 Dec 16 06:45 python3@ -> python3.11
lrwxr-xr-x   1 nedbatchelder  wheel    25 Dec 16 06:45 python3.11@ -> /usr/local/bin/python3.11

$ ls -al /usr/local/bin/python3.11
lrwxr-xr-x  1 nedbatchelder  admin  55 Dec  9 12:23 /usr/local/bin/python3.11@ -> /usr/local/pyenv/pyenv/versions/3.11.0a3/bin/python3.11

$ v311/bin/python -m venv v311-nested
Error: [Errno 2] No such file or directory: '/private/tmp/bpo46028/v311-nested/bin/python'

$ ls -al v311-nested/bin
total 0
drwxr-xr-x  5 nedbatchelder  wheel  160 Dec 16 06:45 ./
drwxr-xr-x  6 nedbatchelder  wheel  192 Dec 16 06:45 ../
lrwxr-xr-x  1 nedbatchelder  wheel   21 Dec 16 06:45 python@ -> /usr/local/bin/python
lrwxr-xr-x  1 nedbatchelder  wheel    6 Dec 16 06:45 python3@ -> python
lrwxr-xr-x  1 nedbatchelder  wheel    6 Dec 16 06:45 python3.11@ -> python

$ ls -al /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory
msg408710 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-12-16 15:08
This PR *might* be a fix, but I think it's only partial. It isn't going to work if you've made a venv and copied "python3"->"python", for example.

It still might be best solved by writing base_executable into pyvenv.cfg, and then simply reading it out.
msg408744 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-12-17 00:51
Thanks, your change looks good.  The exact symlinks in the nested venv's are different for 3.10.1 and your 3.11, but they both work:

$ python3.10 -c "import sys; print(sys.version)"
3.10.1 (main, Dec 14 2021, 08:30:13) [Clang 12.0.0 (clang-1200.0.32.29)]

$ python3.10 -m venv v310

$ ls -al v310/bin/py*
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:48 v310/bin/python@ -> python3.10
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:48 v310/bin/python3@ -> python3.10
lrwxr-xr-x  1 nedbatchelder  wheel  25 Dec 16 18:48 v310/bin/python3.10@ -> /usr/local/bin/python3.10

$ v310/bin/python -m venv v310-nested

$ ls -al v310-nested/bin/py*
lrwxr-xr-x  1 nedbatchelder  wheel  37 Dec 16 18:48 v310-nested/bin/python@ -> /private/tmp/bpo46028/v310/bin/python
lrwxr-xr-x  1 nedbatchelder  wheel   6 Dec 16 18:48 v310-nested/bin/python3@ -> python
lrwxr-xr-x  1 nedbatchelder  wheel   6 Dec 16 18:48 v310-nested/bin/python3.10@ -> python

$

$ python3.11 -c "import sys; print(sys.version)"
3.11.0a3+ (heads/pr/30144:8b260a8606, Dec 16 2021, 14:31:40) [Clang 12.0.0 (clang-1200.0.32.29)]

$ python3.11 -m venv v311

$ ls -al v311/bin/py*
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:50 v311/bin/python@ -> python3.11
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:50 v311/bin/python3@ -> python3.11
lrwxr-xr-x  1 nedbatchelder  wheel  25 Dec 16 18:50 v311/bin/python3.11@ -> /usr/local/bin/python3.11

$ v311/bin/python -m venv v311-nested

$ ls -al v311-nested/bin/py*
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:50 v311-nested/bin/python@ -> python3.11
lrwxr-xr-x  1 nedbatchelder  wheel  10 Dec 16 18:50 v311-nested/bin/python3@ -> python3.11
lrwxr-xr-x  1 nedbatchelder  wheel  33 Dec 16 18:50 v311-nested/bin/python3.11@ -> /usr/local/cpython/bin/python3.11
msg410850 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-01-18 10:13
See also: https://discuss.python.org/t/virtual-environments-vs-nix-python-upgrades/12588 "Virtual environments vs. *nix Python upgrades".
msg410872 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-01-18 15:46
New changeset 7407fe4c25ba0308d49e3e88e4a107ef32251cdc by Steve Dower in branch 'main':
bpo-46028: Calculate base_executable by resolving symlinks in a venv (GH-30144)
https://github.com/python/cpython/commit/7407fe4c25ba0308d49e3e88e4a107ef32251cdc
msg410873 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-01-18 15:47
Merged my PR, but I want to leave this open in commit review for now - I'm not sure it deals with all the issues here, and probably not everything from the Discourse thread linked by Victor (though it might come close).
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90186
2022-01-18 15:47:49steve.dowersetmessages: + msg410873
stage: patch review -> commit review
2022-01-18 15:46:34steve.dowersetmessages: + msg410872
2022-01-18 10:13:33vstinnersetnosy: + vstinner
messages: + msg410850
2021-12-17 00:51:01nedbatsetmessages: + msg408744
2021-12-16 15:08:13steve.dowersetmessages: + msg408710
2021-12-16 15:06:43steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request28362
2021-12-16 11:48:16nedbatsetmessages: + msg408692
2021-12-14 21:41:13steve.dowersetmessages: + msg408569
2021-12-14 13:15:26nedbatsetmessages: + msg408528
2021-12-14 12:57:53steve.dowersetmessages: + msg408526
2021-12-14 12:50:02steve.dowersetmessages: + msg408525
title: 3.11.0a3: sys._base_executable is wrong, breaks venv - it wasn't under 3.11.0a2 -> 3.11.0a3: under tox, sys._base_executable is wrong
2021-12-14 10:09:02vinay.sajipsettitle: 3.11.0a3: under tox, sys._base_executable is wrong -> 3.11.0a3: sys._base_executable is wrong, breaks venv - it wasn't under 3.11.0a2
2021-12-13 23:05:58nedbatsetmessages: + msg408494
2021-12-13 23:00:07steve.dowersetmessages: + msg408493
2021-12-13 22:56:52steve.dowersetmessages: + msg408492
2021-12-13 22:54:17steve.dowersetmessages: + msg408491
2021-12-13 22:50:55nedbatsetmessages: + msg408490
2021-12-13 22:46:01vinay.sajipsetmessages: + msg408489
2021-12-13 22:25:45steve.dowersetpriority: release blocker -> normal

messages: + msg408484
2021-12-13 22:24:47steve.dowersetnosy: + vinay.sajip
messages: + msg408483
2021-12-11 17:14:12saaketpsetmessages: + msg408319
2021-12-11 17:04:58saaketpsetnosy: + saaketp
messages: + msg408318
2021-12-11 16:15:21nedbatsetmessages: + msg408313
2021-12-11 15:47:47steve.dowersetmessages: + msg408312
2021-12-11 15:45:36nedbatsetmessages: + msg408311
2021-12-11 00:10:22steve.dowersetmessages: + msg408278
2021-12-10 13:05:52pablogsalsetmessages: + msg408206
2021-12-10 11:51:25nedbatsetmessages: + msg408195
2021-12-10 09:14:08christian.heimessetnosy: + christian.heimes
messages: + msg408180
2021-12-10 01:22:36pablogsalsetmessages: + msg408168
2021-12-10 01:19:03pablogsalsetpriority: normal -> release blocker
2021-12-10 01:18:18pablogsalsetnosy: + steve.dower
messages: + msg408167
2021-12-10 00:24:21nedbatcreate