This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: sys.prefix is set incorrectly on Mac OS X
Type: behavior Stage: resolved
Components: macOS Versions: Python 3.9
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: mppf, ned.deily, ronaldoussoren
Priority: normal Keywords:

Created on 2020-11-10 15:20 by mppf, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (12)
msg380677 - (view) Author: Michael Ferguson (mppf) Date: 2020-11-10 15:20
I have been trying to create a wrapper script for `python3` in a venv that behaves similarly to a symbolic link. I am able to use `exec -a` in bash to run `python3` with `argv[0]` set to the wrapper script. This allows it to function similarly to the symbolic link in a venv. However, this approach does not work on Mac OS X with a homebrew installation. I think this is a bug.


Here are the simple steps to reproduce (assuming bash shell):

```
cd /tmp
python3 -m venv test-venv
(exec -a test-venv/python3 python3 -c 'import sys; print(sys.executable); print (sys.prefix);')
```

### Good output (Ubuntu 20.04)
/tmp/test-venv/python-wrapper
/tmp

### Bad output (Homebrew on Mac OS X)
/usr/local/opt/python@3.9/bin/python3.9
/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9


Here are some things that might be related:
 * the Mac OS X framework launcher and how it uses `realpath` (and issue22490)
 * `site.py` code in `def venv` and the conditional on `__PYVENV_LAUNCHER__`. The `if` branch is not being run in this configuration.
 * setting the environment variable `PYTHONEXECUTABLE` (e.g. `export PYTHONEXECUTABLE=test-venv/python3` before the other commands) causes the `if` branch in the conditional on `__PYVENV_LAUNCHER__` in `site.py` `def venv` to be run. This allows `sys.executable` to be set as expected but `sys.prefix` is still wrong.



If you are interested in something closer to the use case, the below explains how to get a more user-facing reproducer:

$ python3 -m venv test-venv

-- put this into test-venv/python-wrapper --
#!/usr/bin/env bash

# remove path component to prevent infinite loop
export PATH=${PATH//test-venv/missing}

# Now run the real python3 interpreter but tell it that it
# is being launched at the current path, so it can
# correctly find dependencies in the venv
exec -a "$0" python3 "$@"

$ chmod a+x test-venv/python-wrapper

$ ./test-venv/python-wrapper -c 'import sys; print(sys.executable); print (sys.prefix);'

(and with this script the problematic behavior is exactly the same as the exec commands above)
msg380679 - (view) Author: Michael Ferguson (mppf) Date: 2020-11-10 15:35
In the above I meant to include the `bin` path in the examples, but it does not matter for the behavior

(exec -a test-venv/bin/python3 python3 -c 'import sys; print(sys.executable); print (sys.prefix);')
msg380680 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-11-10 15:47
I'm not sure I understand exactly what you are trying to accomplish but one potential issue strikes me: you may need to ensure you are execing the right python binary by including a more complete path:

$ (exec -a test-venv/bin/python3 test-venv/bin/python3 -c 'import sys; print(sys.executable); print (sys.prefix);')
/private/tmp/test-venv/bin/python3
/private/tmp/test-venv
msg380692 - (view) Author: Michael Ferguson (mppf) Date: 2020-11-10 18:13
> I'm not sure I understand exactly what you are trying to accomplish but one potential issue strikes me: you may need to ensure you are execing the right python binary by including a more complete path:

That does not help with the original problem I was trying to solve, because I was trying to create a wrapper script that used whichever `python3` is available according to the `PATH` variable (other than potentially the one for this venv).

Whether or not you think that is a reasonable thing to do, the examples I showed have a difference in behavior between Mac OS X and linux that is probably undesirable.
msg380695 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-11-10 19:03
Sorry, I didn't intend to criticize what you are trying to do, I just not sure I understand it so I could give a more helpful response. It seems to me that an explanation for the difference in behavior you are seeing between your Ubuntu and macOS setups is in exactly what the value of PATH is when the exec is performed on each and exactly which python3 is found first. For example, when I run the test exec on my macOS system, it is clear that the python3 being invoked is not the venv one but a different python3 altogether that shows up earlier on PATH. Could such differences explain what you are seeing?
msg380697 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-10 19:08
The difference in behaviour is because the executable does not look at argv[0] to find its location on macOS. Your trick therefore doesn't work.

I prefer the current behaviour.

What I don't understand is *why* you want to do this. If I read your report correctly you're trying to run an Python 3 executable outside of the virtual environment as if it is in the virtual environment. Why not just use the binary inside the virtual environment?
msg380702 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-10 19:50
Note that the same is true for python3 outside of a venv, it does not use argv[0] to locate says.prefix (for framework builds).  That’s intentional.
msg380710 - (view) Author: Michael Ferguson (mppf) Date: 2020-11-10 20:21
> For example, when I run the test exec on my macOS system, it is clear that the python3 being invoked is not the venv one but a different python3 altogether that shows up earlier on PATH.

In the test case I am interested in, PATH is not set to the venv at all. You're supposed to be able to run a script in a venv without activating. In my case, I am trying to write a wrapper script that behaves similarly to the symbolic link in test-venv/bin/python3 but that works if the path to the python3 interpreter changes in the future. (For example, one might install and then uninstall pyenv).
msg380714 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-10 21:15
The way sys.prefix is calculated on macOS ensures that the correct sys.prefix is calculated even if you copy the binary to a different location. That's functionality I don't want to drop.
msg380791 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-11 19:34
I've thought about this some more and haven't changed my opinion: the current behaviour is intentional and won't change.

I'm therefore closing this issue.
msg380825 - (view) Author: Michael Ferguson (mppf) Date: 2020-11-12 14:53
> The way sys.prefix is calculated on macOS ensures that the correct sys.prefix is calculated even if you copy the binary to a different location. That's functionality I don't want to drop.

I agree with you that it's important for the Python interpreter to find the libraries it is installed with even if the framework launcher is copied to a different location.

However I think it's possible to support finding the installation while fixing the issue I am bringing up. Finding the installation amounts to getting the correct `sys.base_prefix` (no matter where the launcher is copied). It seems to me that if the launcher is copied to a different directory, `sys.prefix` should change in some cases. That already happens when making a `venv` with copies instead of links. Could the framework launcher consider `argv[0]` without causing problems in this use case?

I think this is a bug and not a feature tradeoff because a platform-specific wrapper should be as transparent as possible and as a result should keep the `argv[0]` behavior that occurs without the wrapper.


In any case, do you think that there is a way to get the behavior I am looking for (basically, invoke a Python interpreter that has sys.prefix set to the venv but without a symbolic link)?
msg380827 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-12 15:15
> In any case, do you think that there is a way to get the behavior I am looking for (basically, invoke a Python interpreter that has sys.prefix set to the venv but without a symbolic link)?

Not that I know.

Using the python binary in the virtual environment is part of the design of this feature. That you can "fake" this on a lot of unix platforms by using a different interpreter with argv[0] set to a different path is a happy coincidence. 

I'm also not convinced that your trick actually accomplishes what you hope, the virtual environment still has a dependency on the Python installation in the original location. That location is mentioned in pyvenv.cfg at the root of the environment, and AFAIK that configuration is used to set up sys.path.
History
Date User Action Args
2022-04-11 14:59:37adminsetgithub: 86478
2020-11-12 15:15:01ronaldoussorensetmessages: + msg380827
2020-11-12 14:53:54mppfsetmessages: + msg380825
2020-11-11 19:34:53ronaldoussorensetstatus: open -> closed
resolution: wont fix
messages: + msg380791

stage: resolved
2020-11-10 21:15:53ronaldoussorensetmessages: + msg380714
2020-11-10 20:21:28mppfsetmessages: + msg380710
2020-11-10 19:50:22ronaldoussorensetmessages: + msg380702
2020-11-10 19:08:16ronaldoussorensetmessages: + msg380697
2020-11-10 19:03:43ned.deilysetmessages: + msg380695
2020-11-10 18:13:05mppfsetmessages: + msg380692
2020-11-10 15:47:46ned.deilysetmessages: + msg380680
2020-11-10 15:35:50mppfsetmessages: + msg380679
2020-11-10 15:20:27mppfcreate