classification
Title: Provide an option to venv to put files in a bin/ directory on Windows
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: DamlaAltun, brett.cannon, uranusjr, vinay.sajip
Priority: normal Keywords:

Created on 2018-10-16 22:12 by brett.cannon, last changed 2019-12-03 18:36 by brett.cannon. This issue is now closed.

Messages (14)
msg327845 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-10-16 22:12
Having venv install files into Scripts/ on Windows but into bin/ on UNIX is troublesome for anything that tries to be cross-platform regarding virtual environments. Having a way to create a virtual environment on Windows where bin/ is used over Scripts/ would be handy.
msg328052 - (view) Author: DamlaAltun (DamlaAltun) * Date: 2018-10-19 16:49
Working on that.
msg328125 - (view) Author: DamlaAltun (DamlaAltun) * Date: 2018-10-20 05:42
I couldn't find a way to prevent being in seperate forms. When i add custom bin, custom include and custom libpaths 4 files are creating. The  normal `bin`, normal `include` and custom `cbin` and custom `cinc`. You can unassign me.
msg357583 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-27 20:19
The problem of bin over Scripts is that IIRC Scripts is baked-in in Windows, not just for venvs (e.g. a standard Python installation puts stuff in Scripts). So venv just sticks with the platform convention.
msg357617 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-28 15:11
There are more differences than Scripts/bin, like Windows use Lib but POSIX uses lib/pythonX.Y. IMO it’s probably better to stick with platform conventions, especially since those can be discovered with sysconfig.get_paths(expand=False).

I wonder whether it’d be a good idea to record what scheme was used to create the venv (in pyvenv.cfg). Any Python runtime can use that value in get_paths(scheme=..., expand=False) to know about the venv’s structure, even if the venv was created on another platform. This would be particularly useful to e.g. inspect a Windows-native venv in a WSL Python.
msg357624 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-11-28 17:29
I've personally never come across Scripts in any other situation than virtual environments, but that isn't saying much about my Windows exposure either. :)

Basically activation is the biggest stumbling block I find with new users when it comes to trying to teach them about using virtual environments and this platform difference doesn't help. I would like to come up with _some_ solution to make it easier, even if it's something I have to put into the Python Launcher (although trying to get people to even agree on a name for virtual environments created beside the code turns out to be a hot topic). I would consider trying to come up with code so we have a `pipenv shell`/`conda shell` equivalent, but I've been told it is not pleasant to try and make work.
msg357625 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-28 17:45
> Basically activation is the biggest stumbling block I find with new users

Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok, and would cover the vast majority of users? (i.e. not including the other, less-common POSIX shells.)
msg357626 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-28 18:22
> Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok [...]?

The if-Windows-X-else-Y part isn’t that hard; it’s the activate part that is :p

I think Brett is thinking about eliminating the manual activate part entirely, but any tool trying to automate that needs to do a lot of platform-specific checks.

---

> I've personally never come across Scripts in any other situation than virtual environments [...].

Windows use a Scripts directory to store… scripts (Setuptools terminology, i.e. console and gui script entry points). So e.g. the global pip.exe would be at "{sys.prefix}\Scripts\pip.exe" (or is it sys.exec_prefix?) `pip install --user` would also install scripts into `%APPDATA%\Programs\Python\PythonXY\Scripts`. So venv’s setup is consistent with the rest of Python.

This directory structure can be expanded from sysconfig. So the proposal in my previous comment is to record the scheme in pyvenv.cfg, so you can have something like

def read_venv_scheme(env_dir):
    with open(os.path.join(env_dir, 'pyvenv.cfg')) as f:
        for line in f:
            key, value = (p.strip() for p in line.split('='))
            if key == 'scheme':
                return value

def get_venv_environ_patch(env_dir):
    scheme = read_venv_scheme(env_dir)
    bin_dir = sysconfig.get_path('scripts', scheme=scheme, expand=False).format(base=env_dir)
    return {'VIRTUAL_ENV': env_dir, 'PATH': bin_dir}

and this would give you the appropriate value on any platform.
msg357630 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-28 22:18
> I think Brett is thinking about eliminating the manual activate part entirely, but any tool trying to automate that needs to do a lot of platform-specific checks.

If you have more than one venv then it seems like some manual step is required to switch between them. What about a tool like Pew (Python Env Wrapper), which from the README "is completely shell-agnostic and thus works on bash, zsh, fish, powershell, etc."

https://github.com/berdario/pew

I haven't used in under Windows, but it works a treat on POSIX for me, YMMV of course.

Of course, scripts installed in venvs never need activation to run - they pick up the correct interpreter from their venv automatically.
msg357649 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-11-29 18:38
> Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok, and would cover the vast majority of users?

Sure, but how many times do we need to make people type, write, or say that exact line instead of a single line of "you activate by doing <command>"?

> So venv’s setup is consistent with the rest of Python.

Right, it's a Python-on-Windows thing, not a Windows thing itself to my knowledge.

> I think Brett is thinking about eliminating the manual activate part entirely

I'm actually after a single command to handle activation of a virtual environment. It's a point of friction on your first day of learning Python and I have seen it solved multiple times at this point by multiple tools. This seemed like a potential simple way to solve it to me, but apparently not everyone agrees. ;)

Now I realize that if we don't worry about the prompt changing it's actually very straight-forward, and so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables and prints out a message about what is happening is enough to do the trick (and if people want the prompt to change they can tweak their shell configs to detect something like `__VENV_PROMPT__` being set and use it appropriately).

> Of course, scripts installed in venvs never need activation to run

Sure, but then that doesn't mean activation isn't convenient. :) Otherwise what is the point of having the activation scripts?
msg357655 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-29 22:22
> so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables

If venv is run in a child process of the shell, how does it set those variables in the parent shell? I thought that wasn't possible (on POSIX, anyway), which is why e.g. pew spawns a subshell.

> Sure, but then that doesn't mean activation isn't convenient

Indeed it is. But many less experienced users think that it's always a necessary step, when it isn't always so.
msg357720 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-02 17:41
> If venv is run in a child process of the shell, how does it set those variables in the parent shell?

You can't. Tools like `pipenv shell` and `conda shell` end up with a new shell running instead of changing the current shell in-place.
msg357728 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-12-02 20:24
> You can't. Tools like `pipenv shell` and `conda shell` end up with a new shell running instead of changing the current shell in-place.

Indeed. I raised it because of what you said earlier:

> and so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables and prints out a message about what is happening is enough to do the trick

So not that simple, then, if it has to handle multiple shells cross-platform. Especially as if tools like pew, pipenv, conda already do this. Are you proposing to bring that functionality, which these third-party tools perform currently, into the stdlib?
msg357757 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-03 18:36
> Are you proposing to bring that functionality, which these third-party tools perform currently, into the stdlib?

I was hoping to, but it turns out you can't make this work under Windows without inspecting the parent process (PowerShell and Command Prompt provide no way to know the path to the shell being run).

So since there's no interest in providing a way to unify the paths I'm going to close this issue.
History
Date User Action Args
2019-12-03 18:36:53brett.cannonsetstatus: open -> closed
resolution: rejected
messages: + msg357757

stage: test needed -> resolved
2019-12-02 20:24:15vinay.sajipsetmessages: + msg357728
2019-12-02 17:41:43brett.cannonsetmessages: + msg357720
2019-11-29 22:22:34vinay.sajipsetmessages: + msg357655
2019-11-29 18:38:57brett.cannonsetmessages: + msg357649
2019-11-28 22:18:48vinay.sajipsetmessages: + msg357630
2019-11-28 18:22:18uranusjrsetmessages: + msg357626
2019-11-28 17:45:44vinay.sajipsetmessages: + msg357625
2019-11-28 17:29:59brett.cannonsetmessages: + msg357624
2019-11-28 15:11:55uranusjrsetnosy: + uranusjr
messages: + msg357617
2019-11-27 20:19:17vinay.sajipsetmessages: + msg357583
2019-11-27 17:56:10brett.cannonsetversions: + Python 3.9, - Python 3.8
2019-11-27 17:56:03brett.cannonsetnosy: + vinay.sajip
2018-10-20 05:43:00DamlaAltunsetmessages: + msg328125
2018-10-19 16:49:27DamlaAltunsetnosy: + DamlaAltun
messages: + msg328052
2018-10-16 22:12:19brett.cannonsetassignee: brett.cannon
2018-10-16 22:12:12brett.cannoncreate