This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Provide an option to venv to put files in a bin/ directory on Windows
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: DamlaAltun, earonesty, uranusjr, vinay.sajip
Priority: normal Keywords:

Created on 2018-10-16 22:12 by brett.cannon, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18083 closed earonesty, 2020-01-20 22:03
Messages (19)
msg327845 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-10-16 22:12
Having venv install files into Scripts/ on Windows but into bin/ on UNIX is troublesome for anything that tries to be cross-platform regarding virtual environments. Having a way to create a virtual environment on Windows where bin/ is used over Scripts/ would be handy.
msg328052 - (view) Author: DamlaAltun (DamlaAltun) * Date: 2018-10-19 16:49
Working on that.
msg328125 - (view) Author: DamlaAltun (DamlaAltun) * Date: 2018-10-20 05:42
I couldn't find a way to prevent being in seperate forms. When i add custom bin, custom include and custom libpaths 4 files are creating. The  normal `bin`, normal `include` and custom `cbin` and custom `cinc`. You can unassign me.
msg357583 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-27 20:19
The problem of bin over Scripts is that IIRC Scripts is baked-in in Windows, not just for venvs (e.g. a standard Python installation puts stuff in Scripts). So venv just sticks with the platform convention.
msg357617 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-28 15:11
There are more differences than Scripts/bin, like Windows use Lib but POSIX uses lib/pythonX.Y. IMO it’s probably better to stick with platform conventions, especially since those can be discovered with sysconfig.get_paths(expand=False).

I wonder whether it’d be a good idea to record what scheme was used to create the venv (in pyvenv.cfg). Any Python runtime can use that value in get_paths(scheme=..., expand=False) to know about the venv’s structure, even if the venv was created on another platform. This would be particularly useful to e.g. inspect a Windows-native venv in a WSL Python.
msg357624 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-11-28 17:29
I've personally never come across Scripts in any other situation than virtual environments, but that isn't saying much about my Windows exposure either. :)

Basically activation is the biggest stumbling block I find with new users when it comes to trying to teach them about using virtual environments and this platform difference doesn't help. I would like to come up with _some_ solution to make it easier, even if it's something I have to put into the Python Launcher (although trying to get people to even agree on a name for virtual environments created beside the code turns out to be a hot topic). I would consider trying to come up with code so we have a `pipenv shell`/`conda shell` equivalent, but I've been told it is not pleasant to try and make work.
msg357625 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-28 17:45
> Basically activation is the biggest stumbling block I find with new users

Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok, and would cover the vast majority of users? (i.e. not including the other, less-common POSIX shells.)
msg357626 - (view) Author: Tzu-ping Chung (uranusjr) * Date: 2019-11-28 18:22
> Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok [...]?

The if-Windows-X-else-Y part isn’t that hard; it’s the activate part that is :p

I think Brett is thinking about eliminating the manual activate part entirely, but any tool trying to automate that needs to do a lot of platform-specific checks.

---

> I've personally never come across Scripts in any other situation than virtual environments [...].

Windows use a Scripts directory to store… scripts (Setuptools terminology, i.e. console and gui script entry points). So e.g. the global pip.exe would be at "{sys.prefix}\Scripts\pip.exe" (or is it sys.exec_prefix?) `pip install --user` would also install scripts into `%APPDATA%\Programs\Python\PythonXY\Scripts`. So venv’s setup is consistent with the rest of Python.

This directory structure can be expanded from sysconfig. So the proposal in my previous comment is to record the scheme in pyvenv.cfg, so you can have something like

def read_venv_scheme(env_dir):
    with open(os.path.join(env_dir, 'pyvenv.cfg')) as f:
        for line in f:
            key, value = (p.strip() for p in line.split('='))
            if key == 'scheme':
                return value

def get_venv_environ_patch(env_dir):
    scheme = read_venv_scheme(env_dir)
    bin_dir = sysconfig.get_path('scripts', scheme=scheme, expand=False).format(base=env_dir)
    return {'VIRTUAL_ENV': env_dir, 'PATH': bin_dir}

and this would give you the appropriate value on any platform.
msg357630 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-28 22:18
> I think Brett is thinking about eliminating the manual activate part entirely, but any tool trying to automate that needs to do a lot of platform-specific checks.

If you have more than one venv then it seems like some manual step is required to switch between them. What about a tool like Pew (Python Env Wrapper), which from the README "is completely shell-agnostic and thus works on bash, zsh, fish, powershell, etc."

https://github.com/berdario/pew

I haven't used in under Windows, but it works a treat on POSIX for me, YMMV of course.

Of course, scripts installed in venvs never need activation to run - they pick up the correct interpreter from their venv automatically.
msg357649 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-11-29 18:38
> Surely "on native Windows you run venv-path\Scripts\activate[.ps1], on POSIX you use source venv-path/bin/activate" isn't *that* hard for new users to grok, and would cover the vast majority of users?

Sure, but how many times do we need to make people type, write, or say that exact line instead of a single line of "you activate by doing <command>"?

> So venv’s setup is consistent with the rest of Python.

Right, it's a Python-on-Windows thing, not a Windows thing itself to my knowledge.

> I think Brett is thinking about eliminating the manual activate part entirely

I'm actually after a single command to handle activation of a virtual environment. It's a point of friction on your first day of learning Python and I have seen it solved multiple times at this point by multiple tools. This seemed like a potential simple way to solve it to me, but apparently not everyone agrees. ;)

Now I realize that if we don't worry about the prompt changing it's actually very straight-forward, and so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables and prints out a message about what is happening is enough to do the trick (and if people want the prompt to change they can tweak their shell configs to detect something like `__VENV_PROMPT__` being set and use it appropriately).

> Of course, scripts installed in venvs never need activation to run

Sure, but then that doesn't mean activation isn't convenient. :) Otherwise what is the point of having the activation scripts?
msg357655 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-11-29 22:22
> so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables

If venv is run in a child process of the shell, how does it set those variables in the parent shell? I thought that wasn't possible (on POSIX, anyway), which is why e.g. pew spawns a subshell.

> Sure, but then that doesn't mean activation isn't convenient

Indeed it is. But many less experienced users think that it's always a necessary step, when it isn't always so.
msg357720 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-02 17:41
> If venv is run in a child process of the shell, how does it set those variables in the parent shell?

You can't. Tools like `pipenv shell` and `conda shell` end up with a new shell running instead of changing the current shell in-place.
msg357728 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2019-12-02 20:24
> You can't. Tools like `pipenv shell` and `conda shell` end up with a new shell running instead of changing the current shell in-place.

Indeed. I raised it because of what you said earlier:

> and so maybe proposing a simple `venv --activate <path>` that does nothing more than set those key environment variables and prints out a message about what is happening is enough to do the trick

So not that simple, then, if it has to handle multiple shells cross-platform. Especially as if tools like pew, pipenv, conda already do this. Are you proposing to bring that functionality, which these third-party tools perform currently, into the stdlib?
msg357757 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-03 18:36
> Are you proposing to bring that functionality, which these third-party tools perform currently, into the stdlib?

I was hoping to, but it turns out you can't make this work under Windows without inspecting the parent process (PowerShell and Command Prompt provide no way to know the path to the shell being run).

So since there's no interest in providing a way to unify the paths I'm going to close this issue.
msg360334 - (view) Author: Erik Aronesty (earonesty) * Date: 2020-01-20 21:12
the single Scripts/activate tool should be simply copied to bin/activate ... this is what you have to do to write a bash script for python now:

source bin/activate || source Scripts/activate

we should not assume that all windows users use things like CMD and PowerShell, and instead just make support for "shell activation" be cross-platform
msg360337 - (view) Author: Erik Aronesty (earonesty) * Date: 2020-01-20 22:04
See https://github.com/python/cpython/pull/18083 for an example of a 'simple copy' for shell script compatibility ... rather than trying to make Scripts move around (which it can't trivially).
msg360367 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-21 08:53
The Scripts/bin thing is not specific to venv - for whatever reason, the original Windows implementation chose to use "Scripts" rather than "bin" for the executables directory under Windows, and I don't think it can be changed now without affecting backward compatibility. The venv code just fits into the existing, wider theme. My guess is you would need to propose a PEP to move *everything* over from "Scripts" to "bin" in the Windows Python world, and show that it would not break any existing code anywhere - seems like a tall order.

This issue was already rejected before you added your PR so I'm not sure why you went to the trouble of creating a PR.
msg360468 - (view) Author: Erik Aronesty (earonesty) * Date: 2020-01-22 14:08
> The Scripts/bin thing is not specific to venv - for whatever reason, the original Windows implementation chose to use "Scripts" rather than "bin" 

That's irrelevant to the PR, which solves the problem in a compatible way.   There's no compatibility issues if a link is made to the activate script, rather than moving the directory at all.

> My guess is you would need to propose a PEP to move *everything* over from "Scripts" to "bin" in the Windows Python world

Certainly not.  That would break everything and would be a bad idea.

> This issue was already rejected before you added your PR so I'm not sure why you went to the trouble of creating a PR.

Because the issue was rejected due to come conflating logic and confusion as to what the underlying problem and issue is.

The venv system produces files specifically for activation on Windows which must and should reside in the Scripts directory.

The venv system also produces files for activation in a bash (or similar) shell.  This *should* reside in the bin directory (there is no o/s dependency here), and it should *also* reside in the Scripts directory ... for compatibility.

Expressed that way, it's clear what the solution is.   Hence the PR.
msg360545 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-01-23 05:40
> This *should* reside in the bin directory

That's your opinion, but there are other views. As Paul Moore said in this comment:

https://github.com/python/cpython/pull/18083#issuecomment-577278587

Having things in bin *and* Scripts is more confusing than "on Windows use Scripts, on POSIX use bin".
History
Date User Action Args
2022-04-11 14:59:07adminsetgithub: 79184
2020-01-23 05:40:08vinay.sajipsetmessages: + msg360545
2020-01-22 18:10:25brett.cannonsetnosy: - brett.cannon
2020-01-22 14:08:46earonestysetmessages: + msg360468
2020-01-21 08:53:51vinay.sajipsetmessages: + msg360367
2020-01-20 22:04:21earonestysetmessages: + msg360337
2020-01-20 22:03:16earonestysetpull_requests: + pull_request17474
2020-01-20 21:12:19earonestysetnosy: + earonesty
messages: + msg360334
2019-12-03 18:36:53brett.cannonsetstatus: open -> closed
resolution: rejected
messages: + msg357757

stage: test needed -> resolved
2019-12-02 20:24:15vinay.sajipsetmessages: + msg357728
2019-12-02 17:41:43brett.cannonsetmessages: + msg357720
2019-11-29 22:22:34vinay.sajipsetmessages: + msg357655
2019-11-29 18:38:57brett.cannonsetmessages: + msg357649
2019-11-28 22:18:48vinay.sajipsetmessages: + msg357630
2019-11-28 18:22:18uranusjrsetmessages: + msg357626
2019-11-28 17:45:44vinay.sajipsetmessages: + msg357625
2019-11-28 17:29:59brett.cannonsetmessages: + msg357624
2019-11-28 15:11:55uranusjrsetnosy: + uranusjr
messages: + msg357617
2019-11-27 20:19:17vinay.sajipsetmessages: + msg357583
2019-11-27 17:56:10brett.cannonsetversions: + Python 3.9, - Python 3.8
2019-11-27 17:56:03brett.cannonsetnosy: + vinay.sajip
2018-10-20 05:43:00DamlaAltunsetmessages: + msg328125
2018-10-19 16:49:27DamlaAltunsetnosy: + DamlaAltun
messages: + msg328052
2018-10-16 22:12:19brett.cannonsetassignee: brett.cannon
2018-10-16 22:12:12brett.cannoncreate