This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python venv inconsistent folder structure on windows
Type: behavior Stage:
Components: Windows Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: darrel.opry, jmoguill2, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-01-08 01:10 by jmoguill2, last changed 2022-04-11 14:59 by admin.

Messages (9)
msg384628 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 01:10
When creating a virtual environment on windows using venv, the folder structure: "Scripts", "Include", "Lib", is inconsistent with other platforms (e.g. "include", "lib", "bin", etc).  

This causes various integration issues.  

For example, suppose we want to build a native C library, and install it to the folder structure generated by the virtual environment.  
The pkg-config file assumes a folder structure of "include", "lib", "bin", and the generated pkg-config files are inconsistent with the python virtual environment folder structure.  

Can we have a consistent folder structure across platforms?
msg384683 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-01-08 17:53
I'm afraid not, at least not without breaking everyone who has hardcoded the paths already. And I'm pretty sure that include directory is never used anymore, either (at least not by distutils, and not by default).

The sysconfig module provides the programmatic interface to the directory structure - I'd suggest using "sysconfig.get_paths()" to find the locations where files may be written to.

I'd also suggest building your native C library to a wheel and then installing that into the virtual environment, as that's the supported way of adding files. In that case, there's other tooling for specifying the destination for particular files, and it all already takes into account the layout on different platforms.
msg384685 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 18:02
To give more context regarding this issue:
I'm currently using meson build system to generate the pkg-config files, and it seems that the paths "include", "lib" are hardcoded.  

From the perspective of the overall system, I think it would simplify integration and reduce complexity if we normalize folder structures across platforms instead of having different folder structures.
msg384686 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-01-08 18:13
> From the perspective of the overall system, I think it would simplify integration and reduce complexity if we normalize folder structures across platforms instead of having different folder structures.

I agree. But from the perspective of not breaking every existing system out there, it makes more sense to not suddenly change it.

If you'd like to champion the 2-3 year (a.k.a. 2-3 release) migration effort to get there, I'm more than happy to make sure you're taken seriously by the rest of the core team. I'm just not prepared to champion it myself :) I have other concerns to use my OSS time for.
msg384687 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 18:34
Maybe we can consider adding additional params and a new code path to python -m venv?  This way we don't break any existing functionality?

e.g. -includedir=... -libdir=... -bindir=...
?
msg384688 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2021-01-08 19:04
Many tools hard code the "if windows then 'Scripts' else 'bin'" logic. They will simply fail to work with environments that don't have that layout. Yes, they probably should use sysconfig, but many don't (and sometimes for arguably good reasons, for example tools run using one Python interpreter that manipulate the environment of a different one).

It's something that *could* be changed, but the cost would be significant, the work needed to manage the transition is non-trivial, and the benefit isn't particularly obvious - "consistency" is never very compelling in isolation, and "works better with meson" is likely to be met with the argument that Python's layout has been around longer than meson, so it's on meson to accommodate Python, not the other way around.

I'm not actively against the idea, but neither do I particularly support it (I'm pretty sure it would require work for pip, for no benefit to pip's existing users, so it's a net loss for me personally).

If you're interested in pushing for this change go for it - I suspect that it would probably need a PEP, and at the very least it would need a community discussion - but be prepared for it to meet with a certain amount of resistance.
msg384689 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 19:13
Thanks for the feedback, I understand
msg415197 - (view) Author: Darrel O'Pry (darrel.opry) Date: 2022-03-14 21:36
steve,

I would be willing to champion this change. The process seems pretty straightforward. Create a release that creates the consistent layout on windows, symlinks the legacy paths, and adds some deprecation warnings.  Then in a later release remove the symlinks and deprecation warnings.. Any way you would be willing to guide me through the process?
msg415202 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2022-03-14 22:25
Considering you've just been encountering issues with mismatched case, you're likely aware of how much harder that will make it to fix.

First you'll need a proposal on how to ensure deprecation warnings reach those who need them. I'd suggest for "Scripts"->"bin", you'll need to generate different stubs in "Scripts" that warn when they are called, but not the ones in "bin".

This alone affects enough of the ecosystem to need a PEP and a lot of coordination. It's not a simple change - sure you're ready for it?
History
Date User Action Args
2022-04-11 14:59:40adminsetgithub: 87029
2022-03-14 22:25:28steve.dowersetmessages: + msg415202
2022-03-14 21:36:11darrel.oprysetnosy: + darrel.opry
messages: + msg415197
2021-01-08 19:13:56jmoguill2setmessages: + msg384689
2021-01-08 19:04:24paul.mooresetmessages: + msg384688
2021-01-08 18:34:18jmoguill2setmessages: + msg384687
2021-01-08 18:13:56steve.dowersetmessages: + msg384686
2021-01-08 18:02:04jmoguill2setmessages: + msg384685
2021-01-08 17:53:26steve.dowersetmessages: + msg384683
2021-01-08 01:10:38jmoguill2create