Title: Python venv inconsistent folder structure on windows
Type: behavior Stage:
Components: Windows Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jmoguill2, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-01-08 01:10 by jmoguill2, last changed 2021-01-08 19:13 by jmoguill2.

Messages (7)
msg384628 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 01:10
When creating a virtual environment on windows using venv, the folder structure: "Scripts", "Include", "Lib", is inconsistent with other platforms (e.g. "include", "lib", "bin", etc).  

This causes various integration issues.  

For example, suppose we want to build a native C library, and install it to the folder structure generated by the virtual environment.  
The pkg-config file assumes a folder structure of "include", "lib", "bin", and the generated pkg-config files are inconsistent with the python virtual environment folder structure.  

Can we have a consistent folder structure across platforms?
msg384683 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-01-08 17:53
I'm afraid not, at least not without breaking everyone who has hardcoded the paths already. And I'm pretty sure that include directory is never used anymore, either (at least not by distutils, and not by default).

The sysconfig module provides the programmatic interface to the directory structure - I'd suggest using "sysconfig.get_paths()" to find the locations where files may be written to.

I'd also suggest building your native C library to a wheel and then installing that into the virtual environment, as that's the supported way of adding files. In that case, there's other tooling for specifying the destination for particular files, and it all already takes into account the layout on different platforms.
msg384685 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 18:02
To give more context regarding this issue:
I'm currently using meson build system to generate the pkg-config files, and it seems that the paths "include", "lib" are hardcoded.  

From the perspective of the overall system, I think it would simplify integration and reduce complexity if we normalize folder structures across platforms instead of having different folder structures.
msg384686 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-01-08 18:13
> From the perspective of the overall system, I think it would simplify integration and reduce complexity if we normalize folder structures across platforms instead of having different folder structures.

I agree. But from the perspective of not breaking every existing system out there, it makes more sense to not suddenly change it.

If you'd like to champion the 2-3 year (a.k.a. 2-3 release) migration effort to get there, I'm more than happy to make sure you're taken seriously by the rest of the core team. I'm just not prepared to champion it myself :) I have other concerns to use my OSS time for.
msg384687 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 18:34
Maybe we can consider adding additional params and a new code path to python -m venv?  This way we don't break any existing functionality?

e.g. -includedir=... -libdir=... -bindir=...
msg384688 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2021-01-08 19:04
Many tools hard code the "if windows then 'Scripts' else 'bin'" logic. They will simply fail to work with environments that don't have that layout. Yes, they probably should use sysconfig, but many don't (and sometimes for arguably good reasons, for example tools run using one Python interpreter that manipulate the environment of a different one).

It's something that *could* be changed, but the cost would be significant, the work needed to manage the transition is non-trivial, and the benefit isn't particularly obvious - "consistency" is never very compelling in isolation, and "works better with meson" is likely to be met with the argument that Python's layout has been around longer than meson, so it's on meson to accommodate Python, not the other way around.

I'm not actively against the idea, but neither do I particularly support it (I'm pretty sure it would require work for pip, for no benefit to pip's existing users, so it's a net loss for me personally).

If you're interested in pushing for this change go for it - I suspect that it would probably need a PEP, and at the very least it would need a community discussion - but be prepared for it to meet with a certain amount of resistance.
msg384689 - (view) Author: Jeff Moguillansky (jmoguill2) Date: 2021-01-08 19:13
Thanks for the feedback, I understand
Date User Action Args
2021-01-08 19:13:56jmoguill2setmessages: + msg384689
2021-01-08 19:04:24paul.mooresetmessages: + msg384688
2021-01-08 18:34:18jmoguill2setmessages: + msg384687
2021-01-08 18:13:56steve.dowersetmessages: + msg384686
2021-01-08 18:02:04jmoguill2setmessages: + msg384685
2021-01-08 17:53:26steve.dowersetmessages: + msg384683
2021-01-08 01:10:38jmoguill2create