I am finding that, using Powershell on Windows 10 and the current version of Python 3.7.3 installed from the Microsoft Store, when I create a virtualenv via "python -m venv <name>" and activate it in Powershell with the Activate.ps1 script that is generated, pip fails with:

pip freeze
Fatal error in launcher: Unable to create process using '"c:\users\<home>\<path-to-venv>\scripts\python.exe" "C:\Users\<home>\<path-to-venv>\Scripts\pip.exe" freeze'

There are no spaces in my python path, and I cannot find any "simple workaround" that actually works online. I am using pip 19.1.1 (current as of now). This only happens with Powershell. cmd.exe is able to use the virtualenv and pip just fine after using the activate.bat script.

If I activate in Powershell and run "python -m pip" or similar commands, including importing pip, I get a message that pip is not installed, which is interesting.

My Powershell version as installed is:
Major Minor Build Revision
----- ----- ----- --------
5 1 18917 1000

I also tried Powershell Core 6 with the same result. This seems to be a consistent behavior of the resultant Activate.ps1 script, which is why I'm creating the issue here. I didn't find quite a duplicate when I searched, but I could be wrong.

I should add that, after testing a bit, it isn't actually working in cmd.exe. That is simply using the overarching python install. It's not using the virtualenv at all. The virtualenv that was created by the venv module appears to be non-functional at least as far as pip goes (which makes it hard to use at all).

Okay, this definitely used to work, but now it's broken for me too.

What version of Windows are you on (copy-paste from cmd.exe's ver command)?

This seems to be a change in Windows at some point, as it still works on one of my other PCs. I've pinged some colleagues to find out what might have happened, but it definitely looks like the lack of Read+Execute permission is working correctly now :(

One potential fix is to return the path under C:\Users\<username>\AppData\Local\Microsoft\WindowsApp as sys.executable (and sys.exec_prefix?), but it only contains the executable launchers and none of the actual Python install. Making this work for venv might require new values in pyvenv.cfg (however, it also seems that the current sys.executable changes with each update, so if we can change it universally to point at the user's WindowsApp directory then that will stop).

The good news is that if you're trying to launch it from within Python it still works, so multiprocessing is not affected. It's only things that write sys.executable out and try to use it later, like venv.

Adding Ned FYI, as if I can come up with a fix for this before we ship 3.7.4 I'd like to include it. No idea yet how possible that will be or what our fix will be though.

OK, I"m bumping it to "deferred blocker" for now so it remains visible.

To answer Steve's question, the ver command gives me:
Microsoft Windows [Version 10.0.18917.1000]

Thank you for looking into this.

So I have a fix for 3.9 for this, I believe (involves making sys.base_executable configurable via PyConfig and overriding it and config->home when we know we're inside an app container).

For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

For 3.7, I'm going to have to come up with a different fix, and possibly expose some new private API to be able to pass through the correct executable path.

Fundamentally, the problem is that Windows is going to start enforcing some ACLs that were previously not being enforced. In theory, when an application is deployed in an app package (as required through the Store), you are not supposed to be able to launch the executable directly. Instead, you should go through your current user's apps directory (C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>, which contains only symlinks to the actual executables).

This has a very good advantage that the package family name does not change with new versions, while the full install path does. Currently, a behind-the-scenes update of Python will actually break environments when their "home" path is no longer correct.

However, it does mean that we need to start reporting a sys.executable that is not inside sys.prefix, and I'm not sure whether that is a good idea.

Looking at venv (+Vinay for confirmation here), it seems to treat "home" as "the directory containing the original Python binary" rather than "the directory containing the Python install". So in theory (and in my simple testing), using the sys.executable-not-in-prefix here is fine. But is that intentional?

you should go through your current user's apps directory
(C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>,
which contains only symlinks to the actual executables).

"%LocalAppData%\Microsoft\WindowsApps" contains IO_REPARSE_TAG_APPEXECLINK reparse points. An application-execution link is not like a regular symlink, i.e. it does not have the name-surrogate bit set in the tag value. In addition to the real application path, an app-exec link also contains the package information that the system uses to create the special access token that's required in order to execute the application.

I stepped through CreateProcessW in 10.0.18362.175 (slow ring), which has the old behavior. The first NtCreateUserProcess system call (the call that actually creates the Process object) fails with STATUS_IO_REPARSE_TAG_NOT_HANDLED. CreateProcessW handles this by getting the package information from the reparse point and creating a new token that contains 3 additional security attributes (WIN://SYSAPPID, WIN://PKG, and WIN://PKGHOSTID). For the subsequent NtCreateUserProcess system call, it uses the real path under "%ProgramFiles%\WindowsApps" and impersonates the new access token.

The real executable has two ACEs for standard users. There's a basic access-allowed ACE that grants read access, but does not grant execute access. There's also an access-allowed callback ACE that grants read and execute access. Normally the kernel security routines ignore callback ACEs in an access check, since they're intended for an application-defined callback in the user-mode AuthZ API. Apparently something is special-casing the access check in this case, to conditionally check the security attributes in the token in order to apply this callback ACE. If I remove the callback ACE, users can no longer execute python.exe, so I know this is the ACE that's granting execute access.

Back to CreateProcessW. If I try to directly execute the real executable in "%ProgramFiles%\WindowsApps", the NtCreateUserProcess system call fails with STATUS_ACCESS_DENIED, as expected. But then CreateProcessW does something that I didn't expect. It parses out the base filename "python.exe" and joins it to "%LocalAppData%\Microsoft\WindowsApps". If it finds the app-exec link, then it reads it to create the special package access token, as if we had run the app-exec link directly. If I had to guess, I would assume this behavior has been disabled in newer versions of Windows 10. I'm using the slow ring for now, so I don't have a way to test this guess.

For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

There is _PyConfig._config_version which is designed to support PyConfig change without losing ABI compatibility... but I didn't implement code for that yet :-)

If you have to modify PyConfig in 3.8, I suggest to directly use "base_executable" "public" name there (don't change it from _base_executable to base_executable from 3.8 to 3.9, it would be painful).

I'm not sure of the purpose of sys._base_executable. Maybe open a separated issue to make the attribute public? I'm not sure if multiprocessing (for example) should use it, or if sys.executble should be used.

New changeset db4d7dd by Steve Dower in branch '3.7':
bpo-37369: Fixes path for sys.executable when running from the Microsoft Store (GH-14450)
db4d7dd

New changeset ed4657b by Steve Dower in branch '3.7':
bpo-37369: Fix venv and test symlinking (GH-14456)
ed4657b

New changeset db4eb2c by Steve Dower in branch '3.7':
bpo-37369: Fix path handling when python.exe is used as a symlink (GH-14461)
db4eb2c

New changeset 9048c49 by Steve Dower in branch 'master':
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14428)
9048c49

New changeset ac14632 by Steve Dower in branch 'master':
bpo-37369: Fixes crash when reporting fatal error (GH-14468)
ac14632

New changeset 323e743 by Steve Dower in branch '3.8':
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14467)
323e743

I'm not sure of the purpose of sys._base_executable. Maybe open a separated issue to make the attribute public? I'm not sure if multiprocessing (for example) should use it, or if sys.executble should be used.

Yeah, it's a bit more complex. Sometimes it's needed and often not, depending on platform. For multiprocessing we need it on Windows because of the handle inheritance that's used, which only goes one subprocess deep,and we use special knowledge to make it work even in a venv.

But at least having it there means we can use it when needed without having to define its semantics for 100% of cases. Venvs make things really hard the way they're currently done...

New changeset 3c34ea9 by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fixes path for sys.executable when running from the Microsoft Store (GH-14450)
3c34ea9

New changeset 57eba3a by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fix venv and test symlinking (GH-14456)
57eba3a

New changeset a88652e by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fix path handling when python.exe is used as a symlink (GH-14461)
a88652e

On Thu, 27 Jun 2019 at 18:08, Steve Dower <report@bugs.python.org> wrote:

However, it does mean that we need to start reporting a sys.executable that is not inside sys.prefix, and I'm not sure whether that is a good idea.

Looking at venv (+Vinay for confirmation here), it seems to treat "home" as "the directory containing the original Python binary" rather than "the directory containing the Python install". So in theory (and in my simple testing), using the sys.executable-not-in-prefix here is fine. But is that intentional?

I'm pretty sure there is code in the wild that expects <sys.prefix> /
python.exe to be the Python executable (when you're not in a virtual
environment). I seem to recall that virtualenv might do that sort of
thing. Sorry I can't be precise here, I've just come back from holiday
and won't be back into the swing of open source for a while yet, but
my recollection is that it gets done in situations where
sys.executable isn't (easily) accessible - for example, when getting
sys.prefix from the registry (um, I don't think that's right, as the
executable path is in the registry too - but you get the idea).

What I'd say is that we should be cautious here, as we risk trading
one breakage for another, rather than actually fixing the issue.
Certainly the assumption I quote above is unwarranted, and code making
it can't complain too much if it breaks, but the problem is that you
*have* to make that assumption as the necessary information isn't
available any other way. So we should do some careful testing of 3rd
party tools like virtualenv, pipenv, pipx, pew, tox, etc, to make sure
any "solution" here doesn't break them, and provide or document
transition options for them if necessary (because such tools can't
realistically avoid continuing to support older versions of Python).

IMO, we should bite the bullet here and actually document the
interfaces involved in virtual environments, so that tools have a
stable contract to work with. That way we'd limit the need for people
to rely on undocumented behaviour in the first place.

Paul

I'm pretty sure there is code in the wild that expects <sys.prefix> /
python.exe to be the Python executable (when you're not in a virtual
environment) ... for example, when getting
sys.prefix from the registry (um, I don't think that's right, as the
executable path is in the registry too - but you get the idea).

Yeah, for sure. Though as you say, when sys.executable is easily accessible, they're probably (hopefully) going to be using that.

Older versions of Python definitely omitted the full executable path from the registry. PEP-514 added that in, though in this case unfortunately I don't think we have any way to fix what goes in the registry :(

What I'd say is that we should be cautious here, as we risk trading
one breakage for another, rather than actually fixing the issue.

Yeah. We probably need to be clear about which approaches are actually supported and which ones are not.

Ultimately though, I think this is a situation where the breakage is forced upon us (by the operating system), and so fixing the behaviour of our built-in functionality pretty clearly outweight preserving the assumptions made by third-parties (especially since those assumptions are going to break).

IMO, we should bite the bullet here and actually document the
interfaces involved in virtual environments, so that tools have a
stable contract to work with. That way we'd limit the need for people
to rely on undocumented behaviour in the first place.

I feel like I've made this argument before, but I suspect if we'd documented them prior to this change, we'd now have no choice but to remain broken.

The interfaces that should be documented are "-m venv", venv.VenvBuilder, and the activation scripts, none of which had to change. But if we start making promises about how "python.exe" actually behaves in a venv then we're going to paint ourselves into a corner very quickly - the venv approach just doesn't scale well to the range of systems people use. (Obligatory mention that PEP-582 would have avoided this whole mess by not requiring us to duplicate python.exe anywhere. But it just wasn't meant to be.)

On Fri, 5 Jul 2019 at 21:33, Steve Dower <report@bugs.python.org> wrote:

Yeah. We probably need to be clear about which approaches are actually supported and which ones are not.

Ultimately though, I think this is a situation where the breakage is forced upon us (by the operating system), and so fixing the behaviour of our built-in functionality pretty clearly outweight preserving the assumptions made by third-parties (especially since those assumptions are going to break).

Well, given that this is an issue specific to the store distribution
of Python, I'd say that it's the release of the store build that
forces it on us, not the OS, and releasing the store build was our
choice, so if we did so with the store build not meeting existing
implicit (i.e. undocumented, I agree) contracts, then we made that
choice. I think it was the right choice, because the store
distribution is a huge plus, but I do think we should be cautious
about impacting the existing user base simply because of a very new,
and explicitly documented as "still experimental" build. (Yes, I'm
still a bit sore about the hassle involved in getting virtualenv to
work - although the final resolution to that issue was IMO clean and a
huge improvement on the previous status quo, so I'm not *really* sore
:-))

> IMO, we should bite the bullet here and actually document the
> interfaces involved in virtual environments, so that tools have a
> stable contract to work with. That way we'd limit the need for people
> to rely on undocumented behaviour in the first place.

I feel like I've made this argument before, but I suspect if we'd documented them prior to this change, we'd now have no choice but to remain broken.

Hardly. We would simply have had to document the changed behaviour and
follow our existing transition/compatibility processes. Documenting
things doesn't lock them in stone, it just requires us to give due
consideration to our users when we change things. Conversely, *not*
documenting things makes it easier for us to change things, at the
cost of not giving users supported ways to do certain things that they
may still need to do.

The interfaces that should be documented are "-m venv", venv.VenvBuilder, and the activation scripts, none of which had to change. But if we start making promises about how "python.exe" actually behaves in a venv then we're going to paint ourselves into a corner very quickly - the venv approach just doesn't scale well to the range of systems people use. (Obligatory mention that PEP-582 would have avoided this whole mess by not requiring us to duplicate python.exe anywhere. But it just wasn't meant to be.)

I assume you're missing my key point here (not unlikely, as I didn't
state it explicitly - I guess I'm relying on undocumented information
myself ;-)). The point here is that venv doesn't exist in a vacuum,
before it existed, virtualenv was the standard way to produce virtual
environments. And because neither virtualenv nor venv document any of
their internal details, they are plagued with incompatibilities, and
work badly together. As a result, tools like pipenv, or pew, or pipx,
can't support both but end up picking just one - and because of
backward compatibility requirements, they pick virtualenv, which has
significantly hindered adoption of venv, which is (for hopefully
obvious reasons) a substantially more robust solution.

It may be that it's just too late to worry about this, and we should
simply accept that tools will be dropping Python 2 support very soon
now, at which point there's no real reason to continue using
virtualenv (apart from no-one having the time to make the change, but
if virtualenv adopts venv as its internal mechanism we can get around
that). But personally, I'd rather not rely on an assumption like that,
just yet (maybe in 3.9?).

(Side issue, I don't really see the relevance of PEP-582 here, as I
don't think it's something that tools like tox and pipenv could
replace virtual environments with - but maybe I'm wrong, if so the
arguments for PEP-582 didn't really get that point across).

A note on the above points, virtualenv has started migrating over to venv via pypa/virtualenv#1377 . Following this, the old ways of virtualenv should be left to the history books. Hopefully, I can get it out within the next month or so.

Hi,

First time reporting so feel free to direct me to a better place if this is not the correct place :)

I am still facing the issue that started this thread with Python 3.8 and 3.9.

Steps to reproduces

1. Clean install of Python 3.9.0b3 (quick install via .exe https://www.python.org/downloads/windows/)

2. create a python virtual environment in powershell

                   python -m venv env
   

or with absolute path: python -m venv C:\Users\<user>\<path>\env

3. activate virtual environment

   .\env\Scripts\Activate.ps1

4. use pip

(env) PS C:\Users\<user>\<path> python --version
Python 3.9.0b3
(env) PS C:\Users\<user>\<path> pip --version
Fatal error in launcher: Unable to create process using '"c:\users\<user>\<path-in-lowercase>\env\scripts\python.exe" "C:\Users\<user>\<path-in-lowercase>\env\Scripts\pip.exe" --version'

However, python and pip are installed in the env\Scripts\*

(env) PS C:\Users\<user>\<path> Get-Command python
python.exe 3.9.113 C:\Users\<user>\<path>\env\Scripts\python.exe

(env) PS C:\Users\<user>\<path> Get-Command pip
pip.exe 0.0.0.0 C:\Users\<user>\<path>\env\Scripts\pip.exe

But the pip module is not actually found by the python interpreter inside the virtual environment.

(env) PS C:\Users\<user>\<path> python -m pip --version
C:\Users\<user>\<path>\env\Scripts\python.exe: No module named pip

Notes:

1. the global pip installation works well and can be used to install packages

PS C:\Users\<user>\<path> pip --version
pip 19.2.3 from c:\users\<user>\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

2. Windows version: Microsoft Windows [Version 10.0.18363.836]

3. Powershell Version: 5.1.18362.752

4. Paths in a python interpreter started in the active env

(env) PS C:\Users\<user>\<path> python
>>> import sys,os

>>> sys.exec_prefix
'C:\\Users\\<user>\\<path>\\env'

>>> sys.base_prefix
'C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python39'

>>> sys.path
['', 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python39\\python39.zip', 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python39\\DLLs', 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python39\\lib', 'C:\\Users\\<user>\\AppData\\Local\\Programs\\Python\\Python39', 'C:\\Users\\<user>\\<path>\\env']

>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pip'

quit()

I did not find more recent bugs related to this and wanted to bring the issue back to your attention. At least to me, based on the discussion here it seemed that the issue should have been fixed. Please let me know if you need any further input.

Hi Miika

Please create a new issue.

It would also be helpful if you gave us some idea of what your username looks like, even if you change it a bit to hide it from the internet. The most common cause of this error is spaces or non-ASCII characters in your username - hiding those makes it impossible to rule out the issue.

I tried to reproduce the issue on a another windows installation (Windows build 10.0.18363.0), but was however unable to do it following the same steps. I fear that my original python/pip installation is just corrupted somehow due to having many python interpreters around previously.

My username with the machine I am having issues is typical C:\Users\miika.makela\

without any nonascii characters and without spaces in the data path.

With the other machine I tried my username is just C:\Users\Miika

I'll open an issue if I sometime find out what really did go wrong, thanks for the reply.

Hello and sorry for reopening this issue but I have it as well.
The problem is that Powershell has case-sensitive commands and pip is trying to call path completely in minor case

PS C:\Users\federico.Tabbo\git\nexus2aci\env\Scripts> .\Activate.ps1
(env) PS C:\Users\federico.Tabbo\git\nexus2aci\env\Scripts>
Fatal error in launcher: Unable to create process using '"c:\users\federico.tabbo\git\nexus2aci\env\scripts\python.exe" "C:\Users\federico.Tabbo\git\nexus2aci\env\Scripts\pip.exe" ': The system cannot find the file specified.

if I run the lowercase path I get an error

(env) PS C:\Users\federico.Tabbo\git\nexus2aci\env\Scripts> c:\users\federico.tabbo\git\nexus2aci\env\scripts\python.exe

c:\users\federico.tabbo\git\nexus2aci\env\scripts\python.exe : The term
'c:\users\federico.tabbo\git\nexus2aci\env\scripts\python.exe' is not recognized as the name of a cmdlet, function,
script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
correct and try again.
At line:1 char:1
+ c:\users\federico.tabbo\git\nexus2aci\env\scripts\python.exe
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (c:\users\federi...ipts\python.exe:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException

Running the correct case path works

(env) PS C:\Users\federico.Tabbo\git\nexus2aci\env\Scripts> c:\Users\federico.Tabbo\git\nexus2aci\env\Scripts\python.exe

Python 3.8.4rc1 (tags/v3.8.4rc1:6c38841, Jun 30 2020, 15:17:30) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>

The problem is that Powershell has case-sensitive commands and pip is trying to call path completely in minor case

This is not normal configuration. I'm not sure what you've done here, but please open a new issue with details about how you enabled case-sensitivity on Windows.

Also please include repro steps that do not involve pip, as we will need an issue in CPython to fix anything. Otherwise you'll have to report it to the pip project at https://github.com/pypa/pip

Case-sensitive file access is potentially possible with FILE_FLAG_POSIX_SEMANTICS (CreateFileW) and FIND_FIRST_EX_CASE_SENSITIVE (FindFirstFileExW). These flags make the API omit the object-manager flag OBJ_CASE_INSENSITIVE in the NtCreateFile or NtOpenFile system call. That said, since Windows XP these API flags are usually ignored because the kernel object manager has a default-enabled global flag that makes it always use OBJ_CASE_INSENSITIVE for object types that are registered as case-insensitive, such as device objects. The global flag is configured as the value "obcaseinsensitive" in "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel".

A filesystem may elect to ignore the OBJ_CASE_INSENSITIVE flag, though it's not particularly elegant. In Windows 10, NTFS ignores the OBJ_CASE_INSENSITIVE flag if the parent directory is flagged as case-sensitive via NtSetInformationFile: FileCaseSensitiveInformation [1], which can be set by WSL or fsutil.exe. This setting should be avoided for directories that are used by Windows programs. An example issue is the way shells use the PATHEXT environment variable to find a file when the extension is omitted. Typically each extension in PATHEXT gets appended to the base name and checked via FindFirstFileW or GetFileAttributesW. But, for example, checking for "spam.BAT" will fail in a case-sensitive directory if the extension is actually ".bat", ".Bat", etc.

---

[1] https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_case_sensitive_information