classification
Title: Issue with pip in venv on Powershell in Windows
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: bstorm, eryksun, gaborbernat, ned.deily, paul.moore, steve.dower, tim.golden, vinay.sajip, vstinner, zach.ware
Priority: deferred blocker Keywords: patch

Created on 2019-06-21 23:54 by bstorm, last changed 2019-07-08 14:34 by gaborbernat. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 14428 merged steve.dower, 2019-06-27 17:10
PR 14450 merged steve.dower, 2019-06-28 15:10
PR 14456 merged steve.dower, 2019-06-28 18:00
PR 14461 merged steve.dower, 2019-06-28 23:22
PR 14467 merged steve.dower, 2019-06-29 17:36
PR 14468 merged steve.dower, 2019-06-29 20:53
Messages (21)
msg346264 - (view) Author: Brooke Storm (bstorm) Date: 2019-06-21 23:54
I am finding that, using Powershell on Windows 10 and the current version of Python 3.7.3 installed from the Microsoft Store, when I create a virtualenv via "python -m venv <name>" and activate it in Powershell with the Activate.ps1 script that is generated, pip fails with:

> pip freeze
Fatal error in launcher: Unable to create process using '"c:\users\<home>\<path-to-venv>\scripts\python.exe"  "C:\Users\<home>\<path-to-venv>\Scripts\pip.exe" freeze'

There are no spaces in my python path, and I cannot find any "simple workaround" that actually works online.  I am using pip 19.1.1 (current as of now).  This only happens with Powershell.  cmd.exe is able to use the virtualenv and pip just fine after using the activate.bat script.

If I activate in Powershell and run "python -m pip" or similar commands, including importing pip, I get a message that pip is not installed, which is interesting.

My Powershell version as installed is:
Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      18917  1000

I also tried Powershell Core 6 with the same result.  This seems to be a consistent behavior of the resultant Activate.ps1 script, which is why I'm creating the issue here.  I didn't find quite a duplicate when I searched, but I could be wrong.
msg346265 - (view) Author: Brooke Storm (bstorm) Date: 2019-06-22 00:03
I should add that, after testing a bit, it isn't actually working in cmd.exe.  That is simply using the overarching python install.  It's not using the virtualenv at all.  The virtualenv that was created by the venv module appears to be non-functional at least as far as pip goes (which makes it hard to use at all).
msg346658 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-26 18:47
Okay, this definitely used to work, but now it's broken for me too.

What version of Windows are you on (copy-paste from cmd.exe's ver command)?
msg346683 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-26 21:28
This seems to be a change in Windows at some point, as it still works on one of my other PCs. I've pinged some colleagues to find out what might have happened, but it definitely looks like the lack of Read+Execute permission is working correctly now :(

One potential fix is to return the path under C:\Users\<username>\AppData\Local\Microsoft\WindowsApp as sys.executable (and sys.exec_prefix?), but it only contains the executable launchers and none of the actual Python install. Making this work for venv might require new values in pyvenv.cfg (however, it also seems that the current sys.executable changes with each update, so if we can change it universally to point at the user's WindowsApp directory then that will stop).

The good news is that if you're trying to launch it from within Python it still works, so multiprocessing is not affected. It's only things that write sys.executable out and try to use it later, like venv.

Adding Ned FYI, as if I can come up with a fix for this before we ship 3.7.4 I'd like to include it. No idea yet how possible that will be or what our fix will be though.
msg346685 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-06-26 21:44
OK, I"m bumping it to "deferred blocker" for now so it remains visible.
msg346694 - (view) Author: Brooke Storm (bstorm) Date: 2019-06-26 22:55
To answer Steve's question, the ver command gives me:
Microsoft Windows [Version 10.0.18917.1000]

Thank you for looking into this.
msg346761 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-27 17:08
So I have a fix for 3.9 for this, I believe (involves making sys.base_executable configurable via PyConfig and overriding it and config->home when we know we're inside an app container).

For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

For 3.7, I'm going to have to come up with a different fix, and possibly expose some new private API to be able to pass through the correct executable path.


Fundamentally, the problem is that Windows is going to start enforcing some ACLs that were previously not being enforced. In theory, when an application is deployed in an app package (as required through the Store), you are not supposed to be able to launch the executable directly. Instead, you should go through your current user's apps directory (C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>, which contains only symlinks to the actual executables).

This has a very good advantage that the package family name does not change with new versions, while the full install path does. Currently, a behind-the-scenes update of Python will actually break environments when their "home" path is no longer correct.

However, it does mean that we need to start reporting a sys.executable that is not inside sys.prefix, and I'm not sure whether that is a good idea.

Looking at venv (+Vinay for confirmation here), it seems to treat "home" as "the directory containing the original Python binary" rather than "the directory containing the Python install". So in theory (and in my simple testing), using the sys.executable-not-in-prefix here is fine. But is that intentional?
msg346783 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-06-27 22:09
> you should go through your current user's apps directory 
> (C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>,
> which contains only symlinks to the actual executables).

"%LocalAppData%\Microsoft\WindowsApps" contains IO_REPARSE_TAG_APPEXECLINK reparse points. An application-execution link is not like a regular symlink, i.e. it does not have the name-surrogate bit set in the tag value. In addition to the real application path, an app-exec link also contains the package information that the system uses to create the special access token that's required in order to execute the application.

I stepped through CreateProcessW in 10.0.18362.175 (slow ring), which has the old behavior. The first NtCreateUserProcess system call (the call that actually creates the Process object) fails with STATUS_IO_REPARSE_TAG_NOT_HANDLED. CreateProcessW handles this by getting the package information from the reparse point and creating a new token that contains 3 additional security attributes (WIN://SYSAPPID, WIN://PKG, and WIN://PKGHOSTID). For the subsequent NtCreateUserProcess system call, it uses the real path under "%ProgramFiles%\WindowsApps" and impersonates the new access token. 

The real executable has two ACEs for standard users. There's a basic access-allowed ACE that grants read access, but does not grant execute access. There's also an access-allowed callback ACE that grants read and execute access. Normally the kernel security routines ignore callback ACEs in an access check, since they're intended for an application-defined callback in the user-mode AuthZ API. Apparently something is special-casing the access check in this case, to conditionally check the security attributes in the token in order to apply this callback ACE. If I remove the callback ACE, users can no longer execute python.exe, so I know this is the ACE that's granting execute access.

Back to CreateProcessW. If I try to directly execute the real executable in "%ProgramFiles%\WindowsApps", the NtCreateUserProcess system call fails with STATUS_ACCESS_DENIED, as expected. But then CreateProcessW does something that I didn't expect. It parses out the base filename "python.exe" and joins it to "%LocalAppData%\Microsoft\WindowsApps". If it finds the app-exec link, then it reads it to create the special package access token, as if we had run the app-exec link directly. If I had to guess, I would assume this behavior has been disabled in newer versions of Windows 10. I'm using the slow ring for now, so I don't have a way to test this guess.
msg346786 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-27 23:29
> For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

There is _PyConfig._config_version which is designed to support PyConfig change without losing ABI compatibility... but I didn't implement code for that yet :-)

If you have to modify PyConfig in 3.8, I suggest to directly use "base_executable" "public" name there (don't change it from _base_executable  to base_executable from 3.8 to 3.9, it would be painful).

I'm not sure of the purpose of sys._base_executable. Maybe open a separated issue to make the attribute public? I'm not sure if multiprocessing (for example) should use it, or if sys.executble should be used.
msg346837 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-28 17:02
New changeset db4d7ddb012ef8f087a8eb2a5b8a672d04a48e1a by Steve Dower in branch '3.7':
bpo-37369: Fixes path for sys.executable when running from the Microsoft Store (GH-14450)
https://github.com/python/cpython/commit/db4d7ddb012ef8f087a8eb2a5b8a672d04a48e1a
msg346849 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-28 18:41
New changeset ed4657bd28432ace671a9e6ae38a3d485c69213d by Steve Dower in branch '3.7':
bpo-37369: Fix venv and test symlinking (GH-14456)
https://github.com/python/cpython/commit/ed4657bd28432ace671a9e6ae38a3d485c69213d
msg346877 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-29 04:30
New changeset db4eb2c57dbfca402a539184c966c449cb1aa8e3 by Steve Dower in branch '3.7':
bpo-37369: Fix path handling when python.exe is used as a symlink (GH-14461)
https://github.com/python/cpython/commit/db4eb2c57dbfca402a539184c966c449cb1aa8e3
msg346890 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-29 17:34
New changeset 9048c49322a5229ff99610aba35913ffa295ebb7 by Steve Dower in branch 'master':
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14428)
https://github.com/python/cpython/commit/9048c49322a5229ff99610aba35913ffa295ebb7
msg346899 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-29 21:28
New changeset ac14632c756fec561e4b868b8793334bd7b22241 by Steve Dower in branch 'master':
bpo-37369: Fixes crash when reporting fatal error (GH-14468)
https://github.com/python/cpython/commit/ac14632c756fec561e4b868b8793334bd7b22241
msg346900 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-29 21:29
New changeset 323e743d4879f1cd861d0b252775797fb7938755 by Steve Dower in branch '3.8':
bpo-37369: Fix initialization of sys members when launched via an app container (GH-14467)
https://github.com/python/cpython/commit/323e743d4879f1cd861d0b252775797fb7938755
msg346915 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-06-30 05:01
> I'm not sure of the purpose of sys._base_executable. Maybe open a separated issue to make the attribute public? I'm not sure if multiprocessing (for example) should use it, or if sys.executble should be used.

Yeah, it's a bit more complex. Sometimes it's needed and often not, depending on platform. For multiprocessing we need it on Windows because of the handle inheritance that's used, which only goes one subprocess deep,and we use special knowledge to make it work even in a venv.

But at least having it there means we can use it when needed without having to define its semantics for 100% of cases. Venvs make things really hard the way they're currently done...
msg347160 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-07-02 22:34
New changeset 3c34ea97a341e4dd80b542c99c593f014a8ae410 by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fixes path for sys.executable when running from the Microsoft Store (GH-14450)
https://github.com/python/cpython/commit/3c34ea97a341e4dd80b542c99c593f014a8ae410

New changeset 57eba3aff16016be84b2fa8532b4f618fec79d97 by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fix venv and test symlinking (GH-14456)
https://github.com/python/cpython/commit/57eba3aff16016be84b2fa8532b4f618fec79d97

New changeset a88652e488edc7664f395ba1a22b5e46539d01d3 by Ned Deily (Steve Dower) in branch '3.7':
bpo-37369: Fix path handling when python.exe is used as a symlink (GH-14461)
https://github.com/python/cpython/commit/a88652e488edc7664f395ba1a22b5e46539d01d3
msg347374 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-07-05 20:10
On Thu, 27 Jun 2019 at 18:08, Steve Dower <report@bugs.python.org> wrote:
> However, it does mean that we need to start reporting a sys.executable that is not inside sys.prefix, and I'm not sure whether that is a good idea.
>
> Looking at venv (+Vinay for confirmation here), it seems to treat "home" as "the directory containing the original Python binary" rather than "the directory containing the Python install". So in theory (and in my simple testing), using the sys.executable-not-in-prefix here is fine. But is that intentional?

I'm pretty sure there is code in the wild that expects <sys.prefix> /
python.exe to be the Python executable (when you're not in a virtual
environment). I seem to recall that virtualenv might do that sort of
thing. Sorry I can't be precise here, I've just come back from holiday
and won't be back into the swing of open source for a while yet, but
my recollection is that it gets done in situations where
sys.executable isn't (easily) accessible - for example, when getting
sys.prefix from the registry (um, I don't think that's right, as the
executable path is in the registry too - but you get the idea).

What I'd say is that we should be cautious here, as we risk trading
one breakage for another, rather than actually fixing the issue.
Certainly the assumption I quote above is unwarranted, and code making
it can't complain too much if it breaks, but the problem is that you
*have* to make that assumption as the necessary information isn't
available any other way. So we should do some careful testing of 3rd
party tools like virtualenv, pipenv, pipx, pew, tox, etc, to make sure
any "solution" here doesn't break them, and provide or document
transition options for them if necessary (because such tools can't
realistically avoid continuing to support older versions of Python).

IMO, we should bite the bullet here and actually document the
interfaces involved in virtual environments, so that tools have a
stable contract to work with. That way we'd limit the need for people
to rely on undocumented behaviour in the first place.

Paul
msg347376 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-07-05 20:33
> I'm pretty sure there is code in the wild that expects <sys.prefix> /
> python.exe to be the Python executable (when you're not in a virtual
> environment) ... for example, when getting
> sys.prefix from the registry (um, I don't think that's right, as the
> executable path is in the registry too - but you get the idea).

Yeah, for sure. Though as you say, when sys.executable is easily accessible, they're probably (hopefully) going to be using that.

Older versions of Python definitely omitted the full executable path from the registry. PEP 514 added that in, though in this case unfortunately I don't think we have any way to fix what goes in the registry :(

> What I'd say is that we should be cautious here, as we risk trading
> one breakage for another, rather than actually fixing the issue.

Yeah. We probably need to be clear about which approaches are actually supported and which ones are not.

Ultimately though, I think this is a situation where the breakage is forced upon us (by the operating system), and so fixing the behaviour of our built-in functionality pretty clearly outweight preserving the assumptions made by third-parties (especially since those assumptions are going to break).

> IMO, we should bite the bullet here and actually document the
> interfaces involved in virtual environments, so that tools have a
> stable contract to work with. That way we'd limit the need for people
> to rely on undocumented behaviour in the first place.

I feel like I've made this argument before, but I suspect if we'd documented them prior to this change, we'd now have no choice but to remain broken.

The interfaces that should be documented are "-m venv", venv.VenvBuilder, and the activation scripts, none of which had to change. But if we start making promises about how "python.exe" actually behaves in a venv then we're going to paint ourselves into a corner very quickly - the venv approach just doesn't scale well to the range of systems people use. (Obligatory mention that PEP 582 would have avoided this whole mess by not requiring us to duplicate python.exe anywhere. But it just wasn't meant to be.)
msg347384 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2019-07-05 21:07
On Fri, 5 Jul 2019 at 21:33, Steve Dower <report@bugs.python.org> wrote:
> Yeah. We probably need to be clear about which approaches are actually supported and which ones are not.
>
> Ultimately though, I think this is a situation where the breakage is forced upon us (by the operating system), and so fixing the behaviour of our built-in functionality pretty clearly outweight preserving the assumptions made by third-parties (especially since those assumptions are going to break).

Well, given that this is an issue specific to the store distribution
of Python, I'd say that it's the release of the store build that
forces it on us, not the OS, and releasing the store build was our
choice, so if we did so with the store build not meeting existing
implicit (i.e. undocumented, I agree) contracts, then we made that
choice. I think it was the right choice, because the store
distribution is a huge plus, but I do think we should be cautious
about impacting the existing user base simply because of a very new,
and explicitly documented as "still experimental" build. (Yes, I'm
still a bit sore about the hassle involved in getting virtualenv to
work - although the final resolution to that issue was IMO clean and a
huge improvement on the previous status quo, so I'm not *really* sore
:-))

> > IMO, we should bite the bullet here and actually document the
> > interfaces involved in virtual environments, so that tools have a
> > stable contract to work with. That way we'd limit the need for people
> > to rely on undocumented behaviour in the first place.
>
> I feel like I've made this argument before, but I suspect if we'd documented them prior to this change, we'd now have no choice but to remain broken.

Hardly. We would simply have had to document the changed behaviour and
follow our existing transition/compatibility processes. Documenting
things doesn't lock them in stone, it just requires us to give due
consideration to our users when we change things. Conversely, *not*
documenting things makes it easier for us to change things, at the
cost of not giving users supported ways to do certain things that they
may still need to do.

> The interfaces that should be documented are "-m venv", venv.VenvBuilder, and the activation scripts, none of which had to change. But if we start making promises about how "python.exe" actually behaves in a venv then we're going to paint ourselves into a corner very quickly - the venv approach just doesn't scale well to the range of systems people use. (Obligatory mention that PEP 582 would have avoided this whole mess by not requiring us to duplicate python.exe anywhere. But it just wasn't meant to be.)

I assume you're missing my key point here (not unlikely, as I didn't
state it explicitly - I guess I'm relying on undocumented information
myself ;-)). The point here is that venv doesn't exist in a vacuum,
before it existed, virtualenv was the standard way to produce virtual
environments. And because neither virtualenv nor venv document any of
their internal details, they are plagued with incompatibilities, and
work badly together. As a result, tools like pipenv, or pew, or pipx,
can't support both but end up picking just one - and because of
backward compatibility requirements, they pick virtualenv, which has
significantly hindered adoption of venv, which is (for hopefully
obvious reasons) a substantially more robust solution.

It may be that it's just too late to worry about this, and we should
simply accept that tools will be dropping Python 2 support very soon
now, at which point there's no real reason to continue using
virtualenv (apart from no-one having the time to make the change, but
if virtualenv adopts venv as its internal mechanism we can get around
that). But personally, I'd rather not rely on an assumption like that,
just yet (maybe in 3.9?).

(Side issue, I don't really see the relevance of PEP 582 here, as I
don't think it's something that tools like tox and pipenv could
replace virtual environments with - but maybe I'm wrong, if so the
arguments for PEP 582 didn't really get that point across).
msg347499 - (view) Author: gaborbernat (gaborbernat) * Date: 2019-07-08 14:34
A note on the above points, virtualenv has started migrating over to venv via https://github.com/pypa/virtualenv/pull/1377 . Following this, the old ways of virtualenv should be left to the history books. Hopefully, I can get it out within the next month or so.
History
Date User Action Args
2019-07-08 14:34:25gaborbernatsetnosy: + gaborbernat
messages: + msg347499
2019-07-05 21:07:57paul.mooresetmessages: + msg347384
2019-07-05 20:33:46steve.dowersetmessages: + msg347376
2019-07-05 20:10:39paul.mooresetmessages: + msg347374
2019-07-02 22:34:02ned.deilysetmessages: + msg347160
2019-07-01 16:13:29steve.dowersetassignee: steve.dower
2019-06-30 05:01:42steve.dowersetmessages: + msg346915
2019-06-30 04:49:04steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-06-29 21:29:02steve.dowersetmessages: + msg346900
2019-06-29 21:28:46steve.dowersetmessages: + msg346899
2019-06-29 20:53:15steve.dowersetpull_requests: + pull_request14285
2019-06-29 17:36:26steve.dowersetpull_requests: + pull_request14284
2019-06-29 17:34:19steve.dowersetmessages: + msg346890
2019-06-29 04:30:56steve.dowersetmessages: + msg346877
2019-06-28 23:22:20steve.dowersetpull_requests: + pull_request14278
2019-06-28 18:41:59steve.dowersetmessages: + msg346849
2019-06-28 18:00:17steve.dowersetpull_requests: + pull_request14273
2019-06-28 17:02:18steve.dowersetmessages: + msg346837
2019-06-28 15:10:46steve.dowersetpull_requests: + pull_request14266
2019-06-27 23:29:15vstinnersetmessages: + msg346786
2019-06-27 22:09:41eryksunsetnosy: + eryksun
messages: + msg346783
2019-06-27 17:10:24steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request14244
2019-06-27 17:08:46steve.dowersetnosy: + vinay.sajip, vstinner

messages: + msg346761
versions: + Python 3.8, Python 3.9
2019-06-26 22:55:21bstormsetmessages: + msg346694
2019-06-26 21:44:00ned.deilysetpriority: critical -> deferred blocker

messages: + msg346685
2019-06-26 21:28:28steve.dowersetpriority: normal -> critical
nosy: + ned.deily
messages: + msg346683

2019-06-26 18:47:36steve.dowersetmessages: + msg346658
2019-06-22 00:03:34bstormsetmessages: + msg346265
2019-06-21 23:54:16bstormcreate