classification
Title: Release Windows Store app containing Python
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: Ethan Smith, jkloth, miss-islington, ned.deily, paul.moore, steve.dower, tim.golden, vinay.sajip, vstinner, xtreak, zach.ware
Priority: Keywords: patch

Created on 2018-10-13 22:43 by steve.dower, last changed 2019-10-18 00:21 by izbyshev. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10245 merged steve.dower, 2018-10-30 15:55
PR 10677 merged steve.dower, 2018-11-23 14:56
PR 11019 merged vstinner, 2018-12-07 11:32
PR 11021 merged vstinner, 2018-12-07 11:51
PR 11027 merged steve.dower, 2018-12-07 15:06
PR 11028 merged steve.dower, 2018-12-07 15:08
PR 11029 merged steve.dower, 2018-12-07 16:03
PR 11030 merged steve.dower, 2018-12-07 16:25
PR 11092 merged steve.dower, 2018-12-11 04:37
PR 11093 merged miss-islington, 2018-12-11 04:56
PR 5812 izbyshev, 2019-10-18 00:21
Messages (42)
msg327679 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-10-13 22:43
The Windows Store can now install unrestricted apps, which means we can provide the Python interpreter in there.

Advantages:
* far more reliable installation
* can have specific executables on PATH
* automatic updates
* other apps can act as extensions (with permission), so things like Mu could also become store apps in the future

Disadvantages:
* only --user installs for pip will work
* harder to find/modify installed files

I think for a (very significant) subset of our users, this will be a much better experience than downloading our current installer. It also has the advantage of making "Python" appear in searches in the start menu, which will link directly to a one-click "Install" button.

The biggest issue is likely to be pip not installing with --user by default, as well as subsequent issues with apps that require users to navigate to their %AppData% directories manually (since these will be redirected to different locations).

But until we get a package available for testing, it will be hard to figure out what needs fixing. Given this is just distribution and not a new platform, I plan to enable it for Python 3.7 (probably 3.7.2, but it may be possible to bundle 3.7.1 as-is for testing).
msg327691 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2018-10-14 06:02
+1 for reliable installation. There were some cases (issue33220, issue30944) where the installer was detected as a Virus by some vendors which would be reduced since this is installed from the App store as a verified app. I hope this will also reduce the scenarios where users need to resort to bug tracker with log files when there are installation failures [1] providing a more smooth one click install.

There is a long standing open issue [2] to make `pip install --user` as a default which I hope gets resolved and there is an issue with using `pip install --user` inside a virtualenv which seems to block the earlier issue [3]

Thanks a lot for this!

[1] https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=windows+msi+install&submit=search&status=-1%2C1%2C2%2C3

[2] https://github.com/pypa/pip/issues/1668

[3] https://github.com/pypa/pip/issues/5702
msg328146 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-10-20 15:19
I'm 99% sure that with issue34725 I'll be able to make "pip --user" the default when using this install, as well as making the commands globally available (i.e. "python[3[.8]]", "pip[3[.8]]", "idle[3[.8]]" will work everywhere).
msg328147 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-10-20 15:19
(And I'll deal with that 1% at the PyPA sprints next weekend. We are due for a new pip insertion anyway, so I guess that'll come for 3.7.2.)
msg330166 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-21 00:29
For people watching the bug, this is ready for review.

Most of the change is adding a "layout" script (roughly the equivalent of "make install" for Windows, but with more flexibility to generate different structures, precompile stuff, etc.)

About the only CPython change is to venv. I added a new launcher variation that looks for pyvenv.cfg and runs the python.exe listed in that, so now we don't have to copy any DLLs into the venv at all (it's needed because the old one would try to LoadLibrary DLLs inside the store package, which is not allowed, but it also fixes my #1 complaint about venv which is that upgrading the base Python breaks all venvs).

There are also a few tweaks to the startup process, including extending the existing __PYVENV_LAUNCHER__ env variable from macOS to Windows as well, though it's handled in a different location. And a couple of minor tweaks to tests - issues discovered now that it's pretty easy to generate a "real" layout to run tests in.

I can break this up into a couple of PRs if that's preferred, though it doesn't really gain anything. If you're not interested in the new layout script, stop reading the PR after you finish launcher.c :)
msg330359 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-23 19:57
If anyone would like to try this out early, here's some instructions for a build I just made.

Enable sideloading apps on your machine (see https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development - may already be done, sideload or developer mode should do)

In a regular (non-admin) PowerShell prompt in a temporary directory, run these commands:

iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34540/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
Expand-Archive .\python37.zip .
Add-AppxPackage appx\python.msix

(The first may take a while as it downloads approx. 80MB. I'm intending to leave the test suite out of the final package, which should bring it down closer to 40MB.)

Now play with it:
$> python.exe
$> python3.exe
$> python3.7.exe
$> pip.exe
$> pip3.exe
$> pip3.7.exe
$> idle.exe
$> idle3.exe
$> idle3.7.exe
(Or press Win+S and search for new entries)

To remove it:

Remove-AppxPackage (Get-AppxPackage *Python.3.7)
(delete the downloaded files)

Right now, the py.exe launcher can't find it, nor can any tools that rely on enumerating the registry - directly requesting the reg keys will return the right values though. VS Code seems to pick it up just fine though, I guess because it's on PATH.
msg330452 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-26 18:41
For my own reference, some issues I found installing this package on a clean machine:

* requires Developer Mode (until I get the sccd file signed - just sideloading is not sufficient)
* requires vcruntime140.dll to be included in the package (workaround is to install the redistributable from https://visualstudio.microsoft.com/downloads/#title-39324)
* venv needs to treat .pdb files as binary
msg330643 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-11-29 00:24
Updated testing instructions - this one should work with just sideloading permissions I think.

iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34760/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
Expand-Archive .\python37.zip .
Add-AppxPackage appx\python.msix

To remove:
Remove-AppxPackage (Get-AppxPackage *Python.3.7)
(delete the downloaded files)
msg331265 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 05:09
New changeset 468a15aaf9206448a744fc5eab3fc21f51966aad by Steve Dower in branch 'master':
bpo-34977: Add Windows App Store package (GH-10245)
https://github.com/python/cpython/commit/468a15aaf9206448a744fc5eab3fc21f51966aad
msg331266 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 05:09
New changeset 253209149389e6793a052034e1f2d97691086f18 by Steve Dower in branch '3.7':
[3.7] bpo-34977: Add Windows App Store package (GH-10245)
https://github.com/python/cpython/commit/253209149389e6793a052034e1f2d97691086f18
msg331267 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 05:15
Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build. As soon as we snap for the RC I'll kick off an update and make the store page public, and hopefully can promote it enough to get eyes on it.

Unfortunately, I discovered as part of a test submission that the minimum Windows version matters, and it's a version that hasn't been rolled out fully yet because of some bugs, so there may not be that many people who can use it this first time. But that will improve over time, and I'm sure I can find enough people to flush out issues before the next release (anyone on the Windows Insiders program should be fine).
msg331300 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 11:46
> Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build.

I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437. It doesn't seem like a good idea to push such change between a release candidate and a final version :-(
msg331301 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 11:57
New changeset cb0b78a070ea3b704416e74f64046178ae0dff3e by Victor Stinner in branch 'master':
Revert "bpo-34977: Add Windows App Store package (GH-10245)" (GH-11019)
https://github.com/python/cpython/commit/cb0b78a070ea3b704416e74f64046178ae0dff3e
msg331304 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 12:49
> I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437.

I reverted the change in master and I will merge PR 11021 once tests pass to revert in 3.7 as well.


> It doesn't seem like a good idea to push such change between a release candidate and a final version :-(

I started a discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2018-December/155932.html
msg331310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 13:31
New changeset 783b794a5e6ea3bbbaba45a18b9e03ac322b3bd4 by Victor Stinner in branch '3.7':
Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)
https://github.com/python/cpython/commit/783b794a5e6ea3bbbaba45a18b9e03ac322b3bd4
msg331311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 13:32
> Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)

That should repair buildbots, so I remove "release blocker" priority.
msg331316 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 14:13
We haven't done RC yet - why is pushing this now so bad? Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.
msg331321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 14:50
> We haven't done RC yet - why is pushing this now so bad?

I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours:
https://pythondev.readthedocs.io/ci.html#revert-on-fail


> Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-) By the way, as I wrote in my email, I would prefer to see the venv+tests change pushed in separated change (and explain why you do these changes).


> I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.

Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

Do you mean that buildbots must upgrade to VS 2017? Can Python 3.6 and 3.7 be compiled without any issue on VS 2017? Maybe send an email to buildbot-status if you want to see more buildbots running VS 2017 rather than VS 2015.

--

There was a small bug in your PR, in PC/launcher.c, fixed by commit 4c49da0cb7434c676d70b9ccf38aca82ac0d64a9 (bpo-35436):

+        if (executable == NULL) {
+            error(RC_NO_MEMORY, L"A memory allocation failed");
+        }
msg331322 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-07 14:51
Since the first attempt broke all Windows buildbot workers, it would be nice to use a custom build before merging a new attempt :-) See instructions in the devguide:
https://devguide.python.org/buildbots/?highlight=custom#custom-builders

Note: Pablo used buildbot-custom to test a revert of the App Store change.
msg331328 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 15:50
>> We haven't done RC yet - why is pushing this now so bad?
> I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours

I don't dispute that policy, but you said it shouldn't go in between RC and a release, which it isn't.

> nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-)

More nitpicking: the venv changes only affect Windows. Which makes it purely Windows and tests, exactly as I said. Adding smiles doesn't save you from having to read :)

> Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

Sure. You can start by reading this thread, but the summary is that App Store apps (regardless of platform) generally have to go through an approval process before they can be installed. Without going through this process, they can't be tested fully in context. I believe we should make a true release and put that through the process, rather than me just doing one out of my branch.

I provided instructions for developer testing above, which covers about 80% of what needs to be checked. The rest can only be tested by real use of a proper release.
msg331350 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-12-07 20:29
Various questions and comments:

0. Steve: For more than one reason, I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org.  Can you assure us that the latter are not going away?  I would have the same concern about adding something to the Apple/Mac store.

1. Ned: Since enhancements are not normally backported, it would be nice to have your approval, in whatever form it took, recorded here and not just in private email.  I give a reason below.

2. Installers are always experimental until tested by multiple users on multiple machines.  Even then, installation specific issues can arise.  As I remember, the current installer, when new in 3.5, needed a shakedown period.  One installer-specific issue was the initial working directory.  To have a Windows Store Python ready for 3.8, we need to start now.

There was once an issue where one of the installers failed to include one idlelib file. I believe it was help.html. I since added idle_test/test_help.py, which would now fail with something like "No such file or directory: 'C:\\Programs\\Python37\\Lib\\idlelib\\help.html'".)
Repository testing does not test the actual binaries that users install, in their production context.

Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously?  So I could test both IDLEs that beginners might install?

3. Anyone: Though it is not essential that I know, I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there.  My search only came up with age restrictions, which python does not need (in my opinion).

4. While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces.  They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear).  I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.
msg331352 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 20:43
Thanks Terry, great questions.

> I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org.  Can you assure us that the latter are not going away?

Absolutely. The current installer is not going anywhere, and there's no point considering the possibility until we've completely dropped support for Windows 10 1803 and earlier (which was only released this year, so dropping support is a long way out yet).

> Installers are always experimental until tested by multiple users on multiple machines.  Even then, installation specific issues can arise. ...  To have a Windows Store Python ready for 3.8, we need to start now.

Thanks for the vote of support for this point.

> Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously?  So I could test both IDLEs that beginners might install?

Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

Also, I've been testing IDLE regularly, since one of the advantages of being in the Store is that we can safely put idle[3[.7]].exe on PATH. It definitely works :)

I've also set up tests that run from an "installed" layout, rather than the source tree, and those are all passing as well (with one of the trivial test fixes in the PR).

> I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there.

In general, all app stores disallow their apps from doing things that may break or corrupt the machines they are installed on. Apps are self-contained and can only operate outside of their boundaries through carefully designed permissions/capabilities (which often prompt the user before continuing).

Recently, Microsoft added support for "full trust" apps, which can bypass many of these restrictions. The aim is to allow traditional applications to swap their MSIs for being installed through the store, which can then ensure trustworthiness of at least the installers. (For example, a Store app is *never* going to install something into System32 or modify operating system files, whether it's full trust or not.) Full trust allows us to do things like launching non-app store processes and creating raw sockets through legacy APIs, so we can very easily move Python into a full trust package rather than having to rewrite it using approved APIs (a project that's been attempted a few times, but always loses low-level functionality that is considered a core part of Python - hence my long-term desire to make the low-level parts optional and have higher-level abstractions be the guaranteed parts).

> While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces.  They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear).  I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.

I agree that it's valuable and I've done it internally plenty of times. Unfortunately, no matter how much people repeatedly say that a PR for CPython is part of a chain, there always seems to be pushback from people who ignore that and won't approve a critical prerequisite because there's "no need".

There's also the point that many of the changes were not known to be needed before the others were made. The pre-squash history is very interleaved between the various changes, and extracting them into separate PRs that totally stand alone is nearly as much work as solving the issues in the first place. I made the call this time to keep them all together and direct reviewers' attention towards the relevant parts. It turns out that didn't attract reviewers, so I'll come up with a new strategy next time I want to get people's attention on a change (maybe Buzzfeed style headlines? :) )
msg331356 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-12-07 21:18
> > Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously?  So I could test both IDLEs that beginners might install?
>
> Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

I'd like to expand on this question a little. The first thing I'll
note is that I'm not aware of any documentation on the Windows Store
installer, and how it works. It's possible there's a docs patch in
here somewhere, but to be honest, there are 8 PRs linked to this issue
and I don't have the time to check them all right now.

How will the Windows Store install work alongside the PSF one? Will it
work the same, in that neither will go on PATH unless the user
explicitly requests it? If I have PSF Python 3.7.1 and Store Python
3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
influence it to choose the other one? Same question for 3rd party
tools like "virtualenv -p 3.7" - I know it's not up to us to deal with
virtualenv issues, but conversely I do think we have a responsibility
to not change assumptions that 3rd party tools rely on. In terms of
PEP 514, will the Store installer use a different tag, or some other
way of having registry entries independent of the PSF installer?

Basically, I'm not clear on how having the Store install and the PSF
install simultaneously present will work - very few tools in my
experience really support PEP 514 (not even the `py` launcher) - and
frankly, if we're going to break tool assumptions that there's only
one official "3.7" present at any time, we're on very shaky ground
claiming that PEP 514 warned them if we haven't even updated our own
supplied tool! (I know that there's lots of ways people can have
multiple versions right now - PSF plus Anaconda, for example - but
it's fine to say "you're mixing distributions, you're on your own" in
that case. If the PSF is publishing two versions of 3.7, which can be
simultaneously installed, we can't really use that argument.

Paul
msg331357 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 21:40
> The first thing I'll note is that I'm not aware of any documentation
> on the Windows Store installer, and how it works.

Right now, correct.

> How will the Windows Store install work alongside the PSF one?

It will go into PATH under the same names as on other platforms (python, python3, python3.7, and same for pip and idle). The Windows Store makes it possible to do this reliably, so it's just on by default (and configurable through OS-provided UI for selecting the app to respond to a certain name).

If you have already installed it traditionally and enabled the PATH modification, that will take precedence. So you'll get the MSI installed Python. If you've modified PATH by hand, it depends on where you put it.

> If I have PSF Python 3.7.1 and Store Python
> 3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
> influence it to choose the other one? Same question for 3rd party
> tools like "virtualenv -p 3.7"

Right now, py.exe will not detect it. This is an unavoidable limitation right now (specifically, enumeration of classic app compatibility keys does not work, though direct access does), but it may become avoidable in the future (and being able to say "Python needs it" is actually going to be useful for removing the limitation).

As usual, our recommended way to influence choice of Python is to specify it. "virtualenv -p 3.7" will find a traditionally installed Python first, but if it's not there and the store one is it *might* find the store one (it depends whether it requests the specific registry key or tries to enumerate all of them - again, I've got people looking into enabling enumeration, but we're talking about adding another feature to Windows specifically for Python, and those take time).

> Basically, I'm not clear on how having the Store install and the PSF
> install simultaneously present will work - very few tools in my
> experience really support PEP 514 (not even the `py` launcher) - and
> frankly, if we're going to break tool assumptions that there's only
> one official "3.7" present at any time, we're on very shaky ground
> claiming that PEP 514 warned them if we haven't even updated our own
> supplied tool! (I know that there's lots of ways people can have
> multiple versions right now - PSF plus Anaconda, for example - but
> it's fine to say "you're mixing distributions, you're on your own" in
> that case. If the PSF is publishing two versions of 3.7, which can be
> simultaneously installed, we can't really use that argument.

In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.
msg331358 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 21:49
> How will the Windows Store install work alongside the PSF one?

Oh, and since I didn't mention it in my direct answer, the Store install goes to a totally different location (per machine, no elevation required), as do packages installed using pip (automatically per user). Nothing is shared between the two installs at all.
msg331363 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-12-07 23:12
Thanks for the clarification.

> In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

Agreed, it does sound like it's probably the correct approach. But I'd
strongly recommend that as much of the behaviour as possible gets
documented as part of this change. There are a *lot* of projects out
there that write code for Windows based on a relatively superficial
understanding of all of the complexities that come into play here -
most registry scanning code is based on guesswork and reverse
engineering of how things worked before PEP 514 (I should know, I
wrote rather a lot of it!!!) and plays fast and loose with even things
like 32-bit/64-bit registry overlaying (it's late and I'm tired, sorry
for not using the accurate term). Settig projects like virtualenv,
tox, nox, pew, pipenv etc up for new bug reports associated with store
installs, without any sort of guidance as to what's going on in the
official docs, seems unfriendly at best. (I'm thinking of a section in
"Python Setup and Usage" covering the store installer - that in
addition to the changes in
https://docs.python.org/3.7/using/windows.html#installation-steps
needed to mention that there's now five installers available).

> A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.

The "pythonup" project (https://github.com/uranusjr/pythonup-windows)
creates versioned copies of python - python3.7.exe. Again, it's a
non-standard project, so not technically our problem, but I'd be
interested in how it would interact with the Store python.

I do appreciate that we need to get the store installer out there in
order to test things like this, and that initially at least there
aren't going to be a huge number of people using the store installer
over the traditional one (and likely vanishingly small numbers of
people using both in parallel). But conversely, there's a class of
problems that will be reported to other projects rather than to us,
and I think we should be trying to support those other projects in
dealing with this change, not simply leaving it up to them to find out
about it (I doubt they'd even know to ask "did you use the store
install or the traditional one" when diagnosing a problem).

In general, I don't think we've done that great a job of publicising
the various install options on Windows - we still get pip users trying
to install packages into the embedded distribution, which they are
using as if it were a standalone interpreter, and as far as I can tell
essentially no-one knows about the nuget packages for Python. Let's
not repeat the mistakes we've made previously with this new
distribution option.
msg331368 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-07 23:45
> In general, I don't think we've done that great a job of publicising
> the various install options on Windows - we still get pip users trying
> to install packages into the embedded distribution, which they are
> using as if it were a standalone interpreter, and as far as I can tell
> essentially no-one knows about the nuget packages for Python. Let's
> not repeat the mistakes we've made previously with this new
> distribution option.

Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?

So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.
msg331377 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2018-12-08 11:06
> Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

Sorry about that - I wasn't trying to make you feel guilty! And yes,
the embeddable package docs are very good. The problem with that one
is mostly that people don't *read* the docs (but having them to point
to makes responding to pip issues much easier, so you have my
gratitude there :-)

> I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?
>
> So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.

In which case, let's have a placeholder note in the docs, saying that
the Windows Store installer exists, but it's experimental and may have
unexpected interactions with other tools that expect a standard Python
install. Users trying the Windows Store installer should be prepared
for such issues, and we'd appreciate reports so that we can address
them. Add that to the release announcement as well for extra
visibility.
msg331387 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-08 15:46
> In which case, let's have a placeholder note in the docs, saying that
> the Windows Store installer exists, but it's experimental and may have
> unexpected interactions with other tools that expect a standard Python
> install. Users trying the Windows Store installer should be prepared
> for such issues, and we'd appreciate reports so that we can address
> them. Add that to the release announcement as well for extra
> visibility.

Sure, but first I need PR 11029 (and 11030 the backport) approved, so that I can rebase and work on redoing the other half of the change. Victor seems to have disappeared for the weekend, but I made all his requested changes.
msg331517 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-10 15:55
I'm just waiting on Victor to sign off on PR 11029 (or someone else to do it) so I can merge it and rebase PR 11027.

Hopefully then someone will review it today so that we don't have to hold up 3.7.2rc1 any longer.
msg331518 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-10 15:57
Sorry, I don't have the bandwidth to provide a proper review.
msg331519 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-10 16:11
Okay, I'll merge the first part now so I can rebase the main one and *hopefully* someone will be willing to review it now it's smaller. Otherwise, I think we're best to go with buildbots and all the testing I've already done.

This is my sole job today, so I'll keep an eye on the buildbots. I'll also get the docs updated as Paul suggested for the main PRs.
msg331520 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-10 16:11
New changeset 1c3de541e64f75046b20cdd27bada1557e550bcd by Steve Dower in branch 'master':
bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
https://github.com/python/cpython/commit/1c3de541e64f75046b20cdd27bada1557e550bcd
msg331521 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-10 16:11
New changeset b264c609853eae9dbb45c6dbee11e84ae3927e88 by Steve Dower in branch '3.7':
[3.7] bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
https://github.com/python/cpython/commit/b264c609853eae9dbb45c6dbee11e84ae3927e88
msg331530 - (view) Author: Jeremy Kloth (jkloth) * Date: 2018-12-10 18:27
See also bpo-35450: venv module doesn't create a copy of python binary by default
msg331534 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-10 20:36
Paul (and anyone else) - the below link should go directly to just the commit with the docs update. I did a slight rearrangement of the install docs to make the options clearer, and wrote up *just* enough info on nuget to help people use it right (I hope). Similarly for the Store package.

Any comments appreciated. Still aiming to get this in for 3.7.2rc1 today.

https://github.com/python/cpython/pull/11027/commits/7d7ddbc41bd30690de8c6227cebc41c4e0f3cf88

I've also kicked off a custom buildbot run against the 3.7 version of the change. The version in master ran earlier and was clean. Plus I'm running test builds on the release machine, so it should be ready to go.
msg331553 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-11 00:25
Thanks, Paul. Appreciate it :)

Doing a final call before I hit merge in an hour or so. Right now:
* full release build works
* full test pass on all CI + custom buildbots work
* prior test package made it to the store
* docs clearly show it may change again

Any last pushback?
msg331560 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-11 02:53
New changeset 0cd6391fd890368ea1743dac50c366b42f2fd126 by Steve Dower in branch 'master':
bpo-34977: Add Windows App Store package (GH-11027)
https://github.com/python/cpython/commit/0cd6391fd890368ea1743dac50c366b42f2fd126
msg331566 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-11 03:58
New changeset 0e4ad88ff8956d9289ab0a1314636ac15b374459 by Steve Dower in branch '3.7':
bpo-34977: Add Windows App Store package (GH-11027)
https://github.com/python/cpython/commit/0e4ad88ff8956d9289ab0a1314636ac15b374459
msg331571 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-11 04:40
Luckily, just noticed that I introduced a bug as part of splitting up the original PR into two.

It wasn't going to have an impact until the package was properly installed from the store, but without the proper directory resolution in there (since the variable I removed is never defined now), in some scenarios it wouldn't be possible to launch Python from a process outside of the app package, such as a virtual environment.

Testing in a proper app container is difficult, but I'll see if I can come up with a way to do it. It might require a custom buildbot, so I'll see whether there's a way I can do that (historically we've not been able to wire internal work machines - including Azure hosted ones - into other CI systems, but I'll try again).
msg331572 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-12-11 04:56
New changeset d5a6a389d492c5e3d7933bafbd5252fd86ac4d49 by Steve Dower in branch 'master':
bpo-34977: Remove unused preprocessor definition (GH-11092)
https://github.com/python/cpython/commit/d5a6a389d492c5e3d7933bafbd5252fd86ac4d49
msg331573 - (view) Author: miss-islington (miss-islington) Date: 2018-12-11 05:15
New changeset 9bb306d586e3f1a48db40bd9519412de4fff3ee8 by Miss Islington (bot) in branch '3.7':
bpo-34977: Remove unused preprocessor definition (GH-11092)
https://github.com/python/cpython/commit/9bb306d586e3f1a48db40bd9519412de4fff3ee8
History
Date User Action Args
2019-10-18 00:21:36izbyshevsetpull_requests: + pull_request16379
2018-12-12 23:45:02steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-12-11 05:15:01miss-islingtonsetnosy: + miss-islington
messages: + msg331573
2018-12-11 04:56:19miss-islingtonsetpull_requests: + pull_request10325
2018-12-11 04:56:11steve.dowersetmessages: + msg331572
2018-12-11 04:40:53steve.dowersetmessages: + msg331571
2018-12-11 04:37:03steve.dowersetpull_requests: + pull_request10324
2018-12-11 03:58:54steve.dowersetmessages: + msg331566
2018-12-11 02:53:09steve.dowersetmessages: + msg331560
2018-12-11 00:25:18steve.dowersetmessages: + msg331553
2018-12-10 20:36:09steve.dowersetmessages: + msg331534
2018-12-10 18:27:24jklothsetnosy: + jkloth
messages: + msg331530
2018-12-10 16:11:37steve.dowersetmessages: + msg331521
2018-12-10 16:11:26steve.dowersetmessages: + msg331520
2018-12-10 16:11:09steve.dowersetmessages: + msg331519
2018-12-10 15:57:28vstinnersetmessages: + msg331518
2018-12-10 15:55:24steve.dowersetmessages: + msg331517
2018-12-08 19:15:31terry.reedysetnosy: - terry.reedy
2018-12-08 15:46:30steve.dowersetmessages: + msg331387
2018-12-08 11:06:52paul.mooresetmessages: + msg331377
2018-12-07 23:45:09steve.dowersetmessages: + msg331368
2018-12-07 23:12:39paul.mooresetmessages: + msg331363
2018-12-07 22:32:49Ethan Smithsetnosy: + Ethan Smith
2018-12-07 21:49:17steve.dowersetmessages: + msg331358
2018-12-07 21:40:09steve.dowersetmessages: + msg331357
2018-12-07 21:18:49paul.mooresetmessages: + msg331356
2018-12-07 20:43:45steve.dowersetmessages: + msg331352
2018-12-07 20:29:03terry.reedysetmessages: + msg331350
2018-12-07 16:25:41steve.dowersetpull_requests: + pull_request10266
2018-12-07 16:03:03steve.dowersetpull_requests: + pull_request10265
2018-12-07 15:50:56steve.dowersetmessages: + msg331328
2018-12-07 15:08:15steve.dowersetpull_requests: + pull_request10264
2018-12-07 15:06:35steve.dowersetpull_requests: + pull_request10263
2018-12-07 14:51:59vstinnersetmessages: + msg331322
2018-12-07 14:50:13vstinnersetmessages: + msg331321
2018-12-07 14:13:53steve.dowersetmessages: + msg331316
2018-12-07 13:32:17vstinnersetpriority: release blocker ->

messages: + msg331311
2018-12-07 13:31:45vstinnersetmessages: + msg331310
2018-12-07 12:49:01vstinnersetmessages: + msg331304
2018-12-07 11:57:46vstinnersetmessages: + msg331301
2018-12-07 11:51:16vstinnersetpull_requests: + pull_request10258
2018-12-07 11:46:22vstinnersetpriority: normal -> release blocker
nosy: + ned.deily, vstinner
messages: + msg331300

2018-12-07 11:32:17vstinnersetstage: commit review -> patch review
pull_requests: + pull_request10255
2018-12-07 05:15:24steve.dowersetstage: patch review -> commit review
2018-12-07 05:15:16steve.dowersetmessages: + msg331267
2018-12-07 05:09:57steve.dowersetmessages: + msg331266
2018-12-07 05:09:24steve.dowersetmessages: + msg331265
2018-11-29 00:24:21steve.dowersetmessages: + msg330643
2018-11-26 18:42:01steve.dowersetmessages: - msg330453
2018-11-26 18:41:44steve.dowersetmessages: + msg330453
2018-11-26 18:41:33steve.dowersetmessages: + msg330452
2018-11-23 19:57:16steve.dowersetnosy: + terry.reedy
messages: + msg330359
2018-11-23 14:56:34steve.dowersetpull_requests: + pull_request9932
2018-11-21 00:29:52steve.dowersetnosy: + vinay.sajip
messages: + msg330166
2018-10-30 15:55:13steve.dowersetkeywords: + patch
stage: patch review
pull_requests: + pull_request9559
2018-10-20 15:19:38steve.dowersetmessages: + msg328147
2018-10-20 15:19:08steve.dowersetmessages: + msg328146
2018-10-14 06:02:43xtreaksetnosy: + xtreak
messages: + msg327691
2018-10-13 22:43:47steve.dowercreate