Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release Windows Store app containing Python #79158

Closed
zooba opened this issue Oct 13, 2018 · 42 comments
Closed

Release Windows Store app containing Python #79158

zooba opened this issue Oct 13, 2018 · 42 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes OS-windows type-feature A feature request or enhancement

Comments

@zooba
Copy link
Member

zooba commented Oct 13, 2018

BPO 34977
Nosy @pfmoore, @vsajip, @vstinner, @tjguk, @jkloth, @ned-deily, @zware, @zooba, @ethanhs, @miss-islington, @tirkarthi
PRs
  • bpo-34977: Add Windows App Store package #10245
  • [3.7] bpo-34977: Add Windows App Store package (GH-10245) #10677
  • Revert "bpo-34977: Add Windows App Store package (GH-10245)" #11019
  • [3.7] Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" #11021
  • bpo-34977: Add Windows App Store package #11027
  • [3.7] bpo-34977: Add Windows App Store package (GH-11027) #11028
  • bpo-34977: Use venv redirector instead of original python.exe on Windows #11029
  • [3.7] bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029) #11030
  • bpo-34977: Remove unused preprocessor definition #11092
  • [3.7] bpo-34977: Remove unused preprocessor definition (GH-11092) #11093
  • bpo-32907: pathlib: Fix test_resolve_common failure on Windows #5812
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2018-12-12.23:45:02.538>
    created_at = <Date 2018-10-13.22:43:47.118>
    labels = ['3.8', 'type-feature', '3.7', 'OS-windows']
    title = 'Release Windows Store app containing Python'
    updated_at = <Date 2019-10-18.00:21:36.712>
    user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2019-10-18.00:21:36.712>
    actor = 'izbyshev'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2018-12-12.23:45:02.538>
    closer = 'steve.dower'
    components = ['Windows']
    creation = <Date 2018-10-13.22:43:47.118>
    creator = 'steve.dower'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 34977
    keywords = ['patch']
    message_count = 42.0
    messages = ['327679', '327691', '328146', '328147', '330166', '330359', '330452', '330643', '331265', '331266', '331267', '331300', '331301', '331304', '331310', '331311', '331316', '331321', '331322', '331328', '331350', '331352', '331356', '331357', '331358', '331363', '331368', '331377', '331387', '331517', '331518', '331519', '331520', '331521', '331530', '331534', '331553', '331560', '331566', '331571', '331572', '331573']
    nosy_count = 11.0
    nosy_names = ['paul.moore', 'vinay.sajip', 'vstinner', 'tim.golden', 'jkloth', 'ned.deily', 'zach.ware', 'steve.dower', 'ethan smith', 'miss-islington', 'xtreak']
    pr_nums = ['10245', '10677', '11019', '11021', '11027', '11028', '11029', '11030', '11092', '11093', '5812']
    priority = None
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue34977'
    versions = ['Python 3.7', 'Python 3.8']

    @zooba
    Copy link
    Member Author

    zooba commented Oct 13, 2018

    The Windows Store can now install unrestricted apps, which means we can provide the Python interpreter in there.

    Advantages:

    • far more reliable installation
    • can have specific executables on PATH
    • automatic updates
    • other apps can act as extensions (with permission), so things like Mu could also become store apps in the future

    Disadvantages:

    • only --user installs for pip will work
    • harder to find/modify installed files

    I think for a (very significant) subset of our users, this will be a much better experience than downloading our current installer. It also has the advantage of making "Python" appear in searches in the start menu, which will link directly to a one-click "Install" button.

    The biggest issue is likely to be pip not installing with --user by default, as well as subsequent issues with apps that require users to navigate to their %AppData% directories manually (since these will be redirected to different locations).

    But until we get a package available for testing, it will be hard to figure out what needs fixing. Given this is just distribution and not a new platform, I plan to enable it for Python 3.7 (probably 3.7.2, but it may be possible to bundle 3.7.1 as-is for testing).

    @zooba zooba added 3.7 (EOL) end of life 3.8 only security fixes labels Oct 13, 2018
    @zooba zooba self-assigned this Oct 13, 2018
    @zooba zooba added OS-windows type-feature A feature request or enhancement labels Oct 13, 2018
    @tirkarthi
    Copy link
    Member

    +1 for reliable installation. There were some cases (bpo-33220, bpo-30944) where the installer was detected as a Virus by some vendors which would be reduced since this is installed from the App store as a verified app. I hope this will also reduce the scenarios where users need to resort to bug tracker with log files when there are installation failures [1] providing a more smooth one click install.

    There is a long standing open issue [2] to make pip install --user as a default which I hope gets resolved and there is an issue with using pip install --user inside a virtualenv which seems to block the earlier issue [3]

    Thanks a lot for this!

    [1] https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=windows+msi+install&submit=search&status=-1%2C1%2C2%2C3

    [2] pypa/pip#1668

    [3] pypa/pip#5702

    @zooba
    Copy link
    Member Author

    zooba commented Oct 20, 2018

    I'm 99% sure that with bpo-34725 I'll be able to make "pip --user" the default when using this install, as well as making the commands globally available (i.e. "python[3[.8]]", "pip[3[.8]]", "idle[3[.8]]" will work everywhere).

    @zooba
    Copy link
    Member Author

    zooba commented Oct 20, 2018

    (And I'll deal with that 1% at the PyPA sprints next weekend. We are due for a new pip insertion anyway, so I guess that'll come for 3.7.2.)

    @zooba
    Copy link
    Member Author

    zooba commented Nov 21, 2018

    For people watching the bug, this is ready for review.

    Most of the change is adding a "layout" script (roughly the equivalent of "make install" for Windows, but with more flexibility to generate different structures, precompile stuff, etc.)

    About the only CPython change is to venv. I added a new launcher variation that looks for pyvenv.cfg and runs the python.exe listed in that, so now we don't have to copy any DLLs into the venv at all (it's needed because the old one would try to LoadLibrary DLLs inside the store package, which is not allowed, but it also fixes my #1 complaint about venv which is that upgrading the base Python breaks all venvs).

    There are also a few tweaks to the startup process, including extending the existing __PYVENV_LAUNCHER__ env variable from macOS to Windows as well, though it's handled in a different location. And a couple of minor tweaks to tests - issues discovered now that it's pretty easy to generate a "real" layout to run tests in.

    I can break this up into a couple of PRs if that's preferred, though it doesn't really gain anything. If you're not interested in the new layout script, stop reading the PR after you finish launcher.c :)

    @zooba
    Copy link
    Member Author

    zooba commented Nov 23, 2018

    If anyone would like to try this out early, here's some instructions for a build I just made.

    Enable sideloading apps on your machine (see https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development - may already be done, sideload or developer mode should do)

    In a regular (non-admin) PowerShell prompt in a temporary directory, run these commands:

    iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34540/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
    Expand-Archive .\python37.zip .
    Add-AppxPackage appx\python.msix

    (The first may take a while as it downloads approx. 80MB. I'm intending to leave the test suite out of the final package, which should bring it down closer to 40MB.)

    Now play with it:
    $> python.exe
    $> python3.exe
    $> python3.7.exe
    $> pip.exe
    $> pip3.exe
    $> pip3.7.exe
    $> idle.exe
    $> idle3.exe
    $> idle3.7.exe
    (Or press Win+S and search for new entries)

    To remove it:

    Remove-AppxPackage (Get-AppxPackage *Python.3.7)
    (delete the downloaded files)

    Right now, the py.exe launcher can't find it, nor can any tools that rely on enumerating the registry - directly requesting the reg keys will return the right values though. VS Code seems to pick it up just fine though, I guess because it's on PATH.

    @zooba
    Copy link
    Member Author

    zooba commented Nov 26, 2018

    For my own reference, some issues I found installing this package on a clean machine:

    • requires Developer Mode (until I get the sccd file signed - just sideloading is not sufficient)
    • requires vcruntime140.dll to be included in the package (workaround is to install the redistributable from https://visualstudio.microsoft.com/downloads/#title-39324)
    • venv needs to treat .pdb files as binary

    @zooba
    Copy link
    Member Author

    zooba commented Nov 29, 2018

    Updated testing instructions - this one should work with just sideloading permissions I think.

    iwr "https://dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_apis/build/builds/34760/artifacts?artifactName=appx&api-version=5.0-preview.5&%24format=zip" -outfile python37.zip
    Expand-Archive .\python37.zip .
    Add-AppxPackage appx\python.msix

    To remove:
    Remove-AppxPackage (Get-AppxPackage *Python.3.7)
    (delete the downloaded files)

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    New changeset 468a15a by Steve Dower in branch 'master':
    bpo-34977: Add Windows App Store package (GH-10245)
    468a15a

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    New changeset 2532091 by Steve Dower in branch '3.7':
    [3.7] bpo-34977: Add Windows App Store package (GH-10245)
    2532091

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build. As soon as we snap for the RC I'll kick off an update and make the store page public, and hopefully can promote it enough to get eyes on it.

    Unfortunately, I discovered as part of a test submission that the minimum Windows version matters, and it's a version that hasn't been rolled out fully yet because of some bugs, so there may not be that many people who can use it this first time. But that will improve over time, and I'm sure I can find enough people to flush out issues before the next release (anyone on the Windows Insiders program should be fine).

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    Making the release experimental as part of the next 3.7 update was approved by Ned (over email), so I merged the build.

    I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437. It doesn't seem like a good idea to push such change between a release candidate and a final version :-(

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    New changeset cb0b78a by Victor Stinner in branch 'master':
    Revert "bpo-34977: Add Windows App Store package (GH-10245)" (GH-11019)
    cb0b78a

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    I suggest to revert this change from Python 3.7: it broke all Windows buildbots, see bpo-35437.

    I reverted the change in master and I will merge PR 11021 once tests pass to revert in 3.7 as well.

    It doesn't seem like a good idea to push such change between a release candidate and a final version :-(

    I started a discussion on python-dev:
    https://mail.python.org/pipermail/python-dev/2018-December/155932.html

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    New changeset 783b794 by Victor Stinner in branch '3.7':
    Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)
    783b794

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    Revert "[3.7] bpo-34977: Add Windows App Store package (GH-10245)" (GH-11021)

    That should repair buildbots, so I remove "release blocker" priority.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    We haven't done RC yet - why is pushing this now so bad? Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

    I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    We haven't done RC yet - why is pushing this now so bad?

    I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours:
    https://pythondev.readthedocs.io/ci.html#revert-on-fail

    Most of the change is purely additive and does not modify the code base, the rest is Windows-only or tests.

    nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-) By the way, as I wrote in my email, I would prefer to see the venv+tests change pushed in separated change (and explain why you do these changes).

    I'll add checks for outdated build machines, but there has to be a release of this or it remains untestable, so it'll either be from my branch or from the release branch.

    Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

    Do you mean that buildbots must upgrade to VS 2017? Can Python 3.6 and 3.7 be compiled without any issue on VS 2017? Maybe send an email to buildbot-status if you want to see more buildbots running VS 2017 rather than VS 2015.

    --

    There was a small bug in your PR, in PC/launcher.c, fixed by commit 4c49da0 (bpo-35436):

    + if (executable == NULL) {
    + error(RC_NO_MEMORY, L"A memory allocation failed");
    + }

    @vstinner
    Copy link
    Member

    vstinner commented Dec 7, 2018

    Since the first attempt broke all Windows buildbot workers, it would be nice to use a custom build before merging a new attempt :-) See instructions in the devguide:
    https://devguide.python.org/buildbots/?highlight=custom#custom-builders

    Note: Pablo used buildbot-custom to test a revert of the App Store change.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    > We haven't done RC yet - why is pushing this now so bad?
    I reverted your change because it broke all Windows buildbots, and we have a policy to revert a change if buildbots turn red and no fix can be found in 2 hours

    I don't dispute that policy, but you said it shouldn't go in between RC and a release, which it isn't.

    nitpicking: Your change modify venv and two tests, it's not purely Windows-only :-)

    More nitpicking: the venv changes only affect Windows. Which makes it purely Windows and tests, exactly as I said. Adding smiles doesn't save you from having to read :)

    Sorry, I don't know Windows App Store, I don't understand the restriction "there has to be a release of this or it remains untestable". Would you mind to elaborate?

    Sure. You can start by reading this thread, but the summary is that App Store apps (regardless of platform) generally have to go through an approval process before they can be installed. Without going through this process, they can't be tested fully in context. I believe we should make a true release and put that through the process, rather than me just doing one out of my branch.

    I provided instructions for developer testing above, which covers about 80% of what needs to be checked. The rest can only be tested by real use of a proper release.

    @terryjreedy
    Copy link
    Member

    Various questions and comments:

    1. Steve: For more than one reason, I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org. Can you assure us that the latter are not going away? I would have the same concern about adding something to the Apple/Mac store.

    2. Ned: Since enhancements are not normally backported, it would be nice to have your approval, in whatever form it took, recorded here and not just in private email. I give a reason below.

    3. Installers are always experimental until tested by multiple users on multiple machines. Even then, installation specific issues can arise. As I remember, the current installer, when new in 3.5, needed a shakedown period. One installer-specific issue was the initial working directory. To have a Windows Store Python ready for 3.8, we need to start now.

    There was once an issue where one of the installers failed to include one idlelib file. I believe it was help.html. I since added idle_test/test_help.py, which would now fail with something like "No such file or directory: 'C:\\Programs\\Python37\\Lib\\idlelib\\help.html'".)
    Repository testing does not test the actual binaries that users install, in their production context.

    Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

    1. Anyone: Though it is not essential that I know, I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there. My search only came up with age restrictions, which python does not need (in my opinion).

    2. While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces. They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear). I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    Thanks Terry, great questions.

    I want this to be and remain an added alternative rather than a replacement of the current installer directly available on python.org. Can you assure us that the latter are not going away?

    Absolutely. The current installer is not going anywhere, and there's no point considering the possibility until we've completely dropped support for Windows 10 1803 and earlier (which was only released this year, so dropping support is a long way out yet).

    Installers are always experimental until tested by multiple users on multiple machines. Even then, installation specific issues can arise. ... To have a Windows Store Python ready for 3.8, we need to start now.

    Thanks for the vote of support for this point.

    Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

    Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

    Also, I've been testing IDLE regularly, since one of the advantages of being in the Store is that we can safely put idle[3[.7]].exe on PATH. It definitely works :)

    I've also set up tests that run from an "installed" layout, rather than the source tree, and those are all passing as well (with one of the trivial test fixes in the PR).

    I am curious what 'restricted' versus 'non restricted' mean to Windows Store, and why Python can now go there.

    In general, all app stores disallow their apps from doing things that may break or corrupt the machines they are installed on. Apps are self-contained and can only operate outside of their boundaries through carefully designed permissions/capabilities (which often prompt the user before continuing).

    Recently, Microsoft added support for "full trust" apps, which can bypass many of these restrictions. The aim is to allow traditional applications to swap their MSIs for being installed through the store, which can then ensure trustworthiness of at least the installers. (For example, a Store app is *never* going to install something into System32 or modify operating system files, whether it's full trust or not.) Full trust allows us to do things like launching non-app store processes and creating raw sockets through legacy APIs, so we can very easily move Python into a full trust package rather than having to rewrite it using approved APIs (a project that's been attempted a few times, but always loses low-level functionality that is considered a core part of Python - hence my long-term desire to make the low-level parts optional and have higher-level abstractions be the guaranteed parts).

    While most PRs should be complete in themselves, reasons such as ease of review may suggest splitting one conceptual Pr into pieces. They just have to be reviewed together, or with expectation of a needed followup (and a revert if it does not appear). I have done this to get readable diffs for major refactorings that moved and edited multiple chunks of code in a large file.

    I agree that it's valuable and I've done it internally plenty of times. Unfortunately, no matter how much people repeatedly say that a PR for CPython is part of a chain, there always seems to be pushback from people who ignore that and won't approve a critical prerequisite because there's "no need".

    There's also the point that many of the changes were not known to be needed before the others were made. The pre-squash history is very interleaved between the various changes, and extracting them into separate PRs that totally stand alone is nearly as much work as solving the issues in the first place. I made the call this time to keep them all together and direct reviewers' attention towards the relevant parts. It turns out that didn't attract reviewers, so I'll come up with a new strategy next time I want to get people's attention on a change (maybe Buzzfeed style headlines? :) )

    @pfmoore
    Copy link
    Member

    pfmoore commented Dec 7, 2018

    > Steve, assuming that I can access the Store, should I be able to install both PSF 3.7.2 and Windows Store 3.7.2 simultaneously? So I could test both IDLEs that beginners might install?

    Yes, absolutely. (To be clear, both are PSF approved and under the PSF name/copyright/etc., so I've been calling them "traditional" and "Store" installers when I talk about them.)

    I'd like to expand on this question a little. The first thing I'll
    note is that I'm not aware of any documentation on the Windows Store
    installer, and how it works. It's possible there's a docs patch in
    here somewhere, but to be honest, there are 8 PRs linked to this issue
    and I don't have the time to check them all right now.

    How will the Windows Store install work alongside the PSF one? Will it
    work the same, in that neither will go on PATH unless the user
    explicitly requests it? If I have PSF Python 3.7.1 and Store Python
    3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
    influence it to choose the other one? Same question for 3rd party
    tools like "virtualenv -p 3.7" - I know it's not up to us to deal with
    virtualenv issues, but conversely I do think we have a responsibility
    to not change assumptions that 3rd party tools rely on. In terms of
    PEP-514, will the Store installer use a different tag, or some other
    way of having registry entries independent of the PSF installer?

    Basically, I'm not clear on how having the Store install and the PSF
    install simultaneously present will work - very few tools in my
    experience really support PEP-514 (not even the py launcher) - and
    frankly, if we're going to break tool assumptions that there's only
    one official "3.7" present at any time, we're on very shaky ground
    claiming that PEP-514 warned them if we haven't even updated our own
    supplied tool! (I know that there's lots of ways people can have
    multiple versions right now - PSF plus Anaconda, for example - but
    it's fine to say "you're mixing distributions, you're on your own" in
    that case. If the PSF is publishing two versions of 3.7, which can be
    simultaneously installed, we can't really use that argument.

    Paul

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    The first thing I'll note is that I'm not aware of any documentation
    on the Windows Store installer, and how it works.

    Right now, correct.

    How will the Windows Store install work alongside the PSF one?

    It will go into PATH under the same names as on other platforms (python, python3, python3.7, and same for pip and idle). The Windows Store makes it possible to do this reliably, so it's just on by default (and configurable through OS-provided UI for selecting the app to respond to a certain name).

    If you have already installed it traditionally and enabled the PATH modification, that will take precedence. So you'll get the MSI installed Python. If you've modified PATH by hand, it depends on where you put it.

    If I have PSF Python 3.7.1 and Store Python
    3.7.2 simultaneously installed, which will py.exe -3.7 select? Can I
    influence it to choose the other one? Same question for 3rd party
    tools like "virtualenv -p 3.7"

    Right now, py.exe will not detect it. This is an unavoidable limitation right now (specifically, enumeration of classic app compatibility keys does not work, though direct access does), but it may become avoidable in the future (and being able to say "Python needs it" is actually going to be useful for removing the limitation).

    As usual, our recommended way to influence choice of Python is to specify it. "virtualenv -p 3.7" will find a traditionally installed Python first, but if it's not there and the store one is it *might* find the store one (it depends whether it requests the specific registry key or tries to enumerate all of them - again, I've got people looking into enabling enumeration, but we're talking about adding another feature to Windows specifically for Python, and those take time).

    Basically, I'm not clear on how having the Store install and the PSF
    install simultaneously present will work - very few tools in my
    experience really support PEP-514 (not even the py launcher) - and
    frankly, if we're going to break tool assumptions that there's only
    one official "3.7" present at any time, we're on very shaky ground
    claiming that PEP-514 warned them if we haven't even updated our own
    supplied tool! (I know that there's lots of ways people can have
    multiple versions right now - PSF plus Anaconda, for example - but
    it's fine to say "you're mixing distributions, you're on your own" in
    that case. If the PSF is publishing two versions of 3.7, which can be
    simultaneously installed, we can't really use that argument.

    In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

    A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    How will the Windows Store install work alongside the PSF one?

    Oh, and since I didn't mention it in my direct answer, the Store install goes to a totally different location (per machine, no elevation required), as do packages installed using pip (automatically per user). Nothing is shared between the two installs at all.

    @pfmoore
    Copy link
    Member

    pfmoore commented Dec 7, 2018

    Thanks for the clarification.

    In all the cases I've tested, regular Python install shows up first. If it's not installed, the Store app shows up much of the time. This feels like the right design (with the caveat about "much of the time" requiring changes outside of our direct control), but without actually giving it to people in all their messy configurations there is literally no way to be sure.

    Agreed, it does sound like it's probably the correct approach. But I'd
    strongly recommend that as much of the behaviour as possible gets
    documented as part of this change. There are a *lot* of projects out
    there that write code for Windows based on a relatively superficial
    understanding of all of the complexities that come into play here -
    most registry scanning code is based on guesswork and reverse
    engineering of how things worked before PEP-514 (I should know, I
    wrote rather a lot of it!!!) and plays fast and loose with even things
    like 32-bit/64-bit registry overlaying (it's late and I'm tired, sorry
    for not using the accurate term). Settig projects like virtualenv,
    tox, nox, pew, pipenv etc up for new bug reports associated with store
    installs, without any sort of guidance as to what's going on in the
    official docs, seems unfriendly at best. (I'm thinking of a section in
    "Python Setup and Usage" covering the store installer - that in
    addition to the changes in
    https://docs.python.org/3.7/using/windows.html#installation-steps
    needed to mention that there's now five installers available).

    A regular install does not create "python3" and "python3.7" shortcuts on PATH, so if you use those you'll prefer the store app. But we've never used those, so there shouldn't be any change to existing behaviour. And I've never really been opposed to adding them to the Python installer, I just haven't done the work (and neither has anyone else) - I'd still be opposed to modifying PATH from the traditional installer though, as all of those problems remain.

    The "pythonup" project (https://github.com/uranusjr/pythonup-windows)
    creates versioned copies of python - python3.7.exe. Again, it's a
    non-standard project, so not technically our problem, but I'd be
    interested in how it would interact with the Store python.

    I do appreciate that we need to get the store installer out there in
    order to test things like this, and that initially at least there
    aren't going to be a huge number of people using the store installer
    over the traditional one (and likely vanishingly small numbers of
    people using both in parallel). But conversely, there's a class of
    problems that will be reported to other projects rather than to us,
    and I think we should be trying to support those other projects in
    dealing with this change, not simply leaving it up to them to find out
    about it (I doubt they'd even know to ask "did you use the store
    install or the traditional one" when diagnosing a problem).

    In general, I don't think we've done that great a job of publicising
    the various install options on Windows - we still get pip users trying
    to install packages into the embedded distribution, which they are
    using as if it were a standalone interpreter, and as far as I can tell
    essentially no-one knows about the nuget packages for Python. Let's
    not repeat the mistakes we've made previously with this new
    distribution option.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 7, 2018

    In general, I don't think we've done that great a job of publicising
    the various install options on Windows - we still get pip users trying
    to install packages into the embedded distribution, which they are
    using as if it were a standalone interpreter, and as far as I can tell
    essentially no-one knows about the nuget packages for Python. Let's
    not repeat the mistakes we've made previously with this new
    distribution option.

    Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

    I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?

    So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.

    @pfmoore
    Copy link
    Member

    pfmoore commented Dec 8, 2018

    Yeah, and these are all essentially my fault (though I documented the embeddable package relatively well, modulo a few uses I hadn't imagined). Maybe my holiday writing project can be some better docs for choosing how to install Python, especially since if/when this merges we'll finally have a good answer for "installing from a build without the installer" (using the PC/layout script, which *sigh* also needs a big fat manual).

    Sorry about that - I wasn't trying to make you feel guilty! And yes,
    the embeddable package docs are very good. The problem with that one
    is mostly that people don't *read* the docs (but having them to point
    to makes responding to pip issues much easier, so you have my
    gratitude there :-)

    I feel like right now there's enough here that isn't quite locked down and I don't want to guarantee it. For example, in a future release of Windows we may be able to enable it to write to the actual %APPDATA% folder instead of an app-private one - should we document the current paths and then "break" it in the future? Or should we document that using the "Reset app" features will clean up everything fully now, knowing that if we change the %APPDATA% location later then _that_ will break?

    So for now, I want an unguaranteed store package that can be used but may change over time (and I'll update the text on the page to reflect that). Now is not the time to lock in all aspects of how it works, exactly as you say. But please hold me accountable for adding specific docs for the first 3.8 release - hopefully it's clear by then (if not earlier) what behavior we're happy with.

    In which case, let's have a placeholder note in the docs, saying that
    the Windows Store installer exists, but it's experimental and may have
    unexpected interactions with other tools that expect a standard Python
    install. Users trying the Windows Store installer should be prepared
    for such issues, and we'd appreciate reports so that we can address
    them. Add that to the release announcement as well for extra
    visibility.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 8, 2018

    In which case, let's have a placeholder note in the docs, saying that
    the Windows Store installer exists, but it's experimental and may have
    unexpected interactions with other tools that expect a standard Python
    install. Users trying the Windows Store installer should be prepared
    for such issues, and we'd appreciate reports so that we can address
    them. Add that to the release announcement as well for extra
    visibility.

    Sure, but first I need PR 11029 (and 11030 the backport) approved, so that I can rebase and work on redoing the other half of the change. Victor seems to have disappeared for the weekend, but I made all his requested changes.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2018

    I'm just waiting on Victor to sign off on PR 11029 (or someone else to do it) so I can merge it and rebase PR 11027.

    Hopefully then someone will review it today so that we don't have to hold up 3.7.2rc1 any longer.

    @vstinner
    Copy link
    Member

    Sorry, I don't have the bandwidth to provide a proper review.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2018

    Okay, I'll merge the first part now so I can rebase the main one and *hopefully* someone will be willing to review it now it's smaller. Otherwise, I think we're best to go with buildbots and all the testing I've already done.

    This is my sole job today, so I'll keep an eye on the buildbots. I'll also get the docs updated as Paul suggested for the main PRs.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2018

    New changeset 1c3de54 by Steve Dower in branch 'master':
    bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
    1c3de54

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2018

    New changeset b264c60 by Steve Dower in branch '3.7':
    [3.7] bpo-34977: Use venv redirector instead of original python.exe on Windows (GH-11029)
    b264c60

    @jkloth
    Copy link
    Contributor

    jkloth commented Dec 10, 2018

    See also bpo-35450: venv module doesn't create a copy of python binary by default

    @zooba
    Copy link
    Member Author

    zooba commented Dec 10, 2018

    Paul (and anyone else) - the below link should go directly to just the commit with the docs update. I did a slight rearrangement of the install docs to make the options clearer, and wrote up *just* enough info on nuget to help people use it right (I hope). Similarly for the Store package.

    Any comments appreciated. Still aiming to get this in for 3.7.2rc1 today.

    7d7ddbc

    I've also kicked off a custom buildbot run against the 3.7 version of the change. The version in master ran earlier and was clean. Plus I'm running test builds on the release machine, so it should be ready to go.

    @zooba
    Copy link
    Member Author

    zooba commented Dec 11, 2018

    Thanks, Paul. Appreciate it :)

    Doing a final call before I hit merge in an hour or so. Right now:

    • full release build works
    • full test pass on all CI + custom buildbots work
    • prior test package made it to the store
    • docs clearly show it may change again

    Any last pushback?

    @zooba
    Copy link
    Member Author

    zooba commented Dec 11, 2018

    New changeset 0cd6391 by Steve Dower in branch 'master':
    bpo-34977: Add Windows App Store package (GH-11027)
    0cd6391

    @zooba
    Copy link
    Member Author

    zooba commented Dec 11, 2018

    New changeset 0e4ad88 by Steve Dower in branch '3.7':
    bpo-34977: Add Windows App Store package (GH-11027)
    0e4ad88

    @zooba
    Copy link
    Member Author

    zooba commented Dec 11, 2018

    Luckily, just noticed that I introduced a bug as part of splitting up the original PR into two.

    It wasn't going to have an impact until the package was properly installed from the store, but without the proper directory resolution in there (since the variable I removed is never defined now), in some scenarios it wouldn't be possible to launch Python from a process outside of the app package, such as a virtual environment.

    Testing in a proper app container is difficult, but I'll see if I can come up with a way to do it. It might require a custom buildbot, so I'll see whether there's a way I can do that (historically we've not been able to wire internal work machines - including Azure hosted ones - into other CI systems, but I'll try again).

    @zooba
    Copy link
    Member Author

    zooba commented Dec 11, 2018

    New changeset d5a6a38 by Steve Dower in branch 'master':
    bpo-34977: Remove unused preprocessor definition (GH-11092)
    d5a6a38

    @miss-islington
    Copy link
    Contributor

    New changeset 9bb306d by Miss Islington (bot) in branch '3.7':
    bpo-34977: Remove unused preprocessor definition (GH-11092)
    9bb306d

    @zooba zooba closed this as completed Dec 12, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants