It would be really useful if you could provide a .zip Windows release of Python that is not crippled/for embedding only.

The reason is simply is that right now, I am having constant pain & trouble with it writing an automated build script for Windows (.bat) that works on a Python-free machine.

Specifically the installer has:

1.) no usable functionality in an automated build environment (%PATH% is very unnecessary or even undesirable to set up for a Python install only temporarily downloaded for a build)

2.) possibly conflicts with existing installs (making an automated build break/affect other Python installs/leaving other side effects, which is extremely undesirable and ugly)

3.) can break into a completely unusable state when interrupted (where the install doesn't complete, but running it again also doesn't, needing complicated special logic to nuke it)

NONE of these would be an issue if there was just a .zip. And there is, of course, in form of the embedded install - but that one is useless to me for an automated build because I need pip, and it doesn't support running pip.

For a "tool-style" installation of Python, see the 'PCBuild\find_python.bat' script in the Python source tree, specifically the nuget section.

Ultimately, it is just:

> nuget install pythonx86 -ExcludeVersion -OutputDirectory "some\path"

which will create an installation of the latest python at "some\path\pythonx86".

Is nuget a standard windows utility? What happens if that process is interrupted? What happens if I install it twice to different folders? What happens if I want a specific python version?

Don't get me wrong maybe it can do all these things, but why on earth do I need this? I can even produce this .zip myself, just the regular installer run with /Q and then copy it, but I need to do that for each new release and everyone else who might need it will also need to do that.

Is it really that much work to do this centrally once so it's available for everyone? After all the embeddable install is already available in such a way, it's just that being the stripped down embeddable version its use is limited

On Sat, Feb 16, 2019 at 8:32 AM jt <report@bugs.python.org> wrote:
> Is nuget a standard windows utility?

No, but if your script can download a zip of Python, it could download
the nuget executable.

> What happens if that process is interrupted?

Same as interrupting an unzip operation.

> What happens if I install it twice to different folders?

NuGet installs are basically just an unzip process, so you simply have
2 different folders containing a Python installation.

> What happens if I want a specific python version?

Use '-Version <version-string>' as an option to 'nuget install'.
<version-string> can be any of the versions listed on
https://www.nuget.org/packages/python/.

> Don't get me wrong maybe it can do all these things, but why on earth do I need this? I can even produce this .zip myself, just the regular installer run with /Q and then copy it, but I need to do that for each new release and everyone else who might need it will also need to do that.
>
> Is it really that much work to do this centrally once so it's available for everyone? After all the embeddable install is already available in such a way, it's just that being the stripped down embeddable version its use is limited

It is available to all and is provided by the Windows release manager,
see https://www.nuget.org/packages/python/.

So is that a no for a regular .zip release here? https://www.python.org/downloads/windows/

Don't get me wrong I'm all for options, I just don't get why there is no simple tool-less binary archive for Windows. Why are the only options different variants of installers or additional tools? Why do I need to fetch another tool just to fetch Python?

If this is the final decision then I recommend at least mentioning NuGet prominently on here: https://www.python.org/downloads/windows/ (as an alternative to the so-so regular windows installer, since while that one works well enough for a manual install, for an automated one it's just not that ideal)

> Why do I need to fetch another tool just to fetch Python?

Well, you still need a tool to fetch the zip and you need another tool
to unzip it.  If not, how are you bootstrapping your script for those
uses?

> If this is the final decision then I recommend at least mentioning NuGet prominently on here: https://www.python.org/downloads/windows/

As far as final say goes, I have no control, but it does seem somewhat
reasonable.

> (as an alternative to the so-so regular windows installer, since while that one works well enough for a manual install, for an automated one it's just not that ideal)

So are you just looking for an automated regular installer where the
end result shows up in the Installed Apps list, or a scriptable
build-tool package?

> Well, you still need a tool to fetch the zip and you need another tool to unzip it

Windows 10 actually has everything necessary around, so I don't! That's kind of the beauty of it, the other point being that every dev knows how to deal with a .zip (more than know NuGet, i would assume)

> So are you just looking for an automated regular installer where the
end result shows up in the Installed Apps list, or a scriptable
build-tool package?

Actually I would explicitly NOT want it to show up in Installed Apps, or any other sort of advanced script. Sorry if I haven't been clear on that, I want to avoid any such side effects as much as possible.

What I want is the TARGET FOLDER result of running the regular windows installer. If you haven't noticed, this folder can be copied onto a system where it was never "installed" and with a simple .bat script (or powershell, ...) that sets %PATH% and %PYTHONPATH% you can use it in-place with no further install required.

That's why I'm asking for this, there doesn't seem to be even any need to install Python, and since for an automated build I want to minimize side effects, why would I want anything more than a .zip in that case? I can only see downsides

> So is that a no for a regular .zip release here? https://www.python.org/downloads/windows/

Correct, it's a no.

The main reason is because people use it wrong and get into trouble, and we don't want to support that. (More after the docs link below)

> If this is the final decision then I recommend at least mentioning NuGet prominently on here: https://www.python.org/downloads/windows/

Agreed. It's already documented at https://docs.python.org/3/using/windows.html and perhaps that's the better page to feature prominently on the download page?

Basically, the breakdown of packages we want to provide are:
* easy for "light interactive use" (Windows Store)
* full developer kit (regular installer)
* app-local redistributable (embeddable package)
* zero-impact CI install (nuget package)
* "install" from source (PC/layout script)

(Note the last is currently undocumented outside of the script itself, but I'm planning to write something soon.)

Adding more than this to our support burden is not desirable until we have more [highly responsive] volunteers (not even core committers, we're just at a point where technical support staff would be helpful).

Okay, I just tested switching to the NuGet package and appear to have hit some sort of issue:

C:\myproject\tools\windows\.python-win\python\tools\python.exe: No module named venv
Traceback (most recent call last):
  File "C:\myproject\tools\windows\\windows-build.py", line 129, in <module>
    do_build()
  File "C:\myproject\tools\windows\\windows-build.py", line 60, in do_build
    ".python-win", "spen-venv")
  File "C:\myproject\tools\windows\.python-win\python\tools\Lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "C:\myproject\tools\windows\.python-win\python\tools\Lib\subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\\myproject\\tools\\windows\\.python-win\\python\\tools\\python.exe', '-m', 'venv', 'C:\\myproject\\tools\\windows\\.python-win\\spen-venv']' returned non-zero exit status 1.

Is this expected with the NuGet variant? Can I somehow install 'venv' on top? I didn't have this issue with the install obtained by running the Windows Installer with /Q

The whole point of a venv is to give you a separate directory with a "stand alone" Python. As that's what you just installed with nuget, there's no point in creating a venv - just pip install directly into it.

Alternatively, you can pip install virtualenv and use that.

> The whole point of a venv is to give you a separate directory with a "stand alone" Python. As that's what you just installed with nuget, there's no point in creating a venv

However, shipping a copy of Python with a non-optional element of the stdlib omitted seems at a minimum user-unfriendly, and possibly even outright broken. (We've had similar discussions in the past over Linux distros splitting parts of the stdlib into optional "devel" packages).

Is there a technical reason why the venv module cannot work with the nuget package? If not, then I'd suggest that we simply include it, and don't pre-judge what people might want to do with it. If there *is* a technical problem, let's document the limitation, and note in the venv docs that it may not be present in all installations.

(I'm specifically interested in venv here because there's a possibility that at some point, virtualenv will change to use the core venv functionality under the hood on Python 3+, and if that happens, your suggestion to use virtualenv would no longer work).

There certainly used to be a problem where venv was totally reliant on Python being in the registry to work properly. I'm not sure if that's still the case - possibly with the new redirector in 3.7.2+ it's fixed.

> As that's what you just installed with nuget, there's no point in creating a venv - just pip install directly into it.

This is not a pip install script or anything, it is more complex code that relies on venv. It is not written to run in this build environment only, even if that is where I use it right now - so technically I could make it not use the venv in this case but I'd rather avoid the code complexity. Why should I be required to add that branching?

I made this ticket precisely because the Windows Installer is a hassle, and the embeddable version isn't a properly complete Python. So as far as I'm concerned, I'll add to the vote that it would be very preferable to add venv to NuGet, and potential other missing modules people might expect to be around. Because otherwise this is kind of the embeddable issue all over again...

Sorry I could have added more detailed use cases, so I'll append them here:

The use case for venv in this particular script is that it's written to also work outside of CI, so it doesn't assume it runs in a "clean" python like this. And I would rather have the exact same process run in CI than outside without too much unnecessary differences like avoiding venv use specifically

Another use case for use of venv in CI is the python-for-android project I contribute to, which uses it for cross compilation and similar and for testing inside CI, will also require venv. Currently I don't think it works on Windows due to some hardcoded Unix tool paths, but there are plans to port it and then this would also be an issue

It would be a great help if you could copy the ensurepip and venv directories from a regular install into an extracted Nuget package and let us know whether your scenario works. As you can see from the discussion, there were problems originally that may have been fixed (since Python 3.5). If they are, or if we have some idea of what doesn't work properly now, it'll be much easier to decide what to do.

Ok, so I just tested this by

1. Installing Python 3.7.2 with the regular installer
2. Copying out "Lib\venv" and "Lib\ensurepip" into another folder
3. Uninstalling Python 3.7.2 (there is now no user-available install of Python on this system)
4. Installing Python via NuGet and copying in venv & ensurepip
5. Creating a virtualenv with subprocess.check_output([sys.executable, "-m", "venv", ...path...]) using the NuGet python
6. Installing things with pip inside that venv

That worked fine for me, I installed Cython and some other packages of mine with no issue. Is there anything else I should test?

Okay there is one issue that looks like it might be venv related, which is that upgrading pip itself doesn't seem to work for some reason (while installing other packages in the venv worked fine):

Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\Jonas\\AppData\\Local\\Temp\\pip-uninstall-l4qh2ucu\\myproject\\tools\\windows\\.python-win\\spen-venv\\scripts\\pip.exe'
Consider using the `--user` option or check the permissions.

Traceback (most recent call last):
  File "C:\myproject\tools\windows\\windows-build.py", line 136, in <module>
    do_build()
  File "C:\myproject\tools\windows\\windows-build.py", line 74, in do_build
    os.path.dirname(__file__), ".python-win", "spen-venv"
  File "C:\myproject\tools\windows\.python-win\python\tools\Lib\subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "C:\myproject\tools\windows\.python-win\python\tools\Lib\subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['C:\\myproject\\tools\\windows\\.python-win\\spen-venv\\Scripts\\pip', 'install', '--upgrade', 'pip']' returned non-zero exit status 1.
C:\myproject>

I would expect that to work even without --user so I'm not sure what is going on

for Python 2.7 you can extract the MSI installer (much like a zip file):
mkdir "c:\targetdir"
msiexec /a "c:\python-2.7.8.amd64.msi" /quiet /norestart TARGETDIR=c:\targetdir

msiexec is part of all Windows editions (except for nanoserver). Side note: extracting the MSI does not enable pip. If you need pip, run this on your targetdir:
c:\targetdir\python.exe -m ensurepip

alternatively, you could perform an unattended installation:
msiexec /i "c:\python-2.7.8.amd64.msi" /quiet /norestart 

Unattended installation also works for Python 3.7:
"c:\python-3.7.2-amd64.exe" /quiet InstallAllUsers=1 PrependPath=1

These options worked fine for me in windows containers for CI stuff with servercore. The unattended installation of Python 3.7 also worked fine for me in nanoserver.

On Sun, 17 Feb 2019 at 14:12, jt <report@bugs.python.org> wrote:
> subprocess.CalledProcessError: Command '['C:\\myproject\\tools\\windows\\.python-win\\spen-venv\\Scripts\\pip', 'install', '--upgrade', 'pip']' returned non-zero exit status 1.
> C:\myproject>
>
> I would expect that to work even without --user so I'm not sure what is going on

That's normal behaviour. You can't upgrade pip in-place with the
wrapper, the correct command is

    python -m pip install --upgrade pip

Paul

> alternatively, you could perform an unattended installation:

I know, I already listed the downsides above. (Basically outside of a container it's a bad way to do things because it has too many side effects, and it likes to break when interrupted after which you will need to repair/reinstall, then uninstall again, only then you can do a clean install... it is quite annoying.) It's not that it's not working, it's just not a very good option in some scenarios which is why I made the ticket

> That's normal behaviour. You can't upgrade pip in-place with the wrapper

Oh, right! Thanks, that explains it. In that case I suppose it works perfectly fine? Does that mean there's a chance venv & ensurepip will be added to the NuGet version? Because then I'm definitely switching over to that one for good

Just to add this, I now tested the proper pip upgrade approach, and it worked fine, like everything else I tried before (installing & using Cython, any other packages, ...). If there's anything else I should test, let me know! NuGet with venv/ensurepip would be awesome and make my life easier, so to me this would be a really cool addition

Thank you for your testing of the venv and pip workflows.  At this point, I believe that all that remains would be for Steve Dower to incorporate the `IncludeVEnv=true` parameter into the nupkg build process.

I personally do not see a problem with including them (given your successful test results), but it is Steve's call as Windows' RM.

Yeah, I'm okay with it for 3.7.3 and future releases. The only change should need to be for the nuget preset in PC/layout/support/options.py

It appears that the venv module did not get added to the 3.7.3 rc1 embedded distribution. Was that an oversight, or had I misunderstood what was needed for this to have happened?

> It appears that the venv module did not get added to the 3.7.3 rc1 
> embedded distribution. Was that an oversight, or had I misunderstood 
> what was needed for this to have happened?

I think it was supposed to be added to the nuget package:

https://github.com/python/cpython/blob/v3.7.3rc1/PC/layout/support/options.py#L54

Ah, yes, you're right. I'd misremembered. Thanks.

Yeah, having this in the NuGet version would be amazing!

I have been too specific with the ticket with my narrow mind set on a solution, but NuGet + complete standard library (including venv and pip) is what I now believe I actually should be using.

Not that I would mind an additional .zip only release, but now it's so trivial to automatically spill this out from an automated NuGet install that I really don't mind!

Apparently, this is not in 3.7.3-rc1:

>nuget install python -version 3.7.3-rc1
Feeds used:
  C:\Users\xxxx\.nuget\packages\
  https://api.nuget.org/v3/index.json



Attempting to gather dependency information for package 'python.3.7.3-rc1' with respect to project 'C:\Work\Scratch\tst', targeting 'Any,Version=v0.0'
Gathering dependency information took 2.14 sec
Attempting to resolve dependencies for package 'python.3.7.3-rc1' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'python.3.7.3-rc1'
Resolved actions to install package 'python.3.7.3-rc1'
Retrieving package 'python 3.7.3-rc1' from 'nuget.org'.
  GET https://api.nuget.org/v3-flatcontainer/python/3.7.3-rc1/python.3.7.3-rc1.nupkg
  OK https://api.nuget.org/v3-flatcontainer/python/3.7.3-rc1/python.3.7.3-rc1.nupkg 517ms
Installing python 3.7.3-rc1.
Adding package 'python.3.7.3-rc1' to folder 'C:\Work\Scratch\tst'
Added package 'python.3.7.3-rc1' to folder 'C:\Work\Scratch\tst'
Successfully installed 'python 3.7.3-rc1' to C:\Work\Scratch\tst
Executing nuget actions took 33.92 sec

>.\python.3.7.3-rc1\tools\python.exe -m venv
C:\Work\Scratch\tst\python.3.7.3-rc1\tools\python.exe: No module named venv

Hmm, I see that the Versions tag here is 3.8. Is this only targeted at 3.8, then?

> Is this only targeted at 3.8, then?

Steve said he was okay with adding it in 3.7.3, but it hasn't been added yet in 3.7.3-rc1. In msg337949 I linked to the 3.7.3-rc1 version of options.py, line 54, where "venv" would need to be added to the nuget "options" list.

Yeah, someone still has to send a PR :)

:-) I assume as we're at rc1, it's too late for 3.7.3, but it could be aimed at 3.7.4. I might try to have a look at it if no-one else does.

> :-) I assume as we're at rc1, it's too late for 3.7.3, but it could be aimed at 3.7.4. I might try to have a look at it if no-one else does.

If Ned's okay with cherry-picking the change I have no problems with 
taking it still. It's just a default option change in the layout script, 
and I know the option works. At worst, we'll go from "venv not found" 
errors to "venv is broken" errors.

The referenced PR isn't working - in the nuget package, python.props still says to exclude venv (and indeed venv is still missing). There's a PC/layout/support/props.py file that includes the offending "remove venv" line. Does that need modifying too?

BTW, that file also removes ensurepip, even though options.py includes "pip" in the nuget package. That seems like it might be a bug?

I'm OK with making focused changes to get this working, but is there anywhere that documents the process by which the nuget distribution is generated (in terms of what the steps are and what the various files do?)

Update: I tried removing venv from props.py, but that didn't work. So I tried removing it from python.props as well, but still no joy. Any suggestions?

The change in the PR should be all that's necessary - the props file you are referring to is for end-users to integrate a Python build into their application (where I still assume they don't want venvs, and if they do there's already a property there to include them).

How do you know your current change isn't working?

I ran Tools/nuget/build.bat manually, and the resulting file didn't have lib\venv in it (I opened it in 7-zip, I don't know how to install a local nuget file...).

Could you build the nuget files and check? Or suggest a better way I can check that the change has worked?

> I ran Tools/nuget/build.bat manually, and the resulting file didn't have lib\venv in it (I opened it in 7-zip, I don't know how to install a local nuget file...).

Huh, Tools/nuget/make_pkg.proj doesn't use the preset for some reason. I 
don't remember why not, so maybe add all the options listed in that file 
into the preset as well and then update the command in make_pkg.proj to 
just have "--preset-nuget".

Thanks! That seems to have done it. Update to the PR incoming once my test build completes.

New changeset e724152796a5a41544f52054506c6c2248242a5d by Steve Dower (Paul Moore) in branch 'master':
bpo-36010: Add venv to the nuget distribution (GH-12367)
https://github.com/python/cpython/commit/e724152796a5a41544f52054506c6c2248242a5d

New changeset 3e78c7c30553baf72b7eb6fe3384d88fff549906 by Steve Dower (Miss Islington (bot)) in branch '3.7':
bpo-36010: Add venv to the nuget distribution (GH-12367)
https://github.com/python/cpython/commit/3e78c7c30553baf72b7eb6fe3384d88fff549906

Thank you so much for making it happen <3