Title: build.bat: register binaries for py launcher
Type: enhancement Stage: test needed
Components: Build, Windows Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, ezio.melotti, paul.moore, steve.dower, terry.reedy, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-05-19 19:46 by terry.reedy, last changed 2017-05-23 16:31 by steve.dower.

Messages (12)
msg293968 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-19 19:46
While working on patches for python, typing path\repository\PCbuild\win32\python-d.exe is an error-prone pain.  I would really like to be able to run locally built binaries with py.    Could pcbuild/build.bat register binaries that it successfully builds?  And then print the -x.y option needed for py?  

#30362 would nicely complement this issue.
msg293970 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-05-19 19:52
I'd be wary of leaving out of date registry entries around. Maybe make it a separate script (or part of build.bat but requiring a command line flag to request it)?

Alternatively (or as well) we could have a cleanup command that removes registry entries for working builds that are no longer wanted. Of course you can handle this by manually editing the registry - but the same could equally be said of the original request...
msg293971 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-05-19 19:57
Hmm, thinking a bit further, are you just suggesting registering the build as -3.7? What if I were to build the 3.6 branch? Would that overwrite my "real" 3.6 registry entries?

As things stand, the launcher can only register two copies of Python for any given minor version (one 32 bit and one 64 bit). So this would only really be useful for trunk (and for builds of versions where you don't already have the release version installed).
msg293982 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-19 22:57
I presume there is already a dead registry entry problem if someone simply deletes the pythonxy directory.  (I know, very bad idea.)  On the other hand, deleting a git repository directory *is* the thing to do.  While listing entries and paths, py could check that the path exists and delete the registry entry if it does not.

I have no idea how easy it would be to expand to more than x.y and x.y-32.  For myself, this is not immediately an issue as I install 64 bit versions and build the default 32 bit debug versions.
msg293985 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-05-19 23:40
Terry: Is python.bat sufficient for your uses?  It's created by PCbuild/python.vcxproj, and always points to whatever interpreter you built last.

If it's not enough, maybe it could be extended to invoke any already-built interpreter?
msg293989 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-05-20 01:36
The launcher could be extended to support additional suffixes in the version string. Currently -32 is supported after the minor version number (e.g. 3.6-32), and (per a recent update) the latest version of the launcher will also support -64, with and without specifying the minor version number. This could be extended to support specifying an [o]fficial / [s]ource and [r]elease / [d]ebug build. A debug build wouldn't be separately registered on the system since it coexists with the release build. Instead, "d" would force the launcher to look for python_d.exe instead of python.exe. A default search would prefer an official build over a source build and a release build over a debug build. The point of supporting "o" and "r" suffixes is to force an error if an official/release build isn't available, rather than defaulting to a source/debug build. 

Under the hood, the INSTALLED_PYTHON struct would be extended with source_build and debug_build boolean values in addition to the bits value (which is an int but may as well be boolean). So there are up to 8 possibilities for every minor version number. MAX_INSTALLED_PYTHONS shouldn't have to be increased from 100. 

For example, `py -3-32d` would look for the highest 3.x 32-bit debug build and prefer an official build over a source build. `py -3-32od` would narrow the search to only official builds. `py -3-32sd` would instead narrow it to only source builds. In the latter case, the registry key would be named like "3.7-32s" to distinguish it from an official install. A build process with admin privileges would register the build in HKLM (taking care to create a "-32" key using KEY_WOW64_32KEY). Otherwise use HKCU.

For convenience, build.bat could set PY_PYTHON after a successful build, to make the current source build the default for the command-line and virtual shebangs, e.g. `set PY_PYTHON=3.7sd` for a registered 64-bit debug build. This would have to be executed after an endlocal statement, since it does most of its work in a setlocal block.
msg293996 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-05-20 03:43
From a couple of tests, the local python.bat should be satisfactory for most of my current uses.  Being part of a #30362 listing would be nice, but I don't often change my set of build binaries.
msg294059 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-05-20 23:52
I am most interested in making py.exe follow PEP 514 and look up tags. Given this, we could register builds as "PythonCore\Dev" and then use "py.exe -dev" to run the most recent build. At worst, we only have one lingering registration (and a positive - it would be available to any tools that follow PEP 514 to locate Python installs).

Alternatively, it could go to "PythonDev\..." and we have another way to look it up.

The default tags used to register Python deliberately match the py.exe launcher (e.g. "3.6" and "3.6-32", though it now also supports "3.6-64" to prevent fallback to the 32-bit interpreter). This should make it *nearly* a non-issue to change the launcher to look up Tags rather than versions. Probably needs a prototype to see just how bad it would be - I haven't had time to do it.
msg294197 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-05-22 22:44
Supporting PEP 514 tags is an interesting idea, but maybe you could go into more detail about what you have in mind. Are you thinking of using the SysVersion, SysArchitecture, and [Windowed]ExecutablePath fields where available instead of parsing version tags and assuming "python[w]/exe"? How would the expansion of candidate installations affect virtual "python" shebangs and command-line arguments such as -3.6-32? If there are several 32-bit 3.6 installations, which one wins?

The PY_PYTHON environment variable could support qualified tags such as "PythonDev/3.6-32". However, arbitrarily named tags can't be supported on the command line without introducing an option reserved for the launcher. It could look for an `X PY_PYTHON` option such as `py -X PY_PYTHON=PythonDev/3.6-32`. That's a bit verbose, but simpler than `cmd /c "set PY_PYTHON=PythonDev/3.6-32 & py`.
msg294204 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-05-22 23:40
My thinking was basically to use an algorithm like this:

    if sys.argv[1].startswith('-'):
        company, _, tag = sys.argv[1][1:].rpartition('\')
        if company:
            # look for Tag under Company
            # look for tag under PythonCore, then other companies

We'd need special handling for Python versions before 3.5 (because of the "-32" change), and possibly for the "-64" suffix as well. But in general, for unique tags you should only have to specify the tag, and the company is used to disambiguate/filter if desired.

SysVersion/SysArchitecture/Version are purely informational according to the PEP (and certainly by intent, if the wording happens to be unclear).
msg294220 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-05-23 05:00
The existing code can be adapted to generalize support for version-number tags, i.e. X.Y[-32|-64]. locate_all_pythons() would need to search every company key to add executable installations that use version-number tags to the installed_pythons array. Whether a tag uses a -32 or -64 suffix is immaterial because the launcher calls GetBinaryType on the executable. A "Company" field would be needed in INSTALLED_PYTHON to allow narrowing a search, such as `py PythonCore\3.6-32`. Also, compare_pythons() would need to order PythonCore ahead of other companies in the installed_pythons array to meet the precedence requirement. 

A new find_python_by_tag(tag, company) function could implement finding non-version tags. It would return the first executable Python installation that exactly matches the tag. find_python_by_tag() would be called in process() in between validate_version() failing and skipping past the command-line options.

(BTW, skipping past the options doesn't work right for options that take an argument such as -X and -W. The launcher sees the optional argument as the script. At best the _wfopen_s call fails, and then it falls back on executing the default Python.)

It's conceivable to generalize maybe_handle_shebang and parse_shebang to support companies in virtual shebangs such as #!/usr/bin/ExampleCorp/python3.6, but that's beyond the intention of the built-in virtual shebangs. They were added to support Unix shebangs in cross-platform scripts. Defining custom commands in py.ini would be a better approach for something like this.

Steve, I gather you don't think it's a problem to use arbitrarily named tags on the command line. The launcher shouldn't look for an "h" tag for `py -h`, so how about using a list of existing command-line options that will never be looked up as unqualified tag names (i.e without a company)? In that case such names would need to be used in -Company\Tag form.
msg294262 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-05-23 16:31
> Steve, I gather you don't think it's a problem to use arbitrarily named tags on the command line. The launcher shouldn't look for an "h" tag for `py -h`

Sure, we can reserve some tags - the -x/y syntax would let you refer to a conflicting one. But other than PythonCore\x.y for (x, y) <= (3, 4), there shouldn't be any need for version or binary checks. Those were needed because we didn't write correct tags, but as of Python 3.5 we write good tags and PEP 514 essentially requires it.
Date User Action Args
2017-05-23 16:31:31steve.dowersetmessages: + msg294262
2017-05-23 05:00:51eryksunsetmessages: + msg294220
2017-05-22 23:40:39steve.dowersetmessages: + msg294204
2017-05-22 22:44:26eryksunsetmessages: + msg294197
2017-05-20 23:52:50steve.dowersetmessages: + msg294059
2017-05-20 03:43:47terry.reedysetmessages: + msg293996
2017-05-20 01:36:05eryksunsettitle: PCbuild.bat: register binaries for py launcher -> build.bat: register binaries for py launcher
nosy: + eryksun

messages: + msg293989

components: + Windows, - Unicode
2017-05-19 23:40:41zach.waresetmessages: + msg293985
2017-05-19 22:57:41terry.reedysetmessages: + msg293982
2017-05-19 19:57:10paul.mooresetmessages: + msg293971
2017-05-19 19:52:33paul.mooresetmessages: + msg293970
2017-05-19 19:46:40terry.reedycreate