classification
Title: Make installed scripts executable on windows
Type: enhancement Stage: resolved
Components: Distutils2 Versions: 3rd party
process
Status: closed Resolution: duplicate
Dependencies: Superseder: packaging: generate scripts from callable (dotted paths)
View: 12394
Assigned To: tarek Nosy List: amaury.forgeotdarc, asvetlov, bebac, eric.araujo, ggenellina, jmfauth, mhammond, phobie, ssbarnea, tarek, techtonik
Priority: normal Keywords: patch

Created on 2008-10-02 12:12 by techtonik, last changed 2011-06-24 12:19 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
create.bat.scripts.on.nt.patch techtonik, 2008-10-02 12:12
executable.scripts.on.nt.patch techtonik, 2009-04-02 10:58 version 2
install_scripts.py techtonik, 2009-04-02 11:13 version 2
hklm_python_extensions.reg phobie, 2010-05-28 13:52 Registry patch to reister new extensions
Messages (24)
msg74159 - (view) Author: anatoly techtonik (techtonik) Date: 2008-10-02 12:12
Distutils contains code to make scripts executable on posix platform.
Here is a patch to for the same feature for nt. It adds .bat file for
every script that doesn't have executable launcher.
msg74286 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-10-04 00:17
As a Windows user, I am not sure I would want this.  A run command
associated with .py makes all .py files executable.  From a command
prompt, which I suspect most Windows users never use, typing 'python' is
not a big deal.  Adding .bat files should at least be optional.
msg74311 - (view) Author: anatoly techtonik (techtonik) Date: 2008-10-04 11:16
1. Associations still do not show Scripts/ among executable files in Run
dialog.

2. Association works only for one version of properly installed Python.
It won't work if Python is installed for different user, if extensions
are not registered or if Python was copied from another machine. 

3. Association will call only the latest Python when you may need to
maintain several installations for your applications. 90% of Python
software is still compiled only for Python 2.5 and when .py association
is set for latest 2.6 users still need to have scripts located in 2.5
dist subdir to be executed with version 2.5

4. Some *nix-developed applications create Python Scripts/ without any
extension at all

Created .bat files guarantee that scripts will be executed with the
version of Python they were installed for. In my opinion this options
should be turned on by default even if made optional to allow users
forget about platform differences.
msg74832 - (view) Author: anatoly techtonik (techtonik) Date: 2008-10-16 11:03
The same issue in "Roundup Tracker" bugtracker
http://sourceforge.net/tracker2/index.php?func=detail&aid=1163804&group_id=31577&atid=402788
msg75198 - (view) Author: Mark Hammond (mhammond) * (Python committer) Date: 2008-10-24 23:35
I can see how this might be useful, but I agree it should not happen by
default, at least until it has been out for a while and feedback is
clear that people do want it by default.

I'd also like to find a way to pass all args, not just the first 9 -
that may well end up a source of bugs for people.  It might also be
worth considering that setuptools has the ability to create a true
executable from a script by using a stub executable.  Wouldn't that be
better still?  In the short term, maybe we should keep this
functionality in setuptools etc and let it filter back to distutils as
it proves itself?
msg83624 - (view) Author: Benny Bach (bebac) Date: 2009-03-15 09:38
I think this should be the default. I am a rookie in python, setup.py in
particular, but I cannot see how you can write portable setup scripts
without this.

I agree that the batch script can be improved. Here is how ruby gems do it:

@ECHO OFF
IF NOT "%~f0" == "~f0" GOTO :WinNT
@"ruby.exe" "C:/ruby/bin/fd" %1 %2 %3 %4 %5 %6 %7 %8 %9
GOTO :EOF
:WinNT
@"ruby.exe" "%~dpn0" %*
msg83698 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-03-17 21:22
I sometimes use this trick on Windows: name the script with a .bat 
extension, and put these lines on top of the file:

@echo off
rem = """
rem run python on this bat file.
rem The -x causes python to skip the first line of the file:
c:\path\to\python -x %~dpnx0 %*
goto :EOF
rem """
msg83747 - (view) Author: Benny Bach (bebac) Date: 2009-03-18 13:50
If you have to name the script with a .bat extension it is not portable
to other platforms or did I misunderstand something?

The point of generating the bat file is to be able to use the same
script on all platforms.
msg83750 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-03-18 14:05
On posix platform, build_scripts already updates the #! line to refer to
the target interpreter, and changes the file mode.
On Windows, it could change the extension as well. Or does it causes
problems?
msg83763 - (view) Author: Benny Bach (bebac) Date: 2009-03-18 17:19
Ok - I see what you mean. I can't see any problems with it. However 
generating a separate bat file has the advantage that you can still 
invoke the original script by calling python explicitly.
msg84516 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2009-03-30 05:58
optional .bat file generating - probably not bad idea.
But I definitely don't want to see this issue as default.

Maybe just tool for generating bat files for desired packages based on 
package metadata for scripts can be solution?
msg84988 - (view) Author: anatoly techtonik (techtonik) Date: 2009-04-01 08:37
The point is not in generating .bat files. The point is to make scripts
executable with exactly the same version of Python the script was
installed. It works well on POSIX, but doesn't work on windows at all.

There is no other way to fix this on windows than generating separate
.exe or .bat launcher. The patch for .bat is ready.

Embedding python code inside of .bat is not a good idea, because runner
script may be complicated, and additional code in header adds probems
with patch submissions and debugging. Not all editors know about magic
-x option. KISS, you know. =)
msg84992 - (view) Author: jmf (jmfauth) Date: 2009-04-01 09:58
It is true, that on Windows the "mime types", .py, .pyw point to a
specific version of Python.

Having Python 2.4, 2.5, 2.6, 3.0, 3.1 installed on my hd and
applications using these (different) versions, I am *very glad* on that
system, all versions, including \site-packages, are isolated from each
other.

Technically, one can argue a Python developer/user should be able to
write a one line batch file if necessary.

Compared to *x systems, I have always found the Windows way as being
very clean and flexible.
msg84996 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-04-01 11:48
> on Windows the "mime types", .py, .pyw point to a 
> specific version of Python.

It could also point to a "python launcher", which reads the first line
of the file and starts the corresponding version of the interpreter.
Visual Studio does this for .sln files. It even displays different icons
depending on the file contents.
msg85001 - (view) Author: Mark Hammond (mhammond) * (Python committer) Date: 2009-04-01 12:01
> It could also point to a "python launcher", which reads the first line

What would that first line look like on Windows? 
o:\src\python-2.6-svn\PCBuild\python.exe would be appropriate for my
machine, but I wouldn't really be happy with installed scripts embedding
this in their first line - if for no better reason than depending on the
Virtual Machine I am using at the time, that exact same directory will
be seen as being on a completely different device, and potentially under
a different 'root' directory too.
msg85017 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-04-01 15:10
I agree.
In any case, double-clicking on a .py file should start an "installed"
interpreter, that is one listed in the registry:
HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\X.Y\InstallPath

Today starting a .py file only open the last installed Python interpreter.
With this proposal, the launcher choose the "best" installed interpreter
for the given script, and falls back to the last installed one.

There may be different definitions of "best". The algorithm could look
like this:
- if the first line is
    #! c:/some/path/to/python.exe
  and the registry contains an installation with 
    InstallPath="c:/some/path/to"
  choose this one.
- if the first line matches
    #! c:/python/([1-9])([0-9])/python.exe
  and if the following registry key exists:
    HKEY_LOCAL_MACHINE/SOFTWARE/Python/PythonCore/\1.\2/InstallPath
  choose this one.
- else, use the last installed interpreter.
msg85021 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2009-04-01 15:35
Maybe also let's look on setuptools solution.It can make windows 
executable for 'entry point scripts'.

Also there are family scripts for single entry point:
* easy_install.exe
* easy_install-2.5.exe
* easy_install-2.5-script.py
* easy_install-script.py

I like it. Exe files executes specific python version. If you are 
installed library in python 2.6 - python 2.6 interpreter will be used, 
not looking in 'default' interpreter etc.

I use setuptools and for me it gives good 'executive python scripts'.  

BTW, double click executes 'default' interpreter, not last installed.
msg85086 - (view) Author: anatoly techtonik (techtonik) Date: 2009-04-01 20:25
The solution with launcher is complex (if not complicated). It will make
scripts unportable - consider using a removable disk with your Python
and application script. The interpreter was not installed on target
system, but with .bat file application is still able to run.

In case there is a problem with registry (backup/restore/profile
migration operation) when recorded version differs from actual installed
file, you will have a great time trying to debug what your script fails
to work, because the version of Python is wrong. I would name this
behaviour implicit.
msg85194 - (view) Author: anatoly techtonik (techtonik) Date: 2009-04-02 11:13
I've updated the script to parse unlimited number of parameters on NT,
to return %errorcode% and to fallback to default Python compiler.

It is based on similar workarounds we've made for SCons in
http://scons.tigris.org/source/browse/scons/branches/core/src/script/scons.bat?view=log
 If you look at the SCons .bat closely you'll notice the difference that
it includes code to launch main application script directly from
site-packages thus removing the requirement to have .py script in Scripts/

In my patch I use Template that has placeholders for product name,
author and email. However, the actual data is not written and hints how
to get it in place are still welcome.
msg91530 - (view) Author: Sorin Sbarnea (ssbarnea) * Date: 2009-08-13 18:37
I totally agree that we must create batch files for commands but not by
including python code inside them.
msg106671 - (view) Author: Per (phobie) Date: 2010-05-28 13:52
On POSIX the interpreter will be read from the first line of a file.
On Windows the interpreter will be read from the Registry HKEY_CLASSES_ROOT\.<file-extension> .

So the correct way to associate a interpreter to a file is to invent a file-extension for every interpreter.
Like /usr/bin/python /usr/bin/python3 and /usr/bin/python3.1 on POSIX, there should be .py .py3 and .py31 on Windows!

I attached a example-registry-patch to register extensions for 2.5, 2.6 and 3.1 .
If you want to use it, you need to adjust the paths!

I propose to change all Python-Windows-installer to install versioned extensions.

If you want a switcher application, it should read the first line of the script and match it against ".*/python(.*)$". So the default POSIX "#!/usr/bin/python3.1" can be kept unchanged. With that rexex the app-path can be read from "HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\<regex-match>\InstallPath\".

BTW.
It would be nice if Python would call itself "Python 3.1" instead of "python" in the "Open with..."-list! The current naming is problematic if you install more than one Python version.
msg106673 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-05-28 13:58
Related to #870479 (should we make that one a meta-bug?)
msg106679 - (view) Author: anatoly techtonik (techtonik) Date: 2010-05-28 17:29
This issue is so old and I do not have time to reread it fully, unfortunately. I believe I wanted to install packages using "easy_install", "pip" or whatever I have and get Scripts/something.bat for  my version of Python. This version is often portable, so registry dependency it is not an option - not a good version at least. Scripts/ can also be located in virtualenv.

The similar patch is used for deploying SCons on Windows.
msg138932 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 12:19
We’re going to follow setuptools’ lead and generate platform-appropriate script or binary files from callables.
History
Date User Action Args
2011-06-24 12:19:46eric.araujosetmessages: + msg138932
2011-06-24 12:18:40eric.araujosetstatus: open -> closed
resolution: duplicate
superseder: packaging: generate scripts from callable (dotted paths)
stage: resolved
2010-11-26 01:36:12eric.araujosetassignee: tarek
nosy: mhammond, amaury.forgeotdarc, ggenellina, techtonik, tarek, eric.araujo, jmfauth, ssbarnea, phobie, bebac, asvetlov
components: + Distutils2, - Distutils
versions: + 3rd party, - Python 3.2
2010-11-12 05:31:10terry.reedysetnosy: - terry.reedy

versions: + Python 3.2, - Python 3.1, Python 2.7
2010-08-02 09:34:55eric.araujolinkissue870479 dependencies
2010-06-06 00:53:39r.david.murraysettitle: [patch] make installed scripts executable on windows -> Make installed scripts executable on windows
2010-05-28 17:29:30techtoniksetmessages: + msg106679
2010-05-28 13:58:11eric.araujosetnosy: + eric.araujo
messages: + msg106673
2010-05-28 13:52:03phobiesetfiles: + hklm_python_extensions.reg
nosy: + phobie
messages: + msg106671

2009-08-14 07:46:24ggenellinasetnosy: + ggenellina
2009-08-13 18:37:13ssbarneasetnosy: + ssbarnea
messages: + msg91530
2009-04-04 08:24:29tareksetpriority: normal
type: enhancement
versions: + Python 3.1, Python 2.7, - Python 2.6
2009-04-02 11:13:23techtoniksetfiles: + install_scripts.py

messages: + msg85194
2009-04-02 10:58:14techtoniksetfiles: + executable.scripts.on.nt.patch
2009-04-01 20:25:44techtoniksetmessages: + msg85086
2009-04-01 15:35:51asvetlovsetmessages: + msg85021
2009-04-01 15:10:22amaury.forgeotdarcsetmessages: + msg85017
2009-04-01 12:01:09mhammondsetmessages: + msg85001
2009-04-01 11:48:05amaury.forgeotdarcsetmessages: + msg84996
2009-04-01 09:58:05jmfauthsetnosy: + jmfauth
messages: + msg84992
2009-04-01 08:37:19techtoniksetmessages: + msg84988
2009-03-30 05:58:52asvetlovsetnosy: + asvetlov
messages: + msg84516
2009-03-18 17:19:06bebacsetmessages: + msg83763
2009-03-18 14:05:58amaury.forgeotdarcsetmessages: + msg83750
2009-03-18 13:50:32bebacsetmessages: + msg83747
2009-03-17 21:22:06amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg83698
2009-03-15 09:38:42bebacsetnosy: + bebac
messages: + msg83624
2009-02-06 01:30:28tareksetnosy: + tarek
2008-10-24 23:35:35mhammondsetnosy: + mhammond
messages: + msg75198
2008-10-16 11:03:51techtoniksetmessages: + msg74832
2008-10-04 11:16:27techtoniksetmessages: + msg74311
2008-10-04 00:17:30terry.reedysetnosy: + terry.reedy
messages: + msg74286
2008-10-02 12:12:04techtonikcreate