At present, packaging support for scripts on Windows is the same as for any other system. This is sub-optimal, for the following reasons:

1. Windows doesn't support #! lines to find the correct executable for a script. On a system with multiple Python versions and scripts written for particular versions, support for locating the correct needs to be present.
2. Windows has two types of executables - console applications and GUI applications - and Windows users expect correct usage of either python.exe or pythonw.exe, depending on the individual script being run.

Setuptools (and therefore Distribute) support these requirements by installing a script "demo", on Windows, as "demo.exe" and "demo-script.py" (or "demo-script.pyw"), where demo.exe is a stock Windows executable (either console or GUI) which invokes the appropriate Python executable on the "demo-script.py[w]" file.

Packaging should provide a similar mechanism, which can be implemented very simply by changing the build_scripts command appropriately. It should work like this:

1. When writing a script, the developer simply provides a #!line as normal, but if intended for deployment on Windows, ensures the executable is named as "pythonw" rather than "python". The script should have no extension, as would be for case for a script Linux or OS X.
2. On Windows, the build-scripts command will build the script as it does now - substituting the correct executable for the #! line - but on Windows, instead of writing the script out as e.g. "demo", it will write it as either "demo-script.py" or "demo-script.pyw" (depending on whether the #! line had "pythonw" in it or not), and will also write a stock executable (either console or GUI, depending) with the corresponding name "demo.exe".
3. Since install_scripts just copies files from the build directory, there shouldn't need to be any changes here.

The stock executables can be the same as Distribute uses (setuptools/cli.exe and setuptools/gui.exe), if there is no licensing (or other) issue with having them in Python. If there is such an issue, they can be written from scratch to do the same job (it's just one C file).

I have this working in the pythonv branch, and if this feature request is accepted then I can work up a patch with test and doc changes. (The build_scripts changes are quite straightforward.)

Are you aware of PEP 397?

http://www.python.org/dev/peps/pep-0397/

@Tim: It had gone under my radar, thanks for the link!

I don't know if/when it will be accepted (i.e. whether before 3.3), so my suggestion could be considered as a fallback alternative which works now. If the PEP 397 launcher is available, then of course we should use that.

Of course PEP 397 does not support 'pythonw' in shebang lines, but the "pythonw" I am suggesting here will not appear in the final output script anyway. It's more of a hint to build_scripts indicating the launcher to use. It seems this choice is still required, since from a quick reading of PEP 397, I couldn't see how the user could just type "demo" for an eponymous script in a virtualenv and get the correct launcher (console or GUI).

Adding Mark H as the author of PEP 397

My GSoC student will work on integrating the scripts generation from setuptools into packaging.

Éric, what will be the scope of that integration? Please bear in mind, I have a working solution, so there's no need to cover this part again unless you think there's a problem with my implementation. 

The changes were straightforward, see

https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf

In the setup.cfg files, scripts will now be a mapping of names to callables, like the setuptools scripts and gui_scripts entry points:

scripts =
    sphinx-build = sphinx.build.run

On UNIX, a Python script named sphinx-build will be created, on Windows, a binary sphinx-build.exe will be created.

Does that mean that you can't just put an arbitrary Python script in your application? You have to structure it as a callable?

Of course, I see the applicability of it for the entry_points functionality of setuptools.

> Does that mean that you can't just put an arbitrary Python script in
> your application?

Arbitrary script files are supported via the resources feature (when the bug you reported is fixed :)  The generation of scripts from callables is a widely-used setuptools feature that will solve a number of problems (Python not on PATH on Windows, unnecessary .py extension on UNIX, such things).

Then for Unix at least, how will the installer know which resources need the execute permission turned on? Just by the destination?

The copy function used will preserve rights.  IOW, the +x will be needed in the source.

Short review of the superseded bugs.

#870479 — Scripts need platform-dependent handling
A request to remove .py on POSIX and create a .cmd file on Windows.  The .cmd part was quickly shot down (see bug thread for problems), and the discussion moved to the setuptools generation.  Interesting comment by a Windows user:
> In principle I don't have a problem with the automatic generation of an
> EXE (I assume it generates a shell script with no extension on Unix?)
> but it should be done in such a way that the EXE is version-independent.
> This is necessary to ensure that pure-python packages, when made into
> bdist_wininst installers, continue to be version-independent. (At the
> moment, distutils generates version-dependent bdist_wininst packages
> *only* for C extensions. Setuptools generates version-dependent
> installers all the time, which is a pain).
> 
> This may mean that a reimplementation is required, rather than copying
> the setuptools code.

The bug links to a thread on distutils-sig, where we can find one more issue by Robert Kern:
> In the generated script, please use "if __name__ == '__main__':" to
> block out the executable bits. Currently, multiprocessing does not
> work on Windows when the application is started by a console_script.
> Because Windows does not have a true fork, multiprocessing will start
> up clean Python subprocesses that import the __main__ of the parent
> process.
(http://mail.python.org/pipermail/distutils-sig/2009-February/010981.html)


#976869 — Stripping script extensions with distutils
A patch by Fred Drake (distutils hacker) to build_scripts to remove the unneeded .py on POSIX.


#1004696 — translate Windows newlines when installing scripts on POSIX
A request that scripts created with CRLF be installed with LF on POSIX, to make the shebang mechanism work.


#4015 — Make installed scripts executable on windows

A patch to add a .bat file for each script (the .bat runs the .py).  Contains the first discussion about a launcher for Windows (PEP 397), and issues with .exe files.


So, Tarek and Fred have expressed support for the setuptools generation in a handful of bug reports and emails, and a number of users report they like it.  IIUC, setuptools supports using python vs. pythonw on Windows (console vs. GUI), but is not flexible enough about which Python version to use, and does not support installing into bin vs. sbin on UNIX (see http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard to learn more about /bin, /sbin, /usr/bin and /usr/sbin).  The cli.exe and gui.exe files look like they can be reused, but the install_scripts code will require more work.

Another interesting thread that was linked from one of the reports: http://mail.python.org/pipermail/distutils-sig/2004-July/004071.html

Finally, when the script generation is implemented and documented, care should be taken to add some doc about old-style script files compatibility.  Projects can have valid reasons to use files for Python scripts, and need a way to install them.  Issues about shebang edition (the distutils behavior in build_scripts) and newlines translation (#1004696) will need thought.

People working on this should probably also look at how zc.buildout's zc.recipe.egg handles script generation.  It's similar to setuptools in that "console_script" entry points are used, but it binds in the desired Python executable as well.

(If you ran the build with an unversioned Python executable name, that's what you get, but if you use a versioned path, it's retained.)

> Short review of the superseded bugs.
> 
> #870479 — Scripts  need platform-dependent handling
> A request to remove .py on POSIX and create  a .cmd file on Windows.  The .cmd 
>part was quickly shot down (see bug  thread for problems), and the discussion 
>moved to the setuptools  generation.  Interesting comment by a Windows user:
> > In principle I  don't have a problem with the automatic generation of an
> > EXE (I assume  it generates a shell script with no extension on Unix?)
> > but it should be  done in such a way that the EXE is version-independent.
> > This is  necessary to ensure that pure-python packages, when made into
> >  bdist_wininst installers, continue to be version-independent. (At the
> >  moment, distutils generates version-dependent bdist_wininst packages
> >  *only* for C extensions. Setuptools generates version-dependent
> >  installers all the time, which is a pain).

I don't see how it makes sense to aim for version independence, especially since 
2.x and 3.x can each raise SyntaxErrors when presented with the other's code. 
Isn't explicit better than implicit here?

> > This may mean that a  reimplementation is required, rather than copying
> > the setuptools  code.

There's not much sense in making a simplistic copy, that's for sure.

> The bug links to a thread on distutils-sig, where we can find one  more issue 
>by Robert Kern:
> > In the generated script, please use "if  __name__ == '__main__':" to
> > block out the executable bits. Currently,  multiprocessing does not
> > work on Windows when the application is started  by a console_script.
> > Because Windows does not have a true fork,  multiprocessing will start
> > up clean Python subprocesses that import the  __main__ of the parent
> > process.
> (http://mail.python.org/pipermail/distutils-sig/2009-February/010981.html)
> 

FYI I have a "scriptize" script that generates scripts for entry points, example 
is at https://gist.github.com/1044799 which is Windows-friendly (it doesn't 
refer to "xxx-script.py" in usage messages, for example), and may be of 
interest.

> So, Tarek and Fred  have expressed support for the setuptools generation in a 
>handful of bug reports  and emails, and a number of users report they like it.  
>IIUC, setuptools  supports using python vs. pythonw on Windows (console vs. 
>GUI), but is not  flexible enough about which Python version to use, and does 
>not support  installing into bin vs. sbin on UNIX (see 
>http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard to learn more  about 
>/bin, /sbin, /usr/bin and /usr/sbin).  The cli.exe and gui.exe files  look like 
>they can be reused, but the install_scripts code will require more  work.
> 

For  the /bin, /sbin, /usr/bin or /usr/sbin issue, sure - but doesn't it make 
sense to do the other stuff in build_scripts?

> Fred L. Drake, Jr. <fdrake@acm.org> added the  comment:

> 
> People working on this should probably also look at how  zc.buildout's 
>zc.recipe.egg handles script generation.  It's similar to  setuptools in that 
>"console_script" entry points are used, but it binds in the  desired Python 
>executable as well.
> 
> (If you ran the build with an  unversioned Python executable name, that's what 
>you get, but if you use a  versioned path, it's  retained.)

And also consider what happens when a script is installed into a virtual env, 
when the virtual env's executable *has* to be used.

[Vinay]
> I don't see how it makes sense to aim for version independence,
> especially since 2.x and 3.x can each raise SyntaxErrors when
> presented with the other's code.
Version independence always means version independence in the same major line of development (i.e. X number in X.Y.Z), or to put it another way, version independence among the versions listed as supported in the Trove classifiers.

> For  the /bin, /sbin, /usr/bin or /usr/sbin issue, sure - but doesn't
> it make sense to do the other stuff in build_scripts?
Of course.  As much as possible should be done by build_scripts, and what’s installation-specific is left to install_scripts.

Fred: Thanks for the pointer!

> In the setup.cfg files, scripts will now be a mapping of names to
> callables, like the setuptools scripts and gui_scripts entry points:
>
> scripts =
>     sphinx-build = sphinx.build.run

> On UNIX, a Python script named sphinx-build will be created, on
> Windows, a binary sphinx-build.exe will be created.

Perhaps some mechanism needs to be provided to indicate additional processing options per script line:

scripts =
    script-name = path.to.callable opt1 opt2 opt3

One specific practical use of this is to determine whether the Windows version will launch using the text mode launcher or the GUI mode launcher. For example:

scripts =
    sphinx-build = sphinx.build.run # No options => text-mode
    sphinx-gui = sphinx.gui.run win # Option "win" => gui-mode

This seems more future proof than an approach like providing for a separate gui-scripts entry.

> Version independence always means version independence in the same
> major line of development (i.e. X number in X.Y.Z), or to put it
> another way, version independence among the versions listed as
> supported in the Trove classifiers.

That can still be a problem with e.g. byte literals: 2.5 will raise SyntaxError where 2.6, 2.7 will not.

> Perhaps some mechanism needs to be provided to indicate additional
> processing options per script line:

Sure.  Higery will have to examine existing usage and think about setuptools→packaging transition.  We’ll start simple and easy (in other words, with something very similar to setuptools) and expand from that.

>> Version independence always means version independence in the same
>> major line of development (i.e. X number in X.Y.Z), or to put it
>> another way, version independence among the versions listed as
>> supported in the Trove classifiers.
> That can still be a problem with e.g. byte literals: 2.5 will raise
> SyntaxError where 2.6, 2.7 will not.

If the code is compatible with 2.6+ only, the “2.5” Trove classifier should not be used.  (Maybe I’m misunderstanding your point.)

> If the code is compatible with 2.6+ only, the “2.5” Trove classifier should not be used.  (Maybe I’m misunderstanding your point.)

I was referring to your comment about the X in X.Y.Z. We also need to bear in mind that you sometimes just get trove classifiers ending in ":: Python :: 2", with no more specific version information. (Of course, there can be a check for this, with perhaps an installation-time warning.)

I hope people can help me test this patch especially on non-Windows platforms.

The main implementation resides in build_scripts.py.

Usage: Just add a 'wrapper-scripts-entries' variable in setup.cfg, which takes a list type as its value. For instance,

wrapper-scripts-entries = ['hello=demo.foo.bar:main']

There is only one entry named 'hello' in current setup.cfg, the 'demo.foo.bar' is the dotted module path, and the 'main' is the main function name which will be called to execute.



Current patch can generte executable script with no extension on POSIX, and .exe file on Windows but have not yet added gui support. 

I also have another problem here looking for help:

In the 'test_command_build_scripts.py', how to test the generated .exe wrapper to see if it can run and generate correct output or not? The only way I can remember is to use os.system(), but if we use os.system() to execute the .exe file, then it will not get the what we want, because it will run in another thread, thus can not get the right sys.path which I have set in my test function.

Great to hear these news!  I will pull from your clone and test on linux2 as soon as possible.

In your Mercurial configuration file, you should set the git option so that diffs can display editions to binary files.  See http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/

> Usage: Just add a 'wrapper-scripts-entries' variable in setup.cfg,
The new scripts feature should reuse the already existing scripts field, in the files section.

> which takes a list type as its value. For instance,
> wrapper-scripts-entries = ['hello=demo.foo.bar:main']
Config files don’t have lists, but they have multi-line values.  See the packaging/setupcfg doc for more info; here’s an example:

  scripts =
      hello = demo.foo.bar.main

(we use dotted paths throughout packaging, not paths with dots and colons, BTW)

> Current patch can generte executable script with no extension on
> POSIX, and .exe file on Windows but have not yet added gui support.
Looks like a great start!  As I said in private email, have a look at the distribute fork of setuptools to see if they have tests for these features.

> In the 'test_command_build_scripts.py', how to test the generated
> .exe wrapper to see if it can run and generate correct output or not?
> The only way I can remember is to use os.system(), but if we use
> os.system() to execute the .exe file, then it will not get the what
> we want, because it will run in another thread, thus can not get the
> right sys.path which I have set in my test function.
os.system runs a process, not a thread.  You should use the subprocess module, it’s better than os.system.  To pass a custom sys.path, set up a PYTHONPATH environment variable in the env argument of subprocess.Popen.  test_sys and test_site have examples of how to use subprocess.

(Before someone suggests using the helpers in test.support, know that I prefer to use only subprocess, to ease the future backport.)

Failure on POSIX (linux2):

ERROR: test_install_wrapper_scripts (packaging.tests.test_command_build_scripts.BuildScriptsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/packaging/tests/test_command_build_scripts.py", line 64, in test_install_wrapper_scripts
    cmd.run()
  File "Lib/packaging/command/build_scripts.py", line 64, in run
    self._check_entries()
  File "Lib/packaging/command/build_scripts.py", line 81, in _check_entries
    raise PackagingOptionError("your specific entry '%s' does not exist!" % entry)
packaging.errors.PackagingOptionError: your specific entry 'script1=foo.bar.main1.main' does not exist!

If you can’t easily test on POSIX, I’ll investigate later.

>>packaging.errors.PackagingOptionError: your specific entry 'script1=foo.bar.main1.main' does not exist!

I think you did not get the latest version of my code.

>>In your Mercurial configuration file, you should set the git option so that diffs can display editions to binary files.  See http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/

Thanks.

>>The new scripts feature should reuse the already existing scripts field, in the files section.

But they have different string syntax, old scripts field just uses a
path string as a valid value, eg. 'demo/script.py', while our
wrapper-scripts field uses a kind of '=' separated value, eg.
'hello=demo.script.main'.

>>Looks like a great start!  As I said in private email, have a look at the distribute fork of setuptools to see if they have tests for these features.

OK.

2011/7/22, higery <shoulderhigher@gmail.com>:
>>>packaging.errors.PackagingOptionError: your specific entry
>>> 'script1=foo.bar.main1.main' does not exist!
>
>
> I think you did not get the latest version of my code.
>

> I think you did not get the latest version of my code.
I pulled and updated again and got a different error :)

ERROR: test_install_wrapper_scripts (packaging.tests.test_command_build_scripts.BuildScriptsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/packaging/tests/test_command_build_scripts.py", line 79, in test_install_wrapper_scripts
    with open(script, "rb") as f:
IOError: [Errno 13] Permission denied: '/tmp/user/1013/tmp2xp9qc/tmpwhzzmg/script1'

(/tmp/user/1013 is my $TMP directory)

>> The new scripts feature should reuse the already existing scripts
>> field, in the files section.
> But they have different string syntax

Yes, you will have to update the config file parsing code in config, code that uses dist.scripts, etc.

>>IOError: [Errno 13] Permission denied: '/tmp/user/1013/tmp2xp9qc/tmpwhzzmg/script1'

I have added an 'ensure_directory()' function to build_script.py, but I'm not sure if it can fix this error.

Same bug.  I’ve added debug prints to find out the rights (UNIX permission system) of the files, and they’re very strange: --wxrw--wt (the read bit is missing, and the t is strange).  It should be -rwxr-xr-x, like other programs.

The bug is likely this change:
-                os.chmod(target, 0755)
+                os.chmod(target, 755)

An octal literal in 3.x is 0o755.  Decimal 755 means 0o1363, which is not good :)

I’ll let you check the Python docs and Wikipedia if you don’t know these rxw or octal codes.

I confirm fixing 0o755 makes the tests pass for me.

Your code gives one warning:
build_scripts.py:241: BytesWarning: str() on a bytes instance
  hdr = "#!%(executable)s%(options)s\n" % locals()

The object with the name executable or options is bytes, which you should explicitly convert to a string with decode.  I also don’t like using locals(), but that’s a personal style thing.

Your test should catch stdout (see other packaging tests for how to do that), so that people or buildbots running the tests don’t see “Hello world!”, and so that you can run asserts for the output.

>>An octal literal in 3.x is 0o755.  Decimal 755 means 0o1363, which is not good :)

Thank you for your reminding. The reason I made this mistake is that I'm not familiar with the right way to set permission code in Python3+ .

>>The object with the name executable or options is bytes, which you should explicitly convert to a string with decode.  I also don’t like using locals(), but that’s a personal style thing.

Thanks for your test, I'll amend it.

>>Your test should catch stdout (see other packaging tests for how to do that), so that people or buildbots running the tests don’t see “Hello world!”, and so that you can run asserts for the output.


Thanks. Got it - captured_stdout

You haven’t set the git option for the diff commands in your config file.

remote repository? It's just a configuration file under the .hg directory...

higery’s message was this:

> I have already set the option as you said earlier, but how to
> 'push' it to remote repository? It's just a configuration file under
> the .hg directory...

I was mistaken; setting the diff git option is important if you generate patches from your clone, but this does not matter with the automatic generation.  I’ll open a report on the metatracker.

Your tests contain this:

+        sys.path.append(source)

When using regrtest (see http://docs.python.org/devguide/runtests#running), you’ll get a warning that test_packaging altered sys.path.  Your code should make sure sys.path is restored to its previous test:

+        self.addCleanup(sys.path.remove, source)
         sys.path.append(source)

See the unittest.TestCase doc for more about addCleanup and tearDown.

Current patch has removed old-style scripts support and just retain new-style wrapper scripts generation support. 

Now, it uses only dotted path string to support kind of 'console_scripts' of setuptools, and uses dotted path with a 'window' or 'win' option to support kind of 'gui_scripts' of setuptools.

Here is a simple example to show these usecases, in setup.cfg:

scripts = 
    foo = a.b.c.main
    foowin = a.b.c.winmain -window

Then a executable 'foo' file will be generated for Posix platform, console programm 'foo.exe' and window programm 'foowin.exe' files are generated for Windows platform. The 'window' option is just used to show that this entry is a kind of 'gui_scripts' entry to support gui programm wrappers generation.

Now, there is an issue to consider when we make this change:

'scripts' belongs to the 'files' section in setup.cfg, it's still ok now to place it in 'files'? Still take the above example to say, 'foo=a.b.c.main' and 'foowin=a.b.c.winmain -window' are just dotted path strings to show main executable entry function, they are not existed files and are only just used to generate files.

In addition, there are two kinds of configuration files supported in packaing - setup.py and setup.cfg, and both of them can exist in a project at the same time , and have different purposes for usage. setup.cfg just offers users a way to change the default configuration, so I think we should write the dotted strings in setup.py, do anyone agree with me? Then the above writting way of 'scripts' maybe changed...

IIUC the support for setup.py is transitional, i.e. "legacy support", for existing packages transitioning from distutils/setuptools/Distribute to packaging. New features should not rely on the existence of setup.py.

BTW higery, did you use any of the build-scripts functionality I developed in the pythonv branch?

Ref. https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf

>>IIUC the support for setup.py is transitional, i.e. "legacy support", for existing packages transitioning from distutils/setuptools/Distribute to packaging. New features should not rely on the existence of setup.py.

I know, the implementation way of scripts has nothing to do with the setup.cfg or setup.py, these two files are just different ways to offer configuration values. So just don't worry about that.

What the I try to express here is :
There are two kinds of configuration files supported in Packaging, and you can say it maybe a transition consideration from  distutils/setuptools to Packaging, but if you look into the documents of Packaging(you can generate it from the /Doc directory), you will get to know that Packaging has a more further and important consideration - setup.cfg and setup.py place different roles in a project, setup.py offers developers to set while setup.cfg offers users to edit in a cheap and easy way... Certainly you can set anyone of these two files to reach the same goal. Just visit /Doc/build/html/packaging/configfile.html to know more about the different roles of these two configuration files.

>>BTW higery, did you use any of the build-scripts functionality I developed in the pythonv branch?

NO. I removed the 'copy_scripts' function, so I did not use your developed functionality. After this change, Packaging module now just builds new-style scripts and old-style scripts will be built by distutils/setuptools. To support the old-style scripts generated by d*/s* in p*, we can use the resource system.

There are two kinds of configuration files supported in Packaging, and you can say it maybe a transition consideration from  distutils/setuptools to Packaging, but if you look into the documents of Packaging(you can generate it from the /Doc directory), you will get to know that Packaging has a more further and important consideration - setup.cfg and setup.py place different roles in a project, setup.py offers developers to set while setup.cfg offers users to edit in a cheap and easy way... Certainly you can set anyone of these two files to reach the same goal.
>
>I think I understand that much. The point of setup.cfg is to do away with the completely ad-hoc nature of code that developers can put into setup.py, which prevents playing nicely with distro package managers. I'm fairly sure I've seen Tarek say that "for developers, no more setup.py" - in fact, I've just found where he said it:

http://tarekziade.wordpress.com/2011/05/22/packaging-has-landed-in-the-stdlib/

and also

http://pycon.tv/#/video/57 (at around 6:55 into the video, and at 8:30 - "there is no more setup.py" - meaning in the new way of doing things)

So the role of setup.py is historical, and the way developers customise installations is through using hooks. These work well enough, and I am currently using them in the nemo project which is a companion to the pythonv branch - see http://www.red-dove.com/screencasts/pythonv/pythonv.html

>________________________________
>NO. I removed the 'copy_scripts' function, so I did not use your developed functionality. After this change, Packaging module now just builds new-style scripts and old-style scripts will be built by distutils/setuptools. To support the old-style scripts generated by d*/s* in p*, we can use the resource system.
>
>Not just as is, I believe. When scripts are installed (as opposed to other resources), it's not enough to copy them across to the configured destination: you also need to change their shebang lines to point to the appropriate Python executable. This is particularly important in virtualenvs where there could be any number of Python executables (of different versions) represented. But how will we know which .py files mentioned in resources are data, and which are actually scripts?

To me, it actually makes more sense to keep those scripts in the [scripts] section, and have some way of recognising which scripts need to be copied/amended and which ones need to be generated from callables.

> scripts = 
>    foo = a.b.c.main
>    foowin = a.b.c.winmain -window

This is great.  About -window: I don’t think using a fake option style (leading -) is useful, and I’d reuse the setuptools name, “gui”.  I also think this good idea of yours can solve our other feature requests:

unit2 = unittest2.main.main
unit2-tk = unittest2.gui.main window
spamd = spamlib.daemon.main sbin

The first example is a regular script; the second one will use pythonw on Windows and Mac OS X; the third one would install to /usr/sbin instead of /usr/bin on POSIX.  What do you think?

> 'scripts' belongs to the 'files' section in setup.cfg, it's still ok
> now to place it in 'files'?
As I said in private email, I don’t think it’s a concern at all, but maybe other people disagree.  An alternate idea: using a new section:

[scripts]
sphinx-build = sphinx.build.main

You should ask on the fellowship ML.

About setup.cfg and setup.py.  In general, packaging totally ignores setup.py.  However, one goal is to support generated setup.py that take all the info from the setup.cfg file, thanks to pysetup generate-setup.  In that case, there’s a compatibility question, as you said: what do we do with scripts as dotted paths?  It would be too much pain to use inspect.getsource to copy our functions that generate scripts into the setup.py file.  I think our documentation should just advise people to create an old-style script file and manually add it in their setup.py.

>> To support the old-style scripts generated by d*/s* in p*, we can use
>> the resource system.
> Not just as is, I believe. When scripts are installed (as opposed to
> other resources), it's not enough to copy them across to the
> configured destination: you also need to change their shebang lines
> to point to the appropriate Python executable.
That is true.  I see too possible solutions: tell people to use a hook, or have special handling for resources registered in the scripts category (IOW a built-in hook).  I prefer the second option.

> But how will we know which .py files mentioned in resources are data,
> and which are actually scripts?
Distutils scripts are not actually restricted to Python scripts.  The shebang updating only happens if a regex matches.  To answer your question: the resources system uses categories, one of which is “scripts”.

I’m -1 on using heuristics to handle both distutils-style files and packaging-style dotted paths (Vinay’s latest message).  It’s much cleaner to have separate fields or sections.

(BTW Vinay, remember that Roundup creates attachments when someone sends HTML email)

Heh, I messed up my example:

unit2-tk = unittest2.gui.main gui

Just to clarify: I'm -1 on heuristics too; it's better to have some way of explicitly declaring the intention. I've no problem with e.g. the [scripts] section being used just for generated scripts, as long as there is a clear way of designating arbitrary .py/.pyw files as scripts in the [resources] section, which would have the shebang line transmutation applied during installation.

Re. the unwanted attachments - sorry, I don't normally use HTML email - I need to change the settings I was using.

FTR, distribute recently committed two fixes for the exe wrappers: https://bitbucket.org/tarek/distribute/issue/238 and https://bitbucket.org/tarek/distribute/issue/207

FYI: In pythonv, the build_scripts functionality provides identical support for dotted callables to what Éric proposed, while preserving existing functionality for ordinary script files. Thus:

scripts = demo1
  demo2 = amodule.main
  demo3 = apackage.asubpackage.main gui

copies demo1 (adjusting its shebang) and creates demo2 and demo3 from the dotted callables (using the appropriate shebang).

While working on this, I came up against a problem with build_scripts in virtual environments: if I install a project into virtual env 1, then scripts are built with shebangs pointing to that env. If I then install the same project into virtual env 2, then the scripts are not re-created, so they would be installed into virtual env 2 with shebangs referencing virtual env 1. One solution is to set the force flag for virtual environments, but then if you later install into the system Python, then files with wrong shebangs could be installed there. I think the force flag should default to True for build_scripts; the extra script build time would be negligible except in pathological cases. Do you agree?

Re. the launcher changes, those improvements by Guy Rozendorn are welcome. I noticed some differences from the approach taken by Mark Hammond and Curt Hagenlocher in the PEP 397 implementation, which I ported to C:

The Ctrl-C is ignored by the PEP 397 launcher. AFAICT it's passed to the child process anyway by Windows. The ignoring prevents the default behaviour (premature termination of the launcher).

The PEP 397 launcher also duplicates the standard handles before creating the child process.

The PEP 397 launcher also associates the launcher and the child together using the Job API.

> FYI: In pythonv, the build_scripts functionality provides identical support for dotted
> callables to what Éric proposed, while preserving existing functionality for ordinary script
> files.

My current preference is to use only new-style generated scripts for the scripts field in the files section, and to let people use the data-files/resources system for old-style file scripts.

We’ll have to think about the shebang munging and decide if we keep it.  I think recommending that people use “/usr/bin/env python” (or python3) and not doing anything to the shebang may be the best thing.

> While working on this, I came up against a problem with build_scripts in virtual environments:
This is also a bug in current distutils and distutils2 scripts.  Scripts should always be regenerated, as they depend on the running Python, not on other files.

> We’ll have to think about the shebang munging and decide if we keep it.  I think 

> recommending that people use “/usr/bin/env python” (or python3) and not doing 
> anything to the shebang may be the best thing.

What about Windows support? Of course there is PEP 397 which brings shebang-line functionality to Windows, but that PEP has not yet been finalised. Without the existing shebang functionality, IMO there will be problems for Windows users with the “/usr/bin/env python” approach you suggest.

> What about Windows support?
Just like with distutils: the file extension is used, not the shebang.

> 

>>  What about Windows support?
> Just like with distutils: the file extension is used, not the shebang.

Please spell out for me how you see this working: I don't see it. Note that scripts have to use the correct Python even if they are invoked using an explicit path pointing into a virtual environment.

To expand on what I said about not seeing how things will work under Windows: are we going to place .exe launchers adjacent to the script, like setuptools does? If the script just has a shebang of "#!/usr/bin/env python", how is the launcher supposed to determine the exact Python to use from that, in a venv scenario where multiple Pythons will be installed? Scripts in virtual envs are supposed to run if invoked, even if the env is not on the PATH.

I’ll get back to this issue later, but now I just wanted to add a link about building the binary exe files: http://mail.python.org/pipermail/python-dev/2006-April/063846.html (it comes from the time where setuptools was supposed to be added to the stdlib)

@Éric: you may also be interested in a standalone launcher which I wrote for the pythonv branch:

https://bitbucket.org/vinay.sajip/simple_launcher/

This is built using Visual Studio and is not based on setuptools code, but uses the same Windows API for child process creation and synchronisation as the PEP 397 launcher (rather than execv/spawnv as the setuptools launcher does). It also links with the runtime statically rather than linking with msvcrt.dll.

I merged default yesterday and produced this patch.  I’ll use the review site to make comments.

Or I won’t, as even a dumb no-git-style diff does not create a review link, maybe because of the binary file change.  *Sigh*  Trying again.