classification
Title: packaging: generate scripts from callable (dotted paths)
Type: enhancement Stage: resolved
Components: Distutils2, Library (Lib) Versions: Python 3.3, 3rd party
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: alexis, ceder, eric.araujo, fdrake, guyrozendorn, higery, mhammond, tarek, tim.golden, vinay.sajip
Priority: normal Keywords: patch

Created on 2011-06-24 09:14 by vinay.sajip, last changed 2014-03-13 03:02 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
6382acfb1685.diff higery, 2011-07-21 12:05
c5692393c621.diff higery, 2011-07-23 14:09
4be1917b2a9e.diff higery, 2011-08-17 15:04
9a7dba6e6f1a.diff eric.araujo, 2012-02-27 12:21 review
Repositories containing patches
https://bitbucket.org/higery/cpython-add-automatic-script-creation-for-packaging#scriptgenerationwork
Messages (58)
msg138896 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 09:14
At present, packaging support for scripts on Windows is the same as for any other system. This is sub-optimal, for the following reasons:

1. Windows doesn't support #! lines to find the correct executable for a script. On a system with multiple Python versions and scripts written for particular versions, support for locating the correct needs to be present.
2. Windows has two types of executables - console applications and GUI applications - and Windows users expect correct usage of either python.exe or pythonw.exe, depending on the individual script being run.

Setuptools (and therefore Distribute) support these requirements by installing a script "demo", on Windows, as "demo.exe" and "demo-script.py" (or "demo-script.pyw"), where demo.exe is a stock Windows executable (either console or GUI) which invokes the appropriate Python executable on the "demo-script.py[w]" file.

Packaging should provide a similar mechanism, which can be implemented very simply by changing the build_scripts command appropriately. It should work like this:

1. When writing a script, the developer simply provides a #!line as normal, but if intended for deployment on Windows, ensures the executable is named as "pythonw" rather than "python". The script should have no extension, as would be for case for a script Linux or OS X.
2. On Windows, the build-scripts command will build the script as it does now - substituting the correct executable for the #! line - but on Windows, instead of writing the script out as e.g. "demo", it will write it as either "demo-script.py" or "demo-script.pyw" (depending on whether the #! line had "pythonw" in it or not), and will also write a stock executable (either console or GUI, depending) with the corresponding name "demo.exe".
3. Since install_scripts just copies files from the build directory, there shouldn't need to be any changes here.

The stock executables can be the same as Distribute uses (setuptools/cli.exe and setuptools/gui.exe), if there is no licensing (or other) issue with having them in Python. If there is such an issue, they can be written from scratch to do the same job (it's just one C file).

I have this working in the pythonv branch, and if this feature request is accepted then I can work up a patch with test and doc changes. (The build_scripts changes are quite straightforward.)
msg138897 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2011-06-24 09:18
Are you aware of PEP 397?

http://www.python.org/dev/peps/pep-0397/
msg138899 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 10:18
@Tim: It had gone under my radar, thanks for the link!

I don't know if/when it will be accepted (i.e. whether before 3.3), so my suggestion could be considered as a fallback alternative which works now. If the PEP 397 launcher is available, then of course we should use that.

Of course PEP 397 does not support 'pythonw' in shebang lines, but the "pythonw" I am suggesting here will not appear in the final output script anyway. It's more of a hint to build_scripts indicating the launcher to use. It seems this choice is still required, since from a quick reading of PEP 397, I couldn't see how the user could just type "demo" for an eponymous script in a virtualenv and get the correct launcher (console or GUI).
msg138900 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2011-06-24 10:33
Adding Mark H as the author of PEP 397
msg138902 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 10:46
My GSoC student will work on integrating the scripts generation from setuptools into packaging.
msg138906 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 10:57
Éric, what will be the scope of that integration? Please bear in mind, I have a working solution, so there's no need to cover this part again unless you think there's a problem with my implementation. 

The changes were straightforward, see

https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf
msg138909 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 11:01
In the setup.cfg files, scripts will now be a mapping of names to callables, like the setuptools scripts and gui_scripts entry points:

scripts =
    sphinx-build = sphinx.build.run

On UNIX, a Python script named sphinx-build will be created, on Windows, a binary sphinx-build.exe will be created.
msg138911 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 11:14
Does that mean that you can't just put an arbitrary Python script in your application? You have to structure it as a callable?

Of course, I see the applicability of it for the entry_points functionality of setuptools.
msg138915 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 11:27
> Does that mean that you can't just put an arbitrary Python script in
> your application?

Arbitrary script files are supported via the resources feature (when the bug you reported is fixed :)  The generation of scripts from callables is a widely-used setuptools feature that will solve a number of problems (Python not on PATH on Windows, unnecessary .py extension on UNIX, such things).
msg138918 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 11:43
Then for Unix at least, how will the installer know which resources need the execute permission turned on? Just by the destination?
msg138922 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 11:53
The copy function used will preserve rights.  IOW, the +x will be needed in the source.
msg138938 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 13:09
Short review of the superseded bugs.

#870479 — Scripts need platform-dependent handling
A request to remove .py on POSIX and create a .cmd file on Windows.  The .cmd part was quickly shot down (see bug thread for problems), and the discussion moved to the setuptools generation.  Interesting comment by a Windows user:
> In principle I don't have a problem with the automatic generation of an
> EXE (I assume it generates a shell script with no extension on Unix?)
> but it should be done in such a way that the EXE is version-independent.
> This is necessary to ensure that pure-python packages, when made into
> bdist_wininst installers, continue to be version-independent. (At the
> moment, distutils generates version-dependent bdist_wininst packages
> *only* for C extensions. Setuptools generates version-dependent
> installers all the time, which is a pain).
> 
> This may mean that a reimplementation is required, rather than copying
> the setuptools code.

The bug links to a thread on distutils-sig, where we can find one more issue by Robert Kern:
> In the generated script, please use "if __name__ == '__main__':" to
> block out the executable bits. Currently, multiprocessing does not
> work on Windows when the application is started by a console_script.
> Because Windows does not have a true fork, multiprocessing will start
> up clean Python subprocesses that import the __main__ of the parent
> process.
(http://mail.python.org/pipermail/distutils-sig/2009-February/010981.html)


#976869 — Stripping script extensions with distutils
A patch by Fred Drake (distutils hacker) to build_scripts to remove the unneeded .py on POSIX.


#1004696 — translate Windows newlines when installing scripts on POSIX
A request that scripts created with CRLF be installed with LF on POSIX, to make the shebang mechanism work.


#4015 — Make installed scripts executable on windows

A patch to add a .bat file for each script (the .bat runs the .py).  Contains the first discussion about a launcher for Windows (PEP 397), and issues with .exe files.


So, Tarek and Fred have expressed support for the setuptools generation in a handful of bug reports and emails, and a number of users report they like it.  IIUC, setuptools supports using python vs. pythonw on Windows (console vs. GUI), but is not flexible enough about which Python version to use, and does not support installing into bin vs. sbin on UNIX (see http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard to learn more about /bin, /sbin, /usr/bin and /usr/sbin).  The cli.exe and gui.exe files look like they can be reused, but the install_scripts code will require more work.

Another interesting thread that was linked from one of the reports: http://mail.python.org/pipermail/distutils-sig/2004-July/004071.html

Finally, when the script generation is implemented and documented, care should be taken to add some doc about old-style script files compatibility.  Projects can have valid reasons to use files for Python scripts, and need a way to install them.  Issues about shebang edition (the distutils behavior in build_scripts) and newlines translation (#1004696) will need thought.
msg138943 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2011-06-24 13:31
People working on this should probably also look at how zc.buildout's zc.recipe.egg handles script generation.  It's similar to setuptools in that "console_script" entry points are used, but it binds in the desired Python executable as well.

(If you ran the build with an unversioned Python executable name, that's what you get, but if you use a versioned path, it's retained.)
msg138944 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 13:59
> Short review of the superseded bugs.
> 
> #870479 — Scripts  need platform-dependent handling
> A request to remove .py on POSIX and create  a .cmd file on Windows.  The .cmd 
>part was quickly shot down (see bug  thread for problems), and the discussion 
>moved to the setuptools  generation.  Interesting comment by a Windows user:
> > In principle I  don't have a problem with the automatic generation of an
> > EXE (I assume  it generates a shell script with no extension on Unix?)
> > but it should be  done in such a way that the EXE is version-independent.
> > This is  necessary to ensure that pure-python packages, when made into
> >  bdist_wininst installers, continue to be version-independent. (At the
> >  moment, distutils generates version-dependent bdist_wininst packages
> >  *only* for C extensions. Setuptools generates version-dependent
> >  installers all the time, which is a pain).

I don't see how it makes sense to aim for version independence, especially since 
2.x and 3.x can each raise SyntaxErrors when presented with the other's code. 
Isn't explicit better than implicit here?

> > This may mean that a  reimplementation is required, rather than copying
> > the setuptools  code.

There's not much sense in making a simplistic copy, that's for sure.

> The bug links to a thread on distutils-sig, where we can find one  more issue 
>by Robert Kern:
> > In the generated script, please use "if  __name__ == '__main__':" to
> > block out the executable bits. Currently,  multiprocessing does not
> > work on Windows when the application is started  by a console_script.
> > Because Windows does not have a true fork,  multiprocessing will start
> > up clean Python subprocesses that import the  __main__ of the parent
> > process.
> (http://mail.python.org/pipermail/distutils-sig/2009-February/010981.html)
> 

FYI I have a "scriptize" script that generates scripts for entry points, example 
is at https://gist.github.com/1044799 which is Windows-friendly (it doesn't 
refer to "xxx-script.py" in usage messages, for example), and may be of 
interest.

> So, Tarek and Fred  have expressed support for the setuptools generation in a 
>handful of bug reports  and emails, and a number of users report they like it.  
>IIUC, setuptools  supports using python vs. pythonw on Windows (console vs. 
>GUI), but is not  flexible enough about which Python version to use, and does 
>not support  installing into bin vs. sbin on UNIX (see 
>http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard to learn more  about 
>/bin, /sbin, /usr/bin and /usr/sbin).  The cli.exe and gui.exe files  look like 
>they can be reused, but the install_scripts code will require more  work.
> 

For  the /bin, /sbin, /usr/bin or /usr/sbin issue, sure - but doesn't it make 
sense to do the other stuff in build_scripts?
msg138945 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-24 14:00
> Fred L. Drake, Jr. <fdrake@acm.org> added the  comment:

> 
> People working on this should probably also look at how  zc.buildout's 
>zc.recipe.egg handles script generation.  It's similar to  setuptools in that 
>"console_script" entry points are used, but it binds in the  desired Python 
>executable as well.
> 
> (If you ran the build with an  unversioned Python executable name, that's what 
>you get, but if you use a  versioned path, it's  retained.)

And also consider what happens when a script is installed into a virtual env, 
when the virtual env's executable *has* to be used.
msg138983 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-24 20:55
[Vinay]
> I don't see how it makes sense to aim for version independence,
> especially since 2.x and 3.x can each raise SyntaxErrors when
> presented with the other's code.
Version independence always means version independence in the same major line of development (i.e. X number in X.Y.Z), or to put it another way, version independence among the versions listed as supported in the Trove classifiers.

> For  the /bin, /sbin, /usr/bin or /usr/sbin issue, sure - but doesn't
> it make sense to do the other stuff in build_scripts?
Of course.  As much as possible should be done by build_scripts, and what’s installation-specific is left to install_scripts.

Fred: Thanks for the pointer!
msg139021 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-25 07:33
> In the setup.cfg files, scripts will now be a mapping of names to
> callables, like the setuptools scripts and gui_scripts entry points:
>
> scripts =
>     sphinx-build = sphinx.build.run

> On UNIX, a Python script named sphinx-build will be created, on
> Windows, a binary sphinx-build.exe will be created.

Perhaps some mechanism needs to be provided to indicate additional processing options per script line:

scripts =
    script-name = path.to.callable opt1 opt2 opt3

One specific practical use of this is to determine whether the Windows version will launch using the text mode launcher or the GUI mode launcher. For example:

scripts =
    sphinx-build = sphinx.build.run # No options => text-mode
    sphinx-gui = sphinx.gui.run win # Option "win" => gui-mode

This seems more future proof than an approach like providing for a separate gui-scripts entry.
msg139022 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-25 07:36
> Version independence always means version independence in the same
> major line of development (i.e. X number in X.Y.Z), or to put it
> another way, version independence among the versions listed as
> supported in the Trove classifiers.

That can still be a problem with e.g. byte literals: 2.5 will raise SyntaxError where 2.6, 2.7 will not.
msg139115 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-25 20:17
> Perhaps some mechanism needs to be provided to indicate additional
> processing options per script line:

Sure.  Higery will have to examine existing usage and think about setuptools→packaging transition.  We’ll start simple and easy (in other words, with something very similar to setuptools) and expand from that.

>> Version independence always means version independence in the same
>> major line of development (i.e. X number in X.Y.Z), or to put it
>> another way, version independence among the versions listed as
>> supported in the Trove classifiers.
> That can still be a problem with e.g. byte literals: 2.5 will raise
> SyntaxError where 2.6, 2.7 will not.

If the code is compatible with 2.6+ only, the “2.5” Trove classifier should not be used.  (Maybe I’m misunderstanding your point.)
msg139122 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-06-25 21:55
> If the code is compatible with 2.6+ only, the “2.5” Trove classifier should not be used.  (Maybe I’m misunderstanding your point.)

I was referring to your comment about the X in X.Y.Z. We also need to bear in mind that you sometimes just get trove classifiers ending in ":: Python :: 2", with no more specific version information. (Of course, there can be a check for this, with perhaps an installation-time warning.)
msg140815 - (view) Author: higery (higery) Date: 2011-07-21 12:33
I hope people can help me test this patch especially on non-Windows platforms.

The main implementation resides in build_scripts.py.

Usage: Just add a 'wrapper-scripts-entries' variable in setup.cfg, which takes a list type as its value. For instance,

wrapper-scripts-entries = ['hello=demo.foo.bar:main']

There is only one entry named 'hello' in current setup.cfg, the 'demo.foo.bar' is the dotted module path, and the 'main' is the main function name which will be called to execute.



Current patch can generte executable script with no extension on POSIX, and .exe file on Windows but have not yet added gui support. 

I also have another problem here looking for help:

In the 'test_command_build_scripts.py', how to test the generated .exe wrapper to see if it can run and generate correct output or not? The only way I can remember is to use os.system(), but if we use os.system() to execute the .exe file, then it will not get the what we want, because it will run in another thread, thus can not get the right sys.path which I have set in my test function.
msg140841 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-22 00:01
Great to hear these news!  I will pull from your clone and test on linux2 as soon as possible.

In your Mercurial configuration file, you should set the git option so that diffs can display editions to binary files.  See http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/

> Usage: Just add a 'wrapper-scripts-entries' variable in setup.cfg,
The new scripts feature should reuse the already existing scripts field, in the files section.

> which takes a list type as its value. For instance,
> wrapper-scripts-entries = ['hello=demo.foo.bar:main']
Config files don’t have lists, but they have multi-line values.  See the packaging/setupcfg doc for more info; here’s an example:

  scripts =
      hello = demo.foo.bar.main

(we use dotted paths throughout packaging, not paths with dots and colons, BTW)

> Current patch can generte executable script with no extension on
> POSIX, and .exe file on Windows but have not yet added gui support.
Looks like a great start!  As I said in private email, have a look at the distribute fork of setuptools to see if they have tests for these features.

> In the 'test_command_build_scripts.py', how to test the generated
> .exe wrapper to see if it can run and generate correct output or not?
> The only way I can remember is to use os.system(), but if we use
> os.system() to execute the .exe file, then it will not get the what
> we want, because it will run in another thread, thus can not get the
> right sys.path which I have set in my test function.
os.system runs a process, not a thread.  You should use the subprocess module, it’s better than os.system.  To pass a custom sys.path, set up a PYTHONPATH environment variable in the env argument of subprocess.Popen.  test_sys and test_site have examples of how to use subprocess.

(Before someone suggests using the helpers in test.support, know that I prefer to use only subprocess, to ease the future backport.)
msg140847 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-22 00:33
Failure on POSIX (linux2):

ERROR: test_install_wrapper_scripts (packaging.tests.test_command_build_scripts.BuildScriptsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/packaging/tests/test_command_build_scripts.py", line 64, in test_install_wrapper_scripts
    cmd.run()
  File "Lib/packaging/command/build_scripts.py", line 64, in run
    self._check_entries()
  File "Lib/packaging/command/build_scripts.py", line 81, in _check_entries
    raise PackagingOptionError("your specific entry '%s' does not exist!" % entry)
packaging.errors.PackagingOptionError: your specific entry 'script1=foo.bar.main1.main' does not exist!

If you can’t easily test on POSIX, I’ll investigate later.
msg140882 - (view) Author: higery (higery) Date: 2011-07-22 14:26
>>packaging.errors.PackagingOptionError: your specific entry 'script1=foo.bar.main1.main' does not exist!

I think you did not get the latest version of my code.
msg140883 - (view) Author: higery (higery) Date: 2011-07-22 14:48
>>In your Mercurial configuration file, you should set the git option so that diffs can display editions to binary files.  See http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/

Thanks.

>>The new scripts feature should reuse the already existing scripts field, in the files section.

But they have different string syntax, old scripts field just uses a
path string as a valid value, eg. 'demo/script.py', while our
wrapper-scripts field uses a kind of '=' separated value, eg.
'hello=demo.script.main'.

>>Looks like a great start!  As I said in private email, have a look at the distribute fork of setuptools to see if they have tests for these features.

OK.

2011/7/22, higery <shoulderhigher@gmail.com>:
>>>packaging.errors.PackagingOptionError: your specific entry
>>> 'script1=foo.bar.main1.main' does not exist!
>
>
> I think you did not get the latest version of my code.
>
msg140888 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-22 15:24
> I think you did not get the latest version of my code.
I pulled and updated again and got a different error :)

ERROR: test_install_wrapper_scripts (packaging.tests.test_command_build_scripts.BuildScriptsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/packaging/tests/test_command_build_scripts.py", line 79, in test_install_wrapper_scripts
    with open(script, "rb") as f:
IOError: [Errno 13] Permission denied: '/tmp/user/1013/tmp2xp9qc/tmpwhzzmg/script1'

(/tmp/user/1013 is my $TMP directory)

>> The new scripts feature should reuse the already existing scripts
>> field, in the files section.
> But they have different string syntax

Yes, you will have to update the config file parsing code in config, code that uses dist.scripts, etc.
msg140925 - (view) Author: higery (higery) Date: 2011-07-23 01:56
>>IOError: [Errno 13] Permission denied: '/tmp/user/1013/tmp2xp9qc/tmpwhzzmg/script1'

I have added an 'ensure_directory()' function to build_script.py, but I'm not sure if it can fix this error.
msg140951 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-23 09:40
Same bug.  I’ve added debug prints to find out the rights (UNIX permission system) of the files, and they’re very strange: --wxrw--wt (the read bit is missing, and the t is strange).  It should be -rwxr-xr-x, like other programs.

The bug is likely this change:
-                os.chmod(target, 0755)
+                os.chmod(target, 755)

An octal literal in 3.x is 0o755.  Decimal 755 means 0o1363, which is not good :)

I’ll let you check the Python docs and Wikipedia if you don’t know these rxw or octal codes.
msg140956 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-23 09:45
I confirm fixing 0o755 makes the tests pass for me.

Your code gives one warning:
build_scripts.py:241: BytesWarning: str() on a bytes instance
  hdr = "#!%(executable)s%(options)s\n" % locals()

The object with the name executable or options is bytes, which you should explicitly convert to a string with decode.  I also don’t like using locals(), but that’s a personal style thing.

Your test should catch stdout (see other packaging tests for how to do that), so that people or buildbots running the tests don’t see “Hello world!”, and so that you can run asserts for the output.
msg140978 - (view) Author: higery (higery) Date: 2011-07-23 12:17
>>An octal literal in 3.x is 0o755.  Decimal 755 means 0o1363, which is not good :)

Thank you for your reminding. The reason I made this mistake is that I'm not familiar with the right way to set permission code in Python3+ .
msg140979 - (view) Author: higery (higery) Date: 2011-07-23 12:18
>>The object with the name executable or options is bytes, which you should explicitly convert to a string with decode.  I also don’t like using locals(), but that’s a personal style thing.

Thanks for your test, I'll amend it.
msg140983 - (view) Author: higery (higery) Date: 2011-07-23 12:45
>>Your test should catch stdout (see other packaging tests for how to do that), so that people or buildbots running the tests don’t see “Hello world!”, and so that you can run asserts for the output.


Thanks. Got it - captured_stdout
msg141047 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-24 15:10
You haven’t set the git option for the diff commands in your config file.
msg141048 - (view) Author: higery (higery) Date: 2011-07-24 15:19
remote repository? It's just a configuration file under the .hg directory...
msg141059 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-24 21:35
higery’s message was this:

> I have already set the option as you said earlier, but how to
> 'push' it to remote repository? It's just a configuration file under
> the .hg directory...

I was mistaken; setting the diff git option is important if you generate patches from your clone, but this does not matter with the automatic generation.  I’ll open a report on the metatracker.
msg141152 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-26 12:38
Your tests contain this:

+        sys.path.append(source)

When using regrtest (see http://docs.python.org/devguide/runtests#running), you’ll get a warning that test_packaging altered sys.path.  Your code should make sure sys.path is restored to its previous test:

+        self.addCleanup(sys.path.remove, source)
         sys.path.append(source)

See the unittest.TestCase doc for more about addCleanup and tearDown.
msg142282 - (view) Author: higery (higery) Date: 2011-08-17 15:22
Current patch has removed old-style scripts support and just retain new-style wrapper scripts generation support. 

Now, it uses only dotted path string to support kind of 'console_scripts' of setuptools, and uses dotted path with a 'window' or 'win' option to support kind of 'gui_scripts' of setuptools.

Here is a simple example to show these usecases, in setup.cfg:

scripts = 
    foo = a.b.c.main
    foowin = a.b.c.winmain -window

Then a executable 'foo' file will be generated for Posix platform, console programm 'foo.exe' and window programm 'foowin.exe' files are generated for Windows platform. The 'window' option is just used to show that this entry is a kind of 'gui_scripts' entry to support gui programm wrappers generation.

Now, there is an issue to consider when we make this change:

'scripts' belongs to the 'files' section in setup.cfg, it's still ok now to place it in 'files'? Still take the above example to say, 'foo=a.b.c.main' and 'foowin=a.b.c.winmain -window' are just dotted path strings to show main executable entry function, they are not existed files and are only just used to generate files.

In addition, there are two kinds of configuration files supported in packaing - setup.py and setup.cfg, and both of them can exist in a project at the same time , and have different purposes for usage. setup.cfg just offers users a way to change the default configuration, so I think we should write the dotted strings in setup.py, do anyone agree with me? Then the above writting way of 'scripts' maybe changed...
msg142302 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-18 04:49
IIUC the support for setup.py is transitional, i.e. "legacy support", for existing packages transitioning from distutils/setuptools/Distribute to packaging. New features should not rely on the existence of setup.py.
msg142303 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-18 04:54
BTW higery, did you use any of the build-scripts functionality I developed in the pythonv branch?

Ref. https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf
msg142322 - (view) Author: higery (higery) Date: 2011-08-18 14:21
>>IIUC the support for setup.py is transitional, i.e. "legacy support", for existing packages transitioning from distutils/setuptools/Distribute to packaging. New features should not rely on the existence of setup.py.

I know, the implementation way of scripts has nothing to do with the setup.cfg or setup.py, these two files are just different ways to offer configuration values. So just don't worry about that.

What the I try to express here is :
There are two kinds of configuration files supported in Packaging, and you can say it maybe a transition consideration from  distutils/setuptools to Packaging, but if you look into the documents of Packaging(you can generate it from the /Doc directory), you will get to know that Packaging has a more further and important consideration - setup.cfg and setup.py place different roles in a project, setup.py offers developers to set while setup.cfg offers users to edit in a cheap and easy way... Certainly you can set anyone of these two files to reach the same goal. Just visit /Doc/build/html/packaging/configfile.html to know more about the different roles of these two configuration files.
msg142323 - (view) Author: higery (higery) Date: 2011-08-18 14:27
>>BTW higery, did you use any of the build-scripts functionality I developed in the pythonv branch?

NO. I removed the 'copy_scripts' function, so I did not use your developed functionality. After this change, Packaging module now just builds new-style scripts and old-style scripts will be built by distutils/setuptools. To support the old-style scripts generated by d*/s* in p*, we can use the resource system.
msg142342 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-18 15:38
There are two kinds of configuration files supported in Packaging, and you can say it maybe a transition consideration from  distutils/setuptools to Packaging, but if you look into the documents of Packaging(you can generate it from the /Doc directory), you will get to know that Packaging has a more further and important consideration - setup.cfg and setup.py place different roles in a project, setup.py offers developers to set while setup.cfg offers users to edit in a cheap and easy way... Certainly you can set anyone of these two files to reach the same goal.
>
>I think I understand that much. The point of setup.cfg is to do away with the completely ad-hoc nature of code that developers can put into setup.py, which prevents playing nicely with distro package managers. I'm fairly sure I've seen Tarek say that "for developers, no more setup.py" - in fact, I've just found where he said it:

http://tarekziade.wordpress.com/2011/05/22/packaging-has-landed-in-the-stdlib/

and also

http://pycon.tv/#/video/57 (at around 6:55 into the video, and at 8:30 - "there is no more setup.py" - meaning in the new way of doing things)

So the role of setup.py is historical, and the way developers customise installations is through using hooks. These work well enough, and I am currently using them in the nemo project which is a companion to the pythonv branch - see http://www.red-dove.com/screencasts/pythonv/pythonv.html
msg142349 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-18 15:56
>________________________________
>NO. I removed the 'copy_scripts' function, so I did not use your developed functionality. After this change, Packaging module now just builds new-style scripts and old-style scripts will be built by distutils/setuptools. To support the old-style scripts generated by d*/s* in p*, we can use the resource system.
>
>Not just as is, I believe. When scripts are installed (as opposed to other resources), it's not enough to copy them across to the configured destination: you also need to change their shebang lines to point to the appropriate Python executable. This is particularly important in virtualenvs where there could be any number of Python executables (of different versions) represented. But how will we know which .py files mentioned in resources are data, and which are actually scripts?

To me, it actually makes more sense to keep those scripts in the [scripts] section, and have some way of recognising which scripts need to be copied/amended and which ones need to be generated from callables.
msg142377 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-18 17:34
> scripts = 
>    foo = a.b.c.main
>    foowin = a.b.c.winmain -window

This is great.  About -window: I don’t think using a fake option style (leading -) is useful, and I’d reuse the setuptools name, “gui”.  I also think this good idea of yours can solve our other feature requests:

unit2 = unittest2.main.main
unit2-tk = unittest2.gui.main window
spamd = spamlib.daemon.main sbin

The first example is a regular script; the second one will use pythonw on Windows and Mac OS X; the third one would install to /usr/sbin instead of /usr/bin on POSIX.  What do you think?

> 'scripts' belongs to the 'files' section in setup.cfg, it's still ok
> now to place it in 'files'?
As I said in private email, I don’t think it’s a concern at all, but maybe other people disagree.  An alternate idea: using a new section:

[scripts]
sphinx-build = sphinx.build.main

You should ask on the fellowship ML.

About setup.cfg and setup.py.  In general, packaging totally ignores setup.py.  However, one goal is to support generated setup.py that take all the info from the setup.cfg file, thanks to pysetup generate-setup.  In that case, there’s a compatibility question, as you said: what do we do with scripts as dotted paths?  It would be too much pain to use inspect.getsource to copy our functions that generate scripts into the setup.py file.  I think our documentation should just advise people to create an old-style script file and manually add it in their setup.py.

>> To support the old-style scripts generated by d*/s* in p*, we can use
>> the resource system.
> Not just as is, I believe. When scripts are installed (as opposed to
> other resources), it's not enough to copy them across to the
> configured destination: you also need to change their shebang lines
> to point to the appropriate Python executable.
That is true.  I see too possible solutions: tell people to use a hook, or have special handling for resources registered in the scripts category (IOW a built-in hook).  I prefer the second option.

> But how will we know which .py files mentioned in resources are data,
> and which are actually scripts?
Distutils scripts are not actually restricted to Python scripts.  The shebang updating only happens if a regex matches.  To answer your question: the resources system uses categories, one of which is “scripts”.

I’m -1 on using heuristics to handle both distutils-style files and packaging-style dotted paths (Vinay’s latest message).  It’s much cleaner to have separate fields or sections.

(BTW Vinay, remember that Roundup creates attachments when someone sends HTML email)
msg142378 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-18 17:36
Heh, I messed up my example:

unit2-tk = unittest2.gui.main gui
msg142400 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-08-18 21:19
Just to clarify: I'm -1 on heuristics too; it's better to have some way of explicitly declaring the intention. I've no problem with e.g. the [scripts] section being used just for generated scripts, as long as there is a clear way of designating arbitrary .py/.pyw files as scripts in the [resources] section, which would have the shebang line transmutation applied during installation.

Re. the unwanted attachments - sorry, I don't normally use HTML email - I need to change the settings I was using.
msg146218 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-23 05:18
FTR, distribute recently committed two fixes for the exe wrappers: https://bitbucket.org/tarek/distribute/issue/238 and https://bitbucket.org/tarek/distribute/issue/207
msg146296 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-24 12:21
FYI: In pythonv, the build_scripts functionality provides identical support for dotted callables to what Éric proposed, while preserving existing functionality for ordinary script files. Thus:

scripts = demo1
  demo2 = amodule.main
  demo3 = apackage.asubpackage.main gui

copies demo1 (adjusting its shebang) and creates demo2 and demo3 from the dotted callables (using the appropriate shebang).

While working on this, I came up against a problem with build_scripts in virtual environments: if I install a project into virtual env 1, then scripts are built with shebangs pointing to that env. If I then install the same project into virtual env 2, then the scripts are not re-created, so they would be installed into virtual env 2 with shebangs referencing virtual env 1. One solution is to set the force flag for virtual environments, but then if you later install into the system Python, then files with wrong shebangs could be installed there. I think the force flag should default to True for build_scripts; the extra script build time would be negligible except in pathological cases. Do you agree?
msg146298 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-24 12:35
Re. the launcher changes, those improvements by Guy Rozendorn are welcome. I noticed some differences from the approach taken by Mark Hammond and Curt Hagenlocher in the PEP 397 implementation, which I ported to C:

The Ctrl-C is ignored by the PEP 397 launcher. AFAICT it's passed to the child process anyway by Windows. The ignoring prevents the default behaviour (premature termination of the launcher).

The PEP 397 launcher also duplicates the standard handles before creating the child process.

The PEP 397 launcher also associates the launcher and the child together using the Job API.
msg146306 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-24 16:26
> FYI: In pythonv, the build_scripts functionality provides identical support for dotted
> callables to what Éric proposed, while preserving existing functionality for ordinary script
> files.

My current preference is to use only new-style generated scripts for the scripts field in the files section, and to let people use the data-files/resources system for old-style file scripts.

We’ll have to think about the shebang munging and decide if we keep it.  I think recommending that people use “/usr/bin/env python” (or python3) and not doing anything to the shebang may be the best thing.

> While working on this, I came up against a problem with build_scripts in virtual environments:
This is also a bug in current distutils and distutils2 scripts.  Scripts should always be regenerated, as they depend on the running Python, not on other files.
msg146311 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-24 17:46
> We’ll have to think about the shebang munging and decide if we keep it.  I think 

> recommending that people use “/usr/bin/env python” (or python3) and not doing 
> anything to the shebang may be the best thing.

What about Windows support? Of course there is PEP 397 which brings shebang-line functionality to Windows, but that PEP has not yet been finalised. Without the existing shebang functionality, IMO there will be problems for Windows users with the “/usr/bin/env python” approach you suggest.
msg146378 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-25 16:08
> What about Windows support?
Just like with distutils: the file extension is used, not the shebang.
msg146382 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-25 16:29
> 

>>  What about Windows support?
> Just like with distutils: the file extension is used, not the shebang.

Please spell out for me how you see this working: I don't see it. Note that scripts have to use the correct Python even if they are invoked using an explicit path pointing into a virtual environment.
msg146383 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-10-25 16:39
To expand on what I said about not seeing how things will work under Windows: are we going to place .exe launchers adjacent to the script, like setuptools does? If the script just has a shebang of "#!/usr/bin/env python", how is the launcher supposed to determine the exact Python to use from that, in a venv scenario where multiple Pythons will be installed? Scripts in virtual envs are supposed to run if invoked, even if the env is not on the PATH.
msg150463 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-02 16:52
I’ll get back to this issue later, but now I just wanted to add a link about building the binary exe files: http://mail.python.org/pipermail/python-dev/2006-April/063846.html (it comes from the time where setuptools was supposed to be added to the stdlib)
msg150502 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-01-03 15:26
@Éric: you may also be interested in a standalone launcher which I wrote for the pythonv branch:

https://bitbucket.org/vinay.sajip/simple_launcher/

This is built using Visual Studio and is not based on setuptools code, but uses the same Windows API for child process creation and synchronisation as the PEP 397 launcher (rather than execv/spawnv as the setuptools launcher does). It also links with the runtime statically rather than linking with msvcrt.dll.
msg154467 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-27 12:13
I merged default yesterday and produced this patch.  I’ll use the review site to make comments.
msg154468 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-27 12:21
Or I won’t, as even a dumb no-git-style diff does not create a review link, maybe because of the binary file change.  *Sigh*  Trying again.
History
Date User Action Args
2014-03-13 03:02:17eric.araujosetstatus: open -> closed
resolution: out of date
stage: needs patch -> resolved
2012-02-27 12:21:59eric.araujosetfiles: - 7099110c8f14.diff
2012-02-27 12:21:44eric.araujosetfiles: - 9a7dba6e6f1a.diff
2012-02-27 12:21:30eric.araujosetfiles: + 9a7dba6e6f1a.diff

messages: + msg154468
2012-02-27 12:13:38eric.araujosetfiles: + 9a7dba6e6f1a.diff

messages: + msg154467
2012-02-19 09:44:05eric.araujolinkissue1173134 superseder
2012-01-03 15:26:36vinay.sajipsetmessages: + msg150502
2012-01-02 16:52:20eric.araujosetmessages: + msg150463
2011-10-25 16:39:38vinay.sajipsetmessages: + msg146383
2011-10-25 16:29:35vinay.sajipsetmessages: + msg146382
2011-10-25 16:08:37eric.araujosetmessages: + msg146378
2011-10-24 17:46:39vinay.sajipsetmessages: + msg146311
2011-10-24 16:26:07eric.araujosetmessages: + msg146306
versions: + 3rd party
2011-10-24 12:35:35vinay.sajipsetnosy: + guyrozendorn
messages: + msg146298
2011-10-24 12:21:50vinay.sajipsetmessages: + msg146296
2011-10-23 05:18:53eric.araujosetmessages: + msg146218
2011-08-22 23:40:19eric.araujosetfiles: + 7099110c8f14.diff
2011-08-18 21:19:33vinay.sajipsetmessages: + msg142400
2011-08-18 17:36:31eric.araujosetmessages: + msg142378
2011-08-18 17:34:14eric.araujosetmessages: + msg142377
2011-08-18 17:19:03eric.araujosetfiles: - unnamed
2011-08-18 17:19:01eric.araujosetfiles: - unnamed
2011-08-18 15:56:52vinay.sajipsetfiles: + unnamed

messages: + msg142349
2011-08-18 15:38:00vinay.sajipsetfiles: + unnamed

messages: + msg142342
2011-08-18 14:27:35higerysetmessages: + msg142323
2011-08-18 14:21:22higerysetmessages: + msg142322
2011-08-18 04:54:28vinay.sajipsetmessages: + msg142303
2011-08-18 04:49:46vinay.sajipsetmessages: + msg142302
2011-08-17 15:22:48higerysetmessages: + msg142282
2011-08-17 15:04:08higerysetfiles: + 4be1917b2a9e.diff
2011-07-26 12:38:01eric.araujosetmessages: + msg141152
2011-07-24 21:35:57eric.araujosetmessages: + msg141059
2011-07-24 21:33:53eric.araujosetfiles: - unnamed
2011-07-24 15:19:08higerysetfiles: + unnamed

messages: + msg141048
2011-07-24 15:10:09eric.araujosetmessages: + msg141047
2011-07-23 14:09:17higerysetfiles: + c5692393c621.diff
2011-07-23 12:45:59higerysetmessages: + msg140983
2011-07-23 12:18:56higerysetmessages: + msg140979
2011-07-23 12:17:02higerysetmessages: + msg140978
2011-07-23 09:45:30eric.araujosetmessages: + msg140956
2011-07-23 09:40:35eric.araujosetmessages: + msg140951
2011-07-23 01:56:21higerysetmessages: + msg140925
2011-07-22 15:24:29eric.araujosetmessages: + msg140888
2011-07-22 14:48:40higerysetmessages: + msg140883
2011-07-22 14:26:40higerysetmessages: + msg140882
2011-07-22 00:33:58eric.araujosetmessages: + msg140847
2011-07-22 00:01:42eric.araujosetmessages: + msg140841
2011-07-21 12:33:42higerysetmessages: + msg140815
2011-07-21 12:05:05higerysetfiles: + 6382acfb1685.diff
keywords: + patch
2011-07-15 16:12:24eric.araujosethgrepos: + hgrepo42
2011-06-25 21:55:10vinay.sajipsetmessages: + msg139122
2011-06-25 20:17:37eric.araujosetmessages: + msg139115
2011-06-25 07:36:35vinay.sajipsetmessages: + msg139022
2011-06-25 07:33:11vinay.sajipsetmessages: + msg139021
2011-06-24 22:00:34cedersetnosy: + ceder
2011-06-24 20:55:46eric.araujosetmessages: + msg138983
2011-06-24 14:00:15vinay.sajipsetmessages: + msg138945
2011-06-24 13:59:03vinay.sajipsetmessages: + msg138944
2011-06-24 13:31:06fdrakesetmessages: + msg138943
2011-06-24 13:09:49eric.araujosetnosy: + fdrake
messages: + msg138938
2011-06-24 12:23:48eric.araujolinkissue976869 superseder
2011-06-24 12:23:09eric.araujolinkissue870479 superseder
2011-06-24 12:20:38eric.araujolinkissue1004696 superseder
2011-06-24 12:18:40eric.araujolinkissue4015 superseder
2011-06-24 12:18:23eric.araujosetassignee: tarek -> eric.araujo

nosy: + higery
stage: needs patch
title: Packaging should provide better support for executable scripts on Windows -> packaging: generate scripts from callable (dotted paths)
2011-06-24 11:53:37eric.araujosetmessages: + msg138922
2011-06-24 11:43:20vinay.sajipsetmessages: + msg138918
2011-06-24 11:27:27eric.araujosetmessages: + msg138915
2011-06-24 11:14:05vinay.sajipsetmessages: + msg138911
2011-06-24 11:01:52eric.araujosetmessages: + msg138909
2011-06-24 10:57:04vinay.sajipsetmessages: + msg138906
2011-06-24 10:46:12eric.araujosetmessages: + msg138902
2011-06-24 10:33:25tim.goldensetnosy: + mhammond
messages: + msg138900
2011-06-24 10:18:09vinay.sajipsetmessages: + msg138899
2011-06-24 09:18:22tim.goldensetnosy: + tim.golden
messages: + msg138897
2011-06-24 09:14:23vinay.sajipcreate