New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
packaging: generate scripts from callable (dotted paths) #56603
Comments
At present, packaging support for scripts on Windows is the same as for any other system. This is sub-optimal, for the following reasons:
Setuptools (and therefore Distribute) support these requirements by installing a script "demo", on Windows, as "demo.exe" and "demo-script.py" (or "demo-script.pyw"), where demo.exe is a stock Windows executable (either console or GUI) which invokes the appropriate Python executable on the "demo-script.py[w]" file. Packaging should provide a similar mechanism, which can be implemented very simply by changing the build_scripts command appropriately. It should work like this:
The stock executables can be the same as Distribute uses (setuptools/cli.exe and setuptools/gui.exe), if there is no licensing (or other) issue with having them in Python. If there is such an issue, they can be written from scratch to do the same job (it's just one C file). I have this working in the pythonv branch, and if this feature request is accepted then I can work up a patch with test and doc changes. (The build_scripts changes are quite straightforward.) |
Are you aware of PEP-397? |
@tim: It had gone under my radar, thanks for the link! I don't know if/when it will be accepted (i.e. whether before 3.3), so my suggestion could be considered as a fallback alternative which works now. If the PEP-397 launcher is available, then of course we should use that. Of course PEP-397 does not support 'pythonw' in shebang lines, but the "pythonw" I am suggesting here will not appear in the final output script anyway. It's more of a hint to build_scripts indicating the launcher to use. It seems this choice is still required, since from a quick reading of PEP-397, I couldn't see how the user could just type "demo" for an eponymous script in a virtualenv and get the correct launcher (console or GUI). |
Adding Mark H as the author of PEP-397 |
My GSoC student will work on integrating the scripts generation from setuptools into packaging. |
Éric, what will be the scope of that integration? Please bear in mind, I have a working solution, so there's no need to cover this part again unless you think there's a problem with my implementation. The changes were straightforward, see https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf |
In the setup.cfg files, scripts will now be a mapping of names to callables, like the setuptools scripts and gui_scripts entry points: scripts =
sphinx-build = sphinx.build.run On UNIX, a Python script named sphinx-build will be created, on Windows, a binary sphinx-build.exe will be created. |
Does that mean that you can't just put an arbitrary Python script in your application? You have to structure it as a callable? Of course, I see the applicability of it for the entry_points functionality of setuptools. |
Arbitrary script files are supported via the resources feature (when the bug you reported is fixed :) The generation of scripts from callables is a widely-used setuptools feature that will solve a number of problems (Python not on PATH on Windows, unnecessary .py extension on UNIX, such things). |
Then for Unix at least, how will the installer know which resources need the execute permission turned on? Just by the destination? |
The copy function used will preserve rights. IOW, the +x will be needed in the source. |
Short review of the superseded bugs. bpo-870479 — Scripts need platform-dependent handling
The bug links to a thread on distutils-sig, where we can find one more issue by Robert Kern:
bpo-976869 — Stripping script extensions with distutils bpo-1004696 — translate Windows newlines when installing scripts on POSIX bpo-4015 — Make installed scripts executable on windows A patch to add a .bat file for each script (the .bat runs the .py). Contains the first discussion about a launcher for Windows (PEP-397), and issues with .exe files. So, Tarek and Fred have expressed support for the setuptools generation in a handful of bug reports and emails, and a number of users report they like it. IIUC, setuptools supports using python vs. pythonw on Windows (console vs. GUI), but is not flexible enough about which Python version to use, and does not support installing into bin vs. sbin on UNIX (see http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard to learn more about /bin, /sbin, /usr/bin and /usr/sbin). The cli.exe and gui.exe files look like they can be reused, but the install_scripts code will require more work. Another interesting thread that was linked from one of the reports: http://mail.python.org/pipermail/distutils-sig/2004-July/004071.html Finally, when the script generation is implemented and documented, care should be taken to add some doc about old-style script files compatibility. Projects can have valid reasons to use files for Python scripts, and need a way to install them. Issues about shebang edition (the distutils behavior in build_scripts) and newlines translation (bpo-1004696) will need thought. |
People working on this should probably also look at how zc.buildout's zc.recipe.egg handles script generation. It's similar to setuptools in that "console_script" entry points are used, but it binds in the desired Python executable as well. (If you ran the build with an unversioned Python executable name, that's what you get, but if you use a versioned path, it's retained.) |
I don't see how it makes sense to aim for version independence, especially since
There's not much sense in making a simplistic copy, that's for sure.
FYI I have a "scriptize" script that generates scripts for entry points, example
For the /bin, /sbin, /usr/bin or /usr/sbin issue, sure - but doesn't it make |
And also consider what happens when a script is installed into a virtual env, |
[Vinay]
Fred: Thanks for the pointer! |
Perhaps some mechanism needs to be provided to indicate additional processing options per script line: scripts =
script-name = path.to.callable opt1 opt2 opt3 One specific practical use of this is to determine whether the Windows version will launch using the text mode launcher or the GUI mode launcher. For example: scripts =
sphinx-build = sphinx.build.run # No options => text-mode
sphinx-gui = sphinx.gui.run win # Option "win" => gui-mode This seems more future proof than an approach like providing for a separate gui-scripts entry. |
That can still be a problem with e.g. byte literals: 2.5 will raise SyntaxError where 2.6, 2.7 will not. |
Sure. Higery will have to examine existing usage and think about setuptools→packaging transition. We’ll start simple and easy (in other words, with something very similar to setuptools) and expand from that.
If the code is compatible with 2.6+ only, the “2.5” Trove classifier should not be used. (Maybe I’m misunderstanding your point.) |
I was referring to your comment about the X in X.Y.Z. We also need to bear in mind that you sometimes just get trove classifiers ending in ":: Python :: 2", with no more specific version information. (Of course, there can be a check for this, with perhaps an installation-time warning.) |
I hope people can help me test this patch especially on non-Windows platforms. The main implementation resides in build_scripts.py. Usage: Just add a 'wrapper-scripts-entries' variable in setup.cfg, which takes a list type as its value. For instance, wrapper-scripts-entries = ['hello=demo.foo.bar:main'] There is only one entry named 'hello' in current setup.cfg, the 'demo.foo.bar' is the dotted module path, and the 'main' is the main function name which will be called to execute. Current patch can generte executable script with no extension on POSIX, and .exe file on Windows but have not yet added gui support. I also have another problem here looking for help: In the 'test_command_build_scripts.py', how to test the generated .exe wrapper to see if it can run and generate correct output or not? The only way I can remember is to use os.system(), but if we use os.system() to execute the .exe file, then it will not get the what we want, because it will run in another thread, thus can not get the right sys.path which I have set in my test function. |
Great to hear these news! I will pull from your clone and test on linux2 as soon as possible. In your Mercurial configuration file, you should set the git option so that diffs can display editions to binary files. See http://hgtip.com/tips/beginner/2009-10-22-always-use-git-diffs/
scripts =
hello = demo.foo.bar.main (we use dotted paths throughout packaging, not paths with dots and colons, BTW)
(Before someone suggests using the helpers in test.support, know that I prefer to use only subprocess, to ease the future backport.) |
Failure on POSIX (linux2): ERROR: test_install_wrapper_scripts (packaging.tests.test_command_build_scripts.BuildScriptsTestCase) Traceback (most recent call last):
File "Lib/packaging/tests/test_command_build_scripts.py", line 64, in test_install_wrapper_scripts
cmd.run()
File "Lib/packaging/command/build_scripts.py", line 64, in run
self._check_entries()
File "Lib/packaging/command/build_scripts.py", line 81, in _check_entries
raise PackagingOptionError("your specific entry '%s' does not exist!" % entry)
packaging.errors.PackagingOptionError: your specific entry 'script1=foo.bar.main1.main' does not exist! If you can’t easily test on POSIX, I’ll investigate later. |
I think you did not get the latest version of my code. |
Thanks for your test, I'll amend it. |
Thanks. Got it - captured_stdout |
You haven’t set the git option for the diff commands in your config file. |
remote repository? It's just a configuration file under the .hg directory... |
higery’s message was this:
I was mistaken; setting the diff git option is important if you generate patches from your clone, but this does not matter with the automatic generation. I’ll open a report on the metatracker. |
Your tests contain this: + sys.path.append(source) When using regrtest (see http://docs.python.org/devguide/runtests#running), you’ll get a warning that test_packaging altered sys.path. Your code should make sure sys.path is restored to its previous test: + self.addCleanup(sys.path.remove, source) See the unittest.TestCase doc for more about addCleanup and tearDown. |
Current patch has removed old-style scripts support and just retain new-style wrapper scripts generation support. Now, it uses only dotted path string to support kind of 'console_scripts' of setuptools, and uses dotted path with a 'window' or 'win' option to support kind of 'gui_scripts' of setuptools. Here is a simple example to show these usecases, in setup.cfg: scripts =
foo = a.b.c.main
foowin = a.b.c.winmain -window Then a executable 'foo' file will be generated for Posix platform, console programm 'foo.exe' and window programm 'foowin.exe' files are generated for Windows platform. The 'window' option is just used to show that this entry is a kind of 'gui_scripts' entry to support gui programm wrappers generation. Now, there is an issue to consider when we make this change: 'scripts' belongs to the 'files' section in setup.cfg, it's still ok now to place it in 'files'? Still take the above example to say, 'foo=a.b.c.main' and 'foowin=a.b.c.winmain -window' are just dotted path strings to show main executable entry function, they are not existed files and are only just used to generate files. In addition, there are two kinds of configuration files supported in packaing - setup.py and setup.cfg, and both of them can exist in a project at the same time , and have different purposes for usage. setup.cfg just offers users a way to change the default configuration, so I think we should write the dotted strings in setup.py, do anyone agree with me? Then the above writting way of 'scripts' maybe changed... |
IIUC the support for setup.py is transitional, i.e. "legacy support", for existing packages transitioning from distutils/setuptools/Distribute to packaging. New features should not rely on the existence of setup.py. |
BTW higery, did you use any of the build-scripts functionality I developed in the pythonv branch? Ref. https://bitbucket.org/vinay.sajip/pythonv/changeset/d2453f281baf |
I know, the implementation way of scripts has nothing to do with the setup.cfg or setup.py, these two files are just different ways to offer configuration values. So just don't worry about that. What the I try to express here is : |
NO. I removed the 'copy_scripts' function, so I did not use your developed functionality. After this change, Packaging module now just builds new-style scripts and old-style scripts will be built by distutils/setuptools. To support the old-style scripts generated by d*/s* in p*, we can use the resource system. |
There are two kinds of configuration files supported in Packaging, and you can say it maybe a transition consideration from distutils/setuptools to Packaging, but if you look into the documents of Packaging(you can generate it from the /Doc directory), you will get to know that Packaging has a more further and important consideration - setup.cfg and setup.py place different roles in a project, setup.py offers developers to set while setup.cfg offers users to edit in a cheap and easy way... Certainly you can set anyone of these two files to reach the same goal.
http://tarekziade.wordpress.com/2011/05/22/packaging-has-landed-in-the-stdlib/ and also http://pycon.tv/#/video/57 (at around 6:55 into the video, and at 8:30 - "there is no more setup.py" - meaning in the new way of doing things) So the role of setup.py is historical, and the way developers customise installations is through using hooks. These work well enough, and I am currently using them in the nemo project which is a companion to the pythonv branch - see http://www.red-dove.com/screencasts/pythonv/pythonv.html |
To me, it actually makes more sense to keep those scripts in the [scripts] section, and have some way of recognising which scripts need to be copied/amended and which ones need to be generated from callables. |
This is great. About -window: I don’t think using a fake option style (leading -) is useful, and I’d reuse the setuptools name, “gui”. I also think this good idea of yours can solve our other feature requests: unit2 = unittest2.main.main
unit2-tk = unittest2.gui.main window
spamd = spamlib.daemon.main sbin The first example is a regular script; the second one will use pythonw on Windows and Mac OS X; the third one would install to /usr/sbin instead of /usr/bin on POSIX. What do you think?
[scripts] You should ask on the fellowship ML. About setup.cfg and setup.py. In general, packaging totally ignores setup.py. However, one goal is to support generated setup.py that take all the info from the setup.cfg file, thanks to pysetup generate-setup. In that case, there’s a compatibility question, as you said: what do we do with scripts as dotted paths? It would be too much pain to use inspect.getsource to copy our functions that generate scripts into the setup.py file. I think our documentation should just advise people to create an old-style script file and manually add it in their setup.py.
I’m -1 on using heuristics to handle both distutils-style files and packaging-style dotted paths (Vinay’s latest message). It’s much cleaner to have separate fields or sections. (BTW Vinay, remember that Roundup creates attachments when someone sends HTML email) |
Heh, I messed up my example: unit2-tk = unittest2.gui.main gui |
Just to clarify: I'm -1 on heuristics too; it's better to have some way of explicitly declaring the intention. I've no problem with e.g. the [scripts] section being used just for generated scripts, as long as there is a clear way of designating arbitrary .py/.pyw files as scripts in the [resources] section, which would have the shebang line transmutation applied during installation. Re. the unwanted attachments - sorry, I don't normally use HTML email - I need to change the settings I was using. |
FTR, distribute recently committed two fixes for the exe wrappers: https://bitbucket.org/tarek/distribute/issue/238 and https://bitbucket.org/tarek/distribute/issue/207 |
FYI: In pythonv, the build_scripts functionality provides identical support for dotted callables to what Éric proposed, while preserving existing functionality for ordinary script files. Thus: scripts = demo1
demo2 = amodule.main
demo3 = apackage.asubpackage.main gui copies demo1 (adjusting its shebang) and creates demo2 and demo3 from the dotted callables (using the appropriate shebang). While working on this, I came up against a problem with build_scripts in virtual environments: if I install a project into virtual env 1, then scripts are built with shebangs pointing to that env. If I then install the same project into virtual env 2, then the scripts are not re-created, so they would be installed into virtual env 2 with shebangs referencing virtual env 1. One solution is to set the force flag for virtual environments, but then if you later install into the system Python, then files with wrong shebangs could be installed there. I think the force flag should default to True for build_scripts; the extra script build time would be negligible except in pathological cases. Do you agree? |
Re. the launcher changes, those improvements by Guy Rozendorn are welcome. I noticed some differences from the approach taken by Mark Hammond and Curt Hagenlocher in the PEP-397 implementation, which I ported to C: The Ctrl-C is ignored by the PEP-397 launcher. AFAICT it's passed to the child process anyway by Windows. The ignoring prevents the default behaviour (premature termination of the launcher). The PEP-397 launcher also duplicates the standard handles before creating the child process. The PEP-397 launcher also associates the launcher and the child together using the Job API. |
My current preference is to use only new-style generated scripts for the scripts field in the files section, and to let people use the data-files/resources system for old-style file scripts. We’ll have to think about the shebang munging and decide if we keep it. I think recommending that people use “/usr/bin/env python” (or python3) and not doing anything to the shebang may be the best thing.
|
What about Windows support? Of course there is PEP-397 which brings shebang-line functionality to Windows, but that PEP has not yet been finalised. Without the existing shebang functionality, IMO there will be problems for Windows users with the “/usr/bin/env python” approach you suggest. |
|
Please spell out for me how you see this working: I don't see it. Note that scripts have to use the correct Python even if they are invoked using an explicit path pointing into a virtual environment. |
To expand on what I said about not seeing how things will work under Windows: are we going to place .exe launchers adjacent to the script, like setuptools does? If the script just has a shebang of "#!/usr/bin/env python", how is the launcher supposed to determine the exact Python to use from that, in a venv scenario where multiple Pythons will be installed? Scripts in virtual envs are supposed to run if invoked, even if the env is not on the PATH. |
I’ll get back to this issue later, but now I just wanted to add a link about building the binary exe files: http://mail.python.org/pipermail/python-dev/2006-April/063846.html (it comes from the time where setuptools was supposed to be added to the stdlib) |
@Éric: you may also be interested in a standalone launcher which I wrote for the pythonv branch: https://bitbucket.org/vinay.sajip/simple_launcher/ This is built using Visual Studio and is not based on setuptools code, but uses the same Windows API for child process creation and synchronisation as the PEP-397 launcher (rather than execv/spawnv as the setuptools launcher does). It also links with the runtime statically rather than linking with msvcrt.dll. |
I merged default yesterday and produced this patch. I’ll use the review site to make comments. |
Or I won’t, as even a dumb no-git-style diff does not create a review link, maybe because of the binary file change. *Sigh* Trying again. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: