classification
Title: "make altinstall" installs many files with incorrect shebangs
Type: Stage:
Components: Build, Documentation Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, allan, belopolsky, benjamin.peterson, docs@python, eric.araujo, eric.smith, ezio.melotti, georg.brandl, jcea, ncoghlan, ned.deily, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2010-11-05 03:10 by allan, last changed 2014-04-14 21:45 by orsenthil.

Files
File name Uploaded Description Edit
env-python3.diff eric.araujo, 2010-11-28 03:59
tools-scripts.diff eric.araujo, 2011-07-05 16:14
pep8-shebangs.diff eric.araujo, 2011-11-22 14:37
no-shebangs-for-stdlib.diff eric.araujo, 2011-11-23 17:04
Messages (26)
msg120468 - (view) Author: Allan McRae (allan) Date: 2010-11-05 03:10
The following files are incorrectly installed with a "#!/usr/bin/env python" shebang when using "make altinstall":

usr/lib/python2.7/base64.py
usr/lib/python2.7/bsddb/dbshelve.py
usr/lib/python2.7/bsddb/test/test_dbtables.py
usr/lib/python2.7/cgi.py
usr/lib/python2.7/cgi.py
usr/lib/python2.7/Cookie.py
usr/lib/python2.7/cProfile.py
usr/lib/python2.7/difflib.py
usr/lib/python2.7/distutils/tests/test_build_scripts.py
usr/lib/python2.7/distutils/tests/test_install_scripts.py
usr/lib/python2.7/distutils/unixccompiler.py
usr/lib/python2.7/encodings/rot_13.py
usr/lib/python2.7/idlelib/PyShell.py
usr/lib/python2.7/keyword.py
usr/lib/python2.7/lib2to3/pgen2/token.py
usr/lib/python2.7/lib2to3/tests/data/different_encoding.py
usr/lib/python2.7/lib2to3/tests/pytree_idempotency.py
usr/lib/python2.7/mailbox.py
usr/lib/python2.7/mimify.py
usr/lib/python2.7/pdb.py
usr/lib/python2.7/platform.py
usr/lib/python2.7/profile.py
usr/lib/python2.7/pydoc.py
usr/lib/python2.7/quopri.py
usr/lib/python2.7/smtpd.py
usr/lib/python2.7/smtplib.py
usr/lib/python2.7/symbol.py
usr/lib/python2.7/tabnanny.py
usr/lib/python2.7/tarfile.py
usr/lib/python2.7/test/curses_tests.py
usr/lib/python2.7/test/pystone.py
usr/lib/python2.7/test/regrtest.py
usr/lib/python2.7/test/re_tests.py
usr/lib/python2.7/test/test_al.py
usr/lib/python2.7/test/test_anydbm.py
usr/lib/python2.7/test/test_array.py
usr/lib/python2.7/test/test_binhex.py
usr/lib/python2.7/test/test_bsddb.py
usr/lib/python2.7/test/test_bz2.py
usr/lib/python2.7/test/test_cd.py
usr/lib/python2.7/test/test_cl.py
usr/lib/python2.7/test/test_cmd.py
usr/lib/python2.7/test/test_codecencodings_cn.py
usr/lib/python2.7/test/test_codecencodings_hk.py
usr/lib/python2.7/test/test_codecencodings_jp.py
usr/lib/python2.7/test/test_codecencodings_kr.py
usr/lib/python2.7/test/test_codecencodings_tw.py
usr/lib/python2.7/test/test_codecmaps_cn.py
usr/lib/python2.7/test/test_codecmaps_hk.py
usr/lib/python2.7/test/test_codecmaps_jp.py
usr/lib/python2.7/test/test_codecmaps_kr.py
usr/lib/python2.7/test/test_codecmaps_tw.py
usr/lib/python2.7/test/test_dl.py
usr/lib/python2.7/test/test_dumbdbm.py
usr/lib/python2.7/test/test_eof.py
usr/lib/python2.7/test/test_errno.py
usr/lib/python2.7/test/test___future__.py
usr/lib/python2.7/test/test_gl.py
usr/lib/python2.7/test/test_gzip.py
usr/lib/python2.7/test/test_imageop.py
usr/lib/python2.7/test/test_imgfile.py
usr/lib/python2.7/test/test_logging.py
usr/lib/python2.7/test/test_marshal.py
usr/lib/python2.7/test/test_multibytecodec.py
usr/lib/python2.7/test/test_multibytecodec_support.py
usr/lib/python2.7/test/test_multiprocessing.py
usr/lib/python2.7/test/test_popen2.py
usr/lib/python2.7/test/test_popen.py
usr/lib/python2.7/test/test_random.py
usr/lib/python2.7/test/test_sets.py
usr/lib/python2.7/test/test_smtpnet.py
usr/lib/python2.7/test/test_socket.py
usr/lib/python2.7/test/test_tcl.py
usr/lib/python2.7/test/test_urllib2_localnet.py
usr/lib/python2.7/test/test_urllib2net.py
usr/lib/python2.7/test/test_urllibnet.py
usr/lib/python2.7/test/test_urlparse.py
usr/lib/python2.7/test/test_userstring.py
usr/lib/python2.7/test/test_whichdb.py
usr/lib/python2.7/test/test_with.py
usr/lib/python2.7/timeit.py
usr/lib/python2.7/token.py
usr/lib/python2.7/trace.py
usr/lib/python2.7/UserString.py
usr/lib/python2.7/uu.py
usr/lib/python2.7/webbrowser.py
usr/lib/python2.7/whichdb.py

These should point to /usr/bin/python2.7 instead.
msg120485 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-11-05 11:39
Is there any reason for the test files to have a shebang line at all? I think those should be removed, which would cut the problem in half.

Also, given "-m", I'm not sure any of the files in the stdlib should have a shebang line. Is there really an expectation that these should be run directly? Is anyone really running UserString.py as a script in order to run its tests?
msg120489 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-05 12:30
Benjamin did some cleanup in this area in at least py3k, so he might have some thoughts, making him nosy.
msg120493 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-11-05 14:06
Removing shebang lines from svn completely and only *adding* them during installation steps as appropriate may be an interesting approach. (I noted that my grep of my local build found only correct references to python3.2 in the built scripts directory)

I'll add the list of Py3k files that are unexpectedly referencing something other than "/usr/bin/env python3" in SVN as well (note that this is a straight grep, without checking to see if any of them are *meant* to be referring to Python 2.x):

Doc/distutils/setupscript.rst does not use python3 in the example
Doc/faq/library.rst (multiple instances)
Doc/howto/unicode.rst
Doc/howto/webservers.rst
Doc/library/cgi.rst
Doc/library/logging.rst
Doc/library/urllib.request.rst
Doc/using/unix.rst

Lib/test/test_logging.py: #! /usr/bin/env python

Lib/cgi.py: #! /usr/bin/env python

Mac/BuildScript/build-installer.py: #! /usr/bin/env python
Mac/Tools/fixapplepython23.py: #! /usr/bin/env python
Mac/Tools/bundlebuilder.py: #! /usr/bin/env python

Tools/gdb/libpython.py: #! /usr/bin/env python
Tools/pybench/clockres.py: #!/usr/bin/env python
Tools/pybench/pybench.py: #!/usr/local/bin/python -O
Tools/pybench/Setup.py: #!python
Tools/pybench/systimes.py: #!/usr/bin/env python
Tools/pynche/pynche: #! /usr/bin/env python
Tools/pynche/pynche.pyw: #! /usr/bin/env python
Tools/scripts/2to3.py: #! /usr/bin/env python
Tools/scripts/gprof2html.py: #! /usr/bin/env python32.3
Tools/scripts/reindent-rst.py: #!/usr/bin/env python
Tools/world/world: #! /usr/bin/env python

The spec file in Misc/RPM also has multiple references to fixing shebang lines, but I don't know anything about spec files, so I didn't even try to check if it was doing the right thing.
msg120494 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-11-05 14:07
For the record, my list is from an svn checkout of r86191
msg120495 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2010-11-05 14:16
A few more deeper in the py3k source tree:

Doc/tools/docutils/_string_template_compat.py
Doc/tools/docutils/readers/python/pynodes.py
Doc/tools/sphinx/pycode/pgen2/token.py
Lib/lib2to3/tests/data/different_encoding.py

Adding Georg, since this affects the docs as well as the source code.
msg120509 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-11-05 15:40
You can ignore those under Doc/tools; they are neither part of the distribution and nor installed.
msg120546 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-11-05 22:20
I agree with Eric's comment about why have shebang lines at all for files in the standard library.  There isn't any use case or recommendation for ever putting /path/to/lib/pythonx.x or its subdirectories directly on a shell search path is there?

WRT the three files found in the Mac directory, I think all of these should be left alone for right now.  Specifically:

Mac/BuildScript/build-installer.py
   is the script used to build OS X installer images; at the moment, it depends on a system Python 2 as a build tool, primarily because of Sphinx, and there has been an effort to keep the Python 2 and Python 3 versions of the script in sync.  Eventually that will need to be changed.  The shebang line could simply be removed.

Mac/Tools/fixapplepython23.py:
    this one needs to be looked at a bit more as it runs during the installation process but only on OS X 10.3, a minor and dwindling niche of the user base.  I think that it actually depends on the Apple-installed system Python at run time.  I'll follow up on it.

Mac/Tools/bundlebuilder.py: #! /usr/bin/env python
    AFAIK, bundlebuilder is neither used during the build process of Python 3 nor is it installed.  It is used in the Python 2 build process.
msg120881 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-09 16:46
I would like to add my +1 to Eric's msg120485 above.  What I really find puzzling is why some scripts in Tools/ have hashbangs, but don't have execute permission.

Tools/scripts/cleanfuture.py
Tools/scripts/combinerefs.py
Tools/scripts/db2pickle.py
Tools/scripts/find_recursionlimit.py
Tools/scripts/md5sum.py
Tools/scripts/pickle2db.py
Tools/scripts/pysource.py
Tools/scripts/svneol.py

I suggest a simple rule: no execute bit set in permissions - no hashbang line.

And I don't think stdlib modules should have execute bit:

-rwxr-xr-x Lib/base64.py
-rwxr-xr-x Lib/cProfile.py
-rwxr-xr-x Lib/cgi.py
-rwxr-xr-x Lib/keyword.py
-rwxr-xr-x Lib/pdb.py
-rwxr-xr-x Lib/platform.py
-rwxr-xr-x Lib/profile.py
-rwxr-xr-x Lib/pydoc.py
-rwxr-xr-x Lib/quopri.py
-rwxr-xr-x Lib/smtpd.py
-rwxr-xr-x Lib/smtplib.py
-rwxr-xr-x Lib/symbol.py
-rwxr-xr-x Lib/tabnanny.py
-rwxr-xr-x Lib/token.py
-rwxr-xr-x Lib/uu.py

-rwxr-xr-x Lib/test/pystone.py
-rwxr-xr-x Lib/test/re_tests.py
-rwxr-xr-x Lib/test/regrtest.py
-rwxr-xr-x Lib/test/test_array.py
-rwxr-xr-x Lib/test/test_binhex.py
-rwxr-xr-x Lib/test/test_dbm_gnu.py
-rwxr-xr-x Lib/test/test_dbm_ndbm.py
-rwxr-xr-x Lib/test/test_errno.py
-rwxr-xr-x Lib/test/test_userstring.py
msg122617 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-28 03:59
Attached patch fixes shebangs to use python3 in py3k.

There are a lot of examples that use a bare “python”; changing all of those would cause merging pains.
msg122618 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-11-28 04:03
-    #!/usr/bin/env python
+    #!/usr/bin/env python3
     # -*- coding: UTF-8 -*-

Not strictly related to this issue, but do we want to recommend
redundant encoding cookie in the docs?
msg122619 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-28 04:05
Good call.  +1 on removing them.
msg134543 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-27 10:04
> There are a lot of examples that use a bare “python”; changing all of
> those would cause merging pains.

I’ve changed my mind.  Given the python/python2/python3 drama with distributions, I now think that we should use “python3” in the 3.x docs.  Merging is not hard.
msg139892 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-05 16:14
This 3.2 patch updates UNIX rights and shebangs in Tools/scripts.

I also edited mailerdaemon, which used a string exception.
msg141348 - (view) Author: Roundup Robot (python-dev) Date: 2011-07-29 12:35
New changeset 65d5ecefd50d by Éric Araujo in branch '3.2':
Fix missing or wrong shebangs and missing executable bits for scripts (#10318)
http://hg.python.org/cpython/rev/65d5ecefd50d

New changeset 5467abaaf5eb by Éric Araujo in branch 'default':
Merge from 3.2 (#10318, #12255, #12043, #12417 and other fixes)
http://hg.python.org/cpython/rev/5467abaaf5eb
msg141358 - (view) Author: Roundup Robot (python-dev) Date: 2011-07-29 12:37
New changeset 7fb0abf928e0 by Éric Araujo in branch '2.7':
Fix missing or shebangs and executable bits for scripts (#10318)
http://hg.python.org/cpython/rev/7fb0abf928e0
msg147945 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-11-19 13:56
Éric, what's the status of this?
msg147989 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-20 14:20
The patches I’ve discussed and committed were actually peripheral.  The original bug reported here is that shebangs shouldn’t use “/usr/bin/env python” with an altinstall installation, as in that case you’re not creating a python (or python3) binary but a pythonx.y, so the shebangs should refer to that exact x.y versions.

The few scripts from Tools/scripts (idle, pydoc, 2to3, python-config) are installed with an x.y suffix and hard-code the Python version and location at build time, by using distutils.  (That’s why Nick reports the only matches for “3.2” found by grep are in the build/scripts directory.)

The stdlib modules do use “/usr/bin/env python[3]”.  I see various ways to handle that:
a) Reject the bug as works for me: these are stdlib modules, not scripts, they can be imported or executed with -m, they’re not symlinked (by us) from anywhere so the bug, while technically valid, has no real effect.

b) Further complicate the build/install machinery to update shebangs in altinstall mode.

c) Remove useless shebangs and execute bit in the stdlib.

My preference is to do c) for 3.3 and nothing for the stable versions.

(About wrongly using python in Python 3 docs: I’ll open another bug for that).
msg148003 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-11-20 17:07
+1 to "c".
msg148018 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-21 00:00
+1 to 'c', but it should come with an update to PEP 8 to say "don't do that".
msg148117 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-22 14:37
Patch to PEP 8 attached.
msg148159 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-23 00:08
Hmm, my initial reaction is that that specific wording is stronger than I had in mind - there's nothing really wrong with having a shebang line and execute bit set on a top level module and symlinking it from /usr/bin. The problem is that we're doing those things for modules that we *don't* install as binaries, and that's silly (particularly when the shebang lines are wrong on altinstall). However, that concern can specifically by addressed by moving the new section down to be a subheading in the "Rules that apply only to the standard library" section.

I'd also mention the justification that this is due to such shebang lines creating a maintenance problem for handling parallel installations of different Python versions.

Also, with PEP 397 a viable candidate, we may as well make the wording OS neutral. Something like:

=======================
Executable Files and Shebang Lines

   Standard library modules (including test modules) should not be
   flagged as executable files nor contain a shebang line.

   Including shebang lines in standard library modules is an issue, as
   they create a maintenance problem when multiple versions of Python
   are installed in parallel. The easiest way to avoid accidentally
   invoking the wrong version of Python is to simply not include such
   lines at all.

   If a module provides command-line functionality, it can be used with
   ``python -m module`` or via a small script (in a different file) that
   imports the module and calls one of its functions (e.g. the ``pydoc``
   script imports the ``pydoc`` module and calls ``pydoc.cli()``).

   Any installed scripts should use a shebang line of the form::

        #!/usr/bin/env pythonX.Y

   where ``X.Y`` is the specific Python version being installed.
   Updating these lines to the correct Python version should be
   automated, either as part of the installation process or as part
   of the version update process (see PEP 101).
=======================

The PEP 8 update should be run by the wider audience of python-dev before it gets committed, though.
msg148192 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-23 17:04
> Hmm, my initial reaction is that that specific wording is stronger than I had in mind -
> there's nothing really wrong with having a shebang line and execute bit set on a top level
> module and symlinking it from /usr/bin.
Okay.  (On that topic, http://lists.debian.org/debian-python/2011/11/msg00058.html may interest you.)

> The problem is that we're doing those things for modules that we *don't* install as binaries,
> and that's silly
Yep.  Attached patch removes them for 3.3.

> I'd also mention the justification that this is due to such shebang lines creating a
> maintenance problem for handling parallel installations of different Python versions.
I’d rather just say that it’s unneeded.  With all due respect to the original poster, I don’t think this really caused problems.


I will move my addition to the stdlib-only section.  I’m not sure about OS-neutrality; the executable bit is Unix-specific and I’d rather use that exact term than a vague “flagged as executable”.  I’ll make the part about shebangs neutral however, it won’t be hard.

About this part of your proposal:
> Any installed scripts should use a shebang line of the form::
>        #!/usr/bin/env pythonX.Y
Due to the use of distutils’ build_scripts that hard-codes one path, I’m not sure it’s time yet to make that recommendation.  For packaging, I intend to launch a discussion about that behavior, which is often unhelpful.

I really appreciate your taking time to review, and will submit the next revision of the patch here before going to python-dev.
msg148210 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-23 22:26
Hmm, interesting mailing list post - I hadn't thought about how the auto-initialisation of sys.path[0] aligns with the Windows vs Unix difference in PATH handling (i.e. whether or not the current directory is considered to be on PATH), with Python coming down in favour of the usually-more-convenient-but-less-secure Windows approach.

We have -S to disable all site processing, -s to disable user site packages and -E to ignore PYTHONPATH and PYTHONHOME - perhaps there should be another flag to skip the auto-initialisation of sys.path[0]. I may include something along those lines in PEP 395.
msg148296 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-11-25 00:47
I created #13475 to discuss the idea of a command line option to override sys.path[0] initialisation.
msg148485 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-28 14:37
Can I removed the shebangs in the 3.3 stdlib or do I need to go through with the PEP 8 patch on python-dev first?
History
Date User Action Args
2014-04-14 21:45:51orsenthilsetnosy: - orsenthil
2011-11-28 14:37:07eric.araujosetmessages: + msg148485
versions: - Python 2.7, Python 3.2
2011-11-27 00:03:47Arfreversetnosy: + Arfrever
2011-11-25 00:47:10ncoghlansetmessages: + msg148296
2011-11-23 22:26:07ncoghlansetmessages: + msg148210
2011-11-23 17:04:56eric.araujosetfiles: + no-shebangs-for-stdlib.diff

messages: + msg148192
2011-11-23 00:08:58ncoghlansetmessages: + msg148159
2011-11-22 14:37:07eric.araujosetfiles: + pep8-shebangs.diff

messages: + msg148117
2011-11-21 00:00:40ncoghlansetmessages: + msg148018
2011-11-20 17:07:28jceasetmessages: + msg148003
2011-11-20 14:20:18eric.araujosetmessages: + msg147989
2011-11-19 13:56:40ezio.melottisetnosy: + ezio.melotti
messages: + msg147945
2011-07-29 12:37:59python-devsetmessages: + msg141358
2011-07-29 12:35:17python-devsetnosy: + python-dev
messages: + msg141348
2011-07-05 16:14:32eric.araujosetfiles: + tools-scripts.diff

messages: + msg139892
versions: - Python 3.1
2011-04-27 10:04:54eric.araujosetmessages: + msg134543
versions: + Python 3.3
2010-11-29 16:27:48eric.araujolinkissue6886 superseder
2010-11-28 05:40:30eric.araujolinkissue6887 superseder
2010-11-28 04:05:15eric.araujosetmessages: + msg122619
2010-11-28 04:03:49belopolskysetmessages: + msg122618
2010-11-28 03:59:20eric.araujosetfiles: + env-python3.diff
keywords: + patch
2010-11-28 03:59:01eric.araujosetmessages: + msg122617
2010-11-16 17:49:36jceasetnosy: + jcea
2010-11-09 16:46:30belopolskysetnosy: + belopolsky
messages: + msg120881
2010-11-09 05:46:40orsenthilsetnosy: + orsenthil
2010-11-05 22:20:46ned.deilysetnosy: + ned.deily
messages: + msg120546
2010-11-05 17:19:55eric.araujosetnosy: + eric.araujo

versions: + Python 3.1
2010-11-05 15:40:45georg.brandlsetmessages: + msg120509
2010-11-05 14:16:24ncoghlansetassignee: docs@python ->
2010-11-05 14:16:13ncoghlansetnosy: + docs@python, georg.brandl
messages: + msg120495

assignee: docs@python
components: + Documentation
2010-11-05 14:07:45ncoghlansetmessages: + msg120494
2010-11-05 14:06:30ncoghlansetnosy: + ncoghlan

messages: + msg120493
versions: + Python 3.2
2010-11-05 12:30:25r.david.murraysetnosy: + r.david.murray, benjamin.peterson
messages: + msg120489
2010-11-05 11:39:25eric.smithsetnosy: + eric.smith
messages: + msg120485
2010-11-05 03:10:06allancreate