classification
Title: Incorrect shared library extension on linux
Type: behavior Stage: committed/rejected
Components: Distutils Versions: Python 3.4, Python 3.3, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: doko Nosy List: Arfrever, Thekent, a.badger, akuchling, amaury.forgeotdarc, barry, bkabrda, dmalcolm, doko, eric.araujo, jafo, jtaylor108, ncoghlan, ned.deily, python-dev, rpetrov, smani, tarek
Priority: normal Keywords: patch

Created on 2012-12-23 12:36 by smani, last changed 2014-03-17 20:32 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
so.diff doko, 2013-03-20 01:42 review
pep.diff doko, 2013-03-20 21:42
Messages (22)
msg177979 - (view) Author: Sandro Mani (smani) Date: 2012-12-23 12:36
I'm using Python3 as available in Fedora rawhide 
(python3-3.3.0-2.fc19.x86_64).

Attempting to build a project using python3/distutils, I noticed that 
find_library_file would not find any library at all. Some investigation 
showed that this was due to the fact that libraries were searched with 
the ".cpython-33m.so" extension. Even more investigation showed that the 
library extension was read being overridden by the one defined in the 
/usr/lib64/python3.3/config-3.3m/Makefile shipped by python3-libs. See 
below for the detailed analysis. The python-versioned extension 
obviously makes no sense for regular shared objects which are not python 
binary modules, so this is clearly wrong. As a workaround I patched 
sysconfig.py to comment out customize_compiler::235 (compiler.shared_lib_extension = 
so_ext, see below), recompiled python (all tests still pass), and things seem to work.


Detailed analysis:

setup.py:
def _find_library_file(self, library):
     return self.compiler.find_library_file(self.compiler.library_dirs, 
library)

---
In function 
/usr/lib64/python3.3/distutils/unixcompiler.py at find_library_file::266:
shared_f = self.library_filename(lib, lib_type='shared')

In function 
/usr/lib64/python3.3/distutils/ccompiler.py at library_filename::882:
ext = getattr(self, lib_type + "_lib_extension")

-> Where does shared_lib_extension get defined?
* At /usr/lib64/python3.3/distutils/ccompiler.py::66
shared_lib_extension = None   -> default for abstract class
* At /usr/lib64/python3.3/distutils/unixcompiler.py::77
shared_lib_extension = ".so"  -> this is the correct value
* In function 
/usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
         compiler.shared_lib_extension = so_ext
by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::194
(cc, cxx, opt, cflags, ccshared, ldshared, so_ext, ar, ar_flags) = \
             get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
                             'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS'))
by /usr/lib64/python3.3/distutils/sysconfig.py at get_config_vars::530
526 global _config_vars
527     if _config_vars is None:
528         func = globals().get("_init_" + os.name)  # -> os.name = posix
529         if func:
530             func()  # -> _init_posix, populates _config_vars
by /usr/lib64/python3.3/distutils/sysconfig.py at _init_posix::439
435     g = {}
436     # load the installed Makefile:
437     try:
438         filename = get_makefile_filename()  # 
/usr/lib64/python3.3/config-3.3m/Makefile
439         parse_makefile(filename, g)
...
476     global _config_vars
477     _config_vars = g  # -> _config_vars["SO"] = ".cpython-33m.so"
by /usr/lib64/python3.3/config-3.3m/Makefile::122
SO=             .cpython-33m.so
msg180887 - (view) Author: Sandro Mani (smani) Date: 2013-01-28 23:15
So, from what I can see, historically the SO extension was taken from sysconfig.py, see [1] lines 24 and 60. Then, the CCompiler class got overhauled, and the value was hardcoded to ".so", see [2], but the "compiler.shared_lib_extension = SO" statement remained, presumably oversight. So technically, this is also affecting python 2.x, though in that case it does not make any difference, since python2 does not use versioned so-names for binary modules.

[1] http://hg.python.org/cpython/file/2802fb52e99b/Lib/distutils/unixccompiler.py
[2] http://hg.python.org/cpython/rev/7922d08426ca
msg183621 - (view) Author: Roumen Petrov (rpetrov) Date: 2013-03-06 22:39
This is issue introduced with implementation of SOABI. Build of standard extensions is protected by following code:
-----
class PyBuildExt(build_ext):

    def __init__(self, dist):
        build_ext.__init__(self, dist)
        self.failed = []

    def build_extensions(self):

        # Detect which modules should be compiled
        old_so = self.compiler.shared_lib_extension
        # Workaround PEP 3149 stuff
        self.compiler.shared_lib_extension = os.environ.get("SO", ".so")
        try:
            missing = self.detect_modules()
        finally:
            self.compiler.shared_lib_extension = old_so
....
------

I think that PEP 3149 is not  accurate . For historical reasons (backward compatibility) SO must remain same as OS specific suffix and and new variable is required for python specific suffix.
msg183628 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2013-03-07 02:36
For reference, quoting PEP 3149:

>>> sysconfig.get_config_var('SO')
'.cpython-32mu.so'
msg184607 - (view) Author: Toshio Kuratomi (a.badger) * Date: 2013-03-19 07:02
Matthias, Barry, and I looked at this at pycon today.  It looks a bit like the original intent was to have
SO = ".so"
SOABI = "cpython-32mu"

and then CPython extension module suffixes would be:

if SOABI:
    so_ext = ''.join(".", SOABI, SO)
else:
    so_ext = SO

This would need to be used in distutils/commands/build_ext.py  We weren't sure if there are other places in the code that would need it as well but a quick build of a module which uses libraries and needs C extensions showed that this seems to work.

The one worrisome question is whether more people have come to rely on the SO variable holding the extension module suffix or if more code was broken by the extension module suffix replacing the library suffix in the SO variable.  Answering that might better show us whether to change these variables back to their original meanings or to create brand new variables that have the correct values.

We also discovered the reason the current version appears to work with python-pillow on Ubuntu boxes but not Fedora.  The find_library_files() code first checks for library names that would match with the "shared" library suffix.  If that fails, it falls back to looking for the "static" library suffix.  On Fedora, there are no static libraries so this function just fails to find the library.  On Ubuntu, the code finds the static libraries and returns that.  This causes the code in python-pillow to attempt to link to the library with -ljpeg, -lpng, etc...  Since the shared libraries actually are present, the compiler and linker use the shared versions even though python only found the static versions.
msg184719 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-03-20 01:42
the patch so.diff

 - introduces two new macros EXT_SUFFIX and SHLIB_SUFFIX, and uses
   these instead of the SO macro.
 - keeps the SO macro, and restores it to it's original value
 - Removes the SHLIB_EXT preprocessor define (only used in
   Python/dynload_hpux.c, and there we know that we will end up
   with .sl anyway.

For trunk maybe consider removing the SO macro, I think it's better to change it back to it's original value for 3.1 and 3.2, knowing that it will probably break some Debian packaging, but that can be fixed too.
msg184791 - (view) Author: Sean Reifschneider (jafo) * (Python committer) Date: 2013-03-20 20:42
Toshio and Matthias: This approach seems sane to me, Nick asked me to review this ticket.  I'm not coming up with any objections.  +1 for retiring SO at some point after 3.2, and EXT_SUFFIX and SHLIB_SUFFIX.

What documentation needs to be changed?  PEP 3149 (maybe just reference this being obsolete)?  What other documentation?
msg184805 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-03-20 21:42
proposed PEP change. Or should the change itself be documented in the PEP?
msg184905 - (view) Author: Roundup Robot (python-dev) Date: 2013-03-21 20:40
New changeset 8f91014c8f00 by doko in branch '3.2':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/8f91014c8f00

New changeset 14f0c9e595a5 by doko in branch '3.3':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/14f0c9e595a5

New changeset f6a6b4eed5b0 by doko in branch 'default':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/f6a6b4eed5b0
msg184909 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-03-21 20:57
fixed, pep update is pending
msg185354 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-27 17:34
So sysconfig.get_config_var('SO') will change in a micro release?

Won't this break working user code? Give unexpected file names?
msg185865 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2013-04-02 23:10
I just tried building trunk on MacOS 10.6.8, and make install fails with this traceback:

Traceback (most recent call last):
  File "./setup.py", line 2175, in <module>
    main()
  File "./setup.py", line 2170, in main
    "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
  File "/Users/akuchling/source/p/cpython/Lib/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install.py", line 566, in run
    self.run_command(cmd_name)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install_lib.py", line 93, in run
    outfiles = self.install()
  File "./setup.py", line 2068, in install
    self.set_file_modes(outfiles, 0o644, 0o755)
  File "./setup.py", line 2079, in set_file_modes
    if filename.endswith(self.shlib_suffix): mode = sharedLibMode
TypeError: endswith first arg must be str or a tuple of str, not NoneType
make: *** [sharedinstall] Error 1

The following patch fixes the traceback, but I don't know why shlib_suffix is None.

-> hg diff
diff -r 9328e2b8a397 setup.py
--- a/setup.py	Tue Apr 02 22:13:49 2013 +0200
+++ b/setup.py	Tue Apr 02 19:10:27 2013 -0400
@@ -2076,7 +2076,9 @@
         for filename in files:
             if os.path.islink(filename): continue
             mode = defaultMode
-            if filename.endswith(self.shlib_suffix): mode = sharedLibMode
+            if (self.shlib_suffix is not None and
+                filename.endswith(self.shlib_suffix)):
+                mode = sharedLibMode
             log.info("changing mode of %s to %o", filename, mode)
             if not self.dry_run: os.chmod(filename, mode)
msg185870 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-04-03 00:43
Did you start with a clean build directory and rerun ./configure?  What ./configure options did you use?  Make sure that an existing install isn't getting in the way during a build, particularly with --enable-shared.  With a current tip of the default branch, I've built all three standard OS X configurations (default unix (non-shared), --enable-shared, and --enable-framework) on OS X 10.8 and "make install" succeeds for all of them with sysconfig.get_config_var("SHLIB_SUFFIX") returning ".so" as expecting.
msg186049 - (view) Author: Julian Taylor (jtaylor108) Date: 2013-04-04 19:26
is SHLIB_SUFFIX=".so" really correct on mac?
shared libraries have .dylib extension, loadable modules have .so (which would be EXT_SUFFIX?)
e.g libpython itself uses .dylib.
msg186053 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-04-04 22:18
are you speculating, or is your comment based on some testing? MacOS had this value before, and apparently it did work before that change.
msg186055 - (view) Author: Julian Taylor (jtaylor108) Date: 2013-04-04 22:26
I'm going by what says in configure:
# SHLIB_SUFFIX is the extension of shared libraries

The extension of shared libraries on macos is .dylib in most cases (e.g libtool based libraries and as mentioned python itself)

Maybe its just a documentation/naming issue.
msg186056 - (view) Author: Julian Taylor (jtaylor108) Date: 2013-04-04 22:40
just to clarify its not any issue in python, python is working fine with .so
The issue is just that theses variables tends to be used by other applications to figure out information on the system (like shared library extension, see numpy.distutils)
You certainly could argue that these applications are broken by even needing this information, but a proper naming of the variables could help reduce confusion and wrong code.
msg188094 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-04-29 20:20
This change may be responsible for #17869
msg188101 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-04-29 20:51
False alarm, disregard my previous message.
msg213887 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-17 19:51
Amaury’s questions are still unanswered:

> So sysconfig.get_config_var('SO') will change in a micro release?
> Won't this break working user code? Give unexpected file names?
msg213889 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2014-03-17 20:07
There were already 5 releases of Python 3.3 (starting with 3.3.1) with this change included.
The number of affected applications was rather small. I had seen only NumPy.
msg213890 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-17 20:32
New changeset ccb679e5ae0e by Éric Araujo in branch 'default':
Update SOABI PEP to reflect config var change (#16754).
http://hg.python.org/peps/rev/ccb679e5ae0e
History
Date User Action Args
2014-03-17 20:32:48python-devsetmessages: + msg213890
2014-03-17 20:07:18Arfreversetstatus: open -> closed

messages: + msg213889
2014-03-17 19:51:38eric.araujosetmessages: + msg213887
2013-11-10 19:48:04barrysetassignee: barry -> doko
2013-04-29 20:51:48eric.araujosetmessages: + msg188101
2013-04-29 20:20:28eric.araujosetmessages: + msg188094
2013-04-20 22:48:36Thekentsetnosy: + Thekent
2013-04-04 22:40:49jtaylor108setmessages: + msg186056
2013-04-04 22:26:10jtaylor108setmessages: + msg186055
2013-04-04 22:18:03dokosetmessages: + msg186053
2013-04-04 19:26:49jtaylor108setnosy: + jtaylor108
messages: + msg186049
2013-04-03 00:43:20ned.deilysetnosy: + ned.deily
messages: + msg185870
2013-04-02 23:10:38akuchlingsetnosy: + akuchling
messages: + msg185865
2013-03-27 17:34:34amaury.forgeotdarcsetstatus: pending -> open
nosy: + amaury.forgeotdarc
messages: + msg185354

2013-03-21 20:57:16dokosetstatus: open -> pending
messages: + msg184909

assignee: eric.araujo -> barry
resolution: fixed
stage: committed/rejected
2013-03-21 20:40:06python-devsetnosy: + python-dev
messages: + msg184905
2013-03-20 22:03:04Arfreversetnosy: + Arfrever
2013-03-20 21:42:38dokosetfiles: + pep.diff
2013-03-20 21:42:06dokosetmessages: + msg184805
2013-03-20 20:42:28jafosetnosy: + jafo
messages: + msg184791
2013-03-20 01:42:51dokosetfiles: + so.diff

nosy: + doko
messages: + msg184719

keywords: + patch
2013-03-20 00:42:09ncoghlansetnosy: + ncoghlan
2013-03-19 07:02:35a.badgersetnosy: + a.badger
messages: + msg184607
2013-03-07 02:36:15dmalcolmsetnosy: + dmalcolm
messages: + msg183628
2013-03-06 22:39:40rpetrovsetnosy: + rpetrov
messages: + msg183621
2013-03-06 12:11:25bkabrdasetnosy: + bkabrda
2013-01-28 23:15:50smanisetmessages: + msg180887
2012-12-24 19:28:45eric.araujosetnosy: + barry

versions: + Python 3.2, Python 3.4
2012-12-23 12:36:06smanicreate