I'm using Python3 as available in Fedora rawhide
(python3-3.3.0-2.fc19.x86_64).

Attempting to build a project using python3/distutils, I noticed that
find_library_file would not find any library at all. Some investigation
showed that this was due to the fact that libraries were searched with
the ".cpython-33m.so" extension. Even more investigation showed that the
library extension was read being overridden by the one defined in the
/usr/lib64/python3.3/config-3.3m/Makefile shipped by python3-libs. See
below for the detailed analysis. The python-versioned extension
obviously makes no sense for regular shared objects which are not python
binary modules, so this is clearly wrong. As a workaround I patched
sysconfig.py to comment out customize_compiler::235 (compiler.shared_lib_extension =
so_ext, see below), recompiled python (all tests still pass), and things seem to work.

Detailed analysis:

setup.py:
def _find_library_file(self, library):
return self.compiler.find_library_file(self.compiler.library_dirs,
library)

---
In function
/usr/lib64/python3.3/distutils/unixcompiler.py at find_library_file::266:
shared_f = self.library_filename(lib, lib_type='shared')

In function
/usr/lib64/python3.3/distutils/ccompiler.py at library_filename::882:
ext = getattr(self, lib_type + "_lib_extension")

-> Where does shared_lib_extension get defined?

• At /usr/lib64/python3.3/distutils/ccompiler.py::66
  shared_lib_extension = None -> default for abstract class
• At /usr/lib64/python3.3/distutils/unixcompiler.py::77
  shared_lib_extension = ".so" -> this is the correct value
• In function
  /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
  by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
  compiler.shared_lib_extension = so_ext
  by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::194
  (cc, cxx, opt, cflags, ccshared, ldshared, so_ext, ar, ar_flags) = \
  get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
  'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS'))
  by /usr/lib64/python3.3/distutils/sysconfig.py at get_config_vars::530
  526 global _config_vars
  527 if _config_vars is None:
  528 func = globals().get("_init_" + os.name) # -> os.name = posix
  529 if func:
  530 func() # -> _init_posix, populates _config_vars
  by /usr/lib64/python3.3/distutils/sysconfig.py at _init_posix::439
  435 g = {}
  436 # load the installed Makefile:
  437 try:
  438 filename = get_makefile_filename() #
  /usr/lib64/python3.3/config-3.3m/Makefile
  439 parse_makefile(filename, g)
  ...
  476 global _config_vars
  477 _config_vars = g # -> _config_vars["SO"] = ".cpython-33m.so"
  by /usr/lib64/python3.3/config-3.3m/Makefile::122
  SO= .cpython-33m.so

So, from what I can see, historically the SO extension was taken from sysconfig.py, see [1] lines 24 and 60. Then, the CCompiler class got overhauled, and the value was hardcoded to ".so", see [2], but the "compiler.shared_lib_extension = SO" statement remained, presumably oversight. So technically, this is also affecting python 2.x, though in that case it does not make any difference, since python2 does not use versioned so-names for binary modules.

[1] http://hg.python.org/cpython/file/2802fb52e99b/Lib/distutils/unixccompiler.py
[2] http://hg.python.org/cpython/rev/7922d08426ca

This is issue introduced with implementation of SOABI. Build of standard extensions is protected by following code:
-----

class PyBuildExt(build_ext):

    def __init__(self, dist):
        build_ext.__init__(self, dist)
        self.failed = []

    def build_extensions(self):

        # Detect which modules should be compiled
        old_so = self.compiler.shared_lib_extension
        # Workaround PEP 3149 stuff
        self.compiler.shared_lib_extension = os.environ.get("SO", ".so")
        try:
            missing = self.detect_modules()
        finally:
            self.compiler.shared_lib_extension = old_so
....

I think that PEP-3149 is not accurate . For historical reasons (backward compatibility) SO must remain same as OS specific suffix and and new variable is required for python specific suffix.

For reference, quoting PEP-3149:

>>> sysconfig.get_config_var('SO')
'.cpython-32mu.so'

Matthias, Barry, and I looked at this at pycon today. It looks a bit like the original intent was to have
SO = ".so"
SOABI = "cpython-32mu"

and then CPython extension module suffixes would be:

if SOABI:
    so_ext = ''.join(".", SOABI, SO)
else:
    so_ext = SO

This would need to be used in distutils/commands/build_ext.py We weren't sure if there are other places in the code that would need it as well but a quick build of a module which uses libraries and needs C extensions showed that this seems to work.

The one worrisome question is whether more people have come to rely on the SO variable holding the extension module suffix or if more code was broken by the extension module suffix replacing the library suffix in the SO variable. Answering that might better show us whether to change these variables back to their original meanings or to create brand new variables that have the correct values.

We also discovered the reason the current version appears to work with python-pillow on Ubuntu boxes but not Fedora. The find_library_files() code first checks for library names that would match with the "shared" library suffix. If that fails, it falls back to looking for the "static" library suffix. On Fedora, there are no static libraries so this function just fails to find the library. On Ubuntu, the code finds the static libraries and returns that. This causes the code in python-pillow to attempt to link to the library with -ljpeg, -lpng, etc... Since the shared libraries actually are present, the compiler and linker use the shared versions even though python only found the static versions.

the patch so.diff

• introduces two new macros EXT_SUFFIX and SHLIB_SUFFIX, and uses
  these instead of the SO macro.
• keeps the SO macro, and restores it to it's original value
• Removes the SHLIB_EXT preprocessor define (only used in
  Python/dynload_hpux.c, and there we know that we will end up
  with .sl anyway.

For trunk maybe consider removing the SO macro, I think it's better to change it back to it's original value for 3.1 and 3.2, knowing that it will probably break some Debian packaging, but that can be fixed too.

Toshio and Matthias: This approach seems sane to me, Nick asked me to review this ticket. I'm not coming up with any objections. +1 for retiring SO at some point after 3.2, and EXT_SUFFIX and SHLIB_SUFFIX.

What documentation needs to be changed? PEP-3149 (maybe just reference this being obsolete)? What other documentation?

proposed PEP change. Or should the change itself be documented in the PEP?

New changeset 8f91014c8f00 by doko in branch '3.2':

• Issue bpo-16754: Fix the incorrect shared library extension on linux. Introduce
  http://hg.python.org/cpython/rev/8f91014c8f00

New changeset 14f0c9e595a5 by doko in branch '3.3':

• Issue bpo-16754: Fix the incorrect shared library extension on linux. Introduce
  http://hg.python.org/cpython/rev/14f0c9e595a5

New changeset f6a6b4eed5b0 by doko in branch 'default':

• Issue bpo-16754: Fix the incorrect shared library extension on linux. Introduce
  http://hg.python.org/cpython/rev/f6a6b4eed5b0

fixed, pep update is pending

So sysconfig.get_config_var('SO') will change in a micro release?

Won't this break working user code? Give unexpected file names?

I just tried building trunk on MacOS 10.6.8, and make install fails with this traceback:

Traceback (most recent call last):
  File "./setup.py", line 2175, in <module>
    main()
  File "./setup.py", line 2170, in main
    "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
  File "/Users/akuchling/source/p/cpython/Lib/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install.py", line 566, in run
    self.run_command(cmd_name)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install_lib.py", line 93, in run
    outfiles = self.install()
  File "./setup.py", line 2068, in install
    self.set_file_modes(outfiles, 0o644, 0o755)
  File "./setup.py", line 2079, in set_file_modes
    if filename.endswith(self.shlib_suffix): mode = sharedLibMode
TypeError: endswith first arg must be str or a tuple of str, not NoneType
make: *** [sharedinstall] Error 1

The following patch fixes the traceback, but I don't know why shlib_suffix is None.

-> hg diff
diff -r 9328e2b8a397 setup.py
--- a/setup.py	Tue Apr 02 22:13:49 2013 +0200
+++ b/setup.py	Tue Apr 02 19:10:27 2013 -0400
@@ -2076,7 +2076,9 @@
         for filename in files:
             if os.path.islink(filename): continue
             mode = defaultMode
-            if filename.endswith(self.shlib_suffix): mode = sharedLibMode
+            if (self.shlib_suffix is not None and
+                filename.endswith(self.shlib_suffix)):
+                mode = sharedLibMode
             log.info("changing mode of %s to %o", filename, mode)
             if not self.dry_run: os.chmod(filename, mode)

Did you start with a clean build directory and rerun ./configure? What ./configure options did you use? Make sure that an existing install isn't getting in the way during a build, particularly with --enable-shared. With a current tip of the default branch, I've built all three standard OS X configurations (default unix (non-shared), --enable-shared, and --enable-framework) on OS X 10.8 and "make install" succeeds for all of them with sysconfig.get_config_var("SHLIB_SUFFIX") returning ".so" as expecting.

is SHLIB_SUFFIX=".so" really correct on mac?
shared libraries have .dylib extension, loadable modules have .so (which would be EXT_SUFFIX?)
e.g libpython itself uses .dylib.

are you speculating, or is your comment based on some testing? MacOS had this value before, and apparently it did work before that change.

I'm going by what says in configure:
# SHLIB_SUFFIX is the extension of shared libraries

The extension of shared libraries on macos is .dylib in most cases (e.g libtool based libraries and as mentioned python itself)

Maybe its just a documentation/naming issue.

just to clarify its not any issue in python, python is working fine with .so
The issue is just that theses variables tends to be used by other applications to figure out information on the system (like shared library extension, see numpy.distutils)
You certainly could argue that these applications are broken by even needing this information, but a proper naming of the variables could help reduce confusion and wrong code.

This change may be responsible for bpo-17869

False alarm, disregard my previous message.

Amaury’s questions are still unanswered:

So sysconfig.get_config_var('SO') will change in a micro release?
Won't this break working user code? Give unexpected file names?

There were already 5 releases of Python 3.3 (starting with 3.3.1) with this change included.
The number of affected applications was rather small. I had seen only NumPy.

New changeset ccb679e5ae0e by Éric Araujo in branch 'default':
Update SOABI PEP to reflect config var change (bpo-16754).
http://hg.python.org/peps/rev/ccb679e5ae0e