msg177979 - (view) |
Author: Sandro Mani (smani) * |
Date: 2012-12-23 12:36 |
I'm using Python3 as available in Fedora rawhide
(python3-3.3.0-2.fc19.x86_64).
Attempting to build a project using python3/distutils, I noticed that
find_library_file would not find any library at all. Some investigation
showed that this was due to the fact that libraries were searched with
the ".cpython-33m.so" extension. Even more investigation showed that the
library extension was read being overridden by the one defined in the
/usr/lib64/python3.3/config-3.3m/Makefile shipped by python3-libs. See
below for the detailed analysis. The python-versioned extension
obviously makes no sense for regular shared objects which are not python
binary modules, so this is clearly wrong. As a workaround I patched
sysconfig.py to comment out customize_compiler::235 (compiler.shared_lib_extension =
so_ext, see below), recompiled python (all tests still pass), and things seem to work.
Detailed analysis:
setup.py:
def _find_library_file(self, library):
return self.compiler.find_library_file(self.compiler.library_dirs,
library)
---
In function
/usr/lib64/python3.3/distutils/unixcompiler.py at find_library_file::266:
shared_f = self.library_filename(lib, lib_type='shared')
In function
/usr/lib64/python3.3/distutils/ccompiler.py at library_filename::882:
ext = getattr(self, lib_type + "_lib_extension")
-> Where does shared_lib_extension get defined?
* At /usr/lib64/python3.3/distutils/ccompiler.py::66
shared_lib_extension = None -> default for abstract class
* At /usr/lib64/python3.3/distutils/unixcompiler.py::77
shared_lib_extension = ".so" -> this is the correct value
* In function
/usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::235
compiler.shared_lib_extension = so_ext
by /usr/lib64/python3.3/distutils/sysconfig.py at customize_compiler::194
(cc, cxx, opt, cflags, ccshared, ldshared, so_ext, ar, ar_flags) = \
get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
'CCSHARED', 'LDSHARED', 'SO', 'AR', 'ARFLAGS'))
by /usr/lib64/python3.3/distutils/sysconfig.py at get_config_vars::530
526 global _config_vars
527 if _config_vars is None:
528 func = globals().get("_init_" + os.name) # -> os.name = posix
529 if func:
530 func() # -> _init_posix, populates _config_vars
by /usr/lib64/python3.3/distutils/sysconfig.py at _init_posix::439
435 g = {}
436 # load the installed Makefile:
437 try:
438 filename = get_makefile_filename() #
/usr/lib64/python3.3/config-3.3m/Makefile
439 parse_makefile(filename, g)
...
476 global _config_vars
477 _config_vars = g # -> _config_vars["SO"] = ".cpython-33m.so"
by /usr/lib64/python3.3/config-3.3m/Makefile::122
SO= .cpython-33m.so
|
msg180887 - (view) |
Author: Sandro Mani (smani) * |
Date: 2013-01-28 23:15 |
So, from what I can see, historically the SO extension was taken from sysconfig.py, see [1] lines 24 and 60. Then, the CCompiler class got overhauled, and the value was hardcoded to ".so", see [2], but the "compiler.shared_lib_extension = SO" statement remained, presumably oversight. So technically, this is also affecting python 2.x, though in that case it does not make any difference, since python2 does not use versioned so-names for binary modules.
[1] http://hg.python.org/cpython/file/2802fb52e99b/Lib/distutils/unixccompiler.py
[2] http://hg.python.org/cpython/rev/7922d08426ca
|
msg183621 - (view) |
Author: Roumen Petrov (rpetrov) * |
Date: 2013-03-06 22:39 |
This is issue introduced with implementation of SOABI. Build of standard extensions is protected by following code:
-----
class PyBuildExt(build_ext):
def __init__(self, dist):
build_ext.__init__(self, dist)
self.failed = []
def build_extensions(self):
# Detect which modules should be compiled
old_so = self.compiler.shared_lib_extension
# Workaround PEP 3149 stuff
self.compiler.shared_lib_extension = os.environ.get("SO", ".so")
try:
missing = self.detect_modules()
finally:
self.compiler.shared_lib_extension = old_so
....
------
I think that PEP 3149 is not accurate . For historical reasons (backward compatibility) SO must remain same as OS specific suffix and and new variable is required for python specific suffix.
|
msg183628 - (view) |
Author: Dave Malcolm (dmalcolm)  |
Date: 2013-03-07 02:36 |
For reference, quoting PEP 3149:
>>> sysconfig.get_config_var('SO')
'.cpython-32mu.so'
|
msg184607 - (view) |
Author: Toshio Kuratomi (a.badger) * |
Date: 2013-03-19 07:02 |
Matthias, Barry, and I looked at this at pycon today. It looks a bit like the original intent was to have
SO = ".so"
SOABI = "cpython-32mu"
and then CPython extension module suffixes would be:
if SOABI:
so_ext = ''.join(".", SOABI, SO)
else:
so_ext = SO
This would need to be used in distutils/commands/build_ext.py We weren't sure if there are other places in the code that would need it as well but a quick build of a module which uses libraries and needs C extensions showed that this seems to work.
The one worrisome question is whether more people have come to rely on the SO variable holding the extension module suffix or if more code was broken by the extension module suffix replacing the library suffix in the SO variable. Answering that might better show us whether to change these variables back to their original meanings or to create brand new variables that have the correct values.
We also discovered the reason the current version appears to work with python-pillow on Ubuntu boxes but not Fedora. The find_library_files() code first checks for library names that would match with the "shared" library suffix. If that fails, it falls back to looking for the "static" library suffix. On Fedora, there are no static libraries so this function just fails to find the library. On Ubuntu, the code finds the static libraries and returns that. This causes the code in python-pillow to attempt to link to the library with -ljpeg, -lpng, etc... Since the shared libraries actually are present, the compiler and linker use the shared versions even though python only found the static versions.
|
msg184719 - (view) |
Author: Matthias Klose (doko) *  |
Date: 2013-03-20 01:42 |
the patch so.diff
- introduces two new macros EXT_SUFFIX and SHLIB_SUFFIX, and uses
these instead of the SO macro.
- keeps the SO macro, and restores it to it's original value
- Removes the SHLIB_EXT preprocessor define (only used in
Python/dynload_hpux.c, and there we know that we will end up
with .sl anyway.
For trunk maybe consider removing the SO macro, I think it's better to change it back to it's original value for 3.1 and 3.2, knowing that it will probably break some Debian packaging, but that can be fixed too.
|
msg184791 - (view) |
Author: Sean Reifschneider (jafo) *  |
Date: 2013-03-20 20:42 |
Toshio and Matthias: This approach seems sane to me, Nick asked me to review this ticket. I'm not coming up with any objections. +1 for retiring SO at some point after 3.2, and EXT_SUFFIX and SHLIB_SUFFIX.
What documentation needs to be changed? PEP 3149 (maybe just reference this being obsolete)? What other documentation?
|
msg184805 - (view) |
Author: Matthias Klose (doko) *  |
Date: 2013-03-20 21:42 |
proposed PEP change. Or should the change itself be documented in the PEP?
|
msg184905 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2013-03-21 20:40 |
New changeset 8f91014c8f00 by doko in branch '3.2':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/8f91014c8f00
New changeset 14f0c9e595a5 by doko in branch '3.3':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/14f0c9e595a5
New changeset f6a6b4eed5b0 by doko in branch 'default':
- Issue #16754: Fix the incorrect shared library extension on linux. Introduce
http://hg.python.org/cpython/rev/f6a6b4eed5b0
|
msg184909 - (view) |
Author: Matthias Klose (doko) *  |
Date: 2013-03-21 20:57 |
fixed, pep update is pending
|
msg185354 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) *  |
Date: 2013-03-27 17:34 |
So sysconfig.get_config_var('SO') will change in a micro release?
Won't this break working user code? Give unexpected file names?
|
msg185865 - (view) |
Author: A.M. Kuchling (akuchling) *  |
Date: 2013-04-02 23:10 |
I just tried building trunk on MacOS 10.6.8, and make install fails with this traceback:
Traceback (most recent call last):
File "./setup.py", line 2175, in <module>
main()
File "./setup.py", line 2170, in main
"Tools/scripts/2to3", "Tools/scripts/pyvenv"]
File "/Users/akuchling/source/p/cpython/Lib/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 917, in run_commands
self.run_command(cmd)
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
cmd_obj.run()
File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install.py", line 566, in run
self.run_command(cmd_name)
File "/Users/akuchling/source/p/cpython/Lib/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
cmd_obj.run()
File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install_lib.py", line 93, in run
outfiles = self.install()
File "./setup.py", line 2068, in install
self.set_file_modes(outfiles, 0o644, 0o755)
File "./setup.py", line 2079, in set_file_modes
if filename.endswith(self.shlib_suffix): mode = sharedLibMode
TypeError: endswith first arg must be str or a tuple of str, not NoneType
make: *** [sharedinstall] Error 1
The following patch fixes the traceback, but I don't know why shlib_suffix is None.
-> hg diff
diff -r 9328e2b8a397 setup.py
--- a/setup.py Tue Apr 02 22:13:49 2013 +0200
+++ b/setup.py Tue Apr 02 19:10:27 2013 -0400
@@ -2076,7 +2076,9 @@
for filename in files:
if os.path.islink(filename): continue
mode = defaultMode
- if filename.endswith(self.shlib_suffix): mode = sharedLibMode
+ if (self.shlib_suffix is not None and
+ filename.endswith(self.shlib_suffix)):
+ mode = sharedLibMode
log.info("changing mode of %s to %o", filename, mode)
if not self.dry_run: os.chmod(filename, mode)
|
msg185870 - (view) |
Author: Ned Deily (ned.deily) *  |
Date: 2013-04-03 00:43 |
Did you start with a clean build directory and rerun ./configure? What ./configure options did you use? Make sure that an existing install isn't getting in the way during a build, particularly with --enable-shared. With a current tip of the default branch, I've built all three standard OS X configurations (default unix (non-shared), --enable-shared, and --enable-framework) on OS X 10.8 and "make install" succeeds for all of them with sysconfig.get_config_var("SHLIB_SUFFIX") returning ".so" as expecting.
|
msg186049 - (view) |
Author: Julian Taylor (jtaylor108) |
Date: 2013-04-04 19:26 |
is SHLIB_SUFFIX=".so" really correct on mac?
shared libraries have .dylib extension, loadable modules have .so (which would be EXT_SUFFIX?)
e.g libpython itself uses .dylib.
|
msg186053 - (view) |
Author: Matthias Klose (doko) *  |
Date: 2013-04-04 22:18 |
are you speculating, or is your comment based on some testing? MacOS had this value before, and apparently it did work before that change.
|
msg186055 - (view) |
Author: Julian Taylor (jtaylor108) |
Date: 2013-04-04 22:26 |
I'm going by what says in configure:
# SHLIB_SUFFIX is the extension of shared libraries
The extension of shared libraries on macos is .dylib in most cases (e.g libtool based libraries and as mentioned python itself)
Maybe its just a documentation/naming issue.
|
msg186056 - (view) |
Author: Julian Taylor (jtaylor108) |
Date: 2013-04-04 22:40 |
just to clarify its not any issue in python, python is working fine with .so
The issue is just that theses variables tends to be used by other applications to figure out information on the system (like shared library extension, see numpy.distutils)
You certainly could argue that these applications are broken by even needing this information, but a proper naming of the variables could help reduce confusion and wrong code.
|
msg188094 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2013-04-29 20:20 |
This change may be responsible for #17869
|
msg188101 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2013-04-29 20:51 |
False alarm, disregard my previous message.
|
msg213887 - (view) |
Author: Éric Araujo (eric.araujo) *  |
Date: 2014-03-17 19:51 |
Amaury’s questions are still unanswered:
> So sysconfig.get_config_var('SO') will change in a micro release?
> Won't this break working user code? Give unexpected file names?
|
msg213889 - (view) |
Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) *  |
Date: 2014-03-17 20:07 |
There were already 5 releases of Python 3.3 (starting with 3.3.1) with this change included.
The number of affected applications was rather small. I had seen only NumPy.
|
msg213890 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2014-03-17 20:32 |
New changeset ccb679e5ae0e by Éric Araujo in branch 'default':
Update SOABI PEP to reflect config var change (#16754).
http://hg.python.org/peps/rev/ccb679e5ae0e
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:39 | admin | set | github: 60958 |
2014-03-17 20:32:48 | python-dev | set | messages:
+ msg213890 |
2014-03-17 20:07:18 | Arfrever | set | status: open -> closed
messages:
+ msg213889 |
2014-03-17 19:51:38 | eric.araujo | set | messages:
+ msg213887 |
2013-11-10 19:48:04 | barry | set | assignee: barry -> doko |
2013-04-29 20:51:48 | eric.araujo | set | messages:
+ msg188101 |
2013-04-29 20:20:28 | eric.araujo | set | messages:
+ msg188094 |
2013-04-20 22:48:36 | Thekent | set | nosy:
+ Thekent
|
2013-04-04 22:40:49 | jtaylor108 | set | messages:
+ msg186056 |
2013-04-04 22:26:10 | jtaylor108 | set | messages:
+ msg186055 |
2013-04-04 22:18:03 | doko | set | messages:
+ msg186053 |
2013-04-04 19:26:49 | jtaylor108 | set | nosy:
+ jtaylor108 messages:
+ msg186049
|
2013-04-03 00:43:20 | ned.deily | set | nosy:
+ ned.deily messages:
+ msg185870
|
2013-04-02 23:10:38 | akuchling | set | nosy:
+ akuchling messages:
+ msg185865
|
2013-03-27 17:34:34 | amaury.forgeotdarc | set | status: pending -> open nosy:
+ amaury.forgeotdarc messages:
+ msg185354
|
2013-03-21 20:57:16 | doko | set | status: open -> pending messages:
+ msg184909
assignee: eric.araujo -> barry resolution: fixed stage: resolved |
2013-03-21 20:40:06 | python-dev | set | nosy:
+ python-dev messages:
+ msg184905
|
2013-03-20 22:03:04 | Arfrever | set | nosy:
+ Arfrever
|
2013-03-20 21:42:38 | doko | set | files:
+ pep.diff |
2013-03-20 21:42:06 | doko | set | messages:
+ msg184805 |
2013-03-20 20:42:28 | jafo | set | nosy:
+ jafo messages:
+ msg184791
|
2013-03-20 01:42:51 | doko | set | files:
+ so.diff
nosy:
+ doko messages:
+ msg184719
keywords:
+ patch |
2013-03-20 00:42:09 | ncoghlan | set | nosy:
+ ncoghlan
|
2013-03-19 07:02:35 | a.badger | set | nosy:
+ a.badger messages:
+ msg184607
|
2013-03-07 02:36:15 | dmalcolm | set | nosy:
+ dmalcolm messages:
+ msg183628
|
2013-03-06 22:39:40 | rpetrov | set | nosy:
+ rpetrov messages:
+ msg183621
|
2013-03-06 12:11:25 | bkabrda | set | nosy:
+ bkabrda
|
2013-01-28 23:15:50 | smani | set | messages:
+ msg180887 |
2012-12-24 19:28:45 | eric.araujo | set | nosy:
+ barry
versions:
+ Python 3.2, Python 3.4 |
2012-12-23 12:36:06 | smani | create | |