classification
Title: Use -Wl,--dynamic-list=x.list, not -Xlinker -export-dynamic
Type: resource usage Stage: patch review
Components: Build Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, dmalcolm, eric.araujo, jankratochvil, loewis, meador.inge, pitrou, tromey, vstinner
Priority: normal Keywords: patch

Created on 2010-10-15 10:16 by jankratochvil, last changed 2019-01-23 15:47 by vstinner.

Files
File name Uploaded Description Edit
hidden4.patch jankratochvil, 2012-02-20 05:43 Fixed patch
Messages (7)
msg118756 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2010-10-15 10:16
FSF GDB (and future Fedora GDBs) became 250KB smaller than before.  Python recently disabled this GDB build optimization so GDB is 250KB larger again.

-rwxr-xr-x 1 4524488 gdb-7.1-19.fc13.x86_64/usr/bin/gdb
-rwxr-xr-x 1 4266728 gdb-7.1-19dynamiclist.fc13.x86_64/usr/bin/gdb
-rw-r--r-- 1 2364656 gdb-7.1-19.fc13.x86_64.rpm
-rw-r--r-- 1 2274300 gdb-7.1-19dynamiclist.fc13.x86_64.rpm

Some Python binaries/libraries could probably also benefit from smaller pageable code image.

GDB regressed due to /usr/lib*/python*/config/Makefile contains:
  LINKFORSHARED=-Xlinker -export-dynamic

and GDB thus has to use it even for its own link, effectively disabling its:
  -Wl,--dynamic-list=./proc-service.list

AFAIK Python already contains the required exports list but it is used only on MS-Windows.  It should be utilized even for GNU/Linux builds.
msg151146 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-01-12 18:18
Here is the implementation.

Python/getargs.c was modified for:
ImportError: /usr/lib64/python2.7/lib-dynload/fcntlmodule.so: undefined symbol: _PyArg_ParseTuple_SizeT
but I guess that patch part should be different.  There is no need to do #ifdef HAVE_DECLSPEC_DLL as PyAPI_FUNC is properly defined even without it.  So the declaration+export block should be unified there.

The Modules/ files were modified for:
*** WARNING: renaming "_ctypes_test" since importing it failed: dynamic module does not define init function (init_ctypes_test)
*** WARNING: renaming "_elementtree" since importing it failed: dynamic module does not define init function (init_elementtree)
*** WARNING: renaming "_hotshot" since importing it failed: dynamic module does not define init function (init_hotshot)
*** WARNING: renaming "_json" since importing it failed: dynamic module does not define init function (init_json)
*** WARNING: renaming "ossaudiodev" since importing it failed: dynamic module does not define init function (initossaudiodev)

without this patch:
-rw-r--r-- 1 jkratoch jkratoch 5775829 Jan 12 17:42 python-libs-2.7.2-18.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1738264 Jan 12 17:44 ./clean/python-libs-2.7.2-18.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1226 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1986161 Jan 12 17:48 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 5018800 Jan 12 19:13 clean/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*

with this patch:
-rw-r--r-- 1 jkratoch jkratoch 5762537 Jan 12 19:04 python-libs-2.7.2-18hidden3.fc17.x86_64.rpm
-r-xr-xr-x 1 jkratoch jkratoch 1720920 Jan 12 19:06 ./hidden3/python-libs-2.7.2-18hidden3.fc17.x86_64/usr/lib64/libpython2.7.so.1.0*
1046 libpython2.7.so.1.0 .dynsym exports
+
-rw-r--r-- 1 jkratoch jkratoch 1886781 Jan 12 19:11 gdb-7.4.50.20120103-8.fc17.x86_64.rpm
-rwxr-xr-x 1 jkratoch jkratoch 4732080 Jan 12 19:13 hidden3/gdb-7.4.50.20120103-8.fc17.x86_64/usr/bin/gdb*
msg153394 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-02-15 08:20
What more can be done to get it fixed?  I do not mind Python but GDB linking is broken due to it.  And while I can workaround it in GDB I am not used for workarounding one Free package in another Free package.  Free software has the advantage problems can be fixed at the right place.
msg153459 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2012-02-16 04:36
When I try to build with the attached patch on the 2.7 branch many (if not all) of the modules fail to build:

Failed to build these modules:
_bisect            _bsddb             _codecs_cn      
_codecs_hk         _codecs_iso2022    _codecs_jp      
_codecs_kr         _codecs_tw         _collections    
_csv               _ctypes            _ctypes_test    
_curses            _curses_panel      _elementtree    
_functools         _hashlib           _heapq          
_hotshot           _io                _json           
_locale            _lsprof            _multibytecodec 
_multiprocessing   _random            _socket         
_sqlite3           _ssl               _struct         
_testcapi          _tkinter           array           
audioop            binascii           bz2             
cmath              cPickle            crypt           
cStringIO          datetime           dbm             
fcntl              future_builtins    gdbm            
grp                itertools          linuxaudiodev   
math               mmap               nis             
operator           ossaudiodev        parser          
pyexpat            readline           resource        
select             spwd               strop           
syslog             termios            time            
unicodedata        zlib                               

I see several warnings in the build output:

*** WARNING: renaming "_codecs_iso2022" since importing it failed: build/lib.lin
ux-x86_64-2.7/_codecs_iso2022.so: undefined symbol: PyExc_ValueError
..
*** WARNING: renaming "linuxaudiodev" since importing it failed: build/lib.linux
-x86_64-2.7/linuxaudiodev.so: undefined symbol: Py_Py3kWarningFlag
...
*** WARNING: renaming "_tkinter" since importing it failed: build/lib.linux-x86_64-2.7/_tkinter.so: undefined symbol: _Py_ZeroStruct
msg153764 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2012-02-20 05:43
There was a bug for non-shared build of Python, thanks.

This patch is an improvement on its but it is not yet complete.
One should also replace "-Xlinker -export-dynamic" by "-Wl,--dynamic-list" so that linking application with libpython is no longer intrusive.

But I found out GDB does not need to be affected by this Bug.  When I use -fvisibility=hidden for GDB .c files then "-Xlinker -export-dynamic" has no longer any negative effect (GDB was using only "-Wl,--dynamic-list" now which works without libpython but not with libpython). Therefore I no longer mind much about this Bug.

Still at least:
1771024 libpython2.7.so.1.0
->
1757776 libpython2.7.so.1.0

Even with clean tree the upstream build generates for me:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               bsddb185
dl                 imageop            sunaudiodev

(so sunaudiodev.c was changed without any testing of it)
msg153810 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-02-20 21:19
The patch is fairly incomplete: it assumes that the compiler supports __attribute__((visibility())), but it really needs to check for that.

Also, there is no chance that this can go into Python 2.7; retargetting for 3.3.
msg237136 - (view) Author: Jan Kratochvil (jankratochvil) Date: 2015-03-03 13:35
It even crashes applications due to pollution of dynamic symbols namespace by application symbols as seen in:
  https://bugzilla.redhat.com/show_bug.cgi?id=1198158
History
Date User Action Args
2019-01-23 15:47:02vstinnersetnosy: + vstinner
2015-03-03 13:35:18jankratochvilsetmessages: + msg237136
2013-10-29 20:38:00tromeysetnosy: + tromey
2012-02-20 21:25:07Arfreversetnosy: + Arfrever
2012-02-20 21:19:08loewissetmessages: + msg153810
versions: + Python 3.3, - Python 2.7
2012-02-20 10:11:22pitrousetnosy: + loewis
2012-02-20 05:43:59jankratochvilsetfiles: - hidden.patch
2012-02-20 05:43:48jankratochvilsetfiles: + hidden4.patch

messages: + msg153764
2012-02-16 04:36:09meador.ingesetmessages: + msg153459
components: + Build, - Library (Lib)
stage: patch review
2012-02-15 08:20:57jankratochvilsetmessages: + msg153394
2012-01-12 18:45:41meador.ingesetnosy: + meador.inge
2012-01-12 18:18:53jankratochvilsetfiles: + hidden.patch
keywords: + patch
messages: + msg151146

versions: + Python 2.7, - Python 3.2
2010-10-15 14:26:51eric.araujosetnosy: + pitrou, eric.araujo
2010-10-15 10:16:24jankratochvilcreate