classification
Title: Cygwin: Makefile does not install DLL import library
Type: behavior Stage:
Components: Extension Modules, Installation Versions: Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: erik.bray, masamoto
Priority: normal Keywords: patch

Created on 2016-06-23 15:20 by erik.bray, last changed 2016-09-23 08:38 by masamoto.

Files
File name Uploaded Description Edit
cygwin-ldlibrary.patch erik.bray, 2016-06-23 15:20 review
3.5-issue27374-distutils-build_ext.patch masamoto, 2016-09-16 03:01
Messages (6)
msg269116 - (view) Author: Erik Bray (erik.bray) * Date: 2016-06-23 15:20
When building Python on Cygwin, both a libpython-X.Y.dll and a
libpython-X.Y.dll.a are created (see https://cygwin.com/cygwin-ug-net/dll.html).

The latter is an "import library"
consisting of stubs for functions in the DLL so that it can be linked
to statically when building, for example, extension modules.

The odd bit is that in the altbininstall target if the
$(DLLLIBRARY) variable (which references the .dll itself) is defined then only it is installed, while $(LDLIBRARY) (which in this cases references the .dll.a import library) is *not* installed, except in $(prefix)/lib/pythonX.Y/config, which is not normally on the linker search path, or even included by
python-config --ldflags.

Therefore static linking to libpython fails, unless the search path is explicitly modified, or a symlink is created from $(prefix)/lib/pythonX.Y/config/libpython.dll.a to $(prefix)/lib.  In fact the Makefile already does the latter, again not if $(DLLLIBRARY) is defined.

In fact Cygwin's own package for Python manually creates the latter symlink in its install script.  But I would consider it a bug in Python's Makefile that it doesn't handle this.

I first asked on Python-dev about this back in April but got no reply (understandably) probably due to the lack of core devs who can comment on Cygwin :)
msg276398 - (view) Author: Masayuki Yamamoto (masamoto) * Date: 2016-09-14 08:25
Hello, Eric.
I tried to build the '_speedups' C extension module of simplejson (third party library) using the python built on cygwin. As you pointed out, the build of '_speedups' C extension module at the stage of the link failed.

build log:
Python3.5.2+ has be used. It has applyed patches to known issues.

$ ./configure --prefix=/opt/py35 --with-threads --with-dbmliborder=gdbm --with-system-ffi --with-system-expat
$ make && make altinstall
$ LANG=C /opt/py35/bin/pip3.5 install simplejson
(snip)
running build_ext
    building 'simplejson._speedups' extension
    creating build/temp.cygwin-2.6.0-i686-3.5
    creating build/temp.cygwin-2.6.0-i686-3.5/simplejson
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/py35/include/python3.5m -c simplejson/_speedups.c -o build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o
    gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o -L/opt/py35/lib/python3.5/config -L/opt/py35/lib -lpython3.5m -o build/lib.cygwin-2.6.0-i686-3.5/simplejson/_speedups.cpython-35m.dll
    /usr/lib/gcc/i686-pc-cygwin/5.4.0/../../../../i686-pc-cygwin/bin/ld: cannot find -lpython3.5m
    collect2: error: ld returned 1 exit status
    ***************************************************************************
    WARNING: The C extension could not be compiled, speedups are not enabled.
    Failure information, if any, is above.
    I'm retrying the build without the C extension now.
    ***************************************************************************
(snip)

In the log, -L option specified to "/opt/py35/lib/python3.5/config". But, the current python has "/opt/py35/lib/python3.5/config-3.5m" instead of "/opt/py35/lib/python3.5/config". And libpython3.5m.dll.a library exists in the config-3.5m. I think that to succeed building a C extension module if -L options is changed to "/opt/py35/lib/python3.5/config-3.5m". 
I'm going to try to find the place that add -L option at build time.
msg276402 - (view) Author: Masayuki Yamamoto (masamoto) * Date: 2016-09-14 08:56
I'm so sorry that I misspelled your name, Erik.
msg276669 - (view) Author: Masayuki Yamamoto (masamoto) * Date: 2016-09-16 03:01
I found -L option addition at Lib/distutils/command/build_ext.py:222.  This place is a process that finalize options for the build_ext command.
Comment in there: for extensions under Cygwin and AtheOS Python's library directory must be appended to library_dirs

After that I searched Makefile rule for installing $(LDLIBRARY) into "$(prefix)/lib/python3.5/config-3.5m" to look for way of simplifying code.  And I came across the $(LIBPL) configuration variable that is used by installation of the configuration for building library.
Hence I made a patch that adds to library_dirs the $(LIBPL) configuration variable instead of the "$(prefix)/lib/python$(VERSION)/config".  I succeeded to build '_speedups' C extension module of simplejson using new python that is applied this patch.
Erik, Would you be able to verify this patch?
Thanks

Build log:
$ patch -p1 <3.5-issue27374-distutils-build_ext.patch
$ ./configure --prefix=/opt/py35 --with-threads --with-dbmliborder=gdbm --with-system-ffi --with-system-expat
$ make touch && make -j2 && make altinstall
$ /opt/py35/bin/python3.5 -V
Python 3.5.2+
$ LANG=C /opt/py35/bin/pip3.5 install -v simplejson
(snip)
running build_ext
building 'simplejson._speedups' extension
creating build/temp.cygwin-2.6.0-i686-3.5
creating build/temp.cygwin-2.6.0-i686-3.5/simplejson
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/py35/include/python3.5m -c simplejson/_speedups.c -o build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o
gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-2.6.0-i686-3.5/simplejson/_speedups.o -L/opt/py35/lib/python3.5/config-3.5m -L/opt/py35/lib -lpython3.5m -o build/lib.cygwin-2.6.0-i686-3.5/simplejson/_speedups.cpython-35m.dll
running install_lib
(snip)
msg277225 - (view) Author: Erik Bray (erik.bray) * Date: 2016-09-22 13:55
Masayuki--concerning your above comments, I think this is similar to, if not the same as #13756
msg277254 - (view) Author: Masayuki Yamamoto (masamoto) * Date: 2016-09-23 08:38
I don't know why Cygwin packaging script (cygport) doesn't copy direct libpythonX.Ym.dll.a to /usr/lib.  Hence I wrote the patch that doesn't conflict with cygport script.

In current makefile rule for Cygwin, Install $(LDLIBRARY) into $(LIBPL).  Therefore I prefer symlink from $(LIBPL) than copying direct library file to $(LIBDIR).
What do you think of makefile rule addition that is same effect (symlink from LIBPL) as cygport script?
History
Date User Action Args
2016-09-23 08:38:29masamotosetmessages: + msg277254
2016-09-22 13:55:43erik.braysetmessages: + msg277225
2016-09-16 03:01:37masamotosetfiles: + 3.5-issue27374-distutils-build_ext.patch

messages: + msg276669
2016-09-14 08:56:24masamotosetmessages: + msg276402
2016-09-14 08:25:42masamotosetversions: + Python 3.5, Python 3.6
nosy: + masamoto

messages: + msg276398

components: + Extension Modules
2016-06-23 15:20:38erik.braycreate