classification
Title: Python 3.7.0 wont compile with SSL Support 1.1.0 > alledged missing X509_VERIFY_PARAM_set1_host() support
Type: Stage:
Components: SSL Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: bkline, chris.jerdonek, christian.heimes, fthommen, hairygristle, kscheidegger, simon@simonfoley.net
Priority: normal Keywords:

Created on 2018-07-03 09:13 by simon@simonfoley.net, last changed 2019-04-04 17:11 by hairygristle.

Messages (10)
msg320947 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-03 09:13
when compiling Python 3.7.0 setup.py is reporting that the ssl module failed to compile due to missing support for X509_VERIFY_PARAM_set1_host()  despite it existing in rsa.h for all versions of OpenSSL 1.1.0.

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

In addition _ssl.o does actually compile.

The issue appears that _ssl is appearing in "missing", "self.failed", "self.failed_on_import"



setup.py

    366         if any('_ssl' in l
    367                for l in (missing, self.failed, self.failed_on_import)):
    368             print()
    369             print("Could not build the ssl module!")
    370             print("Python requires an OpenSSL 1.0.2 or 1.1 compatible "
    371                   "libssl with X509_VERIFY_PARAM_set1_host().")
    372             print("LibreSSL 2.6.4 and earlier do not provide the necessary "
    373                   "APIs, https://github.com/libressl-portable/portable/issues/381")
    374             print()

I havent had time to go through the code yet to find out where the error is gettng flagged and if its a associated with how I have compiled openssl i.e. I need a compilation flag to enabled  X509_VERIFY_PARAM_set1_host() support.
msg320951 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-03 09:30
The function definition should be in openssl/x509_vfy.h, not rsa.h. What's the output of configure on your system? You should see something like

./configure
...
checking for pkg-config... /usr/bin/pkg-config
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python
...

What's your platform and openssl version?
msg320955 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-03 10:43
Apologies, my bad you are correct the function was defined in x509_vfy.h

Im compiling on RHEL
Red Hat Enterprise Linux Server release 7.5 (Maipo)
I have tried Openssl from source versions;
openssl-1.0.2o  (this releaseis a mess and the folder structure has been altered)
openssl-1.1.0h
openssl-1.1.0

I havce tried 
Python-3.7.0
Python-3.6.3

checking for pkg-config... /usr/bin/pkg-config
checking for openssl/ssl.h in /usr/local/ssl... no
checking for openssl/ssl.h in /usr/lib/ssl... no
checking for openssl/ssl.h in /usr/ssl... no
checking for openssl/ssl.h in /usr/pkg... no
checking for openssl/ssl.h in /usr/local... no
checking for openssl/ssl.h in /usr... no
checking whether compiling and linking against OpenSSL works... no
checking for --with-ssl-default-suites... python

My details of Setup are;

SSL=/home/{my_home_folder}/openssl
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL) -lssl -lcrypto

I now spot that the Linker is having issues (-L)

libssl.so & cyypt.so   is in /home/{my_home_folder}/openssl
all the header files are in;

=/home/{my_home_folder}/openssl/include/openssl

including opensslconf.h
however not of the declarations have been commented out including any of the 

DEPRECATEDIN_1_0_0  etc etc
msg320960 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-03 11:18
configure is not able to find OpenSSL. You either have to configure Python to pick up your OpenSSL (./configure --with-openssl=/path/to/openssl) or install the OpenSSL developer packages. RHEL 7.5 comes with OpenSSL 1.0.2, so you are good.
msg321096 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-05 10:54
Thanks 

I have found teh root cause of the problem ...

--with-openssl=[my_dir]

The configure scripts has an assumption you are compiling against a binary packaged version of openssl and that there is a /lib folder under [my_dir]. This simply does not exist under any of the source code releases of openssl. So after I compiled the openssl source code I had to create the lib folder under my openssh build directory and symlink the *.so libraries there for the configure script to work

This is still an issue even if you edit Setup correctlty to compile the module.

>> This is a problem for people like me who are institutional users that have cross platform enterprise softwre deployment platforms (e.g. BladeLogic). There are restricted policies on what packages you can install on a server. In most cases especially for in house developed software) you need to build all dependencies seperatly and bundle them into a package (e.g. /opt RPM) that includes all required depencencies rather than rely on distribution library packages that are hard to manage at an Enterprise level and where you may be sharing the same OS.


To make the code more robust should it not 1st check under the root of [my_dir] before assuming [my_dir]/lib exests or at least report teh full path with the /lib added onto teh end of {my_dir} so you know where confiure has gone wrong ?

Is this not a fair expectation?

no lib folder
checking for openssl/ssl.h in /home/BD7046/openssl... no
checking whether compiling and linking against OpenSSL works... no

with lib folder
checking for openssl/ssl.h in /home/BC7046/openssl... yes
checking whether compiling and linking against OpenSSL works... yes


Thanks for all your help 
#PortingPerltoPython
msg321099 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-05 11:49
autoconf's --with-library options typically don't support build directories and work with installed versions only. The --with-openssl is no different. I suggest that you install OpenSSL to a local directory and then configure Python to fetch OpenSSL from that directory.

The multissltest script in Tools/ssl uses that approach to build Python with multiple OpenSSL versions.
msg322022 - (view) Author: Frank Thommen (fthommen) Date: 2018-07-20 15:47
The configure script doesn't work with a proper openssl installation either.  Even though there is a "lib" directory in the directory given to --with-openssl=<myssl>, libssl.so.1.1 isn't found, because there is still a "-L<myssl>/lib" missing in some of the compiler calls.

LDFLAGS="-L<myssl>/lib" ./configure --with-openssl=<myssl> is required, which seems somehow redundant.  Bug?
msg334701 - (view) Author: Kent Scheidegger (kscheidegger) Date: 2019-02-01 16:34
I was unable to get it working even with all the suggestions in this thread. I have a shared account on a system with only Python 2.7 and an old version of openssl. I have write access only to my user directory. I installed a new openssl in a local directory and pointed to it with both --with-openssl and LDFLAGS, as suggested. The configure step seems to work, but on make the libssl.so.1.1 still isn't found.

I fell back to Python 3.6. Same result. I fell back to 3.4. It finally worked.
msg335133 - (view) Author: Bob Kline (bkline) * Date: 2019-02-09 15:40
I had to add $HOME/usr/lib64 to LD_LIBRARY_PATH to get make to work.
msg339455 - (view) Author: David Chin (hairygristle) Date: 2019-04-04 17:11
OS: RHEL 6.8

I installed OpenSSL 1.1.1b from source into /usr/local. Because it's RHEL, the libs are in /usr/local/lib64 (as set up by default with the OpenSSL "make install") which the configure script does not seem to know about.

My workaround: before running configure for Python, set the environment variable:

LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"

Once that is set, configure manages to find the proper libssl:

checking for openssl/ssl.h in /usr/local... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
History
Date User Action Args
2019-04-04 17:11:19hairygristlesetnosy: + hairygristle
messages: + msg339455
2019-02-16 09:45:08chris.jerdoneksetnosy: + chris.jerdonek
2019-02-09 15:40:41bklinesetnosy: + bkline
messages: + msg335133
2019-02-01 16:34:48kscheideggersetnosy: + kscheidegger
messages: + msg334701
2018-07-20 15:47:53fthommensetnosy: + fthommen
messages: + msg322022
2018-07-05 11:49:17christian.heimessetmessages: + msg321099
2018-07-05 10:54:19simon@simonfoley.netsetmessages: + msg321096
2018-07-03 11:18:14christian.heimessetmessages: + msg320960
2018-07-03 10:43:01simon@simonfoley.netsetmessages: + msg320955
2018-07-03 09:30:39christian.heimessetmessages: + msg320951
2018-07-03 09:13:31simon@simonfoley.netcreate