classification
Title: Python 3.7.0 wont compile with SSL Support 1.1.0 > alledged missing X509_VERIFY_PARAM_set1_host() support
Type: Stage:
Components: SSL Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Phillip Middleton, bkline, cayman, chris.jerdonek, christian.heimes, fthommen, hairygristle, kscheidegger, simon@simonfoley.net
Priority: normal Keywords:

Created on 2018-07-03 09:13 by simon@simonfoley.net, last changed 2019-07-08 17:43 by cayman.

Messages (11)
msg320947 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-03 09:13
when compiling Python 3.7.0 setup.py is reporting that the ssl module failed to compile due to missing support for X509_VERIFY_PARAM_set1_host()  despite it existing in rsa.h for all versions of OpenSSL 1.1.0.

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

In addition _ssl.o does actually compile.

The issue appears that _ssl is appearing in "missing", "self.failed", "self.failed_on_import"



setup.py

    366         if any('_ssl' in l
    367                for l in (missing, self.failed, self.failed_on_import)):
    368             print()
    369             print("Could not build the ssl module!")
    370             print("Python requires an OpenSSL 1.0.2 or 1.1 compatible "
    371                   "libssl with X509_VERIFY_PARAM_set1_host().")
    372             print("LibreSSL 2.6.4 and earlier do not provide the necessary "
    373                   "APIs, https://github.com/libressl-portable/portable/issues/381")
    374             print()

I havent had time to go through the code yet to find out where the error is gettng flagged and if its a associated with how I have compiled openssl i.e. I need a compilation flag to enabled  X509_VERIFY_PARAM_set1_host() support.
msg320951 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-03 09:30
The function definition should be in openssl/x509_vfy.h, not rsa.h. What's the output of configure on your system? You should see something like

./configure
...
checking for pkg-config... /usr/bin/pkg-config
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python
...

What's your platform and openssl version?
msg320955 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-03 10:43
Apologies, my bad you are correct the function was defined in x509_vfy.h

Im compiling on RHEL
Red Hat Enterprise Linux Server release 7.5 (Maipo)
I have tried Openssl from source versions;
openssl-1.0.2o  (this releaseis a mess and the folder structure has been altered)
openssl-1.1.0h
openssl-1.1.0

I havce tried 
Python-3.7.0
Python-3.6.3

checking for pkg-config... /usr/bin/pkg-config
checking for openssl/ssl.h in /usr/local/ssl... no
checking for openssl/ssl.h in /usr/lib/ssl... no
checking for openssl/ssl.h in /usr/ssl... no
checking for openssl/ssl.h in /usr/pkg... no
checking for openssl/ssl.h in /usr/local... no
checking for openssl/ssl.h in /usr... no
checking whether compiling and linking against OpenSSL works... no
checking for --with-ssl-default-suites... python

My details of Setup are;

SSL=/home/{my_home_folder}/openssl
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL) -lssl -lcrypto

I now spot that the Linker is having issues (-L)

libssl.so & cyypt.so   is in /home/{my_home_folder}/openssl
all the header files are in;

=/home/{my_home_folder}/openssl/include/openssl

including opensslconf.h
however not of the declarations have been commented out including any of the 

DEPRECATEDIN_1_0_0  etc etc
msg320960 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-03 11:18
configure is not able to find OpenSSL. You either have to configure Python to pick up your OpenSSL (./configure --with-openssl=/path/to/openssl) or install the OpenSSL developer packages. RHEL 7.5 comes with OpenSSL 1.0.2, so you are good.
msg321096 - (view) Author: simon (simon@simonfoley.net) Date: 2018-07-05 10:54
Thanks 

I have found teh root cause of the problem ...

--with-openssl=[my_dir]

The configure scripts has an assumption you are compiling against a binary packaged version of openssl and that there is a /lib folder under [my_dir]. This simply does not exist under any of the source code releases of openssl. So after I compiled the openssl source code I had to create the lib folder under my openssh build directory and symlink the *.so libraries there for the configure script to work

This is still an issue even if you edit Setup correctlty to compile the module.

>> This is a problem for people like me who are institutional users that have cross platform enterprise softwre deployment platforms (e.g. BladeLogic). There are restricted policies on what packages you can install on a server. In most cases especially for in house developed software) you need to build all dependencies seperatly and bundle them into a package (e.g. /opt RPM) that includes all required depencencies rather than rely on distribution library packages that are hard to manage at an Enterprise level and where you may be sharing the same OS.


To make the code more robust should it not 1st check under the root of [my_dir] before assuming [my_dir]/lib exests or at least report teh full path with the /lib added onto teh end of {my_dir} so you know where confiure has gone wrong ?

Is this not a fair expectation?

no lib folder
checking for openssl/ssl.h in /home/BD7046/openssl... no
checking whether compiling and linking against OpenSSL works... no

with lib folder
checking for openssl/ssl.h in /home/BC7046/openssl... yes
checking whether compiling and linking against OpenSSL works... yes


Thanks for all your help 
#PortingPerltoPython
msg321099 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-07-05 11:49
autoconf's --with-library options typically don't support build directories and work with installed versions only. The --with-openssl is no different. I suggest that you install OpenSSL to a local directory and then configure Python to fetch OpenSSL from that directory.

The multissltest script in Tools/ssl uses that approach to build Python with multiple OpenSSL versions.
msg322022 - (view) Author: Frank Thommen (fthommen) Date: 2018-07-20 15:47
The configure script doesn't work with a proper openssl installation either.  Even though there is a "lib" directory in the directory given to --with-openssl=<myssl>, libssl.so.1.1 isn't found, because there is still a "-L<myssl>/lib" missing in some of the compiler calls.

LDFLAGS="-L<myssl>/lib" ./configure --with-openssl=<myssl> is required, which seems somehow redundant.  Bug?
msg334701 - (view) Author: Kent Scheidegger (kscheidegger) Date: 2019-02-01 16:34
I was unable to get it working even with all the suggestions in this thread. I have a shared account on a system with only Python 2.7 and an old version of openssl. I have write access only to my user directory. I installed a new openssl in a local directory and pointed to it with both --with-openssl and LDFLAGS, as suggested. The configure step seems to work, but on make the libssl.so.1.1 still isn't found.

I fell back to Python 3.6. Same result. I fell back to 3.4. It finally worked.
msg335133 - (view) Author: Bob Kline (bkline) * Date: 2019-02-09 15:40
I had to add $HOME/usr/lib64 to LD_LIBRARY_PATH to get make to work.
msg339455 - (view) Author: David Chin (hairygristle) Date: 2019-04-04 17:11
OS: RHEL 6.8

I installed OpenSSL 1.1.1b from source into /usr/local. Because it's RHEL, the libs are in /usr/local/lib64 (as set up by default with the OpenSSL "make install") which the configure script does not seem to know about.

My workaround: before running configure for Python, set the environment variable:

LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"

Once that is set, configure manages to find the proper libssl:

checking for openssl/ssl.h in /usr/local... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
msg345177 - (view) Author: Phillip Middleton (Phillip Middleton) Date: 2019-06-11 02:30
I have the same issue installing v3.7.3 on RHEL6.8. The standard version came with openssl v1.0.1c, which would not configure. I installed openssl 1.0.2s in /usr/local and created a file /etc/profile.d/openssl.sh adding the following lines: 

# /etc/profile.d/openssl.sh
pathmunge /usr/local/openssl/bin

Exiting and relogging into the shell, the version returns openssl 1.0.2s 28 May 2019. 

As has been mentioned, there is no openssl distro out there that contains the filepath that either configure or make appears to expect. 

To get through configuration, I began with configure, modifying the ssldirs variable to /usr/local/openssl, and also repointing the following to the appropriate subdirs: 

17214     if ! $found; then
17215         OPENSSL_INCLUDES=
17216         for ssldir in $ssldirs; do
17217             { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openssl/ssl.h in $ssldir" >&5
17218 $as_echo_n "checking for openssl/ssl.h in $ssldir... " >&6; }
17219             if test -f "$ssldir/include/openssl/ssl.h"; then
17220                 OPENSSL_INCLUDES="-I$ssldir/include/openssl"
17221                 OPENSSL_LDFLAGS="-L$ssldir/lib"

This apparently wasn't sufficient for configure to recognize this openssl installation. 

Next, I uncommented and modified Modules/Setup.dist to reflect the openssl header and lib paths:
 
211 SSL=/usr/local/openssl
212 _ssl _ssl.c \
213         -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
214         -L$(SSL)/lib -lssl -lcrypto

That gets us to here with ./configure:

checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... no
checking for --with-ssl-default-suites... python

However using ./configure -with-openssl=/usr/local/openssl, configure returns:

checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python

So at least from configure's standpoint, I was able to get configure to pick up the openssl folder and its include/openssl which contains x509_vfy.h. Ok great. 

However make doesn't appear to respect these changes in configure - I get one of the typical variants of: 


*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_lzma                 _tkinter              _uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ssl


Following modules built successfully but were removed because they could not be imported:
_hashlib


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381


What I'd like to know is, what is the difference between what configure is looking for and in what cases would make not necessarily respect the confirmations of configure's checks?

Let me know if there are any dumps/logs you'd be interested in. I realize that in my case we're dealing with an older OS, but this issue doesn't seem restricted per se to that alone.
History
Date User Action Args
2019-07-08 17:43:08caymansetnosy: + cayman
2019-06-11 02:30:39Phillip Middletonsetnosy: + Phillip Middleton
messages: + msg345177
2019-04-04 17:11:19hairygristlesetnosy: + hairygristle
messages: + msg339455
2019-02-16 09:45:08chris.jerdoneksetnosy: + chris.jerdonek
2019-02-09 15:40:41bklinesetnosy: + bkline
messages: + msg335133
2019-02-01 16:34:48kscheideggersetnosy: + kscheidegger
messages: + msg334701
2018-07-20 15:47:53fthommensetnosy: + fthommen
messages: + msg322022
2018-07-05 11:49:17christian.heimessetmessages: + msg321099
2018-07-05 10:54:19simon@simonfoley.netsetmessages: + msg321096
2018-07-03 11:18:14christian.heimessetmessages: + msg320960
2018-07-03 10:43:01simon@simonfoley.netsetmessages: + msg320955
2018-07-03 09:30:39christian.heimessetmessages: + msg320951
2018-07-03 09:13:31simon@simonfoley.netcreate