when compiling Python 3.7.0 setup.py is reporting that the ssl module failed to compile due to missing support for X509_VERIFY_PARAM_set1_host()  despite it existing in rsa.h for all versions of OpenSSL 1.1.0.

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

In addition _ssl.o does actually compile.

The issue appears that _ssl is appearing in "missing", "self.failed", "self.failed_on_import"



setup.py

    366         if any('_ssl' in l
    367                for l in (missing, self.failed, self.failed_on_import)):
    368             print()
    369             print("Could not build the ssl module!")
    370             print("Python requires an OpenSSL 1.0.2 or 1.1 compatible "
    371                   "libssl with X509_VERIFY_PARAM_set1_host().")
    372             print("LibreSSL 2.6.4 and earlier do not provide the necessary "
    373                   "APIs, https://github.com/libressl-portable/portable/issues/381")
    374             print()

I havent had time to go through the code yet to find out where the error is gettng flagged and if its a associated with how I have compiled openssl i.e. I need a compilation flag to enabled  X509_VERIFY_PARAM_set1_host() support.

The function definition should be in openssl/x509_vfy.h, not rsa.h. What's the output of configure on your system? You should see something like

./configure
...
checking for pkg-config... /usr/bin/pkg-config
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python
...

What's your platform and openssl version?

Apologies, my bad you are correct the function was defined in x509_vfy.h

Im compiling on RHEL
Red Hat Enterprise Linux Server release 7.5 (Maipo)
I have tried Openssl from source versions;
openssl-1.0.2o  (this releaseis a mess and the folder structure has been altered)
openssl-1.1.0h
openssl-1.1.0

I havce tried 
Python-3.7.0
Python-3.6.3

checking for pkg-config... /usr/bin/pkg-config
checking for openssl/ssl.h in /usr/local/ssl... no
checking for openssl/ssl.h in /usr/lib/ssl... no
checking for openssl/ssl.h in /usr/ssl... no
checking for openssl/ssl.h in /usr/pkg... no
checking for openssl/ssl.h in /usr/local... no
checking for openssl/ssl.h in /usr... no
checking whether compiling and linking against OpenSSL works... no
checking for --with-ssl-default-suites... python

My details of Setup are;

SSL=/home/{my_home_folder}/openssl
_ssl _ssl.c \
        -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
        -L$(SSL) -lssl -lcrypto

I now spot that the Linker is having issues (-L)

libssl.so & cyypt.so   is in /home/{my_home_folder}/openssl
all the header files are in;

=/home/{my_home_folder}/openssl/include/openssl

including opensslconf.h
however not of the declarations have been commented out including any of the 

DEPRECATEDIN_1_0_0  etc etc

configure is not able to find OpenSSL. You either have to configure Python to pick up your OpenSSL (./configure --with-openssl=/path/to/openssl) or install the OpenSSL developer packages. RHEL 7.5 comes with OpenSSL 1.0.2, so you are good.

Thanks 

I have found teh root cause of the problem ...

--with-openssl=[my_dir]

The configure scripts has an assumption you are compiling against a binary packaged version of openssl and that there is a /lib folder under [my_dir]. This simply does not exist under any of the source code releases of openssl. So after I compiled the openssl source code I had to create the lib folder under my openssh build directory and symlink the *.so libraries there for the configure script to work

This is still an issue even if you edit Setup correctlty to compile the module.

>> This is a problem for people like me who are institutional users that have cross platform enterprise softwre deployment platforms (e.g. BladeLogic). There are restricted policies on what packages you can install on a server. In most cases especially for in house developed software) you need to build all dependencies seperatly and bundle them into a package (e.g. /opt RPM) that includes all required depencencies rather than rely on distribution library packages that are hard to manage at an Enterprise level and where you may be sharing the same OS.


To make the code more robust should it not 1st check under the root of [my_dir] before assuming [my_dir]/lib exests or at least report teh full path with the /lib added onto teh end of {my_dir} so you know where confiure has gone wrong ?

Is this not a fair expectation?

no lib folder
checking for openssl/ssl.h in /home/BD7046/openssl... no
checking whether compiling and linking against OpenSSL works... no

with lib folder
checking for openssl/ssl.h in /home/BC7046/openssl... yes
checking whether compiling and linking against OpenSSL works... yes


Thanks for all your help 
#PortingPerltoPython

autoconf's --with-library options typically don't support build directories and work with installed versions only. The --with-openssl is no different. I suggest that you install OpenSSL to a local directory and then configure Python to fetch OpenSSL from that directory.

The multissltest script in Tools/ssl uses that approach to build Python with multiple OpenSSL versions.

The configure script doesn't work with a proper openssl installation either.  Even though there is a "lib" directory in the directory given to --with-openssl=<myssl>, libssl.so.1.1 isn't found, because there is still a "-L<myssl>/lib" missing in some of the compiler calls.

LDFLAGS="-L<myssl>/lib" ./configure --with-openssl=<myssl> is required, which seems somehow redundant.  Bug?

I was unable to get it working even with all the suggestions in this thread. I have a shared account on a system with only Python 2.7 and an old version of openssl. I have write access only to my user directory. I installed a new openssl in a local directory and pointed to it with both --with-openssl and LDFLAGS, as suggested. The configure step seems to work, but on make the libssl.so.1.1 still isn't found.

I fell back to Python 3.6. Same result. I fell back to 3.4. It finally worked.

I had to add $HOME/usr/lib64 to LD_LIBRARY_PATH to get make to work.

OS: RHEL 6.8

I installed OpenSSL 1.1.1b from source into /usr/local. Because it's RHEL, the libs are in /usr/local/lib64 (as set up by default with the OpenSSL "make install") which the configure script does not seem to know about.

My workaround: before running configure for Python, set the environment variable:

LDFLAGS="-L/usr/local/lib -L/usr/local/lib64 -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/local/lib64"

Once that is set, configure manages to find the proper libssl:

checking for openssl/ssl.h in /usr/local... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes

I have the same issue installing v3.7.3 on RHEL6.8. The standard version came with openssl v1.0.1c, which would not configure. I installed openssl 1.0.2s in /usr/local and created a file /etc/profile.d/openssl.sh adding the following lines: 

# /etc/profile.d/openssl.sh
pathmunge /usr/local/openssl/bin

Exiting and relogging into the shell, the version returns openssl 1.0.2s 28 May 2019. 

As has been mentioned, there is no openssl distro out there that contains the filepath that either configure or make appears to expect. 

To get through configuration, I began with configure, modifying the ssldirs variable to /usr/local/openssl, and also repointing the following to the appropriate subdirs: 

17214     if ! $found; then
17215         OPENSSL_INCLUDES=
17216         for ssldir in $ssldirs; do
17217             { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openssl/ssl.h in $ssldir" >&5
17218 $as_echo_n "checking for openssl/ssl.h in $ssldir... " >&6; }
17219             if test -f "$ssldir/include/openssl/ssl.h"; then
17220                 OPENSSL_INCLUDES="-I$ssldir/include/openssl"
17221                 OPENSSL_LDFLAGS="-L$ssldir/lib"

This apparently wasn't sufficient for configure to recognize this openssl installation. 

Next, I uncommented and modified Modules/Setup.dist to reflect the openssl header and lib paths:
 
211 SSL=/usr/local/openssl
212 _ssl _ssl.c \
213         -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
214         -L$(SSL)/lib -lssl -lcrypto

That gets us to here with ./configure:

checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... no
checking for --with-ssl-default-suites... python

However using ./configure -with-openssl=/usr/local/openssl, configure returns:

checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
checking for --with-ssl-default-suites... python

So at least from configure's standpoint, I was able to get configure to pick up the openssl folder and its include/openssl which contains x509_vfy.h. Ok great. 

However make doesn't appear to respect these changes in configure - I get one of the typical variants of: 


*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_lzma                 _tkinter              _uuid
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ssl


Following modules built successfully but were removed because they could not be imported:
_hashlib


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381


What I'd like to know is, what is the difference between what configure is looking for and in what cases would make not necessarily respect the confirmations of configure's checks?

Let me know if there are any dumps/logs you'd be interested in. I realize that in my case we're dealing with an older OS, but this issue doesn't seem restricted per se to that alone.

I'm having a similar problem.  I'm trying to compile on Red Hat 6.9 using a locally installed OpenSSL library.  I've tried everything here (I think) and I still get this error:

checking for openssl/ssl.h in /home/mf/dp/mpma/ghantousm/apptron/local/... yes   
checking whether compiling and linking against OpenSSL works... no               
checking for --with-ssl-default-suites... python                                 

I've ensured the libssl.* files are all present or linked to in /home/mf/dp/mpma/ghantousm/apptron/local/lib and .../lib64, the include files are present, the OpenSSL build seemed fine, and I compiled version 1.1.1d of openssl.  I've tried setting LDFLAGS and rpath and still no dice; the setup.py module doesn't seem to have any hard-coded paths so I'm really at a loss to understand what is wrong.  I've tried both versions 3.8.0 and 3.7.3 of python with the --with-openssl=/home/mf/dp/mpma/ghantousm/apptron/local option set, and version 3.6.8 (which doesn't accept any ssl related options).

This bug seems to have been preserved for a number of versions, so I'd like to help squash it if I can, but not sure where to go to from here.  I notice that some of the reports here suggest that they've had success with work-arounds, but as I stated none of them seem to work for me and I'm unable to compile with ssl support.

Python uses https://www.gnu.org/software/autoconf-archive/ax_check_openssl.html to detect and check for OpenSSL. Please check config.log for any errors. The log file will contain an error message. 

How did you compile OpenSSL? Did you configure the sources with "./config shared" ?

config.log:
...
configure:17184: checking whether compiling and linking against OpenSSL works
Trying link with OPENSSL_LDFLAGS=-L/home/heimes/dev/python/multissl/openssl/1.1.0l/lib; OPENSSL_LIBS=-lssl -lcrypto; OPENSSL_INCLUDES=-I/home/heimes/dev/python/multissl/openssl/1.1.0l/include
configure:17206: gcc -pthread -o conftest  -I/home/heimes/dev/python/multissl/openssl/1.1.0l/include   -L/home/heimes/dev/python/multissl/openssl/1.1.0l/lib conftest.c -lssl -lcrypto -lcrypt -lpthread -ldl  -lutil -lm >&5
configure:17206: $? = 0
configure:17208: result: yes
configure:17231: checking for X509_VERIFY_PARAM_set1_host in libssl
configure:17260: gcc -pthread -o conftest  -I/home/heimes/dev/python/multissl/openssl/1.1.0l/include   -L/home/heimes/dev/python/multissl/openssl/1.1.0l/lib conftest.c -lssl -lcrypto -lcrypt -lpthread -ldl  -lutil -lm >&5
configure:17260: $? = 0
configure:17271: result: yes
configure:17288: checking for --with-ssl-default-suites
configure:17317: result: python
...

make:
...
building '_ssl' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -I./Include/internal -I/home/heimes/dev/python/multissl/openssl/1.1.0l/include -I./Include -I. -I/usr/local/include -I/home/heimes/dev/python/cpython/Include -I/home/heimes/dev/python/cpython -c /home/heimes/dev/python/cpython/Modules/_ssl.c -o build/temp.linux-x86_64-3.9/home/heimes/dev/python/cpython/Modules/_ssl.o
gcc -pthread -shared build/temp.linux-x86_64-3.9/home/heimes/dev/python/cpython/Modules/_ssl.o -L/home/heimes/dev/python/multissl/openssl/1.1.0l/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.linux-x86_64-3.9/_ssl.cpython-39-x86_64-linux-gnu.so
...

./python
>>> import _ssl
>>> _ssl._OPENSSL_API_VERSION
(1, 1, 0, 12, 15)

Have you also tried $ yum install openssl-devel ?

That should work without requiring to compile openssl from source, unless you want a later version, which isn't advisable to install system-wide, as it could break other things.

Thanks for your responses. Yum is not an option as I'm on a machine without root privileges.  

OK, so I have to be a bit contrite here, I tried everything you suggested, but in the end the solution goes back to this comment, and the LDFLAGS environment variable:

https://bugs.python.org/issue34028?@ok_message=msg%20355232%20cr%C3%A9%C3%A9%0Aissue%2034028%20message_count%2C%20type%2C%20versions%2C%20curieux%2C%20nosy_count%2C%20messages%20modifi%C3%A9%28s%29%20avec%20succ%C3%A8s&@template=item#msg339455

I think I was using the wrong compiler when I tried that solution the first time.  So I'm sorry about that.  

But while we're here, I may as well ask if there isn't another way to do this, without having recourse to the environment variable.  Is there a way to specify these flags using just the configure script?  Based on something I read somewhere on the web, I generally prefer to steer clear of setting environment variables, though perhaps I needn't be so reluctant.  Incidentally, the environment variables OPENSSL_LDFLAGS etc as suggested in config.log (thanks for that, I wouldn't have thought to look there) do not appear to work.

--with-openssl only sets the header location for the pre-processor and library path for the dynamic linker. It does *not* affect the search and lookup paths of the dynamic loader! If you have installed OpenSSL in a non-standard location then you need to tell the dynamic loader how to load the shared libraries. This can be done in three ways:

* configure the dynamic loader globally in /etc/ld.so.conf (not advised for custom OpenSSL)
* Add a rpath to the ELF header of _ssl and _hashlib extensions at link time. This can be accomplished by setting LD_RUN_PATH=/path/to/lib or LDFLAGS=-Wl,-rpath,/path/to/lib
* Add the shared library directory to the runtime lookup path of ld.so by setting LD_LIBRARY_PATH env var.

hello,
I ran over this same problem on Ubuntu 14.04

As per
https://github.com/pyenv/pyenv/wiki/Common-build-problems

"Python 3.7.0 will not compile on RHEL6 because it requires OpenSSL 1.0.2 or 1.1 and RHEL6 provides 1.0.1e"

openssl version confirms this to be the case on Ubuntu 14.04

"On Ubuntu 14.04 on Dreamhost, an extra flag is required for Python 3.7+:
First, follow these instructions: https://help.dreamhost.com/hc/en-us/articles/360001435926-Installing-OpenSSL-locally-under-your-username"

then I ran: 
./configure --with-ensurepip=yes CFLAGS="-I$HOME/openssl/include" LDFLAGS="-L$HOME/openssl/lib"

after that python3.7 was correct

hope that helps, kind regards
Joaquin

I couldn't get this to work at all, python 3.7 compiled fine, but at the end it reports:

'''
*** WARNING: renaming "_ssl" since importing it failed: libssl.so.1.1: cannot open shared object file: No such file or directory
*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.1.1: cannot open shared object file: No such file or directory

Python build finished successfully!

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381
'''

But in the end I got it to work like this:

----- install_python3.7.sh ---- 
#!/bin/bash
set -euo pipefail

mkdir /tmp/openssl
cd /tmp/openssl
wget https://www.openssl.org/source/openssl-1.1.1a.tar.gz
tar -xvf openssl-1.1.1a.tar.gz
cd openssl-1.1.1a
./config --prefix=/usr/local/openssl1.1.1 --openssldir=/usr/local/openssl1.1.1
make
make install
rm -rf /tmp/opensll

echo /usr/local/openssl1.1.1/lib > /etc/ld.so.conf.d/openssl1.1.1.conf
ldconfig 

mkdir /tmp/python37
wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
tar xfz Python-3.7.3.tgz
cd Python-3.7.3
./configure --with-ensurepip=yes --with-openssl=/usr/local/openssl1.1.1 CFLAGS="-I/usr/local/openssl1.1.1/include" LDFLAGS="-L/usr/local/openssl1.1.1/lib" CXX=/usr/bin/g++
make
make install
rm -rf /tmp/python37

ldconfig 
--------------------

This important pieces are:

echo /usr/local/openssl1.1.1/lib > /etc/ld.so.conf.d/openssl1.1.1.conf
ldconfig 

to make it find the .so to load it at runtime and 

./configure --with-ensurepip=yes --with-openssl=/usr/local/openssl1.1.1 CFLAGS="-I/usr/local/openssl1.1.1/include" LDFLAGS="-L/usr/local/openssl1.1.1/lib" CXX=/usr/bin/g++

specifying the non-standard openssl-version specifically.

That's a very dangerous trick and I advise against it. You are modifying the global linker path and inject custom OpenSSL libraries into it. This may affect and disrupt other programs or OS core tools.

Instead compile the _ssl and _hashlib module with rpath, e.g. LD_RUN_PATH. You also don't have to modify CFLAGS or LDFLAGS. --with-openssl does that for you.

$ export LD_RUN_PATH=/home/heimes/dev/python/multissl/openssl/1.1.1f/lib
$ ./configure --with-openssl=/home/heimes/dev/python/multissl/openssl/1.1.1f -C
$ make
$ unset LD_RUN_PATH
$ ldd build/lib.linux-x86_64-3.9/_ssl.cpython-39-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffc124eb000)
        libssl.so.1.1 => /home/heimes/dev/python/multissl/openssl/1.1.1f/lib/libssl.so.1.1 (0x00007fd3d7cab000)
        libcrypto.so.1.1 => /home/heimes/dev/python/multissl/openssl/1.1.1f/lib/libcrypto.so.1.1 (0x00007fd3d7974000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd3d791c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fd3d7753000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fd3d774c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fd3d7d8e000)

Python 3.10 contains various improvements that make it easier to compile and link Python with a custom OpenSSL installation. You can find more information in ticket bpo-43466.