Since 3.7 python depends on OpenSSL>=1.0.2 - this makes it hard to compile Python with SSL on older (yet still vendor supported) linux distributions.

It's easy to compile CPython even on old distributions like RHEL5, RHEL6, Ubuntu14.04 etc. except for ssl module.

Let's add an option to compile SSL staticly (--with-openssl-static) to make it easy to compile Python with SSL on systems with OpenSSL<1.0.2 as you usually don't want to install newer openssl as system libary nor mess with rpath/set LD_LIBRARY_PATH every time you run python.

When --with-openssl-static is not passed to ./configure everything should behave like before.

Installing CPython including ssl on system as old as RHEL5 with this option would be as easy as (after installing required build dependencies from rhel5 repositories and libffi(-devel) libraries):

wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz
tar xf openssl-1.0.2t.tar.gz
cd openssl-1.0.2t
./config --openssldir=/etc/pki/tls -fPIC
make

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar xf Python-3.7.5.tgz
cd Python-3.7.5
./configure --with-openssl-static=path_to_just_compiled_ssl --prefix=prefix_path
make
make install

Since this is a request for a new feature, I'm bumping versions to 3.9 and newer.

There is no need to add more configure flags to build Python with a custom OpenSSL installation. It's sufficient to build OpenSSL as a shared library and run Python's configure with --with-openssl=/path/to/installed/openssl. Then it's just a matter to configure libdl to load the correct libraries, e.g. with a custom rpath. I like to use LD_RUN_PATH to add an rpath to the ELF header of the _ssl and _hashlib module's shared library. Or you can use Tools/ssl/multissl.py to automate this.

This approach works fine on RHEL 6:

wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz
tar -xzf openssl-1.0.2t.tar.gz
pushd openssl-1.0.2t
./config --openssldir=/etc/pki/tls --prefix=/tmp/102t shared
make
make install
popd

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar -xzf Python-3.7.5.tgz
pushd Python-3.7.5
export LD_RUN_PATH=/tmp/102t/lib
./configure --with-openssl=/tmp/102t/ -C
unset LD_RUN_PATH

ldd build/lib.linux-x86_64-3.7/_ssl.cpython-37m-x86_64-linux-gnu.so 
        linux-vdso.so.1 =>  (0x00007ffcefbb7000)
        libssl.so.1.0.0 => /tmp/102t/lib/libssl.so.1.0.0 (0x00007f4c586a7000)
        libcrypto.so.1.0.0 => /tmp/102t/lib/libcrypto.so.1.0.0 (0x00007f4c5826a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4c58047000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4c57cb3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f4c57aae000)
        /lib64/ld-linux-x86-64.so.2 (0x00005639623f7000)

./python 
Python 3.7.5 (default, Nov 15 2019, 04:19:28) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2t  10 Sep 2019'


I'm opposing this PR because it's an unnecessary extension that increases our testing, documentation and support burden. We would have to maintain the feature for at least 5 to 10 years and ensure that it keeps working on all operating systems (also BSD, macOS, AIX, ...) and future versions of OpenSSL.

Fair enough.

Btw. I picked versions 3.7 and up to show this feature should eventually be backported to older CPython versions that depend on OpenSSL >= 1.0.2. 

This feature is to avoid relying on rpath and relying on a seperate installation to be installed in specific location in your target system. Also there is no Tools/ssl/multissl.py in current cpython master.

I believe it's much preferably for administrators to use static linking in this case, but if you think it's not worth it, that's fine. Please close the related PR in the github with your explanation. Thank you.

However, I'm still convinced many would appreciate adding this ./configure option:

https://github.com/pyenv/pyenv/issues/1184#issuecomment-403149437
https://joshspicer.com/python37-ssl-issue
https://superuser.com/questions/1428109/install-python-3-7-3-from-source-with-ssl-failed
https://raspberrypi.stackexchange.com/questions/88150/how-to-install-python-3-7-with-ssl
https://stackoverflow.com/questions/53543477/building-python-3-7-1-ssl-module-failed
https://help.dreamhost.com/hc/en-us/articles/115000702772-Installing-a-custom-version-of-Python-3#3.7.1
https://stackoverflow.com/questions/55928393/compile-python-3-6-statically-with-openssl
https://readthisblog.net/2018/08/13/til-how-to-build-python-3-7-with-statically-linked-libssl-and-libcrypto/
https://gist.github.com/eddy-geek/9604982

If you look at the associated PR you will see it should work fine, and be useful, on other POSIX systems as well. I can test on *BSD if necessary.

Only extra depedency of --with-openssl-static compared to --with-openssl is an assumption of existing files libssl.a libcrypto.a and OpenSSL has provided these files (when compiled static) since the very beginning despite somewhat frequent ABI changes. Option --with-openssl-static works with current libressl as well.

I'm drawing a different conclusion from these posts. It sounds to me like people want a simple way to install a new version of Python on old distros and have a working ssl module. Static linking is one possible way to archive the goal. I argue that it is not necessarily the best way, especially in the light of OpenSSL 3.0.0, dynamic engines and the new crypto providers.

In ideal world OpenSSL would provide stable ABI just like the other libraries Python depends on. That would be, unarguably, the best way to achieve that goal. Agreed.

Aferall frequent OpenSSL ABI breakages are the reason why apple switched to their own cryptography ABI (https://developer.apple.com/library/archive/documentation/Security/Conceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCommunicationAPIs.html):

"""
Although OpenSSL is commonly used in the open source community, it doesn’t provide a stable API from version to version. For this reason, the programmatic interface to OpenSSL is deprecated in macOS and is not provided in iOS. Use of the Apple-provided OpenSSL libraries by apps is strongly discouraged.

To ensure compatibility, if your app depends on OpenSSL, you should compile it yourself and statically link a known version of OpenSSL into your app. Such use works on both iOS and macOS.
"""

However, OpenSSL does not provide stable ABI and Python should deal with that as nicely as possible. If you can think of a better real world solution than static linking OpenSSL please share and let's do it!

> In ideal world OpenSSL would provide stable ABI just like the other libraries Python depends on. That would be, unarguably, the best way to achieve that goal. Agreed.

OpenSSL provides a stable API and stable ABI already. The fact is well documented at https://www.openssl.org/policies/releasestrat.html. Python is not compatible with ancient, unsupported OpenSSL versions 1.0.1 and earlier because Python requires features that were introduced in OpenSSL 1.0.2.

> However, OpenSSL does not provide stable ABI and Python should deal with that as nicely as possible. If you can think of a better real world solution than static linking OpenSSL please share and let's do it!

That's a) incorrect, b) not related to this issue.

You are trying to solve a problem by dictating the solution at the same time. Could you please start with a high level explanation of the problem and let us figure out a viable solution?

Christian, I don't want to argue.

Feel free to close both this bug and the PR as "enhancement rejected". Adding one ./configure option is not really worth arguing over, I can always compile python as I need. I just tried to solve the same problem for others because other people also want a simple way to install a new version of Python on old distros and have a working ssl module. 

If my contribution is not welcome it's fine.

However, I have a feeling the reason for the rejection is not technical because after I stated I am happy to help solve the problem in other ways ("If you can think of a better real world solution than static linking OpenSSL please share and let's do it!") you literally twisted my words 180 degrees ("You are trying to solve a problem by dictating the solution at the same time."). This is very inappropriate.

Not to butt in too much, but I have a related use case that would benefit from being able to statically link to OpenSSL: I have an environment in which dynamic modules are acceptable, but where the entire Python install benefits from being relocatable and used on different hosts with the same *basic* state (same glibc, etc). OpenSSL isn't one of those pieces of basic state (for bad reasons, but ones that I can't control), so having a Python distribution that links it statically would save me some amount of complexity.

I realize this introduces a significant support burden, and that dealing with myraid OpenSSL configurations is already a pain. So I'm content to shoulder this as my own local build patch -- I'm only following up to note another use case that might benefit from a full-fledged `configure` option, should this ever get re-visited.

I introduced several build improvements for better support of custom OpenSSL builds in bpo-43466. The issue mentions a new, undocumented, and unsupported hack to create a shared extension modules with statically linked OpenSSL. The Modules/Setup.local trick works with current Python versions, too. Both the hack and the trick hide symbols, too.

I vetoed against official support for static linking because I simply don't have capacity to test and support another build flavor.

William, I recommend that you do NOT read commit bacefbf41461ab703b8d561f0e3d766427eab367 and do NOT use the undocumented hack. Have fun! :)

Cheers! No promises about not using the hack, but I *will* promise not to complain if it doesn't work for me :-)

For the record, this would have been solved more than a year ago already.

When this change was proposed more than a year ago it was rejected with "There is no need to add more configure flags to build Python with a custom OpenSSL installation. " yet now it's ok to add a new option --with-openssl-rpath https://bugs.python.org/issue43466 ?

And the first comment there, from python core dev nonetheless, is suggesting static linking as well. Emm... this would have been solved year and half ago. I would be happy to completely drop my proposed (and approved on gihub) changes and implement it in a different way.

The maintainer's attitude as demonstrated here can be really harmful in open-source projects (many of us still remember eglibc fork back in the day) but fortunately this is the first time I noticed such attitude among python developers.

Importantly the issue is resolved now (did it take a request from IBM's customer to get this implemented ;-) ?) and hopefully a lesson learnt and Christian will be more welcoming and less judgemental of outsiders' contributions.

I don't think this is a productive or polite response.

If you read the issue linked, you'll note that the other flag added
(--with-openssl-rpath) is in furtherance of any *already* supported linking
scenario (dynamic, with a non-system-default OpenSSL). The proposed change
here is for an *new* linking scenario, i.e. statically linking OpenSSL.

I don't blame the maintainers for not wanting to add another build configuration
to their plate, especially for something like OpenSSL. It's up to us as end
users to make those accommodations on our own, knowing that what we're
doing isn't tested or directly supported. The resolution above is, in my
book, the ideal one.

Best,
William

On Sat, Apr 03, 2021 at 04:45:34PM +0000, Lukas Vacek wrote:
>
> Lukas Vacek <lucas.vacek@gmail.com> added the comment:
>
> For the record, this would have been solved more than a year ago already.
>
> When this change was proposed more than a year ago it was rejected with "There is no need to add more configure flags to build Python with a custom OpenSSL installation. " yet now it's ok to add a new option --with-openssl-rpath https://bugs.python.org/issue43466 ?
>
> And the first comment there, from python core dev nonetheless, is suggesting static linking as well. Emm... this would have been solved year and half ago. I would be happy to completely drop my proposed (and approved on gihub) changes and implement it in a different way.
>
> The maintainer's attitude as demonstrated here can be really harmful in open-source projects (many of us still remember eglibc fork back in the day) but fortunately this is the first time I noticed such attitude among python developers.
>
> Importantly the issue is resolved now (did it take a request from IBM's customer to get this implemented ;-) ?) and hopefully a lesson learnt and Christian will be more welcoming and less judgemental of outsiders' contributions.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue38794>
> _______________________________________

Lukas,
no, some guy called Bill approached me at our last Illuminati meeting in Hollow Earth. He asked me to implement static linking for his next-gen secret brain chip. In exchange for the favor I was allowed to ride his T-Rex. True story!

Thanks, William! :)

Thanks for the explanation of the secret workings behind the scenes.

What a productive and polite response.