ssl/hashlib: Add configure option to set or auto-detect rpath to OpenSSL libs #87632
Comments
|
Python's configure script has the option --with-openssl. It sets a path to a custom OpenSSL installation. Internally it provides OPENSSL_INCLUDES, OPENSSL_LIBS, and OPENSSL_LDFLAGS. The setup.py script turns the variables into include_dirs, library_dirs, and libraries arguments for _ssl and _hashlib extension modules. However neither --with-openssl nor setup.py sets a custom runtime library path (rpath). This makes it confusing and hard for users to use a custom OpenSSL installation. They need to know that a) they have to take care of rpath on the first place, and b) how to set an rpath at compile or runtime. Without an rpath, the dynamic linker either fails to locate libssl/libcrypto or load system-provided shared libraries. Ticket bpo-34028 contains examples of user issues. I propose to include a new option to make it easier for users to use a custom build of OpenSSL: --with-openssl-rpath=<DIR|auto|no> no (default): don't set an rpath The option will only affect the rpath of _ssl and _hashlib modules. The default value "no" is fully backwards compatible with 3.9 and earlier. |
tiran
added
topic-SSL
topic-installation
type-feature
tiran
added
topic-SSL
topic-installation
type-feature
|
I have a suggestion (I am ok with also doing both). We should add also a flag that allows to statically compile openssl into the extension modules (if the .a are available) so there will be no shared object dependency. This allows us to have artifacts that are more self contained (regarding OpenSSL), partially eliminating the problem. We could also go a step ahead and also hide the OpenSSL symbols from the dynamic table, so they don't collide if another shared object is already loaded in the future. |
$ tar -xzf openssl-1.1.1j.tar.gz
$ pushd openssl-1.1.1j
$ ./config \
--prefix=/home/heimes/dev/python/custom-openssl \
--openssldir=\
$(find /etc/ -name openssl.cnf -quit -printf "%h" 2>/dev/null)
$ make
$ make install_sw
$ popd$ pushd cpython
$ ./configure \
--with-openssl=/home/heimes/dev/python/custom-openssl \
--with-openssl-rpath=auto
$ make
$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffde07bc000)
libssl.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libssl.so.1.1 (0x00007f937493a000)
libcrypto.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libcrypto.so.1.1 (0x00007f937465a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9374608000)
libc.so.6 => /lib64/libc.so.6 (0x00007f937443d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f9374436000)
/lib64/ld-linux-x86-64.so.2 (0x00007f93749ff000)
$ ./python
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1j 16 Feb 2021'
>>> import urllib.request
>>> r = urllib.request.urlopen("https://www.python.org")
^^^ No cert validation error! |
|
I would rather not support static linking. OpenSSL uses dynamic linking by default. Static linking is problematic for dynamic engine support. This is going to become an even bigger issue with OSSL providers in OpenSSL 3.0.0. I don't know yet how well OpenSSL 3.0.0 will support static linking. Our macOS and Windows builds are now dynamically linked, too. At least Windows builds used to be statically linked. Dynamic linked OpenSSL has a major benefit: It allows users to update OpenSSL without re-compiling Python. |
As long as OpenSSL provide a .a file (they do) you can always static link. That is a decision of how you integrate the library.
Not sure I follow. What's the problem here? The advantage of static linking here will be to not have a dependency on the shared object, which can be quite beneficial.
Yeah, I agree. That's what should always be the default. But there are still advantages regarding static linking. |
|
Note that I am not proposing to statically link the extensions into the core, I am proposing to statically link openssl into the extensions, so the symbols are in the text segment of the extension. |
The problem is that some features are not baked into the .a files. They are always provided as shared libraries. This included OpenSSL engine extensions such as AFALG engine or external engines like p11-kit, OpenSC, or others. OpenSSL 3.0.0 moves some features into external OSSL provider libraries, for example legacy crypto algorithms. I have not figured out how much functionality we woud loose without engines and external OSSL providers. https://www.openssl.org/docs/manmaster/man3/OSSL_PROVIDER.html |
Oh, I see. So when using modern versions of openssl what shared objects do we expect to be in the NEEDED section? Right now, I see this in python3.8 in Ubuntu with OpenSSL 1.1.1: ❯ ldd /usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so |
|
It's very much the same for OpenSSL 3.0.0: libssl.so and libcrypto.so. $ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffffa3cc000)
libssl.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libssl.so.3 (0x00007f1ab0b66000)
libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007f1ab06b1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1ab065f000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1ab0494000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f1ab048d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1ab0c55000)The external engines and OSSL providers are external plugins. They are very much akin to Python's extension modules. OpenSSL loads them with dlopen(), dlsym()s an init function and finally calls the init function. It uses either RTLD_NOW or RTLD_NOW | RTLD_GLOBAL dlopen() flags. The engines and OSSL providers depend on libcrypto.so. AFAIK this won't play will with static linking. $ ldd ../multissl/openssl/3.0.0-alpha12/lib/engines-3/afalg.so
linux-vdso.so.1 (0x00007fffa417d000)
libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007fbcb3c75000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fbcb3c3e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbcb3c1c000)
libc.so.6 => /lib64/libc.so.6 (0x00007fbcb3a51000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbcb4133000)
$ ldd ../multissl/openssl/3.0.0-alpha12/lib/ossl-modules/legacy.so
linux-vdso.so.1 (0x00007ffd3ccc0000)
libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007f5524f36000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f5524eff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5524edd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f5524d12000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5525419000) |
It should not be any problem as long as we expose the SSL symbols in the dynamic table of the extension (this happens by default). The only nuisance would be that users will still need those shared objects around so they can be dlopened by the extension. |
|
Pablo, Would you mind if I regenerate configure with 2.69 and we stick with stable version for a little bit longer? |
Please, go ahead. I need to remember then to use a docker container or something like that for the release then. What's the problem that you are experiencing? |
|
Should the rpath be set on OpenSSL, on the Python ssl module, or on the Python executable?
If most users get it wrong, why not using "auto: auto-detect rpath from OPENSSL_LDFLAGS (--with-openssl or pkg-config)" by default? The Fedora packaging disallows the usage of rpath. All I need about rpath is that it can be dangerous in some cases, but I forgot about the details. For example, on Fedora, if OpenSSL is installed in the regular /usr/lib64/libssl.so directory, rpath must not be used. Otherwise, it can lead to crash, bugs or vulnerabilities. Fedora, Python and rpath: Tell me if you need more details about the issues when rpath is misued. But I need to ask my colleagues, I forgot the details. |
|
It's a compromise. The default settings for --with-openssl-rpath=no (--without-openssl-rpath) is backwards compatible with previous Python versions. The default behavor stays the same. I don't want to set an rpath *unless* the user specifies that they want an rpath. I lack time and resources to verify that OPENSSL_LDFLAGS and OpenSSL's pkg-config files don't include any -L flags on all Linux, BSD, and macOS distributions. The new flag will make it more obvious to users that they may want an rpath, too. $ ./configure --help
...
--with-openssl=DIR override root of the OpenSSL directory to DIR
--with-openssl-rpath=[DIR|auto|no]
Set runtime library directory (rpath) for OpenSSL
libraries, no (default): don't set rpath, auto:
auto-detect rpath from --with-openssl and
pkg-config, DIR: set an explicit rpath
... |
Ok. That makes sense. |
|
I'm leaving the ticket open as a reminder for me to update whatsnew. |
|
#69177 adds -Wl,--exclude-libs just for libssl.a and libcrypto.a IFF support for -Wl,--exclude-libs,ALL is detected by configure. This puts the symbols from the OpenSSL archive files into the LOCAL segment of ELF binaries. The PR does not set -Wl,--exclude-libs,ALL because I like to keep behavior the same as with 3.9. When OpenSSL is locally build with "no-shared -fPIC", then Python automatically builds a partially static-linked _ssl and _hashlib extension modules that do not pollute the global namespace: $ ./config \
--prefix=/home/heimes/dev/python/multissl/openssl/1.1.1j-static \
--openssldir=/etc/pki/tls \
no-shared -fPIC
...
$ ./configure --with-openssl=/home/heimes/dev/python/multissl/openssl/1.1.1j-static
$ make
$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007fff8dbbc000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa5a533d000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa5a5172000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa5a56ac000)
$ readelf -Ws build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so | grep SSL_CTX_new
5617: 0000000000072a90 1133 FUNC LOCAL DEFAULT 11 SSL_CTX_newI deliberately did not update documentation with instructions for static linking. Static linking of OpenSSL has security and compatibility implications. I don't want to officially support it and deal with bug reports. -Wl,--exclude-libs just enables sane partial static-linking. |
|
There are now multiple ways to build Python with a custom OpenSSL build on Linux and BSD-like platforms:
OPENSSL=/home/heimes/dev/python/multissl/openssl/1.1.1j-static |
|
The new behaviour seems broken on my system. $ PYTHON_VERSION=3.10.0
$ PKG_CONFIG_PATH="/home/chaz/.local/local/sqlite3/lib/pkgconfig:/home/chaz/.local/local/openssl3/lib64/pkgconfig"
$ LLVM_PROFDATA=/home/chaz/.local/local/clang+llvm/bin/llvm-profdata CC=clang CXX=clang++ ./configure --prefix=/home/chaz/.local/local/python-${PYTHON_VERSION} --enable-optimizations --enable-shared --enable-loadable-sqlite-extensions --with-openssl-rpath=auto[..] building '_ssl' extension $ ldd /home/chaz/Downloads/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu_failed.so
linux-vdso.so.1 => (0x00007fffc99f4000)
libssl.so.3 => not found
libcrypto.so.3 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faa70802000)
libc.so.6 => /lib64/libc.so.6 (0x00007faa70434000)
/lib64/ld-linux-x86-64.so.2 (0x00007faa70a1e000)
$ readelf -d /home/chaz/Downloads/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu_failed.so | grep PATHIt appears that RPATH/RUNPATH is not getting set. |
|
If I set LDFLAGS as with previous versions, it works as expected. I do not know if it is related, but I noticed that by default rpath does not get set for the files in DESTDIR/bin/* (e.g. python3.10 or pip3.10) hence the inclusion of /home/chaz/.local/local/python-${PYTHON_VERSION}/lib in the rpath. LDFLAGS="-Wl,-rpath,'/home/chaz/.local/local/openssl3/lib64:/home/chaz/.local/local/sqlite3/lib:/home/chaz/.local/local/python-'${PYTHON_VERSION}'/lib' -Wl,--enable-new-dtags" |
|
Chris, this bug report is closed. Please open a new bug report. I think it's a bug in distutils. UnixCCompiler.runtime_library_dir_option() may not support clang and add wrong option "-R/home/chaz/.local/local/openssl-3.0.0/lib64". |
|
Chris, please try the fix from bpo-45371. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: