classification
Title: Redefinition of HMAC functions prevents static linking
Type: compile error Stage:
Components: Extension Modules Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, christian.heimes, indygreg
Priority: normal Keywords:

Created on 2020-10-06 00:46 by indygreg, last changed 2021-07-17 21:10 by indygreg.

Messages (6)
msg378080 - (view) Author: Gregory Szorc (indygreg) * Date: 2020-10-06 00:46
Commit 54f2898fe7e4ca1f239e96284af3cc5b34d2ae02 (bpo-40645) introduced the functions HMAC_CTX_new, HMAC_CTX_free, and HMAC_CTX_get_md.

These functions share the same names as HMAC functions provided by OpenSSL/LibreSSL. If you attempt to statically link the _hashlib extension as a builtin extension module that is also statically linked against a libcrytpo that provides these functions, the linker may complain about duplicate symbols:

cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_free':
cpython-3.9> hmac.c:(.text+0x7d0): multiple definition of `HMAC_CTX_free';
cpython-3.9> Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c:54: first defined here
cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_get_md':
cpython-3.9> hmac.c:(.text+0xa20): multiple definition of `HMAC_CTX_get_md'; Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c:                 63: first defined here
cpython-3.9> /tools/host/bin/ld: /tools/deps/lib/libcrypto.a(libcrypto_la-hmac.o): in function `HMAC_CTX_new':
cpython-3.9> hmac.c:(.text+0x780): multiple definition of `HMAC_CTX_new'; Modules/_hashopenssl.o:/build/Python-3.9.0/./Modules/_hashopenssl.c:42:                  first defined here
cpython-3.9> clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
cpython-3.9> ln: failed to access 'libpython3.9.so.1.0': No such file or directory
cpython-3.9> make: *** [libpython3.9.so] Error 1
cpython-3.9> Makefile:656: recipe for target 'libpython3.9.so' failed

This log from a build attempting to statically link against LibreSSL 3.1.4. The issue does not reproduce against OpenSSL 1.1.1h for some reason.

While statically linking _hashlib as a built-in extension module and statically linking libcrypto isn't in the default configuration, I believe this use case should be supported.

Perhaps these 3 functions should be renamed to not conflict with symbols provided by libcrypto?
msg378081 - (view) Author: Gregory Szorc (indygreg) * Date: 2020-10-06 01:00
I was going to work around this in python-build-standalone (https://github.com/indygreg/python-build-standalone) by reverting to OpenSSL for builds. But we must use LibreSSL when linking against musl libc due to some random weirdness getting OpenSSL to statically link against musl.

So I'm going to have to work around this via a source patch in python-build-standalone. Not the first time I've had to patch CPython source code to get a working statically linked build working.
msg378082 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-10-06 01:03
Note these are guarded by

#if (OPENSSL_VERSION_NUMBER < 0x10100000L) || defined(LIBRESSL_VERSION_NUMBER)

which suggests that needs to tweaked
msg378084 - (view) Author: Gregory Szorc (indygreg) * Date: 2020-10-06 01:38
My workaround was to remove `|| defined(LIBRESSL_VERSION_NUMBER)`. That works with LibreSSL 3.1.4. I'm unsure if the symbols were introduced in a specific version though. But since python-build-standalone pins all dependencies, this is a safe patch for me.
msg378100 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-06 08:59
It looks like LibreSSL is finally adding OpenSSL 1.1.1 APIs after all. This is a good move but also breaks assumptions.

Please note that compiling Python with LibreSSL is completely untested and unmaintained.
msg397731 - (view) Author: Gregory Szorc (indygreg) * Date: 2021-07-17 21:10
I think this was effectively fixed in 3.10 via commit 39258d3595300bc7b952854c915f63ae2d4b9c3e / bpo-43669, which removed code creating the duplicate symbols. It is still a problem in 3.9. But possibly not worth the backport.
History
Date User Action Args
2021-07-17 21:10:33indygregsetmessages: + msg397731
2020-10-06 08:59:05christian.heimessetmessages: + msg378100
2020-10-06 01:38:26indygregsetmessages: + msg378084
2020-10-06 01:03:32benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg378082
2020-10-06 01:00:25indygregsetmessages: + msg378081
2020-10-06 00:47:32indygregsetnosy: + christian.heimes
2020-10-06 00:46:58indygregsettype: compile error
2020-10-06 00:46:50indygregcreate