This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: FIPS mode causes dead-lock in ssl module
Type: behavior Stage: resolved
Components: SSL Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: alex, christian.heimes, dstufft, janssen
Priority: normal Keywords:

Created on 2016-12-01 20:22 by christian.heimes, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (3)
msg282202 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-12-01 20:22
Python's ssl module is dead-locking when OpenSSL is running in FIPS mode. I first noticed it with pip. The issue is also reproducible with Python's test suite.

$ sudo touch /etc/system-fips

$ OPENSSL_FORCE_FIPS_MODE=1 ./python -m test.regrtest -v test_ssl
== CPython 2.7.12+ (2.7:adb296e4bcaa, Dec 1 2016, 21:14:20) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)]
==   Linux-4.8.8-200.fc24.x86_64-x86_64-with-fedora-24-Twenty_Four little-endian
==   /home/heimes/dev/python/2.7/build/test_python_29991
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
[1/1] test_ssl
test_ssl: testing with 'OpenSSL 1.0.2j-fips  26 Sep 2016' (1, 0, 2, 10, 15)
          under Linux ('Fedora', '24', 'Twenty Four')
          HAS_SNI = True
          OP_ALL = 0x800003ff
          OP_NO_TLSv1_1 = 0x10000000
test__create_stdlib_context (test.test_ssl.ContextTests) ... ok
test__https_verify_certificates (test.test_ssl.ContextTests) ... ok
test__https_verify_envvar (test.test_ssl.ContextTests) ... ok
test_cert_store_stats (test.test_ssl.ContextTests) ... ok
test_check_hostname (test.test_ssl.ContextTests) ... ok
test_ciphers (test.test_ssl.ContextTests) ... ok
test_constructor (test.test_ssl.ContextTests) ... ERROR
test_create_default_context (test.test_ssl.ContextTests) ... ok
test_get_ca_certs (test.test_ssl.ContextTests) ... ok
test_load_cert_chain (test.test_ssl.ContextTests) ... ERROR
test_load_default_certs (test.test_ssl.ContextTests) ... ok
test_load_default_certs_env (test.test_ssl.ContextTests) ... ok
test_load_default_certs_env_windows (test.test_ssl.ContextTests) ... skipped 'Windows specific'
test_load_dh_params (test.test_ssl.ContextTests) ... ok
test_load_verify_cadata (test.test_ssl.ContextTests) ... ok
test_load_verify_locations (test.test_ssl.ContextTests) ... ok
test_options (test.test_ssl.ContextTests) ... ok
test_protocol (test.test_ssl.ContextTests) ... ERROR
test_session_stats (test.test_ssl.ContextTests) ... ERROR
test_set_default_verify_paths (test.test_ssl.ContextTests) ... ok
test_set_ecdh_curve (test.test_ssl.ContextTests) ... 


(gdb) bt
#0  0x00007f0d9f8470c7 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f0d9f847174 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f0d9f84721a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000051e013 in PyThread_acquire_lock (lock=0x27433a0, waitflag=1) at Python/thread_pthread.h:324
#4  0x00007f0d937b0dce in _ssl_thread_locking_function (mode=5, n=18, file=0x7f0d96e417ac "fips_drbg_rand.c", line=124)
    at /home/heimes/dev/python/2.7/Modules/_ssl.c:4000
#5  0x00007f0d96df1d7e in fips_drbg_status () from /lib64/libcrypto.so.10
#6  0x00007f0d96d75b0e in drbg_rand_add () from /lib64/libcrypto.so.10
#7  0x00007f0d96d76645 in RAND_poll () from /lib64/libcrypto.so.10
#8  0x00007f0d96d75237 in ssleay_rand_bytes () from /lib64/libcrypto.so.10
#9  0x00007f0d96d75c33 in drbg_get_entropy () from /lib64/libcrypto.so.10
#10 0x00007f0d96df10c8 in fips_get_entropy () from /lib64/libcrypto.so.10
#11 0x00007f0d96df1226 in drbg_reseed () from /lib64/libcrypto.so.10
#12 0x00007f0d96d75bb8 in drbg_rand_seed () from /lib64/libcrypto.so.10
#13 0x00007f0d96d5da84 in ECDSA_sign_ex () from /lib64/libcrypto.so.10
#14 0x00007f0d96d5db00 in ECDSA_sign () from /lib64/libcrypto.so.10
#15 0x00007f0d96d3bc10 in pkey_ec_sign () from /lib64/libcrypto.so.10
#16 0x00007f0d96d81359 in EVP_SignFinal () from /lib64/libcrypto.so.10
#17 0x00007f0d96def373 in fips_pkey_signature_test () from /lib64/libcrypto.so.10
#18 0x00007f0d96d38fe0 in EC_KEY_generate_key () from /lib64/libcrypto.so.10
#19 0x00007f0d970d7d7c in ssl3_ctx_ctrl () from /lib64/libssl.so.10
#20 0x00007f0d937af934 in set_ecdh_curve (self=0x7f0d92a23f78, name='prime256v1') at /home/heimes/dev/python/2.7/Modules/_ssl.c:3110
#21 0x00000000004d8acc in call_function (pp_stack=0x7ffcc0334730, oparg=1) at Python/ceval.c:4340
#22 0x00000000004d37a9 in PyEval_EvalFrameEx (
    f=Frame 0x7f0d929bf460, for file /home/heimes/dev/python/2.7/Lib/test/test_ssl.py, line 1023, in test_set_ecdh_curve (self=<ContextTests(_resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7f0d9fc281c0>, dots=False, skipped=[(<ContextTests(_resultForDoCleanups=<...>, _type_equality_funcs={<type at remote 0x7fae40>: 'assertMultiLineEqual', <type at remote 0x7f4cc0>: 'assertTupleEqual', <type at remote 0x7f0ee0>: 'assertSetEqual', <type at remote 0x7ebf80>: 'assertListEqual', <type at remote 0x7edd80>: 'assertDictEqual', <type at remote 0x7f09e0>: 'assertSetEqual'}, _testMethodDoc=None, _testMethodName='test_load_default_certs_env_windows', _cleanups=[]) at remote 0x7f0d92a80ae0>, 'Windows specific')], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7f0d9fc281c0>) at remote 0x7f0d93747450>, testsRun=21, buffer=False, _original_stderr=<file at remote 0x7f0d9fc28280>, showAll=True, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], er...(truncated), throwflag=0)
    at Python/ceval.c:2989


Python 3's test suite is dead locking in a different test:

#0  0x00007f2be93dc0c7 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007f2be93dc174 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f2be93dc21a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x0000000000433e5c in PyThread_acquire_lock_timed (lock=0x17c0b40, microseconds=-1, intr_flag=intr_flag@entry=0)
    at Python/thread_pthread.h:352
#4  0x0000000000433f5e in PyThread_acquire_lock (lock=<optimized out>, waitflag=waitflag@entry=1) at Python/thread_pthread.h:556
#5  0x00007f2be0870945 in _ssl_thread_locking_function (mode=<optimized out>, n=<optimized out>, file=<optimized out>, 
    line=<optimized out>) at /home/heimes/dev/python/cpython/Modules/_ssl.c:5069
#6  0x00007f2be02f4d7e in fips_drbg_status () from /lib64/libcrypto.so.10
#7  0x00007f2be0278b0e in drbg_rand_add () from /lib64/libcrypto.so.10
#8  0x00007f2be0279645 in RAND_poll () from /lib64/libcrypto.so.10
#9  0x00007f2be0278237 in ssleay_rand_bytes () from /lib64/libcrypto.so.10
#10 0x00007f2be0278c33 in drbg_get_entropy () from /lib64/libcrypto.so.10
#11 0x00007f2be02f40c8 in fips_get_entropy () from /lib64/libcrypto.so.10
#12 0x00007f2be02f4226 in drbg_reseed () from /lib64/libcrypto.so.10
#13 0x00007f2be0278b39 in drbg_rand_add () from /lib64/libcrypto.so.10
#14 0x00007f2be0868870 in _ssl_RAND_add_impl (module=module@entry=<module at remote 0x7f2be0aa1a58>, view=view@entry=0x7ffd106c1420, 
    entropy=75) at /home/heimes/dev/python/cpython/Modules/_ssl.c:4499
#15 0x00007f2be08688f2 in _ssl_RAND_add (module=<module at remote 0x7f2be0aa1a58>, args=<optimized out>)
    at /home/heimes/dev/python/cpython/Modules/clinic/_ssl.c.h:861
#16 0x00000000004984d3 in _PyCFunction_FastCallDict (
    func_obj=func_obj@entry=<built-in method RAND_add of module object at remote 0x7f2be0aa1a58>, args=args@entry=0x18d54b8, 
    nargs=nargs@entry=2, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:234
#17 0x000000000049872a in _PyCFunction_FastCallKeywords (
    func=func@entry=<built-in method RAND_add of module object at remote 0x7f2be0aa1a58>, stack=stack@entry=0x18d54b8, 
    nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:295
#18 0x000000000052ad5b in call_function (pp_stack=pp_stack@entry=0x7ffd106c15d8, oparg=oparg@entry=2, kwnames=kwnames@entry=0x0)
    at Python/ceval.c:4735
msg282204 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-12-01 20:42
It's a dead lock in OpenSSL. :(

    if (n == CRYPTO_LOCK_RAND) {
        fprintf(stderr, "%s%s %i %s:%i\n",
                (mode & CRYPTO_READ) ? "R" : "W",
                (mode & CRYPTO_LOCK) ? "L" : "U",
                n, file, line);
    }

test_random (test.test_ssl.BasicSocketTests) ... RLCK 18 fips_drbg_rand.c:124
RUNL 18 fips_drbg_rand.c:126

 RAND_status is 1 (sufficient randomness)
WLCK 18 fips_drbg_rand.c:80
WUNL 18 fips_drbg_rand.c:109
WLCK 18 fips_drbg_rand.c:80
WUNL 18 fips_drbg_rand.c:109
WLCK 18 md_rand.c:230
WUNL 18 md_rand.c:262
WLCK 18 md_rand.c:311
WUNL 18 md_rand.c:324
RLCK 18 fips_drbg_rand.c:124
RUNL 18 fips_drbg_rand.c:126
WLCK 18 rand_lib.c:240
RLCK 18 fips_drbg_rand.c:124
msg282381 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-12-04 23:42
It was a downstream bug in Fedora. Tomas Mraz has fixed the issue and will release openssl-1.0.2j-3 soon, https://bugzilla.redhat.com/show_bug.cgi?id=1400922
History
Date User Action Args
2022-04-11 14:58:40adminsetgithub: 73040
2016-12-04 23:42:12christian.heimessetstatus: open -> closed
resolution: third party
messages: + msg282381

stage: resolved
2016-12-01 20:45:44alexsetnosy: + janssen, alex, dstufft
2016-12-01 20:42:50christian.heimessetmessages: + msg282204
2016-12-01 20:22:40christian.heimescreate