OpenSSL 3.0.0 seems to invoke the password callback multiple times under some circumstances. This triggers a fatal error in Python when the first invocation sets an exception.

test_load_cert_chain (test.test_ssl.ContextTests) ... Fatal Python error: _PyEval_EvalFrameDefault: a function returned a result with an exception set
Python runtime state: initialized
ValueError: password cannot be longer than 1023 bytes

Current thread 0x00007fa88787f740 (most recent call first):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1395 in getpass_huge
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1415 in test_load_cert_chain
  File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 549 in _callTestMethod
  File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 592 in run
  File "/home/heimes/dev/python/cpython/Lib/unittest/case.py", line 652 in __call__
  File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 122 in run
  File "/home/heimes/dev/python/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/home/heimes/dev/python/cpython/Lib/unittest/runner.py", line 176 in run
  File "/home/heimes/dev/python/cpython/Lib/test/support/__init__.py", line 959 in _run_suite
  File "/home/heimes/dev/python/cpython/Lib/test/support/__init__.py", line 1082 in run_unittest
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 4836 in test_main
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 246 in _runtest_inner2
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 282 in _runtest_inner
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 154 in _runtest
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/runtest.py", line 194 in runtest
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 321 in rerun_failed_tests
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 698 in _main
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 641 in main
  File "/home/heimes/dev/python/cpython/Lib/test/libregrtest/main.py", line 719 in main
  File "/home/heimes/dev/python/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/home/heimes/dev/python/cpython/Lib/runpy.py", line 86 in _run_code
  File "/home/heimes/dev/python/cpython/Lib/runpy.py", line 196 in _run_module_as_main

Extension modules: _testcapi (total: 1)


(gdb) bt
#0  0x00007ffff7c5d9d5 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c468a4 in abort () from /lib64/libc.so.6
#2  0x000000000051bb53 in fatal_error_exit (status=<optimized out>) at Python/pylifecycle.c:2522
#3  0x000000000051f97e in fatal_error (fd=2, header=header@entry=1, prefix=prefix@entry=0x6c2f60 <__func__.47> "_PyEval_EvalFrameDefault", 
    msg=msg@entry=0x670aa8 "a function returned a result with an exception set", status=status@entry=-1) at Python/pylifecycle.c:2703
#4  0x000000000051f9df in _Py_FatalErrorFunc (func=func@entry=0x6c2f60 <__func__.47> "_PyEval_EvalFrameDefault", 
    msg=msg@entry=0x670aa8 "a function returned a result with an exception set") at Python/pylifecycle.c:2719
#5  0x00000000004d930c in _PyEval_EvalFrameDefault (tstate=0x807060, 
    f=Frame 0x7fffe950e5b0, for file /home/heimes/dev/python/cpython/Lib/test/test_ssl.py, line 1395, in getpass_huge (), throwflag=0) at Python/ceval.c:1733
#6  0x00000000004e640f in _PyEval_EvalFrame (throwflag=0, 
    f=Frame 0x7fffe950e5b0, for file /home/heimes/dev/python/cpython/Lib/test/test_ssl.py, line 1395, in getpass_huge (), tstate=0x807060)
    at ./Include/internal/pycore_ceval.h:46
#7  _PyEval_Vector (tstate=0x807060, con=0x7fffe9377c30, locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>)
    at Python/ceval.c:5109
#8  0x000000000042bf08 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:342
#9  0x00007fffe9cf502c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=0, args=0x0, callable=<function at remote 0x7fffe9377c20>, tstate=0x807060)
    at ./Include/cpython/abstract.h:114
#10 _PyObject_CallNoArg (func=<function at remote 0x7fffe9377c20>) at ./Include/cpython/abstract.h:168
#11 _password_callback (buf=0x7fffffff80c0 "p", size=1023, rwflag=<optimized out>, userdata=0x7fffffff9820) at /home/heimes/dev/python/cpython/Modules/_ssl.c:3935
#12 0x00007fffe9a3bcd2 in ui_read (ui=0xc22d70, uis=0xba6190) at crypto/ui/ui_util.c:111
#13 0x00007fffe9a3a4e0 in UI_process (ui=0xc22d70) at crypto/ui/ui_lib.c:516
#14 0x00007fffe99a3d49 in do_ui_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, prompt_info=0x0, verify=0, ui_method=0xc20050, 
    ui_data=0x7fffffff9820) at crypto/passphrase.c:173
#15 0x00007fffe99a4143 in ossl_pw_get_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, verify=0, data=0xc21cc8)
    at crypto/passphrase.c:269
#16 0x00007fffe99a43ca in ossl_pw_passphrase_callback_dec (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, arg=0xc21cc8)
    at crypto/passphrase.c:324
#17 0x00007fffe99a3f8a in ossl_pw_get_passphrase (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, verify=0, data=0xc0d0c8)
    at crypto/passphrase.c:231
#18 0x00007fffe99a43ca in ossl_pw_passphrase_callback_dec (pass=0x7fffffff87b0 "", pass_size=1024, pass_len=0x7fffffff8bb8, params=0x0, arg=0xc0d0c8)
    at crypto/passphrase.c:324
#19 0x00007fffe9a9d5ca in der2key_decode_p8 (input_der=0x7fffffff8d50, input_der_len=1905, ctx=0xbb2c40, pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, 
    pw_cbarg=0xc0d0c8, key_from_pkcs8=0x7fffe99e8653 <ossl_rsa_key_from_pkcs8>) at providers/implementations/encode_decode/decode_der2key.c:150
#20 0x00007fffe9a9e432 in rsa_d2i_PKCS8 (key=0x0, der=0x7fffffff8d50, der_len=1905, ctx=0xbb2c40, pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc0d0c8)
    at providers/implementations/encode_decode/decode_der2key.c:580
#21 0x00007fffe9a9da9d in der2key_decode (vctx=0xbb2c40, cin=0xc4d0d0, selection=135, data_cb=0x7fffe9931e91 <decoder_process>, data_cbarg=0x7fffffff8de0, 
    pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc0d0c8) at providers/implementations/encode_decode/decode_der2key.c:295
#22 0x00007fffe993240b in decoder_process (params=0x0, arg=0x7fffffff8f00) at crypto/encode_decode/decoder_lib.c:750
#23 0x00007fffe9930f03 in OSSL_DECODER_from_bio (ctx=0xc0d090, in=0xc09700) at crypto/encode_decode/decoder_lib.c:58
#24 0x00007fffe99310b6 in OSSL_DECODER_from_data (ctx=0xc0d090, pdata=0x7fffffff8fc0, pdata_len=0x7fffffff8fb8) at crypto/encode_decode/decoder_lib.c:108
#25 0x00007fffe9a2fc77 in try_key_value (data=0x7fffffff9060, ctx=0xc21c80, cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, cbarg=0xc21cc8, 
    libctx=0x7fffe9c1f880 <default_context_int>, propq=0x0) at crypto/store/store_result.c:288
#26 0x00007fffe9a301cc in try_key (data=0x7fffffff9060, v=0x7fffffff96c0, ctx=0xc21c80, provider=0x97f7d0, libctx=0x7fffe9c1f880 <default_context_int>, propq=0x0)
    at crypto/store/store_result.c:407
#27 0x00007fffe9a2f6e6 in ossl_store_handle_load_result (params=0x7fffffff92f0, arg=0x7fffffff96c0) at crypto/store/store_result.c:152
#28 0x00007fffe9ac4638 in file_load_construct (decoder_inst=0x88b0a0, params=0x7fffffff92f0, construct_data=0x7fffffff9640)
    at providers/implementations/storemgmt/file_store.c:402
#29 0x00007fffe9931fd6 in decoder_process (params=0x7fffffff92f0, arg=0x7fffffff94b0) at crypto/encode_decode/decoder_lib.c:566
#30 0x00007fffe9aa03a8 in pem2der_decode (vctx=0xbb6ef0, cin=0xc04d70, selection=0, data_cb=0x7fffe9931e91 <decoder_process>, data_cbarg=0x7fffffff94b0, 
    pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xbc14f8) at providers/implementations/encode_decode/decode_pem2der.c:243
#31 0x00007fffe993240b in decoder_process (params=0x0, arg=0x7fffffff95d0) at crypto/encode_decode/decoder_lib.c:750
#32 0x00007fffe9930f03 in OSSL_DECODER_from_bio (ctx=0xbc14c0, in=0xc28ab0) at crypto/encode_decode/decoder_lib.c:58
#33 0x00007fffe9ac49db in file_load_file (ctx=0xc282b0, object_cb=0x7fffe9a2f3c2 <ossl_store_handle_load_result>, object_cbarg=0x7fffffff96c0, 
    pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc21cc8) at providers/implementations/storemgmt/file_store.c:522
--Type <RET> for more, q to quit, c to continue without paging--
#34 0x00007fffe9ac4f6f in file_load (loaderctx=0xc282b0, object_cb=0x7fffe9a2f3c2 <ossl_store_handle_load_result>, object_cbarg=0x7fffffff96c0, 
    pw_cb=0x7fffe99a4389 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0xc21cc8) at providers/implementations/storemgmt/file_store.c:682
#35 0x00007fffe9a2c3ff in OSSL_STORE_load (ctx=0xc21c80) at crypto/store/store_lib.c:386
#36 0x00007fffe99c48d0 in pem_read_bio_key (bp=0xbb6050, x=0x0, cb=0x7fffe9cf4f99 <_password_callback>, u=0x7fffffff9820, libctx=0x0, propq=0x0, 
    expected_store_info_type=4, try_secure=1) at crypto/pem/pem_pkey.c:74
#37 0x00007fffe99c4b12 in PEM_read_bio_PrivateKey_ex (bp=0xbb6050, x=0x0, cb=0x7fffe9cf4f99 <_password_callback>, u=0x7fffffff9820, libctx=0x0, propq=0x0)
    at crypto/pem/pem_pkey.c:144
#38 0x00007fffe9c653e7 in SSL_CTX_use_PrivateKey_file (ctx=0xc28350, file=0x7fffea13a060 "/home/heimes/dev/python/cpython/Lib/test/keycert.passwd.pem", type=1)
    at ssl/ssl_rsa.c:372
#39 0x00007fffe9cf82ce in _ssl__SSLContext_load_cert_chain_impl (self=self@entry=0x7fffe94c9be0, certfile='/home/heimes/dev/python/cpython/Lib/test/keycert.passwd.pem', 
    keyfile=0x0, password=<optimized out>) at /home/heimes/dev/python/cpython/Modules/_ssl.c:4032

New changeset d3b73f32ef7c693a6ae8c54eb0e62df3b5315caf by Christian Heimes in branch 'master':
bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303)
https://github.com/python/cpython/commit/d3b73f32ef7c693a6ae8c54eb0e62df3b5315caf

New changeset a188bd44ac3c54dc3bf927f1b10464ab80f37549 by Miss Islington (bot) in branch '3.9':
bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303)
https://github.com/python/cpython/commit/a188bd44ac3c54dc3bf927f1b10464ab80f37549

New changeset a28398e9c60848fc291c83dac44e5212694fb0b2 by Miss Islington (bot) in branch '3.8':
[3.8] bpo-43789: OpenSSL 3.0.0 Don't call passwd callback again in error case (GH-25303) (GH-25306)
https://github.com/python/cpython/commit/a28398e9c60848fc291c83dac44e5212694fb0b2

I'm keeping the bug open as a reminder to investigate the change of behavior more carefully.