Let's consider ssl error `291` (https://bugs.python.org/issue39951):


It was introduced into openssl 2 years ago: https://github.com/openssl/openssl/commit/358ffa05cd3a088822c7d06256bc87516d918798

The documentation states:
SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY:291:\
	application data after close notify

The `ssl.h` header file contains:
# define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY        291

The master branch of openssl contains this definition too:
https://github.com/openssl/openssl/blob/master/include/openssl/sslerr.h
# define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY        291


But what does Python say?
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2629)

What's KRB5? It supposedly stands for Kerberos5, and it too is seemingly present in openssl header file:
/usr/local/Cellar/openssl/1.0.2s/include/openssl/ssl.h
2951:# define SSL_R_KRB5_S_INIT                                291

Moreover, cpython source code contains a fallback, should this value not be defined:
https://github.com/python/cpython/blob/master/Modules/_ssl_data.h
  #ifdef SSL_R_KRB5_S_INIT
    {"KRB5_S_INIT", ERR_LIB_SSL, SSL_R_KRB5_S_INIT},
  #else
    {"KRB5_S_INIT", ERR_LIB_SSL, 291},
  #endif


Thus, today, Python reports an error with wrong *label* but correct *text*:
[SSL: KRB5_S_INIT] application data after close notify


The label and text don't match each other, because... well... I guess that's why we should fix it :)

New changeset 3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a by Benjamin Peterson in branch 'master':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
https://github.com/python/cpython/commit/3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a

New changeset 2714c907df7cfe97911df6ce90364001270d9a43 by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
https://github.com/python/cpython/commit/2714c907df7cfe97911df6ce90364001270d9a43

Got some compiling error of _ssl extension module in my vm after PR19082 merged:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -fPIC -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_ssl.c -o build/temp.linux-x86_64-3.9-pydebug/temp/shihai/cpython/Modules/_ssl.o
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:6:15: error: ‘ERR_LIB_ASYNC’ undeclared here (not in a function); did you mean ‘ERR_LIB_ASN1’?
    6 |     {"ASYNC", ERR_LIB_ASYNC},
      |               ^~~~~~~~~~~~~
      |               ERR_LIB_ASN1
/temp/shihai/cpython/Modules/_ssl_data.h:13:12: error: ‘ERR_LIB_CT’ undeclared here (not in a function); did you mean ‘ERR_LIB_CMS’?
   13 |     {"CT", ERR_LIB_CT},
      |            ^~~~~~~~~~
      |            ERR_LIB_CMS
/temp/shihai/cpython/Modules/_ssl_data.h:19:13: error: ‘ERR_LIB_KDF’ undeclared here (not in a function); did you mean ‘ERR_LIB_BUF’?
   19 |     {"KDF", ERR_LIB_KDF},
      |             ^~~~~~~~~~~
      |             ERR_LIB_BUF
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
  598 |     {"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
      |                            ^~~~~~~~~~~~~
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: error: initializer element is not constant
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:603:32: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
  603 |     {"FAILED_TO_SWAP_CONTEXT", ERR_LIB_ASYNC, 102},

The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL. OpenSSL 1.1.x introduced new error codes or reused existing numbers for different errors codes.

Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8 and 3.9 compatible with the API.

Sorry, I thought I had tested with multissl.

On Sun, Apr 12, 2020, at 06:22, Christian Heimes wrote:
> 
> Christian Heimes <lists@cheimes.de> added the comment:
> 
> The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL. 
> OpenSSL 1.1.x introduced new error codes or reused existing numbers for 
> different errors codes.
> 
> Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8 
> and 3.9 compatible with the API.
> 
> ----------
> nosy: +lukasz.langa
> priority: normal -> release blocker
> resolution: fixed -> 
> status: closed -> open
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>

New changeset 909b87d2bb3d6330d39c48e43f7f50f4d086cc41 by Benjamin Peterson in branch 'master':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
https://github.com/python/cpython/commit/909b87d2bb3d6330d39c48e43f7f50f4d086cc41

New changeset f35e7d3bb0488a15cbb45ff10f02be558a3777cd by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
https://github.com/python/cpython/commit/f35e7d3bb0488a15cbb45ff10f02be558a3777cd

this is still broken even with the latest patch: https://bugs.python.org/issue40266

Could you please give me a chance to review PRs for the SSL module? 

Python is still failing to compile with OpenSSL 1.0.2 and LibreSSL. The new table contains also wrong values for LibreSSL and OpenSSL 1.0.2.

On Mon, Apr 13, 2020, at 17:54, Christian Heimes wrote:
> 
> Christian Heimes <lists@cheimes.de> added the comment:
> 
> Could you please give me a chance to review PRs for the SSL module? 

The original PR was open for 23 days before I merged it. I happy to here feedback at any point during the lifetime of a change, though.

New changeset 584a3cfda4d7a65ea0c1ea1ee541378bb7be46ca by Benjamin Peterson in branch 'master':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
https://github.com/python/cpython/commit/584a3cfda4d7a65ea0c1ea1ee541378bb7be46ca

New changeset c496e29c2bd0c29327c93174d5a40d2dc5a09402 by Miss Islington (bot) in branch '3.8':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
https://github.com/python/cpython/commit/c496e29c2bd0c29327c93174d5a40d2dc5a09402

Do I need to open a new issue?

This breaks building _ssl on AIX.

building '_ssl' extension
xlc_r -O -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 650.28: 1506-045 (S) Undeclared identifier ERR_LIB_ASYNC.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 1510.29: 1506-045 (S) Undeclared identifier ERR_LIB_CT.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 2650.24: 1506-045 (S) Undeclared identifier ERR_LIB_KDF.
"/home/aixtools/python/cpython-master/Modules/_ssl.c", line 579.17: 1506-196 (W) Initialization between types "void*" and "struct _object*(*)(struct {...}*)" is not allowed.



commit 909b87d2bb3d6330d39c48e43f7f50f4d086cc41
Author: Benjamin Peterson <benjamin@python.org>
Date:   Sun Apr 12 13:59:31 2020 -0500

    closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)

commit 3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a
Author: Benjamin Peterson <benjamin@python.org>
Date:   Sat Apr 11 15:36:12 2020 -0500

    closes bpo-39953: Update OpenSSL error codes table. (GH-19082)

    I updated the error codes using the OpenSSL 1.1.1f source tree.

commit 173ad83b074b3bf0c9e86eb8bd101c2841f74297
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Sun Jan 18 17:39:32 2015 +0100

    Issue #23248: Update ssl error codes from latest OpenSSL git master.

commit f7338f65fb8bdb85c52dc54d06d003a82a06bbb3
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Fri Jun 22 21:12:59 2012 +0200

    Add forgotten files for #14837.
$

Also checking with gcc: get the following messages:

Failed to build these modules:
_ssl

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

messages:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
In file included from /home/aixtools/python/cpython-master/Modules/_ssl.c:136:0:
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:650:28: error: 'ERR_LIB_ASYNC' undeclared here (not in a function); did you mean 'ERR_LIB_ASN1'?
     {"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
                            ^~~~~~~~~~~~~
                            ERR_LIB_ASN1
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:1510:29: error: 'ERR_LIB_CT' undeclared here (not in a function); did you mean 'ERR_LIB_CMS'?
     {"BASE64_DECODE_ERROR", ERR_LIB_CT, 108},
                             ^~~~~~~~~~
                             ERR_LIB_CMS
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:2650:24: error: 'ERR_LIB_KDF' undeclared here (not in a function); did you mean 'ERR_LIB_BUF'?
     {"INVALID_DIGEST", ERR_LIB_KDF, 100},
                        ^~~~~~~~~~~
                        ERR_LIB_BUF

And when I use a standard OpenSSL library (on AIX):

building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/freeware/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o -L/opt/freeware/lib -lssl -lcrypto -o build/lib.aix-7200-1543-32-3.9-pydebug/_ssl.so
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_get_ticket_lifetime_hint
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_has_ticket
ld: 0711-317 ERROR: Undefined symbol: .SSL_session_reused
ld: 0711-317 ERROR: Undefined symbol: .COMP_get_type
ld: 0711-317 ERROR: Undefined symbol: .SSL_is_init_finished
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_clear_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_set_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_is_aead
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_cipher_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_digest_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_kx_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_auth_nid
ld: 0711-317 ERROR: Undefined symbol: .X509_STORE_get0_objects
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get0_X509
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_num
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_value
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get_type
ld: 0711-317 ERROR: Undefined symbol: .X509_NAME_ENTRY_set
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb_userdata
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version_num
ld: 0711-317 ERROR: Undefined symbol: .TLS_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_client_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_server_method
ld: 0711-317 ERROR: Undefined symbol: .BIO_up_ref
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_pop_free
ld: 0711-317 ERROR: Undefined symbol: .X509_get_version
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notBefore
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notAfter
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.

$ lslpp -L | grep openssl
  aixtools.openssl.rte      1.0.2.16    C     F    aixtools openssl 27-Aug-2018
  openssl.base             1.0.1.515    CE    F    Open Secure Socket Layer
  openssl.man.en_US        1.0.1.515    C     F    Open Secure Socket Layer
  openssl          1.1.0g-1withsslv2    C     R    Secure Sockets Layer and
  openssl-devel    1.1.0g-1withsslv2    C     R    Secure Sockets Layer and

+++ FYI +++
IBM AIX used some strange version numbers: 1.0.1.515 is actually an OpenSSL 1.0.2 ABI version.

The "aixtools" fileset is 1.0.2p (p == 16th character of alphabet).

In any case - the test for X509_VERIFY_PARAM_set1_host() has been passing.

Michael, could you try with the latest fix in 584a3cfda4?

I did update, and saw that there was one more patch applied. 

I think that fixed the define issues, but there may be a new concern. Ran out of time to document it today. 

Will post tomorrow. 

Sent from my iPhone

> On 15 Apr 2020, at 17:53, SilentGhost <report@bugs.python.org> wrote:
> 
> 
> SilentGhost <ghost.adh@runbox.com> added the comment:
> 
> Michael, could you try with the latest fix in 584a3cfda4?
> 
> ----------
> nosy: +SilentGhost
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>

Checked with latest version - and working as expected. Sorry for the noise.

On 15/04/2020 17:53, SilentGhost wrote:
> SilentGhost <ghost.adh@runbox.com> added the comment:
>
> Michael, could you try with the latest fix in 584a3cfda4?
>
> ----------
> nosy: +SilentGhost
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>