classification
Title: Let's update ssl error codes
Type: compile error Stage: resolved
Components: Extension Modules Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, Dima.Tisnek, Michael.Felt, SilentGhost, alex, benjamin.peterson, christian.heimes, dstufft, eamanu, janssen, lukasz.langa, miss-islington, shihai1991
Priority: release blocker Keywords: patch

Created on 2020-03-13 04:07 by Dima.Tisnek, last changed 2020-04-16 13:35 by Michael.Felt. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19082 merged benjamin.peterson, 2020-03-19 23:37
PR 19478 merged miss-islington, 2020-04-11 20:36
PR 19490 merged benjamin.peterson, 2020-04-12 18:40
PR 19491 merged miss-islington, 2020-04-12 18:59
PR 19507 merged miss-islington, 2020-04-14 03:12
Messages (19)
msg364074 - (view) Author: Dima Tisnek (Dima.Tisnek) * Date: 2020-03-13 04:07
Let's consider ssl error `291` (https://bugs.python.org/issue39951):


It was introduced into openssl 2 years ago: https://github.com/openssl/openssl/commit/358ffa05cd3a088822c7d06256bc87516d918798

The documentation states:
SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY:291:\
	application data after close notify

The `ssl.h` header file contains:
# define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY        291

The master branch of openssl contains this definition too:
https://github.com/openssl/openssl/blob/master/include/openssl/sslerr.h
# define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY        291


But what does Python say?
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2629)

What's KRB5? It supposedly stands for Kerberos5, and it too is seemingly present in openssl header file:
/usr/local/Cellar/openssl/1.0.2s/include/openssl/ssl.h
2951:# define SSL_R_KRB5_S_INIT                                291

Moreover, cpython source code contains a fallback, should this value not be defined:
https://github.com/python/cpython/blob/master/Modules/_ssl_data.h
  #ifdef SSL_R_KRB5_S_INIT
    {"KRB5_S_INIT", ERR_LIB_SSL, SSL_R_KRB5_S_INIT},
  #else
    {"KRB5_S_INIT", ERR_LIB_SSL, 291},
  #endif


Thus, today, Python reports an error with wrong *label* but correct *text*:
[SSL: KRB5_S_INIT] application data after close notify


The label and text don't match each other, because... well... I guess that's why we should fix it :)
msg366218 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-04-11 20:36
New changeset 3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a by Benjamin Peterson in branch 'master':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
https://github.com/python/cpython/commit/3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a
msg366219 - (view) Author: miss-islington (miss-islington) Date: 2020-04-11 20:53
New changeset 2714c907df7cfe97911df6ce90364001270d9a43 by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
https://github.com/python/cpython/commit/2714c907df7cfe97911df6ce90364001270d9a43
msg366229 - (view) Author: hai shi (shihai1991) * Date: 2020-04-12 08:43
Got some compiling error of _ssl extension module in my vm after PR19082 merged:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -fPIC -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_ssl.c -o build/temp.linux-x86_64-3.9-pydebug/temp/shihai/cpython/Modules/_ssl.o
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:6:15: error: ‘ERR_LIB_ASYNC’ undeclared here (not in a function); did you mean ‘ERR_LIB_ASN1’?
    6 |     {"ASYNC", ERR_LIB_ASYNC},
      |               ^~~~~~~~~~~~~
      |               ERR_LIB_ASN1
/temp/shihai/cpython/Modules/_ssl_data.h:13:12: error: ‘ERR_LIB_CT’ undeclared here (not in a function); did you mean ‘ERR_LIB_CMS’?
   13 |     {"CT", ERR_LIB_CT},
      |            ^~~~~~~~~~
      |            ERR_LIB_CMS
/temp/shihai/cpython/Modules/_ssl_data.h:19:13: error: ‘ERR_LIB_KDF’ undeclared here (not in a function); did you mean ‘ERR_LIB_BUF’?
   19 |     {"KDF", ERR_LIB_KDF},
      |             ^~~~~~~~~~~
      |             ERR_LIB_BUF
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
  598 |     {"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
      |                            ^~~~~~~~~~~~~
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: error: initializer element is not constant
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:603:32: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
  603 |     {"FAILED_TO_SWAP_CONTEXT", ERR_LIB_ASYNC, 102},
msg366234 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-04-12 11:22
The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL. OpenSSL 1.1.x introduced new error codes or reused existing numbers for different errors codes.

Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8 and 3.9 compatible with the API.
msg366261 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-04-12 18:37
Sorry, I thought I had tested with multissl.

On Sun, Apr 12, 2020, at 06:22, Christian Heimes wrote:
> 
> Christian Heimes <lists@cheimes.de> added the comment:
> 
> The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL. 
> OpenSSL 1.1.x introduced new error codes or reused existing numbers for 
> different errors codes.
> 
> Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8 
> and 3.9 compatible with the API.
> 
> ----------
> nosy: +lukasz.langa
> priority: normal -> release blocker
> resolution: fixed -> 
> status: closed -> open
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>
msg366262 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-04-12 18:59
New changeset 909b87d2bb3d6330d39c48e43f7f50f4d086cc41 by Benjamin Peterson in branch 'master':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
https://github.com/python/cpython/commit/909b87d2bb3d6330d39c48e43f7f50f4d086cc41
msg366263 - (view) Author: miss-islington (miss-islington) Date: 2020-04-12 19:17
New changeset f35e7d3bb0488a15cbb45ff10f02be558a3777cd by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
https://github.com/python/cpython/commit/f35e7d3bb0488a15cbb45ff10f02be558a3777cd
msg366318 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2020-04-13 16:57
this is still broken even with the latest patch: https://bugs.python.org/issue40266
msg366346 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-04-13 22:54
Could you please give me a chance to review PRs for the SSL module? 

Python is still failing to compile with OpenSSL 1.0.2 and LibreSSL. The new table contains also wrong values for LibreSSL and OpenSSL 1.0.2.
msg366358 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-04-14 02:51
On Mon, Apr 13, 2020, at 17:54, Christian Heimes wrote:
> 
> Christian Heimes <lists@cheimes.de> added the comment:
> 
> Could you please give me a chance to review PRs for the SSL module? 

The original PR was open for 23 days before I merged it. I happy to here feedback at any point during the lifetime of a change, though.
msg366359 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-04-14 03:11
New changeset 584a3cfda4d7a65ea0c1ea1ee541378bb7be46ca by Benjamin Peterson in branch 'master':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
https://github.com/python/cpython/commit/584a3cfda4d7a65ea0c1ea1ee541378bb7be46ca
msg366361 - (view) Author: miss-islington (miss-islington) Date: 2020-04-14 03:31
New changeset c496e29c2bd0c29327c93174d5a40d2dc5a09402 by Miss Islington (bot) in branch '3.8':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
https://github.com/python/cpython/commit/c496e29c2bd0c29327c93174d5a40d2dc5a09402
msg366511 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-15 13:17
Do I need to open a new issue?

This breaks building _ssl on AIX.

building '_ssl' extension
xlc_r -O -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 650.28: 1506-045 (S) Undeclared identifier ERR_LIB_ASYNC.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 1510.29: 1506-045 (S) Undeclared identifier ERR_LIB_CT.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 2650.24: 1506-045 (S) Undeclared identifier ERR_LIB_KDF.
"/home/aixtools/python/cpython-master/Modules/_ssl.c", line 579.17: 1506-196 (W) Initialization between types "void*" and "struct _object*(*)(struct {...}*)" is not allowed.



commit 909b87d2bb3d6330d39c48e43f7f50f4d086cc41
Author: Benjamin Peterson <benjamin@python.org>
Date:   Sun Apr 12 13:59:31 2020 -0500

    closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)

commit 3e0dd3730b5eff7e9ae6fb921aa77cd26efc9e3a
Author: Benjamin Peterson <benjamin@python.org>
Date:   Sat Apr 11 15:36:12 2020 -0500

    closes bpo-39953: Update OpenSSL error codes table. (GH-19082)

    I updated the error codes using the OpenSSL 1.1.1f source tree.

commit 173ad83b074b3bf0c9e86eb8bd101c2841f74297
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Sun Jan 18 17:39:32 2015 +0100

    Issue #23248: Update ssl error codes from latest OpenSSL git master.

commit f7338f65fb8bdb85c52dc54d06d003a82a06bbb3
Author: Antoine Pitrou <solipsis@pitrou.net>
Date:   Fri Jun 22 21:12:59 2012 +0200

    Add forgotten files for #14837.
$
msg366513 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-15 13:23
Also checking with gcc: get the following messages:

Failed to build these modules:
_ssl

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

messages:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
In file included from /home/aixtools/python/cpython-master/Modules/_ssl.c:136:0:
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:650:28: error: 'ERR_LIB_ASYNC' undeclared here (not in a function); did you mean 'ERR_LIB_ASN1'?
     {"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
                            ^~~~~~~~~~~~~
                            ERR_LIB_ASN1
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:1510:29: error: 'ERR_LIB_CT' undeclared here (not in a function); did you mean 'ERR_LIB_CMS'?
     {"BASE64_DECODE_ERROR", ERR_LIB_CT, 108},
                             ^~~~~~~~~~
                             ERR_LIB_CMS
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:2650:24: error: 'ERR_LIB_KDF' undeclared here (not in a function); did you mean 'ERR_LIB_BUF'?
     {"INVALID_DIGEST", ERR_LIB_KDF, 100},
                        ^~~~~~~~~~~
                        ERR_LIB_BUF
msg366515 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-15 13:46
And when I use a standard OpenSSL library (on AIX):

building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/freeware/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o -L/opt/freeware/lib -lssl -lcrypto -o build/lib.aix-7200-1543-32-3.9-pydebug/_ssl.so
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_get_ticket_lifetime_hint
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_has_ticket
ld: 0711-317 ERROR: Undefined symbol: .SSL_session_reused
ld: 0711-317 ERROR: Undefined symbol: .COMP_get_type
ld: 0711-317 ERROR: Undefined symbol: .SSL_is_init_finished
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_clear_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_set_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_is_aead
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_cipher_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_digest_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_kx_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_auth_nid
ld: 0711-317 ERROR: Undefined symbol: .X509_STORE_get0_objects
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get0_X509
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_num
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_value
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get_type
ld: 0711-317 ERROR: Undefined symbol: .X509_NAME_ENTRY_set
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb_userdata
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version_num
ld: 0711-317 ERROR: Undefined symbol: .TLS_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_client_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_server_method
ld: 0711-317 ERROR: Undefined symbol: .BIO_up_ref
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_pop_free
ld: 0711-317 ERROR: Undefined symbol: .X509_get_version
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notBefore
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notAfter
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.

$ lslpp -L | grep openssl
  aixtools.openssl.rte      1.0.2.16    C     F    aixtools openssl 27-Aug-2018
  openssl.base             1.0.1.515    CE    F    Open Secure Socket Layer
  openssl.man.en_US        1.0.1.515    C     F    Open Secure Socket Layer
  openssl          1.1.0g-1withsslv2    C     R    Secure Sockets Layer and
  openssl-devel    1.1.0g-1withsslv2    C     R    Secure Sockets Layer and

+++ FYI +++
IBM AIX used some strange version numbers: 1.0.1.515 is actually an OpenSSL 1.0.2 ABI version.

The "aixtools" fileset is 1.0.2p (p == 16th character of alphabet).

In any case - the test for X509_VERIFY_PARAM_set1_host() has been passing.
msg366527 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2020-04-15 15:53
Michael, could you try with the latest fix in 584a3cfda4?
msg366553 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-15 20:36
I did update, and saw that there was one more patch applied. 

I think that fixed the define issues, but there may be a new concern. Ran out of time to document it today. 

Will post tomorrow. 

Sent from my iPhone

> On 15 Apr 2020, at 17:53, SilentGhost <report@bugs.python.org> wrote:
> 
> 
> SilentGhost <ghost.adh@runbox.com> added the comment:
> 
> Michael, could you try with the latest fix in 584a3cfda4?
> 
> ----------
> nosy: +SilentGhost
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>
msg366596 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-16 13:35
Checked with latest version - and working as expected. Sorry for the noise.

On 15/04/2020 17:53, SilentGhost wrote:
> SilentGhost <ghost.adh@runbox.com> added the comment:
>
> Michael, could you try with the latest fix in 584a3cfda4?
>
> ----------
> nosy: +SilentGhost
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue39953>
> _______________________________________
>
History
Date User Action Args
2020-04-16 13:35:40Michael.Feltsetmessages: + msg366596
2020-04-15 20:36:10Michael.Feltsetmessages: + msg366553
2020-04-15 15:53:32SilentGhostsetnosy: + SilentGhost
messages: + msg366527
2020-04-15 13:46:44Michael.Feltsetmessages: + msg366515
2020-04-15 13:23:00Michael.Feltsetmessages: + msg366513
2020-04-15 13:17:50Michael.Feltsetnosy: + Michael.Felt
messages: + msg366511
2020-04-14 03:31:25miss-islingtonsetmessages: + msg366361
2020-04-14 03:12:08miss-islingtonsetpull_requests: + pull_request18859
2020-04-14 03:11:51benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg366359

stage: patch review -> resolved
2020-04-14 02:51:58benjamin.petersonsetmessages: + msg366358
2020-04-13 22:54:07christian.heimessetstatus: closed -> open
type: compile error
messages: + msg366346

resolution: fixed -> (no value)
stage: resolved -> patch review
2020-04-13 16:57:56Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg366318
2020-04-13 06:13:30SilentGhostlinkissue40266 superseder
2020-04-12 19:17:38miss-islingtonsetmessages: + msg366263
2020-04-12 18:59:47miss-islingtonsetpull_requests: + pull_request18843
2020-04-12 18:59:34benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg366262

stage: patch review -> resolved
2020-04-12 18:40:02benjamin.petersonsetstage: resolved -> patch review
pull_requests: + pull_request18842
2020-04-12 18:37:39benjamin.petersonsetmessages: + msg366261
2020-04-12 11:22:13christian.heimessetstatus: closed -> open
priority: normal -> release blocker

nosy: + lukasz.langa
messages: + msg366234

resolution: fixed -> (no value)
2020-04-12 08:43:10shihai1991setnosy: + shihai1991
messages: + msg366229
2020-04-11 20:53:06miss-islingtonsetmessages: + msg366219
2020-04-11 20:36:27miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request18832
2020-04-11 20:36:18benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg366218

stage: patch review -> resolved
2020-03-19 23:37:48benjamin.petersonsetkeywords: + patch
nosy: + benjamin.peterson

pull_requests: + pull_request18441
stage: patch review
2020-03-13 11:43:19eamanusetnosy: + eamanu
2020-03-13 09:02:52serhiy.storchakasetnosy: + janssen, christian.heimes, alex, dstufft
2020-03-13 04:07:50Dima.Tisnekcreate