Let's update ssl error codes #84134

dimaqq · 2020-03-13T04:07:51Z

BPO	39953
Nosy	@tiran, @benjaminp, @alex, @ambv, @dimaqq, @dstufft, @aixtools, @asottile, @eamanu, @miss-islington, @shihai1991
PRs	closes bpo-39953: Update OpenSSL error codes table. #19082 [3.8] closes bpo-39953: Update OpenSSL error codes table. (GH-19082) #19478 closes bpo-39953: Generate ifdefs around library code definitions. #19490 [3.8] closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490) #19491 [3.8] closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506) #19507

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-14.03:11:51.487>
created_at = <Date 2020-03-13.04:07:50.824>
labels = ['extension-modules', 'build', '3.8', '3.9', 'release-blocker']
title = "Let's update ssl error codes"
updated_at = <Date 2020-04-16.13:35:40.938>
user = 'https://github.com/dimaqq'

bugs.python.org fields:

activity = <Date 2020-04-16.13:35:40.938>
actor = 'Michael.Felt'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-14.03:11:51.487>
closer = 'benjamin.peterson'
components = ['Extension Modules']
creation = <Date 2020-03-13.04:07:50.824>
creator = 'Dima.Tisnek'
dependencies = []
files = []
hgrepos = []
issue_num = 39953
keywords = ['patch']
message_count = 19.0
messages = ['364074', '366218', '366219', '366229', '366234', '366261', '366262', '366263', '366318', '366346', '366358', '366359', '366361', '366511', '366513', '366515', '366527', '366553', '366596']
nosy_count = 13.0
nosy_names = ['janssen', 'christian.heimes', 'benjamin.peterson', 'alex', 'SilentGhost', 'lukasz.langa', 'Dima.Tisnek', 'dstufft', 'Michael.Felt', 'Anthony Sottile', 'eamanu', 'miss-islington', 'shihai1991']
pr_nums = ['19082', '19478', '19490', '19491', '19507']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue39953'
versions = ['Python 3.8', 'Python 3.9']

dimaqq · 2020-03-13T04:07:51Z

Let's consider ssl error 291 (https://bugs.python.org/issue39951):

It was introduced into openssl 2 years ago: openssl/openssl@358ffa0

The documentation states:
SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY:291:\
application data after close notify

The ssl.h header file contains:

define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY 291

The master branch of openssl contains this definition too:
https://github.com/openssl/openssl/blob/master/include/openssl/sslerr.h
# define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY 291

But what does Python say?
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2629)

What's KRB5? It supposedly stands for Kerberos5, and it too is seemingly present in openssl header file:
/usr/local/Cellar/openssl/1.0.2s/include/openssl/ssl.h
2951:# define SSL_R_KRB5_S_INIT 291

Moreover, cpython source code contains a fallback, should this value not be defined:
https://github.com/python/cpython/blob/master/Modules/_ssl_data.h
  #ifdef SSL_R_KRB5_S_INIT
    {"KRB5_S_INIT", ERR_LIB_SSL, SSL_R_KRB5_S_INIT},
  #else
    {"KRB5_S_INIT", ERR_LIB_SSL, 291},
  #endif

Thus, today, Python reports an error with wrong *label* but correct *text*:
[SSL: KRB5_S_INIT] application data after close notify

The label and text don't match each other, because... well... I guess that's why we should fix it :)

benjaminp · 2020-04-11T20:36:19Z

New changeset 3e0dd37 by Benjamin Peterson in branch 'master':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
3e0dd37

miss-islington · 2020-04-11T20:53:07Z

New changeset 2714c90 by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Update OpenSSL error codes table. (GH-19082)
2714c90

shihai1991 · 2020-04-12T08:43:11Z

Got some compiling error of _ssl extension module in my vm after PR19082 merged:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -fPIC -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_ssl.c -o build/temp.linux-x86_64-3.9-pydebug/temp/shihai/cpython/Modules/_ssl.o
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:6:15: error: ‘ERR_LIB_ASYNC’ undeclared here (not in a function); did you mean ‘ERR_LIB_ASN1’?
6 | {"ASYNC", ERR_LIB_ASYNC},
| ^~~~~~~~~~~~~
| ERR_LIB_ASN1
/temp/shihai/cpython/Modules/_ssl_data.h:13:12: error: ‘ERR_LIB_CT’ undeclared here (not in a function); did you mean ‘ERR_LIB_CMS’?
13 | {"CT", ERR_LIB_CT},
| ^~~~~~~~~~
| ERR_LIB_CMS
/temp/shihai/cpython/Modules/_ssl_data.h:19:13: error: ‘ERR_LIB_KDF’ undeclared here (not in a function); did you mean ‘ERR_LIB_BUF’?
19 | {"KDF", ERR_LIB_KDF},
| ^~~~~~~~~~~
| ERR_LIB_BUF
In file included from /temp/shihai/cpython/Modules/_ssl.c:136:
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
598 | {"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
| ^~~~~~~~~~~~~
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: error: initializer element is not constant
/temp/shihai/cpython/Modules/_ssl_data.h:598:28: note: (near initialization for ‘error_codes[112].library’)
/temp/shihai/cpython/Modules/_ssl_data.h:603:32: warning: initialization of ‘int’ from ‘struct py_ssl_library_code *’ makes integer from pointer without a cast [-Wint-conversion]
603 | {"FAILED_TO_SWAP_CONTEXT", ERR_LIB_ASYNC, 102},

tiran · 2020-04-12T11:22:13Z

The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL. OpenSSL 1.1.x introduced new error codes or reused existing numbers for different errors codes.

Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8 and 3.9 compatible with the API.

benjaminp · 2020-04-12T18:37:39Z

Sorry, I thought I had tested with multissl.

On Sun, Apr 12, 2020, at 06:22, Christian Heimes wrote:

Christian Heimes <lists@cheimes.de> added the comment:

The PR broke backwards compatibility with OpenSSL 1.0.2 and LibreSSL.
OpenSSL 1.1.x introduced new error codes or reused existing numbers for
different errors codes.

Although OpenSSL 1.0.2 has reached EOL we should keep keep Python 3.8
and 3.9 compatible with the API.

----------
nosy: +lukasz.langa
priority: normal -> release blocker
resolution: fixed ->
status: closed -> open

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue39953\>

benjaminp · 2020-04-12T18:59:35Z

New changeset 909b87d by Benjamin Peterson in branch 'master':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
909b87d

miss-islington · 2020-04-12T19:17:38Z

New changeset f35e7d3 by Miss Islington (bot) in branch '3.8':
closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)
f35e7d3

asottile · 2020-04-13T16:57:57Z

this is still broken even with the latest patch: https://bugs.python.org/issue40266

tiran · 2020-04-13T22:54:07Z

Could you please give me a chance to review PRs for the SSL module?

Python is still failing to compile with OpenSSL 1.0.2 and LibreSSL. The new table contains also wrong values for LibreSSL and OpenSSL 1.0.2.

benjaminp · 2020-04-14T02:51:59Z

On Mon, Apr 13, 2020, at 17:54, Christian Heimes wrote:

Christian Heimes <lists@cheimes.de> added the comment:

Could you please give me a chance to review PRs for the SSL module?

The original PR was open for 23 days before I merged it. I happy to here feedback at any point during the lifetime of a change, though.

benjaminp · 2020-04-14T03:11:51Z

New changeset 584a3cf by Benjamin Peterson in branch 'master':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
584a3cf

miss-islington · 2020-04-14T03:31:25Z

New changeset c496e29 by Miss Islington (bot) in branch '3.8':
closes bpo-40266, closes bpo-39953: Use numeric lib code if compiling against old OpenSSL. (GH-19506)
c496e29

aixtools · 2020-04-15T13:17:50Z

Do I need to open a new issue?

This breaks building _ssl on AIX.

building '_ssl' extension
xlc_r -O -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 650.28: 1506-045 (S) Undeclared identifier ERR_LIB_ASYNC.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 1510.29: 1506-045 (S) Undeclared identifier ERR_LIB_CT.
"/home/aixtools/python/cpython-master/Modules/_ssl_data.h", line 2650.24: 1506-045 (S) Undeclared identifier ERR_LIB_KDF.
"/home/aixtools/python/cpython-master/Modules/_ssl.c", line 579.17: 1506-196 (W) Initialization between types "void*" and "struct _object*()(struct {...})" is not allowed.

commit 909b87d
Author: Benjamin Peterson <benjamin@python.org>
Date: Sun Apr 12 13:59:31 2020 -0500

closes bpo-39953: Generate ifdefs around library code definitions. (GH-19490)

commit 3e0dd37
Author: Benjamin Peterson <benjamin@python.org>
Date: Sat Apr 11 15:36:12 2020 -0500

closes bpo-39953: Update OpenSSL error codes table. (GH-19082)

I updated the error codes using the OpenSSL 1.1.1f source tree.

commit 173ad83
Author: Antoine Pitrou <solipsis@pitrou.net>
Date: Sun Jan 18 17:39:32 2015 +0100

Issue bpo-23248: Update ssl error codes from latest OpenSSL git master.

commit f7338f6
Author: Antoine Pitrou <solipsis@pitrou.net>
Date: Fri Jun 22 21:12:59 2012 +0200

Add forgotten files for bpo-14837.

$

aixtools · 2020-04-15T13:23:00Z

Also checking with gcc: get the following messages:

Failed to build these modules:
_ssl

Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, libressl/portable#381

messages:
building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/aixtools/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
In file included from /home/aixtools/python/cpython-master/Modules/_ssl.c:136:0:
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:650:28: error: 'ERR_LIB_ASYNC' undeclared here (not in a function); did you mean 'ERR_LIB_ASN1'?
{"FAILED_TO_SET_POOL", ERR_LIB_ASYNC, 101},
^~~~~~~~~~~~~
ERR_LIB_ASN1
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:1510:29: error: 'ERR_LIB_CT' undeclared here (not in a function); did you mean 'ERR_LIB_CMS'?
{"BASE64_DECODE_ERROR", ERR_LIB_CT, 108},
^~~~~~~~~~
ERR_LIB_CMS
/home/aixtools/python/cpython-master/Modules/_ssl_data.h:2650:24: error: 'ERR_LIB_KDF' undeclared here (not in a function); did you mean 'ERR_LIB_BUF'?
{"INVALID_DIGEST", ERR_LIB_KDF, 100},
^~~~~~~~~~~
ERR_LIB_BUF

aixtools · 2020-04-15T13:46:44Z

And when I use a standard OpenSSL library (on AIX):

building '_ssl' extension
gcc -pthread -Wno-unused-result -Wsign-compare -g -Og -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I/opt/freeware/include -I./Include -I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_ssl.c -o build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9-pydebug/home/aixtools/python/cpython-master/Modules/_ssl.o -L/opt/freeware/lib -lssl -lcrypto -o build/lib.aix-7200-1543-32-3.9-pydebug/_ssl.so
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_get_ticket_lifetime_hint
ld: 0711-317 ERROR: Undefined symbol: .SSL_SESSION_has_ticket
ld: 0711-317 ERROR: Undefined symbol: .SSL_session_reused
ld: 0711-317 ERROR: Undefined symbol: .COMP_get_type
ld: 0711-317 ERROR: Undefined symbol: .SSL_is_init_finished
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_clear_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_set_options
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_is_aead
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_cipher_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_digest_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_kx_nid
ld: 0711-317 ERROR: Undefined symbol: .SSL_CIPHER_get_auth_nid
ld: 0711-317 ERROR: Undefined symbol: .X509_STORE_get0_objects
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get0_X509
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_num
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_value
ld: 0711-317 ERROR: Undefined symbol: .X509_OBJECT_get_type
ld: 0711-317 ERROR: Undefined symbol: .X509_NAME_ENTRY_set
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb
ld: 0711-317 ERROR: Undefined symbol: .SSL_CTX_get_default_passwd_cb_userdata
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version_num
ld: 0711-317 ERROR: Undefined symbol: .TLS_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_client_method
ld: 0711-317 ERROR: Undefined symbol: .TLS_server_method
ld: 0711-317 ERROR: Undefined symbol: .BIO_up_ref
ld: 0711-317 ERROR: Undefined symbol: .OPENSSL_sk_pop_free
ld: 0711-317 ERROR: Undefined symbol: .X509_get_version
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notBefore
ld: 0711-317 ERROR: Undefined symbol: .X509_getm_notAfter
ld: 0711-317 ERROR: Undefined symbol: .OpenSSL_version
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.

$ lslpp -L | grep openssl
  aixtools.openssl.rte      1.0.2.16    C     F    aixtools openssl 27-Aug-2018
  openssl.base             1.0.1.515    CE    F    Open Secure Socket Layer
  openssl.man.en_US        1.0.1.515    C     F    Open Secure Socket Layer
  openssl          1.1.0g-1withsslv2    C     R    Secure Sockets Layer and
  openssl-devel    1.1.0g-1withsslv2    C     R    Secure Sockets Layer and

+++ FYI +++
IBM AIX used some strange version numbers: 1.0.1.515 is actually an OpenSSL 1.0.2 ABI version.

The "aixtools" fileset is 1.0.2p (p == 16th character of alphabet).

In any case - the test for X509_VERIFY_PARAM_set1_host() has been passing.

SilentGhost · 2020-04-15T15:53:32Z

Michael, could you try with the latest fix in 584a3cf?

aixtools · 2020-04-15T20:36:11Z

I did update, and saw that there was one more patch applied.

I think that fixed the define issues, but there may be a new concern. Ran out of time to document it today.

Will post tomorrow.

Sent from my iPhone

On 15 Apr 2020, at 17:53, SilentGhost <report@bugs.python.org> wrote:

SilentGhost <ghost.adh@runbox.com> added the comment:

Michael, could you try with the latest fix in 584a3cf?

----------
nosy: +SilentGhost

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue39953\>

aixtools · 2020-04-16T13:35:41Z

Checked with latest version - and working as expected. Sorry for the noise.

On 15/04/2020 17:53, SilentGhost wrote:

SilentGhost <ghost.adh@runbox.com> added the comment:

Michael, could you try with the latest fix in 584a3cf?

----------
nosy: +SilentGhost

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue39953\>

dimaqq mannequin added 3.8 only security fixes 3.9 only security fixes extension-modules C modules in the Modules dir labels Mar 13, 2020

benjaminp closed this as completed Apr 11, 2020

tiran reopened this Apr 12, 2020

tiran added the release-blocker label Apr 12, 2020

tiran reopened this Apr 12, 2020

tiran added the release-blocker label Apr 12, 2020

benjaminp closed this as completed Apr 12, 2020

tiran reopened this Apr 13, 2020

tiran added the build The build process and cross-build label Apr 13, 2020

tiran reopened this Apr 13, 2020

tiran added the build The build process and cross-build label Apr 13, 2020

benjaminp closed this as completed Apr 14, 2020

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let's update ssl error codes #84134

Let's update ssl error codes #84134

dimaqq mannequin commented Mar 13, 2020

dimaqq mannequin commented Mar 13, 2020

benjaminp commented Apr 11, 2020

miss-islington commented Apr 11, 2020

shihai1991 commented Apr 12, 2020

tiran commented Apr 12, 2020

benjaminp commented Apr 12, 2020

benjaminp commented Apr 12, 2020

miss-islington commented Apr 12, 2020

asottile mannequin commented Apr 13, 2020

tiran commented Apr 13, 2020

benjaminp commented Apr 14, 2020

benjaminp commented Apr 14, 2020

miss-islington commented Apr 14, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 15, 2020

SilentGhost mannequin commented Apr 15, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 16, 2020

Let's update ssl error codes #84134

Let's update ssl error codes #84134

Comments

dimaqq mannequin commented Mar 13, 2020

dimaqq mannequin commented Mar 13, 2020

define SSL_R_APPLICATION_DATA_AFTER_CLOSE_NOTIFY 291

benjaminp commented Apr 11, 2020

miss-islington commented Apr 11, 2020

shihai1991 commented Apr 12, 2020

tiran commented Apr 12, 2020

benjaminp commented Apr 12, 2020

benjaminp commented Apr 12, 2020

miss-islington commented Apr 12, 2020

asottile mannequin commented Apr 13, 2020

tiran commented Apr 13, 2020

benjaminp commented Apr 14, 2020

benjaminp commented Apr 14, 2020

miss-islington commented Apr 14, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 15, 2020

SilentGhost mannequin commented Apr 15, 2020

aixtools commented Apr 15, 2020

aixtools commented Apr 16, 2020