[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() #87416

vstinner · 2021-02-18T11:21:35Z

BPO	43250
Nosy	@vstinner, @njsmith, @corona10, @shihai1991

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-18.11:21:34.559>
labels = ['expert-C-API', '3.10']
title = '[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT()'
updated_at = <Date 2021-02-22.12:20:04.947>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-02-22.12:20:04.947>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-02-18.11:21:34.559>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43250
keywords = []
message_count = 8.0
messages = ['387213', '387215', '387218', '387219', '387220', '387225', '387461', '387467']
nosy_count = 4.0
nosy_names = ['vstinner', 'njs', 'corona10', 'shihai1991']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue43250'
versions = ['Python 3.10']

vstinner · 2021-02-18T11:21:34Z

In Python 3.6, there was a WANT_SIGFPE_HANDLER macro defined if Python was built with ./configure --with-fpectl. By default, the macro was not defined. The fpectl module installed a SIGFPE signal handler which uses longjmp(). If the WANT_SIGFPE_HANDLER macro was defined, PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros used setjmp() to catch SIGFPE.

All of this machinery was removed by bpo-29137 to fix ABI issues:

commit 735ae8d
Author: Nathaniel J. Smith <njs@pobox.com>
Date: Fri Jan 5 23:15:34 2018 -0800

bpo-29137: Remove fpectl module (bpo-4789)

This module has never been enabled by default, never worked correctly
on x86-64, and caused ABI problems that caused C extension
compatibility. See bpo-29137 for details/discussion.

The PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros still exist for backward compatibility with Python 3.6 and older. Python/pyfpe.c contains the comment:

/* These variables used to be used when Python was built with --with-fpectl,

but support for that was dropped in 3.7. We continue to define them,
though, because they may be referenced by extensions using the stable ABI.
*/

I'm not convinced that these macros are part of the stable ABI. The PEP-384 doesn't exclude pyfpe.h explicitly, but it's also unclear if it's part of the stable ABI or not. "PyFPE_START_PROTECT" and "PyFPE_END_PROTECT" symbols are not part of the stable ABI at least: they are missing from PC/python3dll.c and Doc/data/stable_abi.dat. Well, these are macros which emitted directly code rather than calling an opaque function. That sounds really bad in term of stable ABI :-(

Python 3.6 macros, when WANT_SIGFPE_HANDLER macro is defined:

---

#include <signal.h>
#include <setjmp.h>
#include <math.h>
extern jmp_buf PyFPE_jbuf;
extern int PyFPE_counter;
extern double PyFPE_dummy(void *);

#define PyFPE_START_PROTECT(err_string, leave_stmt) \
if (!PyFPE_counter++ && setjmp(PyFPE_jbuf)) { \
	PyErr_SetString(PyExc_FloatingPointError, err_string); \
	PyFPE_counter = 0; \
	leave_stmt; \
}

#define PyFPE_END_PROTECT(v) PyFPE_counter -= (int)PyFPE_dummy(&(v));

I propose to either deprecate these macros or remove them. These macros are not documented.

In Python 3.6, the PyFPE macros were used in the Python stdlib, by files:

Doc/library/fpectl.rst
Include/pyfpe.h
Modules/_tkinter.c
Modules/clinic/cmathmodule.c.h
Modules/cmathmodule.c
Modules/fpetestmodule.c
Modules/mathmodule.c
Objects/complexobject.c
Objects/floatobject.c
Python/bltinmodule.c
Python/pystrtod.c

But it doesn't seem to be used outside Python.

Note: In master, they are not used anywhere, they are only defined by Include/pyfpe.h.

vstinner · 2021-02-18T11:23:21Z

Latest commit related to these macros:

commit be143ec
Author: Victor Stinner <vstinner@python.org>
Date: Wed Nov 20 02:51:30 2019 +0100

bpo-38835: Don't use PyFPE_START_PROTECT and PyFPE_END_PROTECT (GH-17231)

The PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros are empty:
they have been doing nothing for the last year  (since commit
735ae8d139a673b30b321dc10acfd3d14f0d633b), so stop using them.

vstinner · 2021-02-18T11:25:37Z

Another related change from bpo-37474:

commit 5e0ea75
Author: Victor Stinner <vstinner@python.org>
Date: Tue Oct 1 13:12:29 2019 +0200

bpo-37474: Don't call fedisableexcept() on FreeBSD (GH-16515)

On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.

vstinner · 2021-02-18T11:27:26Z

I explicitly excluded the PyFPE macros from the limited C API in Python 3.9:

commit 488d02a
Author: Victor Stinner <vstinner@python.org>
Date: Wed Nov 20 12:17:09 2019 +0100

bpo-38835: Exclude PyFPE macros from the stable API (GH-17228)

Exclude PyFPE_START_PROTECT() and PyFPE_END_PROTECT() macros of
pyfpe.h from Py_LIMITED_API (stable API).

vstinner · 2021-02-18T11:28:01Z

See also bpo-38835: "pyfpe.h: Exclude PyFPE_START_PROTECT and PyFPE_END_PROTECT from the Py_LIMITED_API".

vstinner · 2021-02-18T12:15:45Z

Using INADA-san recipe https://github.com/methane/notes/tree/master/2020/wchar-cache I found the 62 projects on the PyPI top 4000 which contain "PyFPE" pattern:

asyncpg-0.22.0.tar.gz
auditwheel-3.3.1.tar.gz
av-8.0.3.tar.gz
bcolz-1.2.1.tar.gz
BDQuaternions-0.2.15.tar.gz
bx-python-0.8.9.tar.gz
ConfigSpace-0.4.18.tar.gz
Cython-0.29.21.tar.gz
cytoolz-0.11.0.tar.gz
ddtrace-0.46.0.tar.gz
dedupe-hcluster-0.3.8.tar.gz
dependency-injector-4.23.5.tar.gz
fastavro-1.3.2.tar.gz
fastdtw-0.3.4.tar.gz
Fiona-1.8.18.tar.gz
fuzzysearch-0.7.3.tar.gz
fuzzyset-0.0.19.tar.gz
gensim-3.8.3.tar.gz
grpcio-1.35.0.tar.gz
gssapi-1.6.12.tar.gz
hdbscan-0.8.27.tar.gz
hmmlearn-0.2.5.tar.gz
imagecodecs-2021.1.28.tar.gz
implicit-0.4.4.tar.gz
lightfm-1.16.tar.gz
lupa-1.9.tar.gz
lxml-4.6.2.tar.gz
mujoco-py-2.0.2.13.tar.gz
neobolt-1.7.17.tar.gz
orderedset-2.0.3.tar.gz
peewee-3.14.1.tar.gz
Pillow-8.1.0.tar.gz
Pillow-SIMD-7.0.0.post3.tar.gz
pmdarima-1.8.0.tar.gz
pomegranate-0.14.2.tar.gz
pycapnp-1.0.0.tar.gz
pydevd-2.2.0.tar.gz
pygame-2.0.1.tar.gz
pyhacrf-datamade-0.2.5.tar.gz
pyjq-2.5.1.tar.gz
pysam-0.16.0.1.tar.gz
pystan-2.19.1.1.tar.gz
PyWavelets-1.1.1.tar.gz
rasterio-1.2.0.tar.gz
ruptures-1.1.3.tar.gz
s2sphere-0.2.5.tar.gz
scikit-image-0.18.1.tar.gz
scipy-1.6.1.tar.gz
Shapely-1.7.1.tar.gz
simplejson-3.17.2.tar.gz
spacy-3.0.3.tar.gz
statsmodels-0.12.2.tar.gz
tables-3.6.1.tar.gz
Theano-1.0.5.tar.gz
thinc-8.0.1.tar.gz
tinycss-0.4.tar.gz
tslearn-0.5.0.5.tar.gz
uvloop-0.15.1.tar.gz
weighted_levenshtein-0.2.1.tar.gz
wordcloud-1.8.1.tar.gz
wsaccel-0.6.3.tar.gz
wxPython-4.1.1.tar.gz

Example with asyncpg:

$ rg PyFPE
asyncpg/pgproto/pgproto.c
39841:            PyFPE_START_PROTECT("add", return NULL)
39843:            PyFPE_END_PROTECT(result)

asyncpg/protocol/protocol.c
85260: PyFPE_START_PROTECT("add", return NULL)
85262: PyFPE_END_PROTECT(result)
85563: PyFPE_START_PROTECT("subtract", return NULL)
85565: PyFPE_END_PROTECT(result)
85734: PyFPE_START_PROTECT("add", return NULL)
85736: PyFPE_END_PROTECT(result)

So it doesn't sound like a good idea to immediately remove these two macros.

shihai1991 · 2021-02-21T15:32:32Z

So it doesn't sound like a good idea to immediately remove these two macros.

MAYBE we can update the annotation in pyfpe.h to address that we will remove those macros in python 4.0.

The fpe have been removed, so those macros will be removed finally.

njsmith · 2021-02-21T17:25:11Z

If I remember correctly, cython-generated C code calls those macros at
appropriate places, so there are probably a bunch of projects that are
using them and the developers have no idea.

On Thu, Feb 18, 2021, 04:16 STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor <vstinner@python.org> added the comment:

Using INADA-san recipe
https://github.com/methane/notes/tree/master/2020/wchar-cache I found the
62 projects on the PyPI top 4000 which contain "PyFPE" pattern:

asyncpg-0.22.0.tar.gz
auditwheel-3.3.1.tar.gz
av-8.0.3.tar.gz
bcolz-1.2.1.tar.gz
BDQuaternions-0.2.15.tar.gz
bx-python-0.8.9.tar.gz
ConfigSpace-0.4.18.tar.gz
Cython-0.29.21.tar.gz
cytoolz-0.11.0.tar.gz
ddtrace-0.46.0.tar.gz
dedupe-hcluster-0.3.8.tar.gz
dependency-injector-4.23.5.tar.gz
fastavro-1.3.2.tar.gz
fastdtw-0.3.4.tar.gz
Fiona-1.8.18.tar.gz
fuzzysearch-0.7.3.tar.gz
fuzzyset-0.0.19.tar.gz
gensim-3.8.3.tar.gz
grpcio-1.35.0.tar.gz
gssapi-1.6.12.tar.gz
hdbscan-0.8.27.tar.gz
hmmlearn-0.2.5.tar.gz
imagecodecs-2021.1.28.tar.gz
implicit-0.4.4.tar.gz
lightfm-1.16.tar.gz
lupa-1.9.tar.gz
lxml-4.6.2.tar.gz
mujoco-py-2.0.2.13.tar.gz
neobolt-1.7.17.tar.gz
orderedset-2.0.3.tar.gz
peewee-3.14.1.tar.gz
Pillow-8.1.0.tar.gz
Pillow-SIMD-7.0.0.post3.tar.gz
pmdarima-1.8.0.tar.gz
pomegranate-0.14.2.tar.gz
pycapnp-1.0.0.tar.gz
pydevd-2.2.0.tar.gz
pygame-2.0.1.tar.gz
pyhacrf-datamade-0.2.5.tar.gz
pyjq-2.5.1.tar.gz
pysam-0.16.0.1.tar.gz
pystan-2.19.1.1.tar.gz
PyWavelets-1.1.1.tar.gz
rasterio-1.2.0.tar.gz
ruptures-1.1.3.tar.gz
s2sphere-0.2.5.tar.gz
scikit-image-0.18.1.tar.gz
scipy-1.6.1.tar.gz
Shapely-1.7.1.tar.gz
simplejson-3.17.2.tar.gz
spacy-3.0.3.tar.gz
statsmodels-0.12.2.tar.gz
tables-3.6.1.tar.gz
Theano-1.0.5.tar.gz
thinc-8.0.1.tar.gz
tinycss-0.4.tar.gz
tslearn-0.5.0.5.tar.gz
uvloop-0.15.1.tar.gz
weighted_levenshtein-0.2.1.tar.gz
wordcloud-1.8.1.tar.gz
wsaccel-0.6.3.tar.gz
wxPython-4.1.1.tar.gz

Example with asyncpg:

$ rg PyFPE
asyncpg/pgproto/pgproto.c
39841: PyFPE_START_PROTECT("add", return NULL)
39843: PyFPE_END_PROTECT(result)

asyncpg/protocol/protocol.c
85260: PyFPE_START_PROTECT("add", return NULL)
85262: PyFPE_END_PROTECT(result)
85563: PyFPE_START_PROTECT("subtract", return NULL)
85565: PyFPE_END_PROTECT(result)
85734: PyFPE_START_PROTECT("add", return NULL)
85736: PyFPE_END_PROTECT(result)

So it doesn't sound like a good idea to immediately remove these two
macros.

----------

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue43250\>

vstinner · 2022-11-03T17:53:02Z

I don't have the bandwidth to update Cython and mark the macro as deprecated. So I just close the issue.

da-woods · 2023-11-18T16:30:03Z

I don't have the bandwidth to update Cython and mark the macro as deprecated. So I just close the issue.

I'm about to kill them off in Cython - cython/cython#5841.

vstinner added 3.10 only security fixes topic-C-API labels Feb 18, 2021

rhettinger changed the title ~~[C API] Depreate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT()~~ [C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() Feb 22, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

vstinner closed this as completed Nov 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() #87416

[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() #87416

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

shihai1991 commented Feb 21, 2021

njsmith commented Feb 21, 2021

vstinner commented Nov 3, 2022

da-woods commented Nov 18, 2023

[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() #87416

[C API] Deprecate or remove PyFPE_START_PROTECT() and PyFPE_END_PROTECT() #87416

Comments

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

vstinner commented Feb 18, 2021

shihai1991 commented Feb 21, 2021

njsmith commented Feb 21, 2021

vstinner commented Nov 3, 2022

da-woods commented Nov 18, 2023