classification
Title: Raised exception in Enum keeping user objects alive unnecessarily
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: efroemling, ethan.furman, gerald.dalley2, mark.dickinson, pitrou
Priority: normal Keywords: patch

Created on 2020-11-02 20:46 by efroemling, last changed 2021-11-13 19:10 by ethan.furman. This issue is now closed.

Files
File name Uploaded Description Edit
enum_ref_loop_example.py efroemling, 2020-11-02 20:46 script to demonstrate the issue
Pull Requests
URL Status Linked Edit
PR 25350 merged ethan.furman, 2021-04-12 04:13
PR 25369 merged ethan.furman, 2021-04-12 18:53
PR 25370 merged ethan.furman, 2021-04-12 19:05
Messages (12)
msg380249 - (view) Author: Eric Froemling (efroemling) * Date: 2020-11-02 20:46
I've run into an issue where exceptions thrown by Enum constructors are keeping my objects alive. The underlying issue seems to be the same as https://bugs.python.org/issue36820

The same method used to fix the issue above seems to work here: simply adding a try/finally clause around the error handling at the end of enum.Enum.__new__() which sets ve_exc and exc to None does the trick.

I've attached a short script which demonstrates the issue. I realize that the cyclic garbage collector will eventually handle this case, but its a bummer to lose determinism in the destruction of my objects.

I'd be happy to create a PR for this or whatever I can do to help; just let me know if I should (I'm new here).
msg382523 - (view) Author: Gerald Dalley (gerald.dalley2) Date: 2020-12-04 20:09
I and a few others have run into issues with the Enum constructors producing spurious reference cycles. This can cause memory explosions if large objects like numpy arrays are held in any of the relevant stack frames. Based on https://bugs.python.org/issue36820, it looks like the maintainers of CPython are open to fixing similar issues, and PRs look like the way to make progress.
msg390093 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-02 19:26
I'm running into this, too, on Python 3.9.2; in my case it's causing segmentation faults in a wxPython-based application. 

Those segfaults aren't the fault of the reference cycle, of course; they're a result of wxPython being finicky about object cleanup order, but the existence of the enum reference cycle does make it harder to maneuver wxPython into a place where it doesn't crash.
msg390095 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-02 20:22
Here's a cut-down example, at the prompt:

Python 3.9.2 (default, Mar 31 2021, 05:47:22) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal
>>> class App: pass
... 
>>> def create_app():
...     app = App()
...     signal.signal(signal.SIGINT, signal.SIG_DFL)
... 
>>> create_app()

At this point, since the App() instance was local to create_app, and wasn't returned, I'd expect there to be no App objects alive in the system. But it turns out there's still an App object being kept alive:

>>> import gc
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[<__main__.App object at 0x10acb3d90>]

The cause is a call to _int_to_enum in signal.py which attempts to coerce the default signal handler to an element of Handlers. That coercion fails, leaving an exception

  ValueError('<built-in function default_int_handler> is not a valid Handlers')

The traceback on that exception leads to the frame containing the call to Enum.__new__, which in turn contains a reference ve_exc back to the exception.

[In the real code, App was a wx.App object, and the App.__init__ method played the role of create_app.]
msg390859 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 15:51
New changeset 8c14f5a787b21d5a1eae5d5ee981431d1c0e055f by Ethan Furman in branch 'master':
bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350)
https://github.com/python/cpython/commit/8c14f5a787b21d5a1eae5d5ee981431d1c0e055f
msg390889 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 19:16
New changeset f396a1a940f8608a4be2a9ac4ef82e37c198ecd3 by Ethan Furman in branch '3.8':
[3.8] bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350). (GH-25369)
https://github.com/python/cpython/commit/f396a1a940f8608a4be2a9ac4ef82e37c198ecd3
msg390913 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 22:03
New changeset 6379924ecd51e346b42b0293da0f4442a0f67707 by Ethan Furman in branch '3.9':
[3.9] bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350). (GH-25370)
https://github.com/python/cpython/commit/6379924ecd51e346b42b0293da0f4442a0f67707
msg391285 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-17 12:00
Thank you for the quick fix!
msg403860 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-10-13 17:38
It appears this is *not* fixed in 3.10.0:

Python 3.10.0 (default, Oct 13 2021, 08:45:17) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal, gc
>>> class App: pass
... 
>>> def create_app():
...   app = App()
...   signal.signal(signal.SIGINT, signal.SIG_DFL)
... 
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[]
>>> create_app()
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[<__main__.App object at 0x7f2d3f3f1c30>]
msg403861 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-10-13 17:40
Indeed, changeset  8c14f5a787b21d5a1eae5d5ee981431d1c0e055f is part of git main, but is not on the 3.10 branch AFAICT.
msg403862 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-10-13 17:42
It seems the problem is that bpo-44559 reset the enum module to a previous state without the bugfix.
msg403875 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-10-13 21:33
Going back through the various bug fixes that got cut from 3.10 to re-add to 3.10.1.
History
Date User Action Args
2021-11-13 19:10:16ethan.furmansetstatus: open -> closed
resolution: fixed
2021-10-13 21:33:39ethan.furmansetmessages: + msg403875
2021-10-13 17:42:40pitrousetmessages: + msg403862
2021-10-13 17:40:25pitrousetmessages: + msg403861
2021-10-13 17:38:53pitrousetstatus: closed -> open

nosy: + pitrou
messages: + msg403860

resolution: fixed -> (no value)
2021-04-17 12:00:11mark.dickinsonsetmessages: + msg391285
2021-04-15 14:25:51ethan.furmansetstatus: open -> closed
versions: + Python 3.9, Python 3.10
resolution: fixed
assignee: ethan.furman
type: enhancement -> behavior
stage: patch review -> resolved
2021-04-12 22:03:39ethan.furmansetmessages: + msg390913
2021-04-12 19:16:54ethan.furmansetmessages: + msg390889
2021-04-12 19:05:32ethan.furmansetpull_requests: + pull_request24103
2021-04-12 18:53:57ethan.furmansetpull_requests: + pull_request24102
2021-04-12 15:51:28ethan.furmansetmessages: + msg390859
2021-04-12 04:13:06ethan.furmansetkeywords: + patch
stage: patch review
pull_requests: + pull_request24085
2021-04-02 20:22:45mark.dickinsonsetmessages: + msg390095
2021-04-02 19:26:15mark.dickinsonsetnosy: + ethan.furman, mark.dickinson
messages: + msg390093
2020-12-04 20:09:11gerald.dalley2settype: enhancement

messages: + msg382523
nosy: + gerald.dalley2
2020-11-02 20:46:49efroemlingcreate