Title: Raised exception in Enum keeping user objects alive unnecessarily
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: efroemling, ethan.furman, gerald.dalley2, mark.dickinson
Priority: normal Keywords: patch

Created on 2020-11-02 20:46 by efroemling, last changed 2021-04-17 12:00 by mark.dickinson. This issue is now closed.

File name Uploaded Description Edit efroemling, 2020-11-02 20:46 script to demonstrate the issue
Pull Requests
URL Status Linked Edit
PR 25350 merged ethan.furman, 2021-04-12 04:13
PR 25369 merged ethan.furman, 2021-04-12 18:53
PR 25370 merged ethan.furman, 2021-04-12 19:05
Messages (8)
msg380249 - (view) Author: Eric Froemling (efroemling) * Date: 2020-11-02 20:46
I've run into an issue where exceptions thrown by Enum constructors are keeping my objects alive. The underlying issue seems to be the same as

The same method used to fix the issue above seems to work here: simply adding a try/finally clause around the error handling at the end of enum.Enum.__new__() which sets ve_exc and exc to None does the trick.

I've attached a short script which demonstrates the issue. I realize that the cyclic garbage collector will eventually handle this case, but its a bummer to lose determinism in the destruction of my objects.

I'd be happy to create a PR for this or whatever I can do to help; just let me know if I should (I'm new here).
msg382523 - (view) Author: Gerald Dalley (gerald.dalley2) Date: 2020-12-04 20:09
I and a few others have run into issues with the Enum constructors producing spurious reference cycles. This can cause memory explosions if large objects like numpy arrays are held in any of the relevant stack frames. Based on, it looks like the maintainers of CPython are open to fixing similar issues, and PRs look like the way to make progress.
msg390093 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-02 19:26
I'm running into this, too, on Python 3.9.2; in my case it's causing segmentation faults in a wxPython-based application. 

Those segfaults aren't the fault of the reference cycle, of course; they're a result of wxPython being finicky about object cleanup order, but the existence of the enum reference cycle does make it harder to maneuver wxPython into a place where it doesn't crash.
msg390095 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-02 20:22
Here's a cut-down example, at the prompt:

Python 3.9.2 (default, Mar 31 2021, 05:47:22) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal
>>> class App: pass
>>> def create_app():
...     app = App()
...     signal.signal(signal.SIGINT, signal.SIG_DFL)
>>> create_app()

At this point, since the App() instance was local to create_app, and wasn't returned, I'd expect there to be no App objects alive in the system. But it turns out there's still an App object being kept alive:

>>> import gc
>>> [obj for obj in gc.get_objects() if type(obj) is App]
[<__main__.App object at 0x10acb3d90>]

The cause is a call to _int_to_enum in which attempts to coerce the default signal handler to an element of Handlers. That coercion fails, leaving an exception

  ValueError('<built-in function default_int_handler> is not a valid Handlers')

The traceback on that exception leads to the frame containing the call to Enum.__new__, which in turn contains a reference ve_exc back to the exception.

[In the real code, App was a wx.App object, and the App.__init__ method played the role of create_app.]
msg390859 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 15:51
New changeset 8c14f5a787b21d5a1eae5d5ee981431d1c0e055f by Ethan Furman in branch 'master':
bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350)
msg390889 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 19:16
New changeset f396a1a940f8608a4be2a9ac4ef82e37c198ecd3 by Ethan Furman in branch '3.8':
[3.8] bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350). (GH-25369)
msg390913 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2021-04-12 22:03
New changeset 6379924ecd51e346b42b0293da0f4442a0f67707 by Ethan Furman in branch '3.9':
[3.9] bpo-42248: [Enum] ensure exceptions raised in ``_missing_`` are released (GH-25350). (GH-25370)
msg391285 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2021-04-17 12:00
Thank you for the quick fix!
Date User Action Args
2021-04-17 12:00:11mark.dickinsonsetmessages: + msg391285
2021-04-15 14:25:51ethan.furmansetstatus: open -> closed
versions: + Python 3.9, Python 3.10
resolution: fixed
assignee: ethan.furman
type: enhancement -> behavior
stage: patch review -> resolved
2021-04-12 22:03:39ethan.furmansetmessages: + msg390913
2021-04-12 19:16:54ethan.furmansetmessages: + msg390889
2021-04-12 19:05:32ethan.furmansetpull_requests: + pull_request24103
2021-04-12 18:53:57ethan.furmansetpull_requests: + pull_request24102
2021-04-12 15:51:28ethan.furmansetmessages: + msg390859
2021-04-12 04:13:06ethan.furmansetkeywords: + patch
stage: patch review
pull_requests: + pull_request24085
2021-04-02 20:22:45mark.dickinsonsetmessages: + msg390095
2021-04-02 19:26:15mark.dickinsonsetnosy: + ethan.furman, mark.dickinson
messages: + msg390093
2020-12-04 20:09:11gerald.dalley2settype: enhancement

messages: + msg382523
nosy: + gerald.dalley2
2020-11-02 20:46:49efroemlingcreate