classification
Title: enum.Enum reference count leaks
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: barry, dan.gass@gmail.com, eli.bendersky, ethan.furman, hongweipeng
Priority: low Keywords:

Created on 2020-01-25 06:30 by dan.gass@gmail.com, last changed 2020-09-14 21:00 by ethan.furman. This issue is now closed.

Files
File name Uploaded Description Edit
sample.py dan.gass@gmail.com, 2020-01-25 06:30 demonstrate refcount leak and likely root cause pattern
Messages (3)
msg360670 - (view) Author: Dan Gass (dan.gass@gmail.com) Date: 2020-01-25 06:30
Given
(1) instantiation of an enumeration class with an invalid value
(2) a try/except around the instantiation where the exception is ignored

Then:
An unneeded reference to the bad value is lost (as well as other values that I suspect are local variables within a participating method)

When run, the attached sample script shows before and after reference counts which demonstrate the potential resource leaks. The sample script includes the output from running the script on Python version 3.7.5 within the module docstring.

The root cause appears to be in the exception handling in the Enum.__new__ method (in the area where it calls the _missing_ hook). The attached sample script includes a simplified version of those methods that should help pinpoint the code in question and confirm the root cause.

Not being an exception nitty-gritty expert, I have suspicions that users should be warned about using this pattern of exception handling. I suspect this pattern would be worth avoiding in the Enum implementation.

I am willing to take a stab at submitting a patch for Enum. I hesitate slightly not knowing if there are specific reasons for the code existing in its current form. Alternatively, I plan on being at PyCon2020 for the sprints and could be available then to work on it.
msg360730 - (view) Author: Dan Gass (dan.gass@gmail.com) Date: 2020-01-26 17:26
Sorry for not thinking of trying this sooner. Running garbage collection, import gc; gc.collect(), releases the resources and restores the expected reference counts.

From my perspective, this is satisfactory and could justify closing this bug report. 

The exception logic in Enum.__new__ could potentially be cleaned up for this use case, just not sure if that would introduce risk for other use cases. 

I am guessing (from the perspective of the larger Python exception handling design and implementation) what this bug report demonstrates had been previously thought through.
msg376905 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2020-09-14 21:00
Thanks, Dan, for checking this out.  As you noted, a garbage collection solved the issue, so I'm not going to worry about it.
History
Date User Action Args
2020-09-14 21:00:10ethan.furmansetstatus: open -> closed
resolution: not a bug
messages: + msg376905

stage: resolved
2020-02-10 04:26:22hongweipengsetnosy: + hongweipeng
2020-01-26 17:26:33dan.gass@gmail.comsetmessages: + msg360730
2020-01-25 21:27:05rhettingersetpriority: normal -> low
assignee: ethan.furman
2020-01-25 07:05:52xtreaksetnosy: + barry, eli.bendersky, ethan.furman
2020-01-25 06:30:33dan.gass@gmail.comcreate