classification
Title: Raising an exception raised in a "future" instance will create reference cycles
Type: Stage: resolved
Components: Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jcea Nosy List: jcea, miss-islington, ned.deily
Priority: normal Keywords: patch

Created on 2019-02-07 14:25 by jcea, last changed 2021-04-26 12:25 by jcea. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24995 merged jcea, 2021-03-23 16:48
PR 25070 merged miss-islington, 2021-03-29 17:22
PR 25071 merged miss-islington, 2021-03-29 17:22
PR 25072 closed miss-islington, 2021-03-29 17:22
PR 25073 closed miss-islington, 2021-03-29 17:22
Messages (9)
msg335022 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2019-02-07 14:25
Try this in a terminal:

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""

You will see (python 3.7) that 23 objects are collected when cleaning the cycle.

The problem is the attribute "future._exception". If the exception provided by the "future" is raised somewhere else, we will have reference cycles because we have the same exception/traceback in two different places in the traceback framestack.

I commonly do this in my code:

"""
try:
  future.result()  # This will raise an exception if the future did it
except Exception:
   ... some clean up ...
  raise  # Propagate the "future" exception
"""

This approach will create reference cycles. They will eventually cleaned up, but I noticed this issue because the cycle clean up phase was touching big objects with many references but unused for a long time, so they were living in the SWAP. The cycle collection was hugely slow because of this and the interpreter is completely stopped until done.

Not sure about what to do about this. I am currently doing something like:

"""
try:
  future.result()  # This will raise an exception if the future did it
except Exception:
   if future.done():
       del future._exception
  raise  # Propagate the exception
"""

I am breaking the cycle manually. I do not use "future.set_exception(None) because side effects like notifying waiters.

I think this is a bug to be solved. Not sure how to do it cleanly.

What do you think? Ideas?.
msg383099 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2020-12-15 21:37
The corrected test case in the terminal: (a "del" was missing)

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
del future  # Variable vanish, but data is still there because the cycle
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""
msg383112 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2020-12-16 01:21
Even more reproductible case, now 100%:

"""
import gc
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(999)

def a():
  1/0

future=executor.submit(a)
future.result()
# An exception is raised here. That is normal
del future  # Variable vanish, but data is still there because the cycle
1/0  # Raises another exception drop references to the future one
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
gc.garbage
"""
msg389714 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2021-03-29 17:22
New changeset 32430aadadf6e012e39167d3c18a24e49fb84874 by Jesús Cea in branch 'master':
bpo-35930: Raising an exception raised in a "future" instance will create reference cycles (#24995)
https://github.com/python/cpython/commit/32430aadadf6e012e39167d3c18a24e49fb84874
msg389717 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2021-03-29 17:53
New changeset dae1963cf38f730291126b7dadfda89ffb21cefd by Miss Islington (bot) in branch '3.8':
bpo-35930: Raising an exception raised in a "future" instance will create reference cycles (GH-24995) (#25071)
https://github.com/python/cpython/commit/dae1963cf38f730291126b7dadfda89ffb21cefd
msg389718 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2021-03-29 17:53
New changeset d914813a7a9cee3b42e9c91f91ac491f3bbfe118 by Miss Islington (bot) in branch '3.9':
bpo-35930: Raising an exception raised in a "future" instance will create reference cycles (GH-24995) (#25070)
https://github.com/python/cpython/commit/d914813a7a9cee3b42e9c91f91ac491f3bbfe118
msg389779 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-30 00:43
@jcea, I see you have created backports for 3.7 and 3.6 as well. Those release as in their security-fix-only phase of their life cycles. This doesn't seem like a security issue but am I missing something?
msg389780 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-30 00:44
"Those releases are in their"
msg391907 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2021-04-26 12:25
I retired the backporting request. Thanks, Ned.
History
Date User Action Args
2021-04-26 12:25:49jceasetmessages: + msg391907
versions: - Python 3.6, Python 3.7
2021-03-30 00:44:27ned.deilysetmessages: + msg389780
2021-03-30 00:43:44ned.deilysetnosy: + ned.deily
messages: + msg389779
2021-03-29 17:55:16jceasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-03-29 17:53:59jceasetmessages: + msg389718
2021-03-29 17:53:21jceasetmessages: + msg389717
2021-03-29 17:22:49miss-islingtonsetpull_requests: + pull_request23823
2021-03-29 17:22:44miss-islingtonsetpull_requests: + pull_request23822
2021-03-29 17:22:42jceasetmessages: + msg389714
2021-03-29 17:22:37miss-islingtonsetpull_requests: + pull_request23821
2021-03-29 17:22:28miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23820
2021-03-23 16:55:46jceasetversions: + Python 3.10
2021-03-23 16:55:35jceasetassignee: jcea
2021-03-23 16:48:02jceasetkeywords: + patch
stage: patch review
pull_requests: + pull_request23753
2020-12-16 01:22:01jceasetversions: + Python 3.8, Python 3.9
2020-12-16 01:21:48jceasetmessages: + msg383112
2020-12-15 21:37:40jceasetmessages: + msg383099
2019-02-07 14:25:30jceacreate