This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: py34 OrderedDict is using weakref for root reference
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: eric.snow, piotr.dobrogost, pitrou, rhettinger, ssbarnea, tim.peters
Priority: normal Keywords:

Created on 2015-04-01 12:01 by ssbarnea, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (5)
msg239789 - (view) Author: Sorin Sbarnea (ssbarnea) * Date: 2015-04-01 12:01
The standard library implementation of an OrderedDict (which is included in Python 3.4) uses a weakref for the root reference. 
https://hg.python.org/cpython/file/534b26837a13/Lib/collections/__init__.py#l74

This was reported initially on https://github.com/kennethreitz/requests/issues/2303 and at the moment there are no known workarounds.
msg240101 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-04-05 03:09
Antoine, I could use a second pair of eyes to see what is going on there.  It looks like an upstream __del__() method is trying to iterate over an OrderedDict that was already being shutdown (the hardroot link no longer exists).  That said, I don't see how the OD can be partially shutdown if the upstream code still has a reference to the OD in the adapters.
msg240114 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-05 11:39
weakrefs are traditionally cleared by the cyclic GC before calling any __del__ method. This used to be mandatory to eschew situations where a weakref callback could see cleared objects, but produces the side effect that __del__ methods can see dead weakrefs. This is also true pre-3.4, by the way, but perhaps the OP's __del__ wasn't called at all (if it was part of the cycle, the object would end up in gc.garbage instead)?

We could perhaps reverse the order, but then weakref callbacks may see __del__'ed objects. Hard to say which one is better, and keeping the traditional behaviour means less compatibility hassles.

Some of this is discussed in Modules/gc_weakref.txt.

In general, __del__ methods may see strange errors. For example, at interpreter shutdown, your __del__ method can be called while some modules have started unloading. Python 3.4 definitely tries to improve on all this, but there's no perfect solution, and some of the improvements might also backfire in some rare cases :-)
msg240115 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-05 11:42
Concretely, the possible workarounds are:

- don't do anything complex in your __del__
- be prepared to deal with unexpected errors in your __del__
- starting from Python 3.4, don't define __del__ and use weakref.finalize() instead: https://docs.python.org/3/library/weakref.html#weakref.finalize
msg240246 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-04-08 03:11
Closing as not-a-bug.  It seems that the __del__ method in requests needs to make one of the adjustments suggested by Antoine.
History
Date User Action Args
2022-04-11 14:58:15adminsetgithub: 68029
2015-04-08 03:11:08rhettingersetpriority: high -> normal
status: open -> closed
resolution: not a bug
messages: + msg240246
2015-04-05 11:42:09pitrousetmessages: + msg240115
2015-04-05 11:40:00pitrousetnosy: + tim.peters

messages: + msg240114
versions: + Python 3.5
2015-04-05 03:09:27rhettingersetnosy: + pitrou, eric.snow
messages: + msg240101
2015-04-02 06:55:36rhettingersetpriority: normal -> high
2015-04-02 06:53:35rhettingersetassignee: rhettinger
2015-04-01 12:26:40piotr.dobrogostsetnosy: + piotr.dobrogost
2015-04-01 12:05:36serhiy.storchakasetnosy: + rhettinger
type: compile error -> behavior
2015-04-01 12:01:47ssbarneacreate