I am seeing the following non-deterministic behavior:

My code processes DeviceTree, a tree-based specification format for hardware descriptions, that includes cross-references ("phandles").  For all intents and purposes, this format is similar to XML; phandles are analog to ID/IDREFS.

To prevent reference cycles and avoid the need for garbage collection, my code uses weakref.proxy for parent pointers and weakref.ref for cross-references.

My goal is to provide a "projection" operation on a DeviceTree: creating derived DeviceTrees that model subsets of the hardware (this is to partition the hardware into multiple independent sub-machines).  The projection is specified by newly introduced nodes and attributes (aka properties) in the tree; phandles are used to indicate which part belongs to which partition.

Python weak references provide a callback interface to indicate the demise of their referents and my code uses that to prune the tree: e.g., if a node modeling a partition is deleted, nodes that reference that node (i.e., indicate they belong to that partition) are deleted in the corresponding weakref callback.  So technically, the code implicitly uses the interpreters list of weak referrers (__weakref__) to find and execute code on them when the referent's state changes.

This works exactly as envisioned when single-stepping in PDB.

When running at full speed however, I see that weak reference callbacks are being triggered after the corresponding weak reference has been deleted with del (the weak reference is a value of a Python dict holding a node's attributes.)

I suspect that this is because of some batching or deferred processing in the Python interpreter.

Ultimately, this is a violation of the semantics and must be classified as a bug.

However, in my case, it would suffice to have a "memory barrier" type of operation that flushes the queue of deferred deletions before continuing.  Something like that must exist, because single stepping in PDB is successful.

Initial tests of calling the garbage collector to this end were inconclusive, unfortunately.