Author mark.dickinson
Recipients bquinlan, mark.dickinson, schmir
Date 2012-10-20.12:30:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1350736200.64.0.0405408744633.issue16284@psf.upfronthosting.co.za>
In-reply-to
Content
A new patch (with tests), and a fuller explanation:

At work, we've got Python talking to a customer's existing COM library; we're using Thomas Heller's 'comtypes' library to do that.  Unfortunately, comtypes depends quite a lot on __del__-time cleanup, so reference counting matters.  (I'm well aware that this isn't the recommended way to deal with resource cleanup in Python, but rewriting the existing infrastructure isn't a realistic option here.)

Anyway, it turned out that the concurrent.futures executors were indirectly holding onto references to COM objects, causing issues with our application.

The attached patch adds a few 'del' statements to remove references that are no longer needed.  For the ProcessExecutor, some of those 'del' statements had to go into the multiprocessing.Queue implementation.

The troublesome pattern (in both multiprocessing and futures) takes the form (simplified):


def my_worker_function(...):
    ...
    while <exit_condition_not_satisfied>:
        obj = blocking_wait_for_next_item()
        do_processing(obj)
    ...

The issue is that the reference to obj is kept until the completion of the next blocking wait call.  I'm suggesting just adding an extra 'del obj' after 'do_processing(obj)'.
History
Date User Action Args
2012-10-20 12:30:00mark.dickinsonsetrecipients: + mark.dickinson, bquinlan, schmir
2012-10-20 12:30:00mark.dickinsonsetmessageid: <1350736200.64.0.0405408744633.issue16284@psf.upfronthosting.co.za>
2012-10-20 12:30:00mark.dickinsonlinkissue16284 messages
2012-10-20 12:30:00mark.dickinsoncreate