This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: memory leak with threads and enhancement of the timer class
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.2
process
Status: closed Resolution: duplicate
Dependencies: Superseder: concurrent.futures: add ScheduledExecutor
View: 17956
Assigned To: Nosy List: ajaksu2, neologix, pitrou, r.david.murray, thaar, yael
Priority: normal Keywords: patch

Created on 2004-07-22 13:16 by thaar, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
mywork.patch yael, 2013-04-14 16:17 Patch implementing Threading.TimerPool + tests. review
scheduled.diff neologix, 2013-05-10 17:08
test.py neologix, 2013-05-10 17:08
Messages (11)
msg21751 - (view) Author: Tobias Haar (thaar) Date: 2004-07-22 13:16
If i use the Timer cyclically the  memory becomes
always less.


I found following problems::

1.) The thread is not clean deleted.=>
file:threading.py  class: Thread methode:__delete

del _active[_get_ident()] only delete the thead from
the list, not the thead self.  I think the call of the
destructor of the c++ based library need a explicit del.

The problem will be fixed with following lines:

    def __delete(self):
        _active_limbo_lock.acquire()
        t=_active[_get_ident()]
        del _active[_get_ident()]
        del t
        _active_limbo_lock.release()

2.) A cyclic timer is a needed feature and it should
not use a new thread every time.

So i made following enhancement (parameter cyclic) in the 
file:threading.py  class: _Timer  

class _Timer(Thread):
    """Call a function after a specified number of seconds:

    t = Timer(30.0, f, args=[], kwargs={})
    t.start()
    t.cancel() # stop the timer's action if it's still
waiting
    """

    def __init__(self, interval, function, cyclic=0,
args=[], kwargs={}):
        Thread.__init__(self)
        self.interval = interval
        self.function = function
        self.args = args
        self.kwargs = kwargs
        self.finished = Event()
        self.cyclic= Event()
        if cyclic:
            self.cyclic.set()

    def cancel(self):
        """Stop the timer if it hasn't finished yet"""
        self.cyclic.clear()
        self.finished.set()

    def run(self):
        flag =1
        while flag:
            self.finished.wait(self.interval)
            if not self.finished.isSet():
                self.function(*self.args, **self.kwargs)
            if not self.cyclic.isSet():
                self.finished.set()
                flag = 0         



msg82079 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-02-14 14:46
Can't quite understand what the problem is supposed to be, comments?
msg82090 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-14 16:52
I can't understand problem number 1 either. I suspect it is a
misunderstanding by the poster, since he is talking about a "destructor
of the c++ based library" and there's no c++ code in the interpreter.

As for number 2, it is a legitimate feature request. However, since the
current Timer class is very inefficient (one separate thread for each
timer), it would make sense to rephrase it as a more general suggestion
to re-implement the Timer class using a single background thread and a
heapq-based priority queue. It would be a good project for someone
wanting to start contributing to the interpreter.
msg186829 - (view) Author: Yael (yael) * Date: 2013-04-13 20:30
I am working on a patch for a new class that uses a single background thread, it should be ready soon. One unintended consequence of this change is that with one thread, multiple timers that have the same timeout will no longer run in parallel, but one after the other.
msg186927 - (view) Author: Yael (yael) * Date: 2013-04-14 16:17
Added a class Threading.TimerPool. This new class spawns one thread, and that thread is running as long as there are active timers.
msg188438 - (view) Author: Yael (yael) * Date: 2013-05-05 12:04
Can you please review the patch? thanks!
msg188700 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-05-08 01:51
Review comments added.
msg188710 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-08 08:22
IMO, this shouldn't be implemented atop thread, but ought to be a regular thread pool: this way, you won't get behind if some task takes too long to execute, the thread pool can start new threads as needed, and we get the general work submit/cancel (through future) for free.
Also, it would probably deserve a new interface in concurrent.futures, as ScheduledExecutor, with new schedule(delay, fn, *args, **kwargs) and schedule_periodic(delay, fn, *args, **kwargs) for one-shot and periodic calls.

It would be much more consistant than an ad-hoc implementation in the threading module.
msg188712 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-05-08 10:29
I take your point; I knew there was something bothering me about how the tasks were handled but I didn't consciously see the bug.  I like the idea of a ScheduledExecutor.

Yael, thanks a lot for working through this, but I think we should probably close this issue and open a new one for adding a ScheduledExecutor to concurrent.futures, and make see-also link from the Timer class to it.  Would you be interested in working on it?
msg188851 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-10 17:08
I'm attaching a proof of concept code for a ScheduledExecutor
interface, and a ScheduledThreadPoolExecutor implementation
(unfortunately I can't upload it as a mercurial diff for now).

Here's what the API looks like:

"""
from concurrent.futures import ScheduledThreadPoolExecutor
import time

def say(text):
    print("{}: {}".format(time.ctime(), text))

with ScheduledThreadPoolExecutor(5) as p:
    p.schedule(1, say, 'hello 1')
    f = p.schedule_fixed_rate(0, 2, say, 'hello 2')
    p.schedule_fixed_delay(0, 3, say, 'hello 3')
    time.sleep(6)
    say("cancelling: %s" % f)
    f.cancel()
    time.sleep(10)
    say("shutting down")
"""

schedule() is for one-shot, schedule_fixed_rate() for fixed rate
scheduling (i.e. there will be no drift due to the task execution
time), and schedule_fixed_delay() is for fixed delay (i.e. there will
always be a fixed amount of time between two invokations).

Random notes:
- the scheduling is handled by a new SchedQueue in the queue module:
sched would have been useful, but actually it can't be used here: it
stops as soon as the queue is empty, when it calls the wait function
it won't wake up if a new task is enqueued, etc. Also, I guess such a
queue could be useful in general.
- I had to create a DelayedFuture subclass, which is returned by
schedule_XXX methods. The main differences with raw Future are that it
has a scheduled time and period attributes, and supports
reinitialization (a future can only be run once). It can be cancelled,
and also supports result/exception retrieval.
- I don't know if a process-based counterpart
(ScheduledProcessPoolExecutor) is really useful. I didn't look at it
for now.
msg188936 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-05-11 17:46
OK, I just created #17956 for ScheduledExecutor, closing this one.
History
Date User Action Args
2022-04-11 14:56:05adminsetgithub: 40619
2013-05-11 17:46:49neologixsetstatus: open -> closed
superseder: concurrent.futures: add ScheduledExecutor
messages: + msg188936

resolution: duplicate
stage: test needed -> resolved
2013-05-10 17:08:21neologixsetfiles: + scheduled.diff, test.py

messages: + msg188851
2013-05-08 10:29:33r.david.murraysetmessages: + msg188712
2013-05-08 08:22:39neologixsetnosy: + neologix
messages: + msg188710
2013-05-08 01:51:26r.david.murraysetnosy: + r.david.murray
messages: + msg188700
2013-05-05 12:04:56yaelsetmessages: + msg188438
2013-04-14 16:17:32yaelsetfiles: + mywork.patch
keywords: + patch
messages: + msg186927
2013-04-13 20:30:41yaelsetnosy: + yael
messages: + msg186829
2010-08-19 15:28:34BreamoreBoysetversions: + Python 3.2, - Python 2.7
2009-02-14 16:52:19pitrousetnosy: + pitrou
messages: + msg82090
2009-02-14 14:46:07ajaksu2settype: enhancement
stage: test needed
messages: + msg82079
nosy: + ajaksu2
versions: + Python 2.7, - Python 2.3
2004-07-22 13:16:52thaarcreate