classification
Title: queue example doesn't stop worker threads
Type: Stage:
Components: Documentation Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: docs@python, pitrou, python-dev, rhettinger, terry.reedy, vstinner
Priority: low Keywords:

Created on 2011-05-23 10:22 by vstinner, last changed 2015-03-18 13:06 by vstinner. This issue is now closed.

Messages (5)
msg136601 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-23 10:22
The queue doc contains the following example:
------------------
def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done
------------------
http://docs.python.org/library/queue.html

It doesn't define do_work(), num_worker_threads or do_work(), but my concern is that it doesn't stop worker threads.

I consider "t.daemon = True" as an hack to not care about stopping threads.

The example should pass a special value to each worker to stop it. For example:

<worker>

while True:
  job = queue.get()
  if job is None:
     break
  audio.play(*job)
  queue.task_done()

Main thread:

...
threads = []
for i in range(num_worker_threads):
     t = Thread(target=worker)
     threads.append(t)
     t.start()
...
for i in range(num_worker_threads):
     queue.put(None)
queue.join()
for thread in threads:
    thread.join()
msg136710 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-05-24 01:17
> It doesn't define do_work(), num_worker_threads or do_work()

Is it unclear to you what those mean?  They are placeholders for the user's actual task at hand.

> my concern is that it doesn't stop worker threads.

Stopping the threads wasn't the intent of the example.

> I consider "t.daemon = True" as an hack to not care 
> about stopping threads.

If you post a high-quality self-contained example somewhere on the net, I would be happy to link to it.
msg136724 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-24 07:32
> Is it unclear to you what those mean?

Well, it's clear, but I like when I can simply copy/paste the example and it does just work:

> If you post a high-quality self-contained example somewhere 
> on the net, I would be happy to link to it.

I just propose to change the example to stop the threads:
------------------
def worker():
    while True:
        item = q.get()
        if item is None:
            break
        do_work(item)
        q.task_done()

q = Queue()
threads = []
for i in range(num_worker_threads):
     t = Thread(target=worker)
     threads.append(t)
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done
for i in range(num_worker_threads):
    q.put(None)
for t in threads: t.join()
------------------
msg137153 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-28 20:55
The proposed change adds about 7 lines to show the 'trick' of putting num-worker Nones on the queue. I think that is worth it.
msg238426 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-03-18 13:06
New changeset b44ec269abda by Victor Stinner in branch 'default':
Issue #12155: Fix queue doc example to join threads
https://hg.python.org/cpython/rev/b44ec269abda
History
Date User Action Args
2015-03-18 13:06:34vstinnersetstatus: open -> closed
resolution: fixed
versions: + Python 3.5, - Python 3.1, Python 2.7, Python 3.2, Python 3.3
2015-03-18 13:06:20python-devsetnosy: + python-dev
messages: + msg238426
2011-05-28 20:55:18terry.reedysetnosy: + terry.reedy
messages: + msg137153
2011-05-24 07:32:36vstinnersetmessages: + msg136724
2011-05-24 01:17:49rhettingersetpriority: normal -> low

messages: + msg136710
2011-05-24 01:08:33rhettingersetassignee: docs@python -> rhettinger
2011-05-23 10:49:17pitrousetnosy: + rhettinger
2011-05-23 10:22:30vstinnercreate