This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Vyacheslav Grigoryev
Recipients Vyacheslav Grigoryev
Date 2016-10-17.15:38:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1476718684.05.0.447400261666.issue28462@psf.upfronthosting.co.za>
In-reply-to
Content
I'm creating a master stand-alone module on Python which should run some children via subprocess module. Working with children is done in separate worker threads. Additionally I need to receive real-time output from a child so in a worker thread I also create reader helper thread which reads on a pipe from the child.

All is working correctly while I have only one worker thread and run only one child. When the child finishes the reader helper thread gets EOF and exits. But when I have two worker threads and run two children, a pipe from early child doesn't see EOF until the second child is finished. Though they are completely unrelated.

Let's take a look on simplest example for reproducing the problem. There is a simplest child:
------------------------------
import time, sys

# first arg is an ID. Second arg is how long to work in seconds
sys.stdout.write("start slave %s\n" % sys.argv[1])
sys.stdout.flush()

time.sleep(int(sys.argv[2]))

sys.stdout.write("finish slave %s\n" % sys.argv[1])
sys.stdout.flush()
------------------------------

And there is a master module:
------------------------------
import subprocess, sys, os, threading, time

g_logLock = threading.Lock()

def log(msg):
    with g_logLock:
        t = time.time()
        print "%s.%03d %-5s %s" % \
            (time.strftime('%H:%M:%S', time.localtime(t)), int((t - t // 1) * 1000), threading.currentThread().name, msg)

def thread1Proc():
    def reader(stdout):
        while True:
            line = stdout.readline()
            if not line:
                break
            log('slave said: %s' % line.strip())
        log('finish slave reader thread')

    log('thread 1 started')
    timeToWork = '1'
    util = subprocess.Popen((sys.executable, 'slave.py', '1', timeToWork), stdout=subprocess.PIPE)
    readerT = threading.Thread(target=reader, args=(util.stdout,), name='t1-r')
    readerT.start()
    log('slave 1 returned %d' % util.wait())
    readerT.join()
    log('thread 1 finished')

def thread2Proc():
    log('thread 2 started')
    timeToWork = '3'
    util = subprocess.Popen((sys.executable, 'slave.py', '2', timeToWork))
    log('slave 2 returned %d' % util.wait())
    log('thread 2 finished')

#---------------------------
log('starting test')

threads = (threading.Thread(target=thread1Proc, name='t1'), threading.Thread(target=thread2Proc, name='t2'))
for t in threads:
    t.start()
for t in threads:
    t.join()

log('finished test')
------------------------------

Here is what I see on the output (note - slave 1 outputs to the master via pipe, while slave 2 outputs to a console because its output is not redirected):

>master.py
08:57:31.342 MainThread starting test
08:57:31.342 t1    thread 1 started
08:57:31.342 t2    thread 2 started
08:57:31.405 t1-r  slave said: start slave 1
start slave 2
08:57:32.420 t1-r  slave said: finish slave 1
08:57:32.420 t1    slave 1 returned 0
finish slave 2
08:57:34.415 t1-r  finish slave reader thread
08:57:34.415 t2    slave 2 returned 0
08:57:34.415 t1    thread 1 finished
08:57:34.431 t2    thread 2 finished
08:57:34.431 MainThread finished test

Here you can see that even if the slave 1 finishes at 32.420, its reader thread receives EOF and exits only when the slave 2 finishes also - at 34.415 (slave 1 works 1 second, slave 2 - 3 seconds).

Why the reader thread doesn't see EOF just in time?

The issue is reproduced in Python 2.7.12 x86 on Windows 7. On Linux Ubuntu 16.04 with system Python 2.7 all works as expected.
History
Date User Action Args
2016-10-17 15:38:04Vyacheslav Grigoryevsetrecipients: + Vyacheslav Grigoryev
2016-10-17 15:38:04Vyacheslav Grigoryevsetmessageid: <1476718684.05.0.447400261666.issue28462@psf.upfronthosting.co.za>
2016-10-17 15:38:04Vyacheslav Grigoryevlinkissue28462 messages
2016-10-17 15:38:02Vyacheslav Grigoryevcreate