Author Sebastian.Kreft.Deezer
Recipients Sebastian.Kreft.Deezer, bquinlan, glangford, mark.dickinson, pitrou, python-dev, sbt, tim.peters, vstinner
Date 2014-06-03.09:35:41
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1401788141.6.0.392997184129.issue20319@psf.upfronthosting.co.za>
In-reply-to
Content
I'm using the Python 3.4.1 compiled from source and I'm may be hitting this issue.

My workload is launching two subprocess in parallel, and whenever one is ready, launches another one. In one of the runs, the whole process got stuck after launching about 3K subprocess, and the underlying processes had in fact finished.

To wait for the finished subprocesses, I'm using FIRST_COMPLETED. Below is the core of my workload:

for element in element_generator:
    while len(running) >= max_tasks:
        done, pending = concurrent.futures.wait(running, timeout=15.0, return_when=concurrent.futures.FIRST_COMPLETED)
        process_results(done)
        running = pending

    running.add(executor.submit(exe_subprocess, element)) 
 

I don't really know what's the best way to reproduce this, as I've run the same workload with different executables, more concurrency and faster response times, and I haven't seen the issue.
History
Date User Action Args
2014-06-03 09:35:41Sebastian.Kreft.Deezersetrecipients: + Sebastian.Kreft.Deezer, tim.peters, bquinlan, mark.dickinson, pitrou, vstinner, python-dev, sbt, glangford
2014-06-03 09:35:41Sebastian.Kreft.Deezersetmessageid: <1401788141.6.0.392997184129.issue20319@psf.upfronthosting.co.za>
2014-06-03 09:35:41Sebastian.Kreft.Deezerlinkissue20319 messages
2014-06-03 09:35:41Sebastian.Kreft.Deezercreate