Author Jonathan.Gossage
Recipients Jonathan.Gossage, Windson Yang, dzhu, nconway, pitrou
Date 2018-12-07.01:44:06
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1544147050.65.0.788709270274.issue35267@psf.upfronthosting.co.za>
In-reply-to
Content
This is a great example of abusing the multi-processing API and thus
creating timing errors that lead to locks not being released. What is
happening is that the example attempts to transmit data that is too
big for the underlying pipe and this creates the timing errors since
the pipe that returns the result of invoking apply_async does not have
the capacity of transmitting the result of the worker process in a single
operation and accordingly the worker process returns prematurely while
still holding the pipe lock. This can be seen by testing the ready status
of the result returned by apply_async which will show whether the complete
result of the worker process has been received.
  
 The solution to this situation is simple, simply invoke get() on
 the asynchronous result returned by apply_async. Using this call has the
 effect of correctly synchronizing the pipe used by the low-level queues
 used by the Pool Manager and there will be no lock left active when it
 should'nt be. The script lock1.py contains code that implements this fix
 as well as verifying that everything has worked properly. The output from
 the script is found in the file lock1.result.txt.
  
 Because there is an API based solution to the problem plus the
 behavior of apply_async makes sense in that the process of transferring
 multi-buffer data is very CPU intensive and should be delegated to the
 worker process rather than to the main-line process, I do not recommend that
 any changes be made to the multiprocessing code in Python, rather the
 solution should use the available multiprocessing API correctly.
History
Date User Action Args
2018-12-07 01:44:10Jonathan.Gossagesetrecipients: + Jonathan.Gossage, pitrou, Windson Yang, dzhu, nconway
2018-12-07 01:44:10Jonathan.Gossagesetmessageid: <1544147050.65.0.788709270274.issue35267@psf.upfronthosting.co.za>
2018-12-07 01:44:10Jonathan.Gossagelinkissue35267 messages
2018-12-07 01:44:09Jonathan.Gossagecreate