classification
Title: Multiprocessing creates incorrect pids
Type: crash Stage:
Components: Versions: Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: cfriedline, pitrou
Priority: normal Keywords:

Created on 2012-08-09 15:03 by cfriedline, last changed 2012-08-09 20:20 by cfriedline. This issue is now closed.

Messages (7)
msg167788 - (view) Author: Chris (cfriedline) Date: 2012-08-09 15:03
http://stackoverflow.com/questions/11884864/python-multiprocessing-creates-incorrect-pids

Jesse Noller suggested that I submit this stack overflow as a bug so it can be examined.  As the problem is kind of domain specific, it's not easy to come up with a test case that can reproduce the issue.

Please let me know what more information I can provide.  

Thanks!
msg167793 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-09 17:02
Your report is rather ambiguous. Do you claim that said pids don't exist, or that they don't appear in "top"?

Also, do you manage to reproduce without R, or do you need to use R to reproduce? I would suspect an issue (or a misunderstanding on your part) with the R-Python bindings, rather than a bug in multiprocessing.
msg167797 - (view) Author: Chris (cfriedline) Date: 2012-08-09 17:14
That they don't appear in top.  I was using that as a proxy for existence.  I wrote a test case without R and realized (accidentally) that tracebacks will now flow up to the main process from the worker processes (which makes perfect sense, actually).  I added explicit try/except to __get_valid_triplets.  Will report back.
msg167798 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-09 17:24
> Chris added the comment:
>
> That they don't appear in top.  I was using that as a proxy for existence.

Well, I don't know in which way you use "top", but by default it will 
only show you the N most CPU-consuming processes. If the R bindings, for 
example (here begins the wild speculation on my part :-)), fork in the 
background so as to run the R engine in a separate process, the 
CPU-consuming processes would not be those directly created by 
multiprocessing.
msg167799 - (view) Author: Chris (cfriedline) Date: 2012-08-09 17:32
Using top/'ps aux' is not the issue.  I have code running now with the try/except.  Will report back ASAP.  I suspect that you're correct about something in the R pipeline failing.  Hopefully, I'll get a good traceback.
msg167801 - (view) Author: Chris (cfriedline) Date: 2012-08-09 17:37
Also, when I said "...that tracebacks will now flow up to the main process from the worker processes (which makes perfect sense, actually)", I meant to say "...that tracebacks will NOT flow up to the main process from the worker processes (which makes perfect sense, actually)"  ;-)
msg167830 - (view) Author: Chris (cfriedline) Date: 2012-08-09 20:20
Ugh, a server reboot seems to have cleared this all up (FML).  It's been running well past the point of previous failure.  I suppose I'll never know why it was failing.  I'm going to close this out.  Thanks for your time in helping me think this through.
History
Date User Action Args
2012-08-09 20:20:36cfriedlinesetstatus: open -> closed
resolution: works for me
messages: + msg167830
2012-08-09 17:37:49cfriedlinesetmessages: + msg167801
2012-08-09 17:32:46cfriedlinesetmessages: + msg167799
2012-08-09 17:24:26pitrousetmessages: + msg167798
2012-08-09 17:14:38cfriedlinesetmessages: + msg167797
2012-08-09 17:02:52pitrousetnosy: + pitrou
messages: + msg167793
2012-08-09 15:03:18cfriedlinecreate