classification
Title: Python VM deadlock
Type: behavior Stage:
Components: Interpreter Core Versions: Python 2.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: vinay.sajip Nosy List: marcin.bachry, pcdinh, vinay.sajip
Priority: normal Keywords:

Created on 2011-02-09 08:25 by pcdinh, last changed 2011-02-21 22:44 by vinay.sajip. This issue is now closed.

Messages (9)
msg128206 - (view) Author: Pham Cong Dinh (pcdinh) Date: 2011-02-09 08:25
Hi,

I was implementing a multiple process compatible logging which child processes are expected to send all messages to the parent process via a queue and then the parent process serializes and flushes them into a log file. The idea is built around multiprocessing.Queue()

The problem is the first child process is always blocked while subsequent ones can run smoothly. It seems to be blocked on semaphore_wait_signal_trap

Environment
===========
Mac OS X 10.6.5 
Python 2.6.6

Reproduction script:
====================
https://gist.github.com/818114

Command to run
====================

python log.py

Console message
===============
Dinh-Phams-MacBook-Pro:task dinhpham$ python log.py
Parent PID 90401
90403
Parent PID 90401
Parent PID 90401
90404
Parent PID 90401
90405
No child process exits
90406
Child process ID 90406
Child process ID 90404
Child process ID 90405
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
Child process PID 90404 will exits now
Child process PID 90406 will exits now
Child process PID 90405 will exits now
The child process 90406 exits
The child process 90405 exits
The child process 90404 exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
^C[90403]
Parent process exits
Dinh-Phams-MacBook-Pro:task dinhpham$ 
 


=========> All the process exits except the first child process one (90403)
msg128232 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-02-09 19:16
I don't believe this is an interpreter or stdlib problem. See this post from September last year:

http://plumberjack.blogspot.com/2010/09/using-logging-with-multiprocessing.html

which shows how to use multiprocessing and logging together with rotating logs. The post includes the code for a complete working script which runs fine with Python 2.6 on Windows and Linux, but which I can't test on Mac OS X/Python 2.6 as I have no access to a Snow Leopard system.

You're welcome to run that script on your system and update this issue with your results - so I'm marking as "invalid" for now (as I don't believe there's a valid Python issue) and "pending" awaiting your feedback. If I don't hear from you in the next week or so, I'll close this issue.
msg128262 - (view) Author: Pham Cong Dinh (pcdinh) Date: 2011-02-10 05:03
I just simplified my script into a smaller one that can be easier to understand and find out what happens

https://gist.github.com/819979

Thanks
msg128263 - (view) Author: Pham Cong Dinh (pcdinh) Date: 2011-02-10 05:05
Sorry, just ignore my latest comment. It is for another issue
msg128265 - (view) Author: Pham Cong Dinh (pcdinh) Date: 2011-02-10 05:10
Hi Vinay,

Your example use a child process to receive log messages. I am implementing a thread-based solution. Do you think that multiprocessing.Queue() has major issues with threads or this problem happens only on Mac OSX implementations?

I still reproduce it quite easily on my system: Mac OS 10.6 and Python 2.6.6
msg128268 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-02-10 08:37
I don't believe that the problem is about threads vs. processes. In the following gist:

https://gist.github.com/820132

I modified the script so that the main process uses a listener thread, rather than a listener process, to collect the logs. (Basically, I changed multiprocessing.Process to threading.Thread, changed the name of the function to listener_thread from listener_process, added an import for threading and updated the comments. No other changes were made.)

The script in the above gist runs fine on Linux Mint Debian and Windows 7 (Python 2.6 in both cases). Please run it on Snow Leopard and report your findings.
msg128269 - (view) Author: Marcin Bachry (marcin.bachry) Date: 2011-02-10 09:07
I think it may be related to this bug: http://bugs.python.org/issue6721
msg128286 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-02-10 12:34
Issue 6721 may well be valid - if this is a symptom of the same issue, then this issue should be closed. I'd like to know if the gist script I referred to works (or not) on Pham Cong Dinh's Snow Leopard machine.
msg129008 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-02-21 22:44
As I've not had any response to my latest comment for over a week, I'm closing this issue, as I said I would.
History
Date User Action Args
2011-02-21 22:44:10vinay.sajipsetstatus: pending -> closed

messages: + msg129008
2011-02-10 12:34:56vinay.sajipsetstatus: open -> pending

messages: + msg128286
2011-02-10 09:07:06marcin.bachrysetstatus: pending -> open
nosy: + marcin.bachry
messages: + msg128269

2011-02-10 08:37:33vinay.sajipsetstatus: open -> pending

messages: + msg128268
resolution: not a bug
2011-02-10 05:10:55pcdinhsetmessages: + msg128265
2011-02-10 05:05:14pcdinhsetmessages: + msg128263
2011-02-10 05:03:01pcdinhsetstatus: pending -> open

messages: + msg128262
resolution: not a bug -> (no value)
2011-02-09 19:44:06vinay.sajipsetstatus: open -> pending
2011-02-09 19:17:05vinay.sajipsetstatus: pending -> open
assignee: vinay.sajip
2011-02-09 19:16:39vinay.sajipsetstatus: open -> pending

messages: + msg128232
resolution: not a bug
2011-02-09 12:44:09pitrousetnosy: + vinay.sajip
2011-02-09 08:41:26pcdinhsettitle: Parent process did not quit after sys.exit() -> Python VM deadlock
2011-02-09 08:25:48pcdinhcreate