classification
Title: Hanging bug with multiprocessing + sqlite3 + tkinter (OS X 10.9 only)
Type: crash Stage: resolved
Components: macOS, Tkinter Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Craig.Silverstein, PyAcrisel, eitan.adler, erlendaasland, evan.jones@bluecore.com, ned.deily, ronaldoussoren, spicyj, terry.reedy
Priority: normal Keywords:

Created on 2014-01-22 18:24 by Craig.Silverstein, last changed 2020-07-06 08:43 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
hang.py Craig.Silverstein, 2014-01-22 18:24 Small repro of the hang (OS X 10.9 only)
Messages (8)
msg208838 - (view) Author: Craig Silverstein (Craig.Silverstein) Date: 2014-01-22 18:24
Don't ask me how, but our code managed to combine multiprocessing, sqlite, and tkinter all in one app, and got a hanging behavior, though only on a particular OS.  Removing any of these removes the hang.  I filed this under tkinter, but the fault could well be elsewhere.

This happens for us only when run under OS X 10.9 (Mavericks).  We are using python 2.7.5; I don't know if other versions are affected.
msg208872 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-01-23 01:19
I can reproduce a hang (but not a crash) when using a Python 2.7.x or 3.4.x built from source and linking with the OS X 10.9 system libsqlite3.  Using the Apple-supplied Python 2.7 on 10.9 causes a crash in libdispatch.  Using Python 2.7.6 or 3.4.0 from the current python.org 64-bit installers, the test does not hang; these Pythons are linked with a static local version of libsqlite3, not the system-supplied one.  If I modify the test to include a call to sqlite3.connect() in the top-level code before calling multiprocessing apply_async, the test also does not hang anymore. Perhaps you can use that as a workaround.  I'm not sure why the combination of sqlite and _tkinter are necessary to hang.  I'm not going to have time to look further into this right away.
msg208874 - (view) Author: Craig Silverstein (Craig.Silverstein) Date: 2014-01-23 01:37
Thanks -- we managed to work around it by removing the tkinter dependency, so this isn't time-critical for us.  It is curious, though.

One thing I forgot to mention before is that playing around with the hung process in gdb, it seems like sqlite was waiting on a lock.  I don't know what lock it might be though.
msg240102 - (view) Author: Arnon Sela (PyAcrisel) Date: 2015-04-05 03:56
I ran into similar issue on OSX.  Multiprocessing system where processes issue sqlite3.connect().  Periodically it hangs.  
System is using Python 3.4.3 and sqlite3; it doesn't use tkinter 

Noticed the following:
1. This doesn't happen on Ubuntu
2. It happens even if URL is invalid - which means that it happens before referring to URL as DB.

Workaround didn't solve the problem. But it seems to reduce the frequency.  And unfortunately, the system is too large and complex for it to be sent.

I tried to set my environment to debug, but with no luck yet :)

Thanks,
msg240103 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-04-05 04:51
Arnon, what version of sqlite3 is the Python linked with?  Try:

python3.4 -c "import sqlite3;print(sqlite3.sqlite_version)"

What kind of database access is happening in your program, i.e. strictly multi-read, one writer many reads, multiple-writers?

Also, regarding the workaround, if you do call sqlite3.connect in the main process, check that you keep a reference to it (by assigning the result to a variable) so that the open connection doesn't get garbage-collected.
msg266397 - (view) Author: Evan Jones (evan.jones@bluecore.com) Date: 2016-05-25 23:07
I believe the root cause of this bug is the following issue: http://bugs.python.org/issue27126 ; a workaround is to run the following code before forking any subprocesses: sqlite3.connect(':memory:').close()
msg321658 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-14 18:39
#33111 claims that importing tkinter is enough to hang multiprocessing on MacOS up through 3.6.
msg370342 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-29 23:25
hang.py uses the default multiprocessing start method, which was then 'fork'.  It is now 'spawn' on macOS.  And there have been many other changes.  When I run the following from IDLE, it finishes immediately.

import multiprocessing, sqlite3
def hang():
   sqlite3.connect('/tmp/foo')
if __name__ == '__main__':
   multiprocessing.Pool(2).apply_async(hang, []).get(999)
print('done')

Unless I am missing something, this should be closed as 'out of date'.
History
Date User Action Args
2020-07-06 08:43:04terry.reedysetstatus: open -> closed
resolution: out of date
stage: resolved
2020-05-29 23:25:34terry.reedysetmessages: + msg370342
2020-05-29 19:41:13erlendaaslandsetnosy: + erlendaasland
2018-07-14 18:39:07terry.reedysetnosy: + terry.reedy, ronaldoussoren
messages: + msg321658
components: + macOS
2017-12-12 23:15:49eitan.adlersetnosy: + eitan.adler
2016-05-25 23:07:09evan.jones@bluecore.comsetnosy: + evan.jones@bluecore.com
messages: + msg266397
2015-04-05 04:51:58ned.deilysetmessages: + msg240103
2015-04-05 03:56:19PyAcriselsetnosy: + PyAcrisel
messages: + msg240102
2014-01-23 06:41:42spicyjsetnosy: + spicyj
2014-01-23 01:37:32Craig.Silversteinsetmessages: + msg208874
2014-01-23 01:19:41ned.deilysetnosy: + ned.deily
messages: + msg208872
2014-01-22 18:24:27Craig.Silversteincreate