classification
Title: Multiprocessing dict sharing between forked processes
Type: crash Stage: resolved
Components: Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: André Neto, Leonard Lausen, christian.heimes, davin, gus.goulart, pablogsal, pitrou, taleinat
Priority: normal Keywords:

Created on 2018-01-03 09:44 by André Neto, last changed 2018-11-09 20:10 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing_crash.py André Neto, 2018-01-03 09:44 Snippet that demonstrates the issue
Messages (7)
msg309402 - (view) Author: André Neto (André Neto) Date: 2018-01-03 09:44
I'm working on a project where I need to share state between several processes (forked using gunicorn). I'm using dictionaries obtained from a multiprocessing SyncManager to achieve this.

The issue is that if I have multiple forked processes concurrently accessing to different dictionaries created by the same SyncManager, the code will randomly crash while accessing (read-only and read-write) to any of the dictionaries. This behaviour is independent of the way I access the dictionary (i.e. it happens both using the dictionary functions (e.g. has_key) or the built-in keywords (e.g. key in dict).

The attached snippet demonstrates the issue (tested only in python2.7): the function test_manyForkedProcessesSingleThreaded will crash.

The issue is also being discussed here: https://stackoverflow.com/questions/48052148/python-multiprocessing-dict-sharing-between-processes
msg309403 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-03 09:55
Does it it actually crash (segfault) or are you merely getting an exception? I'm just getting bunch of exception:

Traceback (most recent call last):
  File "multiprocessing_crash.py", line 155, in <module>
    test_manyForkedProcessesSingleThreaded(inst1, inst2, nRuns, nProcesses)
  File "multiprocessing_crash.py", line 76, in test_manyForkedProcessesSingleThreaded
    run(inst1, nRuns)
  File "multiprocessing_crash.py", line 28, in run
    inst.run()
  File "multiprocessing_crash.py", line 23, in run
    self.d[self.key] = 0
  File "<string>", line 2, in __setitem__
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod
  File "multiprocessing_crash.py", line 155, in <module>
    test_manyForkedProcessesSingleThreaded(inst1, inst2, nRuns, nProcesses)
  File "multiprocessing_crash.py", line 82, in test_manyForkedProcessesSingleThreaded
    run(inst2, nRuns)
  File "multiprocessing_crash.py", line 28, in run
    inst.run()
  File "multiprocessing_crash.py", line 19, in run
    item = self.d.get(self.key)
  File "<string>", line 2, in get
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod
    kind, result = conn.recv()
    kind, result = conn.recv()
cPickle.UnpicklingErrorMemoryError
: invalid load key, '#'.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 268, in _run_finalizers
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 268, in _run_finalizers
    finalizer()
    finalizer()
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 201, in __call__
  File "/usr/lib64/python2.7/multiprocessing/util.py", line 201, in __call__
    res = self._callback(*self._args, **self._kwargs)
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 609, in _finalize_manager
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 609, in _finalize_manager
    if process.is_alive():
    if process.is_alive():
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 155, in is_alive
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 155, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
msg309405 - (view) Author: André Neto (André Neto) Date: 2018-01-03 10:12
You are right, it does not segfault (sorry for the abuse of language). It raises an exception while accessing the shared dictionary. The exception varies but typically is:
Traceback (most recent call last):
  File "multiprocessbug.py", line 156, in <module>
    test_manyForkedProcessesSingleThreaded(inst1, inst2, nRuns, nProcesses)
  File "multiprocessbug.py", line 77, in test_manyForkedProcessesSingleThreaded
    run(inst1, nRuns)
  File "multiprocessbug.py", line 29, in run
    inst.run()
  File "multiprocessbug.py", line 18, in run
    if (self.d.has_key(self.key)):
  File "<string>", line 2, in has_key
  File "/usr/local/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
    kind, result = conn.recv()
cPickle.UnpicklingError: invalid load key, '#'.
msg309410 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-03 11:29
Don't worry about your choice of words. We usually use the C definition of the term crash. An exception is just a bug, but a segfault may be a security issue.
msg329452 - (view) Author: Gus Goulart (gus.goulart) * Date: 2018-11-08 00:38
This is probably a bug that was fixed sometime in Python 3x.
Testing on 3.7.1, Python gracefully errors out:

```
➜  cpython git:(bpo-32485) ✗ python issue_32485.py
Same manager, same mux
Starting test_sameProcessSameThread
Traceback (most recent call last):
  File "issue_32485.py", line 130, in <module>
    test_sameProcessSameThread(inst1, inst2, nRuns)
  File "issue_32485.py", line 48, in test_sameProcessSameThread
    run(inst1, nRuns)
  File "issue_32485.py", line 30, in run
    inst.run()
  File "issue_32485.py", line 19, in run
    if (self.d.has_key(self.key)):
  File "<string>", line 2, in has_key
  File "/home/gus/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/managers.py", line 811, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gus/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/managers.py", line 251, in serve_client
    function = getattr(obj, methodname)
AttributeError: 'dict' object has no attribute 'has_key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gus/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/managers.py", line 271, in serve_client
    fallback_func = self.fallback_mapping[methodname]
KeyError: 'has_key'
---------------------------------------------------------------------------

```
msg329479 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2018-11-08 16:28
Is this supposed to work at all? Does multiprocessing support sharing resources between processes forked outside of its own abstractions?

Reading the docs[1], calling os.fork() directly doesn't seem to be the way that the multiprocessing module is supposed to be used.

[1] https://docs.python.org/library/multiprocessing.html
msg329552 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-11-09 20:10
Gus, dict don't have a has_key() method in Python 3.

multiprocessing objects are generally not fork-safe: multiprocessing has its own mechanisms to duplicate objects between processes, but you cannot assume that after calling fork() yourself, objects will still be usable in the child.

I'm closing this issue as not a bug.
History
Date User Action Args
2018-11-09 20:10:51pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg329552

stage: resolved
2018-11-09 19:32:53ned.deilysetnosy: + pitrou
2018-11-08 16:28:40taleinatsetmessages: + msg329479
2018-11-08 00:38:49gus.goulartsetnosy: + taleinat, gus.goulart
messages: + msg329452
2018-08-14 22:02:37pablogsalsetnosy: + pablogsal
2018-05-15 06:01:49Leonard Lausensetnosy: + Leonard Lausen
2018-01-03 21:05:02rhettingersetnosy: + davin
2018-01-03 11:29:45christian.heimessetmessages: + msg309410
2018-01-03 10:12:58André Netosetmessages: + msg309405
2018-01-03 09:55:58christian.heimessetnosy: + christian.heimes
messages: + msg309403
2018-01-03 09:44:21André Netocreate