This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Shelve works inconsistently when carried over to child processes
Type: Stage:
Components: Extension Modules, Interpreter Core Versions: Python 3.4, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Paul Ellenbogen, jnoller, mkellogg, sbt
Priority: normal Keywords:

Created on 2016-04-15 19:43 by Paul Ellenbogen, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
shelve_process.py Paul Ellenbogen, 2016-04-15 19:43 Example python script which can trigger the shelve error.
Messages (4)
msg263522 - (view) Author: Paul Ellenbogen (Paul Ellenbogen) Date: 2016-04-15 19:43
If a shelve is opened, then the processed forked, sometime the shelve will appear to work in the child, and other times it will throw a KeyError. I suspect the order of element access may trigger the issue. I have included a python script that will exhibit the error. It may need to be run a few times.

If shelve is not meant to be inherited by the child process in this way, it should consistently throw an error (probably not a KeyError) on any use, including the first. This way it can be caught in the child, and the shelve can potentially be reopened in the child.

A current workaround is to find all places where a process may fork, and reopen any shelves in the child process after the fork. This may work for most smaller scripts. This could become tedious in more complex applications that fork in multiple places and open shelves in multiple places.

-------------------------------------------------------

Running

#!/usr/bin/env python3

import multiprocessing
import platform
import sys

print(sys.version)
print(multiprocessing.cpu_count())
print(platform.platform())


outputs:
3.4.3+ (default, Oct 14 2015, 16:03:50) 
[GCC 5.2.1 20151010]
8
Linux-4.2.0-34-generic-x86_64-with-Ubuntu-15.10-wily
msg263618 - (view) Author: Paul Ellenbogen (Paul Ellenbogen) Date: 2016-04-17 17:28
I think this behavior is due to the underlying behavior of the dbm. The same code using dbm, rather than shelve, also throws KeyErrors:

from multiprocessing import Process
import dbm

db = dbm.open("example.dbm", "c")
for i in range(100):
    db[str(i)] = str(i ** 2)


def parallel():
    for i in range(100):
        print(db[str(i)])

a = Process(target = parallel)
b = Process(target = parallel)
a.start()
b.start()
a.join()
b.join()
msg264037 - (view) Author: Mark Kellogg (mkellogg) Date: 2016-04-23 01:54
I have also been running into this issue. 

I am using Debian GNU/Linux 8, it was also reproduced on ubuntu by a coworker. 

It however was not reproducible on Gentoo in our testing.
msg264039 - (view) Author: Mark Kellogg (mkellogg) Date: 2016-04-23 02:19
The gentoo user's version of gdbm is: gdbm-1.11
History
Date User Action Args
2022-04-11 14:58:29adminsetgithub: 70960
2016-04-23 02:19:05mkelloggsetmessages: + msg264039
2016-04-23 01:54:17mkelloggsetnosy: + mkellogg
messages: + msg264037
2016-04-17 17:28:16Paul Ellenbogensetmessages: + msg263618
2016-04-15 19:50:32SilentGhostsetnosy: + jnoller, sbt
components: + Extension Modules
2016-04-15 19:43:36Paul Ellenbogencreate