classification
Title: shelve module is not thread-safe during accessing different databases from different threads
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, gardener.willy, ncoghlan, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2020-12-17 11:05 by gardener.willy, last changed 2020-12-18 18:20 by serhiy.storchaka.

Messages (4)
msg383229 - (view) Author: (gardener.willy) Date: 2020-12-17 11:05
Shelve module uses "import dbm" instruction while opening database. Dbm module has global dictionary "_modules". This dictionary modifies during database opening operation. When different threads simultaneously try to open different databases, unexpected behavior occurred. In my case I've got such message: 

    with shelve.open('some_file') as f:
  File "/usr/local/lib/python3.6/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/local/lib/python3.6/shelve.py", line 227, in init
    Shelf.init(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/local/lib/python3.6/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
AttributeError: module 'dbm.ndbm' has no attribute 'open'

Behavior is the same on python 3.6 and 3.7. Error is spontaneous.
msg383258 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-17 18:55
I added import experts because the issue looks relates to race condition in import.

Willy, can the issue be reproduced on more modern Python releases (3.8 or 3.9)? It could help if you provide minimal script which reproduces it.
msg383312 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-12-18 17:47
I suggest removing the global state from dbm and just recomputing the underlying database for every call to open().
msg383314 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-18 18:20
Loading modules every time without using a cache (sys.modules) can add significant overhead.
History
Date User Action Args
2020-12-18 18:20:24serhiy.storchakasetmessages: + msg383314
2020-12-18 17:47:19rhettingersetnosy: + rhettinger
messages: + msg383312
2020-12-17 18:55:25serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg383258
2020-12-17 18:00:36brett.cannonsetnosy: - brett.cannon
2020-12-17 12:01:17serhiy.storchakasetnosy: + brett.cannon, ncoghlan, eric.snow
2020-12-17 11:05:17gardener.willycreate