classification
Title: shelve.open fails with error "anydbm.error: db type could not be determined"
Type: behavior Stage: resolved
Components: Documentation Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: docs@python Nosy List: chillaranand, docs@python, krichter, r.david.murray, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2015-01-06 00:05 by krichter, last changed 2017-04-17 11:44 by rhettinger. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1164 closed python-dev, 2017-04-17 05:38
Messages (13)
msg233488 - (view) Author: Karl Richter (krichter) Date: 2015-01-06 00:05
`shelve.open(tempfile.mkstemp()[1])` fails with error "anydbm.error: db type could not be determined" which is not explainable with the docs. Traceback is

    Traceback (most recent call last):
      File "./cudaminer_param_checker.py", line 720, in <module>
        plac.call(cudaminer_param_checker)
      File "/usr/local/lib/python2.7/dist-packages/plac_core.py", line 309, in call
        cmd, result = parser_from(obj).consume(arglist)
      File "/usr/local/lib/python2.7/dist-packages/plac_core.py", line 195, in consume
        return cmd, self.func(*(args + varargs + extraopts), **kwargs)
      File "./cudaminer_param_checker.py", line 715, in cudaminer_param_checker
        visualize_cudaminer_param_checker_results_wxpython_gui()
      File "./cudaminer_param_checker.py", line 365, in visualize_cudaminer_param_checker_results_wxpython_gui
        frame = CudaminerParamChecker(None, )
      File "./cudaminer_param_checker.py", line 378, in __init__
        self.generator = CudaminerParamCheckerGenerator()
      File "./cudaminer_param_checker.py", line 160, in __init__
        self.result_dict= shelve.open(storage_file_path) 
      File "/usr/lib/python2.7/shelve.py", line 239, in open
        return DbfilenameShelf(filename, flag, protocol, writeback)
      File "/usr/lib/python2.7/shelve.py", line 223, in __init__
        Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
      File "/usr/lib/python2.7/anydbm.py", line 82, in open
        raise error, "db type could not be determined"
    anydbm.error: db type could not be determined
msg233490 - (view) Author: Karl Richter (krichter) Date: 2015-01-06 00:12
EDIT 1: other examples, e.g.

    import os
    import shelve

    curdir = os.path.dirname(__file__)
    passwords = shelve.open(os.path.join(curdir, 'password_db'))

work, so there's need for usable error messages.
msg233497 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-06 00:58
You are opening a just-created empty file.  The db type of the file cannot, therefore, be determined.  Which is what the error message says. 

How would you suggest the error message be improved?
msg233498 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-06 01:07
Sorry, I mean "an empty file with no recognized extension".  I doubt supplying a suffix is what you want, though, since you probably want shelve to pick the persistent backend on db creation in order to be portable.
msg233499 - (view) Author: Karl Richter (krichter) Date: 2015-01-06 01:58
Then, let the error message say "You are opening a just-created empty file.  The db type of the file cannot, therefore, be determined." which is much clearer than "anydbm.error: db type could not be determined" which sounds like a generic fallback error message in "an error occured"-style.

It seems to be necessary that the filename passed to `shelve.open` has a suffix which is not very intuitive for Linux users. It'd be great to have validation of the suffix and raise a `ValueError` with an error message like `"filename '%s' doesn't contain a suffix" % (filename,)` when it isn't supplied. If there's another issue regarding the fact that the file is "just-created" and empty, this is independent of the issue above.
msg233507 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-06 08:22
The problem is that this is not the only situation in which that error can be generated.  If someone passed in an "old" (and possibly non-empty) file with no extension, the same code path would get executed.

The shelve/anydbm API is database agnostic, so there is no way to know a-priori what the valid extensions are.  The whichdbm module (which anydbm uses) could provide this information in theory, but it doesn't currently, and if it did that would be a new API.

Your problem is that the shelve API is designed so that you either pass it the base name of an existing database, *or* you pass it the base name of a *non* existent database, which it then creates all the right files for (there may be more than one file).  The fact that you are passing it a new existing file is the problem, and I'm not sure the error message can be effectively improved to handle that particular mistaken use of the API without breaking the message for the places it is more meaningful.

The best I can think of is to add the filename to the error message and list all the extensions and (if there are any) the 'magic' type guesses that whichdbm tried, but as I said that would be a new feature and thus could only go into 3.5.
msg233511 - (view) Author: Karl Richter (krichter) Date: 2015-01-06 10:22
That's nice, thanks. Considering your last comment, some points

  * If the issue can't go into the error message, than the essence of the discussion here should go into the docs (in 0.5 to 1.5 sentences).
  * It'd be nice if it was clear from the error message that shelve or underlying modules check the extension in filename, i.e. do `anydbm.error: db type could not be determined by file extension '%s' in filename '%s'` rather than `anydbm.error: db type could not be determined`.
msg233547 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-06 16:36
What is it you want the docs to say?

For your second point, yes, that is what I was saying would be an enhancement (the extension is a list of possible extensions that varies per-platform and per how python is built/installed).
msg235522 - (view) Author: Karl Richter (krichter) Date: 2015-02-07 13:07
For example, it should be clear why `shelve.open(tempfile.mkstemp()[1])` fails with the mentioned exception and `shelve.open("/tmp/bla")` fails. I still haven't figured out the constraints to create a working `shelve.Shelve` at all. It should be clear why `shelve.open("/tmp/tmphTTQLd")` fails and `shelve.open("/tmp/tmphTTQLda")` succeeds. There has to be something unrelated to extensions.
msg235527 - (view) Author: Karl Richter (krichter) Date: 2015-02-07 13:58
After checking the code, I think that it'd make more sense to document `whichdb.py`. It needs to be enhanced with references to criteria for the determination of the database type. Currently there're only function comments and the fact that some variables are named magic speaks for itself, i.e. it's ok for the implementation of the module to be a black box, but the requirements for the input, i.e. the potential database file, which is per se not part of the black box, needs to be specified. Then just link that in the `shelve` docs stating that `shelve.open` is basically a wrapper around `whichdb`.
msg291784 - (view) Author: Anand Reddy Pandikunta (chillaranand) * Date: 2017-04-17 05:38
dbm.whichdb mentions that it returns empty string ('') if the file’s format can’t be guessed. Enhancing exception message should suffice.
msg291785 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-04-17 05:48
Proposed error message isn't correct. The file extension is not the only method for determining db type.

Current error message looks good to me and I don't see how it can be enhanced.
msg291796 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-04-17 11:44
[Serhiy]
> Current error message looks good to me and I don't see how
> it can be enhanced.

I concur with Serhiy and David.
History
Date User Action Args
2017-04-17 11:44:53rhettingersetstatus: open -> closed

nosy: + rhettinger
messages: + msg291796

resolution: not a bug
stage: resolved
2017-04-17 05:48:13serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg291785
2017-04-17 05:38:58chillaranandsetnosy: + chillaranand
messages: + msg291784
2017-04-17 05:38:40python-devsetpull_requests: + pull_request1291
2015-02-07 13:58:17krichtersetmessages: + msg235527
2015-02-07 13:07:57krichtersetmessages: + msg235522
2015-01-06 16:36:41r.david.murraysetmessages: + msg233547
2015-01-06 10:22:53krichtersetmessages: + msg233511
2015-01-06 08:22:58r.david.murraysetmessages: + msg233507
2015-01-06 01:58:55krichtersetmessages: + msg233499
2015-01-06 01:07:05r.david.murraysetmessages: + msg233498
2015-01-06 00:58:51r.david.murraysetnosy: + r.david.murray
messages: + msg233497
2015-01-06 00:12:21krichtersetmessages: + msg233490
2015-01-06 00:05:22krichtercreate