dbm.whichdb (python 3) and whichdb.whichdb (python 2) only check for "magic == 0x13579ace" to recognize gdbm databases. The recently released gdbm 1.9 series adds 0x13579acd (for "32bit") and 0x13579acf (for "64bit") magics (see src/gdbmconst.h in the gdbm-1.9 or gdbm-1.9.1 source tree). Python's gdbm linked to gdbm 1.9 creates databases with the new magic, which causes the whichdb tests to fail. To reproduce this just build python against the new libgdbm and run test/test_dbm.py.

whichdb should probably just check for the two new magic numbers too. This will cause it to recognize the new gdbm databases even if python's gdbm extension is linked to an older version of gdbm that cannot open them, but that is probably less likely to cause problems than the current behavior, where a database created by python's gdbm is not recognized as such.

(I've left the "Versions" field untouched as I do not know which versions of python this should be backported to, but it affects all of them.)

I have a question... What about the endianness of the data?. For instance, if an Intel machine (little endian) creates a GDBM file, can it a) be recognized as such in Sparc (big endian)? b) Can be open and used in Sparc?

New changeset 14cafb8d1480 by Jesus Cea in branch '2.7':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/14cafb8d1480

New changeset 7a41855b6196 by Jesus Cea in branch '3.2':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/7a41855b6196

New changeset b194af345cb5 by Jesus Cea in branch 'default':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/b194af345cb5

This fix should be included in RC2, right? Because I can't get it to work.

Amarok:roller niklas$ ls
data.db		roller.py
Amarok:roller niklas$ python3-32 
Python 3.2.3rc2 (v3.2.3rc2:428f05cb7277, Mar 18 2012, 00:08:43) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> file = shelve.open("data.db")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py", line 232, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py", line 216, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py", line 83, in open
    raise error[0]("db type could not be determined")
dbm.error: db type could not be determined

It is, yes.

Can you do some debugging and see why it is failing?  It should be simple enough to add a print to see what magic number Python is seeing.

Oh, actually...are you sure you are running 3.2.3 against the 3.2.3 stdlib?  It looks like you might be running against the Apple default installed library, but I don't know enough about OSX to be sure.

I'm new to python so please , how do I print what? This?


>>> shelve
<module 'shelve' from '/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py'>

accidentally a or two word there :)

…please have patience…

was what I wanted to say

Oh, I meant sticking a print statement into the stdlib code.

But I really think your problem is that you aren't running the 3.2.3 stdlib code.

Try opening up the file /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py in your favorite editor, and see if there is a line that says:

  if magic in (0x13579ace, 0x13579acd, 0x13579acf):

If it instead says:

  if magic == 0x13579ace:

Then you are using the 3.2 stdlib (which from the traceback filenames it looks like you are).

In other words, I'm pretty sure what you have here is a problem with correctly installing or running the 3.2.3RC2 version of python, and you should ask for help on python-list.

Found this:

# Check for GNU dbm
if magic in (0x13579ace, 0x13579acd, 0x13579acf):
    return "dbm.gnu"

I suppose the .dmg pre-packaged installer was insufficient, so I tried to download the source and compile locally, but that didn't help either. Thanks for taking time to check in on this! I will look for help elsewhere and not clog up this Issue anymore.

Well, that's the correct line.  So if that is what is in your /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py, then there is a problem.  If so, please put the line:

  print(magic)

just before that if statement, and let us know what it says why you try to open your db file.

Niklas, the Python 2 and 3 binaries for Mac OS X provided by the python.org installers are not built with support for gdbm, nor are the Apple-supplied system Pythons; Apple does not ship gdbm with OS X.  So you can't open a shelve file created in another Python that does use gdbm.

David, print(magic) returns 

    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
AttributeError: 'module' object has no attribute 'open'


Ned, I tried the dmg first, but then I downloaded the source and built it locally, does this mean that the source does not contain it either?

Ned: that shouldn't prevent whichdb from figuring out the type of the db file, though, if I understand correctly.

Niklas: that's unexpected, so I suspect something is not right about how you inserted the print. You did say you were a Python newbie...I think perhaps the best thing to do would be to post with your issue to the python mailing list and see if folks there can help you sort out whether there really is a bug in 3.2.3rc2.

As for gdbm support in a source build, it can only be built if you have the right header files available for gdbm on your system when you do the compile.  From what Ned said I suspect you have to do extra work to make that happen, which is again a better question for python-list rather than the bug tracker.

David, right you are. I verified that the current 3.2.3rc2 without gdbm does correctly recognize a shelve from another Python backed with gdbm 1.10:
  dbm.error: db type is dbm.gnu, but the module is not available
Niklas, yes, to build a Python from source on OS X with gdbm support, you will need to build and install the GNU gdbm library on your system first.  There are also third-party Python distributions for Mac OS X that optionally include gdbm support, like MacPorts.