classification
Title: gdbm 1.9 has new magic that whichdb does not recognize
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jcea Nosy List: Arfrever, Niklas.Brunberg, jcea, marienz, ned.deily, python-dev, r.david.murray
Priority: normal Keywords:

Created on 2011-09-19 14:18 by marienz, last changed 2012-03-31 16:21 by ned.deily. This issue is now closed.

Messages (15)
msg144277 - (view) Author: Marien Zwart (marienz) Date: 2011-09-19 14:18
dbm.whichdb (python 3) and whichdb.whichdb (python 2) only check for "magic == 0x13579ace" to recognize gdbm databases. The recently released gdbm 1.9 series adds 0x13579acd (for "32bit") and 0x13579acf (for "64bit") magics (see src/gdbmconst.h in the gdbm-1.9 or gdbm-1.9.1 source tree). Python's gdbm linked to gdbm 1.9 creates databases with the new magic, which causes the whichdb tests to fail. To reproduce this just build python against the new libgdbm and run test/test_dbm.py.

whichdb should probably just check for the two new magic numbers too. This will cause it to recognize the new gdbm databases even if python's gdbm extension is linked to an older version of gdbm that cannot open them, but that is probably less likely to cause problems than the current behavior, where a database created by python's gdbm is not recognized as such.

(I've left the "Versions" field untouched as I do not know which versions of python this should be backported to, but it affects all of them.)
msg144281 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2011-09-19 14:37
I have a question... What about the endianness of the data?. For instance, if an Intel machine (little endian) creates a GDBM file, can it a) be recognized as such in Sparc (big endian)? b) Can be open and used in Sparc?
msg144288 - (view) Author: Roundup Robot (python-dev) Date: 2011-09-19 15:13
New changeset 14cafb8d1480 by Jesus Cea in branch '2.7':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/14cafb8d1480

New changeset 7a41855b6196 by Jesus Cea in branch '3.2':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/7a41855b6196

New changeset b194af345cb5 by Jesus Cea in branch 'default':
Close #13007: whichdb should recognize gdbm 1.9 magic numbers
http://hg.python.org/cpython/rev/b194af345cb5
msg157138 - (view) Author: Niklas Br (Niklas.Brunberg) Date: 2012-03-30 14:55
This fix should be included in RC2, right? Because I can't get it to work.

Amarok:roller niklas$ ls
data.db		roller.py
Amarok:roller niklas$ python3-32 
Python 3.2.3rc2 (v3.2.3rc2:428f05cb7277, Mar 18 2012, 00:08:43) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shelve
>>> file = shelve.open("data.db")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py", line 232, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py", line 216, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py", line 83, in open
    raise error[0]("db type could not be determined")
dbm.error: db type could not be determined
msg157139 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-30 15:01
It is, yes.

Can you do some debugging and see why it is failing?  It should be simple enough to add a print to see what magic number Python is seeing.
msg157140 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-30 15:03
Oh, actually...are you sure you are running 3.2.3 against the 3.2.3 stdlib?  It looks like you might be running against the Apple default installed library, but I don't know enough about OSX to be sure.
msg157141 - (view) Author: Niklas Br (Niklas.Brunberg) Date: 2012-03-30 15:16
I'm new to python so please , how do I print what? This?


>>> shelve
<module 'shelve' from '/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/shelve.py'>
msg157142 - (view) Author: Niklas Br (Niklas.Brunberg) Date: 2012-03-30 15:17
accidentally a or two word there :)

…please have patience…

was what I wanted to say
msg157143 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-30 15:28
Oh, I meant sticking a print statement into the stdlib code.

But I really think your problem is that you aren't running the 3.2.3 stdlib code.

Try opening up the file /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py in your favorite editor, and see if there is a line that says:

  if magic in (0x13579ace, 0x13579acd, 0x13579acf):

If it instead says:

  if magic == 0x13579ace:

Then you are using the 3.2 stdlib (which from the traceback filenames it looks like you are).

In other words, I'm pretty sure what you have here is a problem with correctly installing or running the 3.2.3RC2 version of python, and you should ask for help on python-list.
msg157144 - (view) Author: Niklas Br (Niklas.Brunberg) Date: 2012-03-30 16:17
Found this:

# Check for GNU dbm
if magic in (0x13579ace, 0x13579acd, 0x13579acf):
    return "dbm.gnu"

I suppose the .dmg pre-packaged installer was insufficient, so I tried to download the source and compile locally, but that didn't help either. Thanks for taking time to check in on this! I will look for help elsewhere and not clog up this Issue anymore.
msg157146 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-30 17:32
Well, that's the correct line.  So if that is what is in your /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dbm/__init__.py, then there is a problem.  If so, please put the line:

  print(magic)

just before that if statement, and let us know what it says why you try to open your db file.
msg157165 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-03-31 04:15
Niklas, the Python 2 and 3 binaries for Mac OS X provided by the python.org installers are not built with support for gdbm, nor are the Apple-supplied system Pythons; Apple does not ship gdbm with OS X.  So you can't open a shelve file created in another Python that does use gdbm.
msg157168 - (view) Author: Niklas Br (Niklas.Brunberg) Date: 2012-03-31 09:47
David, print(magic) returns 

    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
AttributeError: 'module' object has no attribute 'open'


Ned, I tried the dmg first, but then I downloaded the source and built it locally, does this mean that the source does not contain it either?
msg157194 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-31 14:59
Ned: that shouldn't prevent whichdb from figuring out the type of the db file, though, if I understand correctly.

Niklas: that's unexpected, so I suspect something is not right about how you inserted the print. You did say you were a Python newbie...I think perhaps the best thing to do would be to post with your issue to the python mailing list and see if folks there can help you sort out whether there really is a bug in 3.2.3rc2.

As for gdbm support in a source build, it can only be built if you have the right header files available for gdbm on your system when you do the compile.  From what Ned said I suspect you have to do extra work to make that happen, which is again a better question for python-list rather than the bug tracker.
msg157201 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-03-31 16:21
David, right you are. I verified that the current 3.2.3rc2 without gdbm does correctly recognize a shelve from another Python backed with gdbm 1.10:
  dbm.error: db type is dbm.gnu, but the module is not available
Niklas, yes, to build a Python from source on OS X with gdbm support, you will need to build and install the GNU gdbm library on your system first.  There are also third-party Python distributions for Mac OS X that optionally include gdbm support, like MacPorts.
History
Date User Action Args
2012-03-31 16:21:35ned.deilysetmessages: + msg157201
2012-03-31 14:59:43r.david.murraysetmessages: + msg157194
2012-03-31 09:47:35Niklas.Brunbergsetmessages: + msg157168
2012-03-31 04:15:40ned.deilysetnosy: + ned.deily
messages: + msg157165
2012-03-30 17:32:30r.david.murraysetmessages: + msg157146
2012-03-30 16:17:39Niklas.Brunbergsetmessages: + msg157144
2012-03-30 15:28:35r.david.murraysetmessages: + msg157143
2012-03-30 15:17:08Niklas.Brunbergsetmessages: + msg157142
2012-03-30 15:16:13Niklas.Brunbergsetmessages: + msg157141
2012-03-30 15:03:58r.david.murraysetmessages: + msg157140
2012-03-30 15:01:42r.david.murraysetnosy: + r.david.murray
messages: + msg157139
2012-03-30 14:55:19Niklas.Brunbergsetnosy: + Niklas.Brunberg
messages: + msg157138
2011-09-19 15:13:28python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg144288

resolution: fixed
stage: resolved
2011-09-19 15:04:56Arfreversetnosy: + Arfrever
2011-09-19 14:37:22jceasetmessages: + msg144281
2011-09-19 14:28:23jceasetassignee: jcea
versions: + Python 2.7, Python 3.2, Python 3.3
nosy: + jcea
2011-09-19 14:18:09marienzcreate