classification
Title: Error to build _dbm module during make
Type: compile error Stage:
Components: Extension Modules Versions: Python 3.0, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: skip.montanaro Nosy List: AndyP, ddvento@ucar.edu, flub, legerf, rpetrov, skip.montanaro, tlesher
Priority: normal Keywords: easy, patch

Created on 2008-12-01 22:35 by legerf, last changed 2013-02-08 16:04 by ddvento@ucar.edu. This issue is now closed.

Files
File name Uploaded Description Edit
dbm.diff skip.montanaro, 2008-12-02 00:47 Possible Ubuntu/Lenny fix
dbm2.diff skip.montanaro, 2008-12-02 00:56 Python 2.x patch for Ubuntu/Lenny
dbm2.diff legerf, 2008-12-02 17:32
dbm3.diff AndyP, 2008-12-05 00:07 Patch allowing Python 3 to build on Debian Lenny
dbm.diff skip.montanaro, 2008-12-05 00:24
dbm.diff skip.montanaro, 2008-12-05 01:16
Messages (30)
msg76709 - (view) Author: Leger (legerf) Date: 2008-12-01 22:35
Under Debian/Lenny/i486, I install python3.0rc3.
After untar the Python3.0rc3 package, I replace in the configure
"/usr/local" by "/usr". After I do :
"configure --with-pydebug --with-doc-strings --enable-shared
--enable-profiling --enable-ipv6 --with-threads --with-tsc ; make"

The result during make is :
"Failed to find the necessary bits to build these modules:
_dbm                                                  
To find the necessary bits, look in setup.py in detect_modules() for the
module's name."

The release of the dbm library is : 
libgdbm-dev   1.8.3-3  GNU dbm database routines (development files)

The header for dbm : "_gdbmmodule.c" request "gdbm.h", and is here
"/usr/include/gdbm.h"

The "Modules/_dbmmodule.c" use :
#include <gdbm/ndbm.h>" ... #include <ndbm.h> ...
but I have only "/usr/include/gdbm-ndbm.h"

Can you patch the "_dbmmodule.c" to use it ?
msg76720 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-02 00:43
Patching _dbmmodule.c alone isn't sufficient.  At minimum setup.py would
have to be adjusted to detect the presence of the odd ndbm.h file.
msg76721 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-02 00:47
Can you try the attached patch and let us know if it works for you?
msg76722 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-02 00:56
Here's a similarly untested patch for Python 2.x.  Can you try it as well?
msg76760 - (view) Author: Leger (legerf) Date: 2008-12-02 17:32
- I modify your dbm2.diff because the module name is "_dbmmodule.c" and
not "dbmmodule.c". 
- Patching success
- new test "configure ... ; make" and make stop with error :
"...
gcc -pthread -shared
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/_ctypes.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/callbacks.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/callproc.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/stgdict.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/cfield.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/malloc_closure.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/prep_cif.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/x86/ffi.o
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_ctypes/libffi/src/x86/sysv.o
-L/usr/local/lib -L. -lpython3.0 -o
build/lib.linux-i686-3.0-pydebug/_ctypes.so

Failed to build these modules:
_dbm                                                  

running build_scripts
creating build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/pydoc ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/idle ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Tools/scripts/2to3 ->
build/scripts-3.0
copying and adjusting /install/python-3.0rc3.diff/Lib/smtpd.py ->
build/scripts-3.0
changing mode of build/scripts-3.0/pydoc from 644 to 755
changing mode of build/scripts-3.0/idle from 644 to 755
changing mode of build/scripts-3.0/2to3 from 644 to 755
changing mode of build/scripts-3.0/smtpd.py from 644 to 755
[88798 refs]"
msg76761 - (view) Author: Leger (legerf) Date: 2008-12-02 17:51
I retest with light configure modification and dbm2.diff with file name
"_dbmmodule.c" :

"configure --with-pydebug --with-doc-strings --enable-shared
--enable-profiling --enable-ipv6 --with-threads --with-tsc --prefix=/usr
; make"

"...
gcc -pthread -shared
build/temp.linux-i686-3.0-pydebug/install/python-3.0rc3.diff/Modules/_dbmmodule.o
-L/usr/local/lib -L. -lgdbm -lpython3.0 -o
build/lib.linux-i686-3.0-pydebug/_dbm.so
*** WARNING: renaming "_dbm" since importing it failed:
build/lib.linux-i686-3.0-pydebug/_dbm.so: undefined symbol: dbm_firstkey

Failed to build these modules:
_dbm                                                  

running build_scripts
[89618 refs]"
msg76763 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-02 18:12
Leger> - I modify your dbm2.diff because the module name is
    Leger>   "_dbmmodule.c" and not "dbmmodule.c".

I think you missed the point of the dbm2.diff file.  It should be applied to
the Python 2.x code, not to Python 3.x.  I don't have access to the
necessary platform to run the test.

To summarize:

    * dbm.diff should be applied to the py3k branch (or to the 3.0rc3 release
      code).
    * dbm2.diff should be applied to the code on the trunk (or to the 2.6
      release code).

Skip
msg76766 - (view) Author: Leger (legerf) Date: 2008-12-02 19:13
Skip, Sorry for my mistake with dbm2.diff ;)

From the tar 3.0rc3 I apply the patch (dbm.diff). Configure ... and
make. But the result is always wrong :
"Failed to build these modules:
_dbm"

Best regards, Frederic
msg76769 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-02 19:51
Frederic> From the tar 3.0rc3 I apply the patch (dbm.diff). Configure
    Frederic> ... and make. But the result is always wrong :

    > "Failed to build these modules:
    > _dbm"

Why does it fail?  Is there compile error?  Does linking fail?  I guessed
that the library would still be "gdbm", but I don't really know.  Does your
system maybe install a library file named "libgdbm-ndbm.so" or something
similarly obscure?

I'm happy to try and get this resolved, but not that there is only another
day or two before 3.0final is released.  (This might not make the release
anyway.)
msg76770 - (view) Author: Leger (legerf) Date: 2008-12-02 20:06
- Only the make generate last error sentence.
- Debian/Lenny provide only "libgdbm-dev-1.8.3-3" provide :
root@xxx:/install# ll /usr/lib/libgdbm.so*
lrwxrwxrwx 1 root root    16 2008-12-01 21:45 /usr/lib/libgdbm.so ->
libgdbm.so.3.0.0
lrwxrwxrwx 1 root root    16 2008-03-08 21:38 /usr/lib/libgdbm.so.3 ->
libgdbm.so.3.0.0
-rw-r--r-- 1 root root 19940 2006-04-24 05:25 /usr/lib/libgdbm.so.3.0.0
msg76873 - (view) Author: Leger (legerf) Date: 2008-12-04 06:05
I install the Python3.0 (final), the test make is the same and the
result too. I join the output.
msg76937 - (view) Author: Andrew Price (AndyP) Date: 2008-12-04 22:39
I'm running the same distro as Leger and I was having the same problem.
Now I've applied dbm.diff and with a clean build I'm seeing this:

building '_dbm' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_GDBM_DASH_NDBM_H -I.
-I/home/andy/src/python3/Python-3.0/./Include -I. -IInclude -I../Include
-I/usr/local/include -I/home/andy/src/python3/Python-3.0/debug/Include
-I/home/andy/src/python3/Python-3.0/debug -c
/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.c -o
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
gcc -pthread -shared
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
-L/usr/local/lib -lgdbm -o build/lib.linux-i686-3.0/_dbm.so
*** WARNING: renaming "_dbm" since importing it failed:
build/lib.linux-i686-3.0/_dbm.so: undefined symbol: dbm_firstkey
msg76948 - (view) Author: Andrew Price (AndyP) Date: 2008-12-05 00:07
In Debian Lenny libgbdm-dev provides two libs, libgdbm and libgdbm_compat:

andy@plato:~$ objdump -t /usr/lib/libgdbm.a | grep dbm_firstkey
00000000         *UND*	00000000 gdbm_firstkey
00000140 g     F .text	00000056 gdbm_firstkey

andy@plato:~$ objdump -t /usr/lib/libgdbm_compat.a | grep dbm_firstkey
00000000         *UND*	00000000 gdbm_firstkey
00000000 g     F .text	00000060 dbm_firstkey
00000000         *UND*	00000000 gdbm_firstkey

So it looks like we need to link against -lgdbm_compat instead of -lgdbm
for the _dbm module. I'm attaching dbm3.diff which seems to fix the
problem (I did a test build with it).
msg76949 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-05 00:11
Andrew> I'm running the same distro as Leger and I was having the same
    Andrew> problem.  Now I've applied dbm.diff and with a clean build I'm
    Andrew> seeing this:

    ...

    Andrew> *** WARNING: renaming "_dbm" since importing it failed:
    Andrew> build/lib.linux-i686-3.0/_dbm.so: undefined symbol: dbm_firstkey

It looks like we might need to be linking with -lgdbm_compat as well as
-lgdbm.

Can you attach the output of these commands

    objdump -T /usr/lib/libgdbm.so.N
    objdump -T /usr/lib/libgdbm_compat.so.N

to this issue (where N is the highest version available)?

Thanks,
msg76951 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-05 00:24
Here's a new version of the patch for Python 3.0.  It appends gdbm_compat
to the gdbm libs if that's where dbm_firstkey is defined.  Please back
out the previous patch against setup.py and Modules/_dbmmodule.c and
apply this one.

Thanks...
msg76956 - (view) Author: Andrew Price (AndyP) Date: 2008-12-05 01:00
Skip, the new patch makes it fail with (highlights):

...
  File "/home/andy/src/python3/Python-3.0/Lib/distutils/ccompiler.py",
line 844, in has_function
    import tempfile
  File "/home/andy/src/python3/Python-3.0/Lib/tempfile.py", line 35, in
<module>
    from random import Random as _Random
  File "/home/andy/src/python3/Python-3.0/Lib/random.py", line 42, in
<module>
    from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil
as _ceil
ImportError: No module named math
make: *** [sharedmods] Error 1


The has_function source in Lib/distutils/ccompiler.py has this comment:

        # this can't be included at module scope because it tries to
        # import math which might not be available at that point - maybe
        # the necessary logic should just be inlined?
msg76958 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-05 01:16
Andrew> ImportError: No module named math
    Andrew> make: *** [sharedmods] Error 1

    Andrew> The has_function source in Lib/distutils/ccompiler.py has this
    Andrew> comment:

    Andrew> # this can't be included at module scope because it tries to
    Andrew> # import math which might not be available at that point - maybe
    Andrew> # the necessary logic should just be inlined?

*sigh* I guess that would explain why it wasn't used anywhere in setup.py.

Can you try this simplified version?  (It's probably almost exactly what you
came up with.)

Thanks,

Skip
msg76962 - (view) Author: Andrew Price (AndyP) Date: 2008-12-05 01:35
Skip, your simplified patch works for me. Makes it build fine with the
following snippet:

building '_dbm' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -DHAVE_GDBM_DASH_NDBM_H -I.
-I/home/andy/src/python3/Python-3.0/./Include -I. -IInclude -I../Include
-I/usr/local/include -I/home/andy/src/python3/Python-3.0/debug/Include
-I/home/andy/src/python3/Python-3.0/debug -c
/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.c -o
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
gcc -pthread -shared
build/temp.linux-i686-3.0/home/andy/src/python3/Python-3.0/Modules/_dbmmodule.o
-L/usr/local/lib -lgdbm -lgdbm_compat -o build/lib.linux-i686-3.0/_dbm.so

Thanks.
msg77074 - (view) Author: Leger (legerf) Date: 2008-12-05 21:35
This patch resolve the problem. Evidence with "make test" :
...
test_dbm
test_dbm_dumb
test_dbm_gnu
test_dbm_ndbm
...

Many thanks Skip. You can close this bug I think.
msg77122 - (view) Author: Floris Bruynooghe (flub) Date: 2008-12-06 13:25
Hi, I'd like to confirm that Skip's last patch fixes the issue.  Hope it
gets included soon!

Thanks
msg77154 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-06 17:53
Fix checked in for py3k (r67612), release26-maint (r67613),
release30-maint (r67615) branches as well as trunk (r67614).
msg77168 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-06 20:59
now 2008-12-06 trunk fail on linux
msg77169 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-06 21:07
may be patch is not applied correctly :
http://svn.python.org/view/python/trunk/setup.py?rev=67614&view=auto

        libraries = gdbm_libs ) )
?   exts.append( Extension('dbm', ['dbmmodule.c'],
?     define_macros=[('HAVE_GDBM_NDBM_H',None)],
?     libraries = ['gdbm'] ) )
  elif db_incs is not None:
msg77205 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-07 01:41
Roumen> may be patch is not applied correctly :
    Roumen> http://svn.python.org/view/python/trunk/setup.py?rev=67614&view=auto

    Roumen>         libraries = gdbm_libs ) )
    Roumen> ?   exts.append( Extension('dbm', ['dbmmodule.c'],
    Roumen> ?     define_macros=[('HAVE_GDBM_NDBM_H',None)],
    Roumen> ?     libraries = ['gdbm'] ) )
    Roumen>   elif db_incs is not None:

Yeah, muffed it.  I should have a fix checked in shortly.

Skip
msg77256 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-07 21:07
I'm not sure that recent commits in trunk are correct.

Please confirm that build of dbm module with "Berkeley DB" is deprecated.

Note that exist another module gdbm[gdbmmodule.c].
msg77269 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-07 22:10
Roumen> I'm not sure that recent commits in trunk are correct.

    Roumen> Please confirm that build of dbm module with "Berkeley DB" is
    Roumen> deprecated.

Can you explain how you think they are incorrect?  The code to decide how to
build the dbm module looks like this:

    if we have ndbm.h:
        build dbm using ndbm
    else if we have a gdbm library:
        if we have gdbm/ndbm.h:
            build dbm using gdbm
        else if we have gdbm-ndbm.h:
            build dbm using gdbm
    else if we found Berkeley includes:
        build dbm using Berkeley db.

This overall structure agrees with that in the Python 2.4 and 2.5 setup.py.
The only thing that changed (in theory) was that there's an extra check for
gdbm-ndbm.h after we've already decided to build dbm with gdbm.  The
preference is still ndbm before gdbm before BerkDB.  That's not to say it's
correct, but nothing's changed as far as I can tell.

Maybe it should look more like this:

    dbmext = None
    if we have ndbm.h:
        ...
        dbmext = Extension(...)

    if dbmext is None and we have a gdbm library:
        if we have gdbm/ndbm.h:
            ...
            dbmext = Extension(...)
        else if we have gdbm-ndbm.h:
            ...
            dbmext = Extension(...)

    if dbmext is None and we found Berkeley includes:
        ...
        dbmext = Extension(...)

    if dbmext is not None:
        exts.append(dbmext)

That still leaves the order ndbm before gdbm before BerkDB.

Skip
msg77275 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-12-07 22:52
The old if statement was "flat":
if find_file("ndbm.h", inc_dirs, []) is not None:
   <CODE>
elif (self.compiler.find_library_file(lib_dirs, 'gdbm')
      and find_file("gdbm/ndbm.h", inc_dirs, []) is not None):
   <CODE>
elif db_incs is not None:
   <CODE>
else:
   miss
If the second case fail, try third and if  succeed build dbm with
"berkeley db".

The new if statement contain nested if:
Now the second case is only "self.compiler.find_library_file(lib_dirs,
'gdbm'):" and if succeed (my case) =>
Failed to find the necessary bits to build these modules:
... dbm ...
Note the build system lack headers "gdbm/ndbm.h", "gdbm-ndbm.h".


To restore previous I add copy of code from third case as new case in
nested if:
===============
                         'dbm', ['dbmmodule.c'],
                         define_macros=[('HAVE_GDBM_DASH_NDBM_H',None)],
                         libraries = gdbm_libs ) )
+                elif db_incs is not None:
+                    exts.append( Extension('dbm', ['dbmmodule.c'],
+                        library_dirs=dblib_dir,
+                        runtime_library_dirs=dblib_dir,
+                        include_dirs=db_incs,
+                        define_macros=[('HAVE_BERKDB_H',None),
+                                       ('DB_DBM_HSEARCH',None)],
+                        libraries=dblibs))
                 else:
                     missing.append('dbm')
             elif db_incs is not None:
===============
Note that above is not proposed patch as I don't like nested ifs and
code duplicate.
Is possible the checks to be in a new separate function ?
msg77279 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-08 01:16
Roumen> The old if statement was "flat":
    ...

Okay, I understand now.

    Roumen> The new if statement contain nested if:
    Roumen> Now the second case is only "self.compiler.find_library_file(lib_dirs,
    Roumen> 'gdbm'):" and if succeed (my case) =>
    Roumen> Failed to find the necessary bits to build these modules:
    Roumen> ... dbm ...
    Roumen> Note the build system lack headers "gdbm/ndbm.h", "gdbm-ndbm.h".

In my mind that simply means we haven't accounted for how your system has
organized its header files.  It has a libgdbm so it probably has some sort
of gdbm header file unless you have failed to install some package.

I'm beginning to think this area needs more work.  Let's leave this ticket
closed.  I'll open a new one so we can figure out the best way to tackle
this.  In the meantime can you figure out why your system has a gdbm library
but apparently no gdbm headers (or different ones than we are currently
testing for)?

Thx,

Skip
msg77282 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-12-08 01:50
Skip> I'm beginning to think this area needs more work.  Let's leave
    Skip> this ticket closed.  I'll open a new one so we can figure out the
    Skip> best way to tackle this.

http://bugs.python.org/issue4587

Skip
msg181679 - (view) Author: (ddvento@ucar.edu) Date: 2013-02-08 16:04
This is still an issue in Python 2.7.3 but there is a quick manual workaround. I know it's trivial and one can easily develop it from what is said in the thread or maybe looking at the patches, but for reference this is a nice recipe as oppose to digging through many messages

1) build as usual, but redirect the output/error stream in a file, for example if your shell is bash (I find this to always be a good idea):

make > make.log 2>&1

2) Towards the end of make.log there will be the following message. If you don't see this message, you don't have this problem, but possibly a different one:

Failed to build these modules:
dbm


3) execute
grep "\-o .*/dbmmodule.o" make.log

This will find a compiler line. Cut, paste and and execute that command

4) grep "\-o .*/dbm.so" make.log

This will find another compiler line. Cut, paste and and execute that command, ADDING (this is essential) -lgdbm_compat

5) (optional) you may want to remove dbm_failed.so (in the same directory where the previous bullet creates dbm.so)
History
Date User Action Args
2013-02-08 16:04:34ddvento@ucar.edusetnosy: + ddvento@ucar.edu
messages: + msg181679
2008-12-08 01:50:14skip.montanarosetmessages: + msg77282
2008-12-08 01:16:35skip.montanarosetmessages: + msg77279
2008-12-07 22:52:08rpetrovsetmessages: + msg77275
2008-12-07 22:10:26skip.montanarosetmessages: + msg77269
2008-12-07 21:07:52rpetrovsetmessages: + msg77256
2008-12-07 01:41:45skip.montanarosetmessages: + msg77205
2008-12-06 21:07:00rpetrovsetmessages: + msg77169
2008-12-06 20:59:09rpetrovsetnosy: + rpetrov
messages: + msg77168
versions: + Python 2.7
2008-12-06 17:53:34skip.montanarosetstatus: open -> closed
resolution: fixed
messages: + msg77154
2008-12-06 13:25:09flubsetnosy: + flub
messages: + msg77122
2008-12-05 21:35:44legerfsetmessages: + msg77074
2008-12-05 01:35:23AndyPsetmessages: + msg76962
2008-12-05 01:16:38skip.montanarosetfiles: + dbm.diff
messages: + msg76958
2008-12-05 01:00:20AndyPsetmessages: + msg76956
2008-12-05 00:24:47skip.montanarosetfiles: + dbm.diff
messages: + msg76951
2008-12-05 00:11:59skip.montanarosetmessages: + msg76949
2008-12-05 00:07:37AndyPsetfiles: + dbm3.diff
messages: + msg76948
2008-12-04 22:39:19AndyPsetmessages: + msg76937
2008-12-04 22:14:21AndyPsetnosy: + AndyP
2008-12-04 22:06:17tleshersetnosy: + tlesher
2008-12-04 06:05:19legerfsetmessages: + msg76873
2008-12-02 20:06:31legerfsetmessages: + msg76770
2008-12-02 19:51:28skip.montanarosetmessages: + msg76769
2008-12-02 19:13:14legerfsetmessages: + msg76766
2008-12-02 18:12:37skip.montanarosetmessages: + msg76763
2008-12-02 17:51:56legerfsetmessages: + msg76761
2008-12-02 17:32:12legerfsetfiles: + dbm2.diff
messages: + msg76760
2008-12-02 00:56:33skip.montanarosetfiles: + dbm2.diff
messages: + msg76722
2008-12-02 00:47:13skip.montanarosetkeywords: + patch, easy
assignee: skip.montanaro
messages: + msg76721
files: + dbm.diff
2008-12-02 00:43:14skip.montanarosetnosy: + skip.montanaro
messages: + msg76720
2008-12-01 22:35:11legerfcreate