classification
Title: configure options don't trickle down to distutils
Type: behavior Stage: resolved
Components: Build, Distutils Versions: Python 3.2
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: tarek Nosy List: akitada, eric.araujo, jcea, rpetrov, speno, steve.dower, tarek
Priority: normal Keywords: patch

Created on 2008-10-01 16:43 by skip.montanaro, last changed 2021-02-03 18:30 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
runtime.diff skip.montanaro, 2008-10-01 20:44
py-issue-4010.patch rpetrov, 2009-01-04 19:56 LDFLAGS as is for distutils
issue4010.diff akitada, 2009-02-28 17:56
Messages (25)
msg74137 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-01 16:43
If you are fortunate enough to have all your third-party libraries in
a single quasi-standard location, say, /usr/local/lib, you will
probably have never encountered this problem, but setting environment
variables like LDFLAGS don't get translated into the relevant args for
the distutils build_ext command.  The evidence for this is this output
at the end of the make process:

    Failed to find the necessary bits to build these modules:
    _hashlib           _sqlite3           _ssl
    bsddb185           linuxaudiodev      ossaudiodev
    To find the necessary bits, look in setup.py in detect_modules() for
the module's name.


    Failed to build these modules:
    _curses            _curses_panel      _tkinter
    gdbm               readline

Here's the specific failure for readline:

    >>> import readline_failed
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
	ImportError: ld.so.1: python: fatal: libreadline.so.4: open failed: No
such file or directory

yet if you poke around in my config.status file you'll see that I set
the relevant -L and -R flags:

    s,@LDFLAGS@,|#_!!_#|-L/opt/app/nonc++/ncurses-5.6/lib
-R/opt/app/nonc++/ncurses-5.6/lib -L/opt/app/nonc++/gdbm-1.8/lib
-R/opt/app/nonc++/gdbm-1.8/lib -L/opt/app/nonc++/readline-4.3/lib
-R/opt/app/nonc++/readline-4.3/lib -L/opt/app/nonc++/tcl-8.4/lib
-R/opt/app/nonc++/tcl-8.4/lib -L/opt/app/nonc++/BerkleyDB-4.3/lib
-R/opt/app/nonc++/BerkleyDB-4.3/lib,g

The workaround is to run the build_ext command separately:

    ./python ../setup.py build_ext --library-dirs=... --rpath=...

It's not hard to work around this problem, but it's tedious to build
the command line args from the massive list of directories I have to
educate distutils about.
msg74140 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-10-01 19:25
Well, all of my modules are in a non-standard location and I have no
build issues on OS X 10.5. If you look at PyBuildExt.detect_modules()
you will see that the paths are at least added for the search path from
LDFLAGS and CPPFLAGS.
msg74143 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-01 20:27
Brett> Well, all of my modules are in a non-standard location and I have
    Brett> no build issues on OS X 10.5. If you look at
    Brett> PyBuildExt.detect_modules() you will see that the paths are at
    Brett> least added for the search path from LDFLAGS and CPPFLAGS.

Ah, there's the problem.  It picks through LDFLAGS looking for -L but fails
to pay attention to -R.  I'll see if I can work up a patch.

Also, I'm seeing this problem on Solaris.  I suppose it's possible that
linkage on OS X records the location of the libraries it finds, sort of
implicitly doing a -R.

Skip
msg74146 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-01 20:44
Here's a patch.  Works for me on Solaris 10.  I'll try to check
it out on OS X 10.5.  Would appreciate it if someone on Linux can
kick the tires too.
msg74152 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-02 02:24
Confirmed that nothing seems broken on my OS X 10.5 laptop (doing a
unix-style build, not a framework build).
msg74154 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-02 03:31
Seems to work for framework builds as well.
msg74194 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-10-02 20:57
One of the problems that I see in that LDFLAGS is Makefile variable and
Makefile is part of distribution for posix build systems.
If you set specific LDFLAGS and you want to distribute own python build
user will get you specific settings.
One another environment variable is OPT (if compiler don't accept -R XXX
may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
variable is also Makefile variable and user specific settings will go
into distribution too :( .

I quick look into distutils code show that sysconfig.py don't parse
LDFLAGS and append environment variable LDFLAGS as is to the ldshared
variable.
So the first question is why setup.py parse only -L flags instead to
pass variable value as is (aka sysconfig.py).
The second question is why LDFLAGS is a Makefile variable ? 


For now I would like to know if you set on Solaris suitable
LD_RUN_PATH(for the build) and LD_LIBRARY_PATH (when run) did you
succeed to build and run python ?
msg74199 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-10-02 22:03
On Thu, Oct 2, 2008 at 1:57 PM, Roumen Petrov <report@bugs.python.org> wrote:
>
> Roumen Petrov <bugtrack@roumenpetrov.info> added the comment:
>
> One of the problems that I see in that LDFLAGS is Makefile variable and
> Makefile is part of distribution for posix build systems.
> If you set specific LDFLAGS and you want to distribute own python build
> user will get you specific settings.
> One another environment variable is OPT (if compiler don't accept -R XXX
> may be -Wl,-rpath,XXX ?) and we may use it. Also OPT environment
> variable is also Makefile variable and user specific settings will go
> into distribution too :( .
>
> I quick look into distutils code show that sysconfig.py don't parse
> LDFLAGS and append environment variable LDFLAGS as is to the ldshared
> variable.
> So the first question is why setup.py parse only -L flags instead to
> pass variable value as is (aka sysconfig.py).

Because that's the use case I was supporting when I added the support;
I just needed the -L flags for compiling the extensions. If someone
writes a patch that makes it more generic it has a chance of being
checked in.

> The second question is why LDFLAGS is a Makefile variable ?

Because there is no guarantee that someone specified the envvar when
running setup.py, but if they were set for configure you know that
Python was built using those values.
msg74346 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-10-05 16:40
Lets see method customize_compiler(from sysconfig.py) 
The method search variables 'CPP', 'LDFLAGS' and 'CPPFLAGS' only in
environment. The variable CPP is not a Makefile variable. 'LDFLAGS' and
'CPPFLAGS' are makefile variables(macros). Usually makefile will export
to commands a macro only if exists environment variable with same name.
So to pass to distutils LDFLAGS is enough to to run as example
"LDFLAGS=fake make".

Issue3678 is about LDFLAGS for python shared library and now  command
start with $(LDSHARED) $(LDFLAGS) ...

If we modify Makefile to use  "...LDFLAGS=$(LDFLAGS) ... setup.py"
customize_compiler will append LDFLAGS to LDSHARED.
So this issue is about to use LDFLAGS for python modules.

Another point is that configure script append LDFLAGS to LDSHARED only
on certain platforms (all BSD based) and ignore all others.

What is preferred patch:
- a simple patch that just export LDFLAGS for setup.py;
- a patch that export LDFLAGS for all python programs, i.e. in Makefile
to append to RUNSHARED LDFLAGS=@LDFLAGS@ ;
- a patch that append LDFLAGS to LDSHARED in configure for all platforms
and remove use of LDFLAGS in Makefile and setup.py ?
msg74420 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2008-10-07 02:03
I checked in r66823 to add the -R flag extraction like Brett's -L
extraction.  I'll leave the issue open while folks discuss Roumen's
proposal.  I have no particular desire to delve deeper into 
distutils.
msg74650 - (view) Author: Roumen Petrov (rpetrov) * Date: 2008-10-10 22:03
I don't have rights to derive a new issue.
May be a new new title about distutils and makefile integration is more
appropriate.
msg76507 - (view) Author: Akira Kitada (akitada) * Date: 2008-11-27 17:33
I'm having similar problem with distutils.
http://mail.python.org/pipermail/python-dev/2008-November/083670.html

Is there any reason customize_compiler 
- only get CPPFLAGS in env, not ones from sysconfig?
- doesn't get OPT from from sysconfig, but it's used when env has CFLAGS?

I think this is a bug.
CPPFLAGS and OPT both should be used when building extension modules.
Just imagine a system that has include files in a non-standard locations
and explicitly added -I compiler flags to CPPFLAGS to adjust it.
When building an extension modules, the system should requires
the same CPPFLAGS settings. Without it, it won't compile.

In my opinion, this is rather serious bug.
So it would be nice to fix this before releasing Python 3.0/2.6.1/2.5.3
msg77649 - (view) Author: Akira Kitada (akitada) * Date: 2008-12-12 00:14
Is there anyone knowing the historical reason for this?
In that case I will take this and submit a patch here soon.
msg79073 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-01-04 18:03
May I propose a patch. The patch is only for linker flags(LDFLAGS). If
is accepted I will prepare another one for compiler flags.
The patch is not minimal. It is mostly cleanup patch.

For minimal patch - only change in Makefile.pre.in is enough.
Cleanup patch remove work-arounds for the issue.
msg79082 - (view) Author: Roumen Petrov (rpetrov) * Date: 2009-01-04 19:56
correct patch uploaded
msg80580 - (view) Author: Akira Kitada (akitada) * Date: 2009-01-26 17:03
Attached patch changes distutils to pass CPPFLAGS to compiler.
msg81278 - (view) Author: Akira Kitada (akitada) * Date: 2009-02-06 16:27
s/get_config_vars/get_config_var/
msg82930 - (view) Author: Akira Kitada (akitada) * Date: 2009-02-28 17:56
Updated issue4010 to honor os.environ['CPPFLAGS'].
msg91839 - (view) Author: John P. Speno (speno) Date: 2009-08-21 20:07
Hi. We encountered this issue on a Solaris 10 while building python 2.6.2.

There's a problem with reptrov's patch from 2009-01-04. The LDFLAGS in the 
Makefile.pre.in patch need to be quoted also. Like so:

LDFLAGS='$(LDFLAGS)'

This allows for multiple flags. Other than that, it works for us.
msg91847 - (view) Author: John P. Speno (speno) Date: 2009-08-21 21:48
Furthermore, there's another bug in setup.py  When extensions are built 
and CPPFLAGS (or other arguments) has multiple arguments, the order of 
the arguments are reversed. This is the wrong behavior. The specified 
directories should be used in the given order.

For example, if CPPFLAGS="-I/one -I/two -I/three", then when an 
extention is build, the order of the compile would be:

cc -I/three -I/two -I/one ...

There's this bit that seems reponsible in setup.py:

                    for directory in reversed(options.dirs):
                        add_dir_to_list(dir_list, directory)
msg121397 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-18 00:55
The reversed call was added by Brett in r60537.  Brett, can you comment?
msg121401 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-11-18 01:15
When I did the patch they were coming in reversed, so I reversed the reversal. Obviously something changed so now the manual reversal is not necessary anymore.
msg121403 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-18 01:19
I don’t think there can be a unit test for that, so do we agree on just removing the reversed call?
msg121409 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-11-18 01:49
On Wed, Nov 17, 2010 at 17:19, Éric Araujo <report@bugs.python.org> wrote:
>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> I don’t think there can be a unit test for that, so do we agree on just removing the reversed call?

As long as the person removing it verifies it and keeps an eye on the
buildbots, sure.
msg386437 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:30
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:30:40steve.dowersetstatus: open -> closed

nosy: + steve.dower
messages: + msg386437

resolution: out of date
stage: patch review -> resolved
2013-02-01 21:59:16brett.cannonsetnosy: - brett.cannon
2012-02-10 17:07:35jceasetnosy: + jcea
2010-11-18 01:49:20brett.cannonsetmessages: + msg121409
2010-11-18 01:19:33eric.araujosetmessages: + msg121403
2010-11-18 01:15:43brett.cannonsetmessages: + msg121401
2010-11-18 00:57:25eric.araujosetkeywords: - easy
2010-11-18 00:55:54eric.araujosetnosy: + brett.cannon
messages: + msg121397
2010-07-31 21:14:27eric.araujosetnosy: + eric.araujo
stage: patch review
type: compile error -> behavior

versions: + Python 3.2, - Python 2.6, Python 3.0, Python 3.1, Python 2.7
2010-05-20 20:31:41skip.montanarosetnosy: - skip.montanaro
2009-08-21 21:48:21spenosetmessages: + msg91847
2009-08-21 20:07:42spenosetnosy: + speno
messages: + msg91839
2009-02-28 17:56:08akitadasetfiles: + issue4010.diff
messages: + msg82930
2009-02-28 17:54:39akitadasetfiles: - issue4010.diff
2009-02-06 16:27:25akitadasetfiles: + issue4010.diff
messages: + msg81278
versions: + Python 3.1
2009-02-06 16:26:04akitadasetfiles: - issue4010.diff
2009-01-26 17:08:30tareksetassignee: tarek
2009-01-26 17:03:51akitadasetfiles: + issue4010.diff
nosy: + tarek
messages: + msg80580
2009-01-04 19:56:43rpetrovsetfiles: - py-issue-4010.patch
2009-01-04 19:56:32rpetrovsetfiles: + py-issue-4010.patch
messages: + msg79082
2009-01-04 18:03:19rpetrovsetfiles: + py-issue-4010.patch
messages: + msg79073
2008-12-20 14:37:41loewissetversions: - Python 2.5.3
2008-12-12 01:18:39brett.cannonsetnosy: - brett.cannon
2008-12-12 00:14:09akitadasetmessages: + msg77649
versions: + Python 2.7, Python 2.5.3
2008-11-27 17:33:09akitadasettype: behavior -> compile error
messages: + msg76507
nosy: + akitada
versions: + Python 2.6, Python 3.0
2008-10-10 22:03:30rpetrovsetmessages: + msg74650
2008-10-07 02:03:56skip.montanarosetmessages: + msg74420
2008-10-05 16:40:50rpetrovsetmessages: + msg74346
2008-10-02 22:03:41brett.cannonsetmessages: + msg74199
2008-10-02 20:57:22rpetrovsetnosy: + rpetrov
messages: + msg74194
2008-10-02 03:31:30skip.montanarosetmessages: + msg74154
2008-10-02 02:24:31skip.montanarosetmessages: + msg74152
2008-10-01 20:44:39skip.montanarosetkeywords: + patch, easy
files: + runtime.diff
messages: + msg74146
2008-10-01 20:27:17skip.montanarosetmessages: + msg74143
2008-10-01 19:25:58brett.cannonsetnosy: + brett.cannon
messages: + msg74140
2008-10-01 16:43:52skip.montanarocreate