classification
Title: configure: allow user-provided CFLAGS to override AC_PROG_CC defaults
Type: Stage:
Components: Build Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lemburg Nosy List: Arfrever, lemburg, loewis, mark.dickinson, ned.deily, vstinner
Priority: normal Keywords: patch

Created on 2010-03-23 12:14 by vstinner, last changed 2010-05-05 13:26 by lemburg. This issue is now closed.

Files
File name Uploaded Description Edit
configure_cflags.patch vstinner, 2010-03-23 12:14
Messages (27)
msg101578 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-23 12:14
configure.in uses AC_PROG_CC, extract of the autoconf manual:
(http://www.delorie.com/gnu/docs/autoconf/autoconf_64.html)
 
 If using the GNU C compiler, set shell variable GCC to `yes'. If output variable CFLAGS was not already set, set it to `-g -O2' for the GNU C compiler (`-O2' on systems where GCC does not accept `-g'), or `-g' for other compilers.

Python does already set the optimization level in its OPT variable: for gcc, it uses -O3 by default, and not -O option (in this case, gcc disables all optimisations, it's like -O0) if --with-pydebug is used.

Because of AC_PROG_CC, Python is compiled with -O2 even if --with-pydebug is used, which is bad because it's harder to debug an optimized program: most variable are unavailable (gcc prints "<optimized out>").

Another problem is that AC_PROG_CC eats user CFLAGS. It's not possible to specify: ./configure CFLAGS="myflags".

On the autoconf mailing list, I saw a simple "trick": 

   Save CFLAGS before you call AC_PROG_CC, and restore it after, 
   if you don't want "-g -O2".

Attached patch implements that. Results:
 * ./configure: CFLAGS=$(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS)
 * ./configure --with-pdebug CFLAGS="-O0": CFLAGS=$(BASECFLAGS) -O0 $(OPT) $(EXTRA_CFLAGS)

It works :-)
msg101589 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-03-23 18:07
Setting CFLAGS is broken in Python configure system, so it's better not to rely on it (or to fix it, but that's a major task - the whole CFLAGS and LDFLAGS system used in Python's configure has over the years turned into a complete mess).

You should get the same result by using ./configure OPT="-O0".
msg101592 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-23 18:53
> (or to fix it, but that's a major task - the whole CFLAGS 
> and LDFLAGS system used in Python's configure has over the
> years turned into a complete mess).

What do you mean by "a complete mess"? Did you try my patch? Is it enough to fix this issue?

A least, my patch is enough to disable all compilation optimization if --with-pydebug option is used (which was my first goal).
msg101593 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-03-23 19:17
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> (or to fix it, but that's a major task - the whole CFLAGS 
>> and LDFLAGS system used in Python's configure has over the
>> years turned into a complete mess).
> 
> What do you mean by "a complete mess"? 

There's a discussion on some other ticket about this. Basically,
Python's configure script doesn't really allow setting CFLAGS
in a meaningful way via env vars (which it should to be standards
conform).

It adds all sorts of non-standard variables which are then combined
to build the final CFLAGS variable: BASECFLAGS, EXTRA_CFLAGS and
OPT.

> Did you try my patch? Is it enough to fix this issue?

No. It doesn't allow overriding the CFLAGS settings actually
used by the system. You'd also need to set BASECFLAGS="", OPT="" and
EXTRA_CFLAGS="" in addition to applying your patch and setting
CFLAGS to "-O0".

> A least, my patch is enough to disable all compilation optimization if --with-pydebug option is used (which was my first goal).

It unconditionally overrides CFLAGS - even if it is not set and
defined by AC_PROG_CC as "-g -O2". That would need to be corrected.

Other than that it does help a little work around the mess :-)
msg101660 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-25 00:26
MaL> It unconditionally overrides CFLAGS - even if it is not 
MaL> set and defined by AC_PROG_CC as "-g -O2". That would need 
MaL> to be corrected.
MaL>
MaL> Other than that it does help a little work around the mess :-)

I commited my patch: r79392 (trunk). I'm waiting for the buildbots before porting to other branches :-)

On my Linux, .c files are now compiled with:
 - "-fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall ..." (default)
 - "-fno-strict-aliasing -g -Wall ..." (--with-pydebug): no more compiler optimization hurting gdb ;-)
msg101664 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-25 01:06
> I commited my patch: r79392 (trunk). I'm waiting for the buildbots before porting to other branches :-)

The buildbots look happy => r79401 (py3k), blocked in 2.6 and 3.1

The issue title was "configure: ignore AC_PROG_CC hardcoded CFLAGS", and not "fix configure", so I can close the issue.

Marc-Andre: open a new issue if you are motivated to fix "configure mess" :-)
msg101673 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-03-25 07:42
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> MaL> It unconditionally overrides CFLAGS - even if it is not 
> MaL> set and defined by AC_PROG_CC as "-g -O2". That would need 
> MaL> to be corrected.
> MaL>
> MaL> Other than that it does help a little work around the mess :-)
> 
> I commited my patch: r79392 (trunk). I'm waiting for the buildbots before porting to other branches :-)
> 
> On my Linux, .c files are now compiled with:
>  - "-fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall ..." (default)
>  - "-fno-strict-aliasing -g -Wall ..." (--with-pydebug): no more compiler optimization hurting gdb ;-)

The patch you checked in still unconditionally overrides the
CFLAGS setting applied by AC_PROG_CC in case no CFLAGS variable
is set.

Please fix that or revert the patch.
msg101674 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-03-25 07:43
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
>> I commited my patch: r79392 (trunk). I'm waiting for the buildbots before porting to other branches :-)
> 
> The buildbots look happy => r79401 (py3k), blocked in 2.6 and 3.1
> 
> The issue title was "configure: ignore AC_PROG_CC hardcoded CFLAGS", and not "fix configure", so I can close the issue.

The issue now is: AC_PROG_CC no longer initializes CFLAGS if not set.

> Marc-Andre: open a new issue if you are motivated to fix "configure mess" :-)

Some other day...
msg101680 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-03-25 09:14
MaL> The patch you checked in still unconditionally overrides the
MaL> CFLAGS setting applied by AC_PROG_CC in case no CFLAGS variable
MaL> is set.
MaL>
MaL> The issue now is: AC_PROG_CC no longer initializes CFLAGS 
MaL> if not set.

Sorry, but I don't understand. Why should it be initialized?

I don't like the default value because it enables optimization when --with-pydebug is used and I consider that as a bug. If no configure option is used, Python adds -O3 as before. About -g: Python always add it, so the -g from AC_PROG_CC was redundant.

In Makefile.pre.in, prefixes and suffixes are added to the CFLAGS: "CFLAGS= $(BASECFLAGS) @CFLAGS@ $(OPT) $(EXTRA_CFLAGS)". But I consider this as a separate issue.
msg101689 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-03-25 10:31
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> MaL> The patch you checked in still unconditionally overrides the
> MaL> CFLAGS setting applied by AC_PROG_CC in case no CFLAGS variable
> MaL> is set.
> MaL>
> MaL> The issue now is: AC_PROG_CC no longer initializes CFLAGS 
> MaL> if not set.
> 
> Sorry, but I don't understand. Why should it be initialized?

Sorry, perhaps I wasn't clear enough.

The purpose of AC_PROG_CC is to setup the env vars depending on
the found C compiler. One of these settings is the CFLAGS env
var and you patch causes those settings to be overwritten
if CFLAGS has not been defined as env var before starting
./configure.

Since CFLAGS will most of the time not be set by the user
running ./configure, the patch effectively disables the
default settings determined based on the compiler by AC_PROG_CC.

> I don't like the default value because it enables optimization when --with-pydebug is used and I consider that as a bug. If no configure option is used, Python adds -O3 as before. About -g: Python always add it, so the -g from AC_PROG_CC was redundant.

Right, but it's possible that AC_PROG_CC adds other flags
as well (it currently doesn't, but that may well change in
future versions of autoconf).

I'd suggest to only override the CFLAGS setting in case it
was defined before running the AC_PROG_CC code.

In addition, it may be useful to have --with-pydebug replace OPT
with "-O0" or add "-O0" to it the end of it.

> In Makefile.pre.in, prefixes and suffixes are added to the CFLAGS: "CFLAGS= $(BASECFLAGS) @CFLAGS@ $(OPT) $(EXTRA_CFLAGS)". But I consider this as a separate issue.

Right. That's for historic reasons. OPT is very old, BASECFLAGS
and EXTRA_CFLAGS are newer additions.
msg102811 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-04-11 06:48
Note these changes to restore CFLAGS have the side effect of breaking OS X universal builds; see Issue8366.
msg102862 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-11 18:35
Victor, could you please fix the patch or revert it ?

Thanks.
msg102864 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-11 18:39
Reopening the ticket: it shouldn't have been closed.

I'm also making this a release blocker, since this needs to be fixed before the next release - the CC variable has to be initialized by the build system and breaking this in general for all default builds just to get a debug build without optimizations is not warranted.
msg102868 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-04-11 19:00
To be totally fair, it is likely that part of the OS X breakage was caused by the original code inadvertently working around the original CFLAGS misbehavior. From an OS X perspective, it may be best to just fix the new issue and move on.
msg103952 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-22 11:30
Issue #8366 was caused by a fix of issue #1628484 (and ok, indirectly by my change). Issue #8366 is now fixed. Can I close this issue again or do you think that there is still something to do?
msg103981 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-04-22 21:48
I think the issue can be closed again.
msg103983 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-22 21:54
Even if this issue doesn't fix all the configure "complete mess", I think that it improves it a little bit. Open new issues if you would like to fix other parts of the configure script.
msg103989 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-23 07:12
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Even if this issue doesn't fix all the configure "complete mess", I think that it improves it a little bit. Open new issues if you would like to fix other parts of the configure script.

Viktor, you are still missing the point and please stop closing the issue again and again.

Please change the configure.in script to only override the CFLAGS in case they were set before entering the AC_PROG_CC part !

I've explained that over and over again and I don't understand why you are being completely ignorant of the implications of your change.

The Mac OS X fix is unrelated to this change.
msg103992 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-23 07:38
Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
> 
> Please change the configure.in script to only override the CFLAGS in case they were set before entering the AC_PROG_CC part !

Sorry, this should have read:

... if CFLAGS *were* set before entering AC_PROG_CC ...

> I've explained that over and over again and I don't understand why you are being completely ignorant of the implications of your change.

One of the main purposes of AC_PROG_CC is to setup CFLAGS in case
they are not and your change defeats this purpose.
msg104649 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2010-04-30 16:14
Sorry but i don't really understand the problem of my patch, and I don't want to spend time of this. Revert my patch if you think that it introduced a regression.
msg104653 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-04-30 17:34
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> Sorry but i don't really understand the problem of my patch, and I don't want to spend time of this. Revert my patch if you think that it introduced a regression.

See the new implementation for what I meant...

r80665 - in python/trunk: configure configure.in
r80666 - in python/branches/py3k: configure	configure.in

[issue8211] configure: ignore AC_PROG_CC hardcoded CFLAGS

Only override the AC_PROG_CC determined CFLAGS if they were set by the user.
This restores the default behavior in the common case of not having CFLAGS
defined when running configure.
msg105015 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-05 12:17
Since r80665, a

./configure --with-pydebug

seems to give compilation with -O2 (tested on OS X and Linux).  Is this intentional?

I'm getting, e.g.,

gcc -c  -g -O2 -g -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c
msg105019 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-05-05 12:34
Mark Dickinson wrote:
> 
> Mark Dickinson <dickinsm@gmail.com> added the comment:
> 
> Since r80665, a
> 
> ./configure --with-pydebug
> 
> seems to give compilation with -O2 (tested on OS X and Linux).  Is this intentional?

Yes. I've restored the previous behavior of configure to
have AC_PROG_CC determine CFLAGS defaults.

Please open a new ticket to have --with-pydebug disable use
of any optimization flags. We need to find a different
solution for that. Unconditionally ignoring the AC_PROG_CC
CFLAGS defaults is not solution.
msg105020 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-05-05 12:40
Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
> 
> Mark Dickinson wrote:
>>
>> Mark Dickinson <dickinsm@gmail.com> added the comment:
>>
>> Since r80665, a
>>
>> ./configure --with-pydebug
>>
>> seems to give compilation with -O2 (tested on OS X and Linux).  Is this intentional?
> 
> Yes. I've restored the previous behavior of configure to
> have AC_PROG_CC determine CFLAGS defaults.
> 
> Please open a new ticket to have --with-pydebug disable use
> of any optimization flags. We need to find a different
> solution for that. Unconditionally ignoring the AC_PROG_CC
> CFLAGS defaults is not solution.

Note that using the following line will disable the AC_PROG_CC
defaults:

./configure --with-pydebug CFLAGS=''

This is a new feature that was introduced previously by Victor
and that I corrected in r80665.
msg105023 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-05 13:00
> Yes. I've restored the previous behavior of configure to
> have AC_PROG_CC determine CFLAGS defaults.

Just to be clear, the previous behaviour has *not* been restored.  Up until r80665, a '--with-pydebug' build did not include optimization.  Since r80665, it does.
msg105025 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-05-05 13:23
Ah, I understand now: -O2 started appearing in debug builds in r79218, which changed the Makefile to respect CFLAGS.  I tested a variety of revision, but failed to test revision in between that and Victor's change...
msg105026 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2010-05-05 13:26
Mark Dickinson wrote:
> 
> Mark Dickinson <dickinsm@gmail.com> added the comment:
> 
> Ah, I understand now: -O2 started appearing in debug builds in r79218, which changed the Makefile to respect CFLAGS.  I tested a variety of revision, but failed to test revision in between that and Victor's change...

Right. I was referring to r79391, ie. the state before Victor checked
in his patch.
History
Date User Action Args
2010-05-05 13:26:51lemburgsetmessages: + msg105026
2010-05-05 13:23:59mark.dickinsonsetmessages: + msg105025
2010-05-05 13:00:33mark.dickinsonsetmessages: + msg105023
2010-05-05 12:40:28lemburgsetmessages: + msg105020
2010-05-05 12:35:14lemburgsetstatus: open -> closed
2010-05-05 12:34:49lemburgsetmessages: + msg105019
2010-05-05 12:17:56mark.dickinsonsetstatus: closed -> open
nosy: + mark.dickinson
messages: + msg105015

2010-04-30 17:35:28lemburgsetstatus: open -> closed
assignee: lemburg
title: configure: ignore AC_PROG_CC hardcoded CFLAGS -> configure: allow user-provided CFLAGS to override AC_PROG_CC defaults
resolution: fixed
priority: release blocker -> normal
2010-04-30 17:34:16lemburgsetmessages: + msg104653
2010-04-30 16:14:33vstinnersetmessages: + msg104649
2010-04-23 07:38:06lemburgsetmessages: + msg103992
2010-04-23 07:12:24lemburgsetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg103989
2010-04-22 21:54:48vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg103983
2010-04-22 21:48:09ned.deilysetmessages: + msg103981
2010-04-22 11:30:17vstinnersetmessages: + msg103952
2010-04-11 19:00:09ned.deilysetmessages: + msg102868
2010-04-11 18:39:37lemburgsetresolution: fixed -> (no value)
2010-04-11 18:39:07lemburgsetstatus: closed -> open
priority: release blocker
messages: + msg102864
2010-04-11 18:35:09lemburgsetmessages: + msg102862
2010-04-11 06:48:26ned.deilysetnosy: + ned.deily
messages: + msg102811
2010-03-25 10:31:31lemburgsetmessages: + msg101689
2010-03-25 09:14:35vstinnersetmessages: + msg101680
2010-03-25 07:43:59lemburgsetmessages: + msg101674
2010-03-25 07:42:25lemburgsetmessages: + msg101673
2010-03-25 01:06:19vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg101664
2010-03-25 00:26:30vstinnersetmessages: + msg101660
2010-03-23 19:17:45lemburgsetmessages: + msg101593
2010-03-23 18:53:42vstinnersetmessages: + msg101592
2010-03-23 18:09:23Arfreversetnosy: + Arfrever
2010-03-23 18:07:35lemburgsetmessages: + msg101589
2010-03-23 17:50:22pitrousetnosy: + lemburg, loewis
2010-03-23 12:14:06vstinnercreate