classification
Title: sysconfig generation uses some env variables multiple times
Type: behavior Stage: resolved
Components: Build Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: bkabrda, frougon, ned.deily, serhiy.storchaka, strogon14, vstinner
Priority: normal Keywords: patch

Created on 2013-04-09 13:51 by bkabrda, last changed 2019-05-17 12:01 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
00178-dont-duplicate-flags-in-sysconfig.patch bkabrda, 2013-04-10 08:59 review
install.diff vstinner, 2018-12-18 14:28
build.diff vstinner, 2018-12-18 14:29
Messages (8)
msg186408 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-04-09 13:51
When compiling Python 3.3.1, I noticed that some variables like LDFLAGS or CFLAGS in sysconfig have some flags multiple times. (Which BTW breaks distutils.tests.{test_sysconfig_compiler_vars,test_sysconfig_module}) This is caused by interpretation of Makefile in sysconfig._parse_makefile(), which seems to evaluate the variables in Makefile - but some variables in Makefile are formed by expanding env variables multiple times, e.g.:
PY_LDFLAGS=        $(CONFIGURE_LDFLAGS) $(LDFLAGS)
CONFIGURE_LDFLAGS=        @LDFLAGS@
so when doing the build from scratch with configure & make, PY_LDFLAGS gets the content of LDFLAGS twice  (as far as I remember autotools...), CFLAGS gets expanded like 5 times at least.
I think that this is not the correct behaviour, but not sure, maybe I'm doing something wrong.

Thanks.
msg186467 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-04-09 22:35
There definitely are configurations where some values do get duplicated in CFLAGS and LDFLAGS.  In my experience this is generally harmless for builds but, as you point out, it can break tests that expect particular values.  It would be nice to clean this up.
msg186483 - (view) Author: Bohuslav "Slavek" Kabrda (bkabrda) * Date: 2013-04-10 08:59
I'm attaching a patch that I'm currently using to solve this. It works, but it's a bit aggressive - in the sense that it only adds a string to the sysconfig variable iff this string is not a substring of current variable value. So it may corrupt some values, e.g. it wouldn't add "python" to variable if that variable already had "/usr/lib/python". But it seems to get all the values just fine for me.
msg217210 - (view) Author: Christopher Arndt (strogon14) Date: 2014-04-26 20:04
Another solution may be to make the test more relaxed and regard the value returned by sysconfig.get_config_var() as a _set_ of shell tokens, whose elements may occur more than once, e.g.


    def test_sysconfig_module(self):
        import sysconfig as global_sysconfig
        from shlex import split
        self.assertEqual(
            set(split(global_sysconfig.get_config_var('CFLAGS'))),
            set(split(sysconfig.get_config_var('CFLAGS'))))
        self.assertEqual(
            set(split(global_sysconfig.get_config_var('LDFLAGS'))),
            set(split(sysconfig.get_config_var('LDFLAGS'))))
msg232499 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-12-11 22:02
The patch doesn't look good to me. If the value contains "-lfoo-lbar $(name)" then substituting name="-lfoo" or name="-lbar" doesn't work.
msg332062 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-18 14:28
I don't think that the attached patch is correct. See attached install.diff: difference without/with 00178-dont-duplicate-flags-in-sysconfig.patch on Python installed in /usr/bin/python3.

Example of bug:

  'TESTRUNNER': 'LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.7.1/build/optimized '
-               './python '
-               '/builddir/build/BUILD/Python-3.7.1/Tools/scripts/run_tests.py',
+               './python /Tools/scripts/run_tests.py',

The /Tools directory doesn't exist :-/


> I think that this is not the correct behaviour, but not sure, maybe I'm doing something wrong.

Technically, it's perfectly fine to pass the same flag multiple times. It's common to pass -O0 -Og to gcc for example: gcc uses the last -O option value (which overrides the previous ones).

--

This patch is used in the python3 package of Fedora:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00178-dont-duplicate-flags-in-sysconfig.patch

Patch added by:

commit 58f477b403222ea6c13d5d7358551b606cddc0f8
Author: Bohuslav Kabrda <bkabrda@redhat.com>
Date:   Wed Apr 10 14:30:09 2013 +0200

    Updated to Python 3.3.1.
    
    - Refreshed patches: 55 (systemtap), 111 (no static lib), 146 (hashlib fips),
    153 (fix test_gdb noise), 157 (uid, gid overflow - fixed upstream, just
    keeping few more downstream tests)
    - Removed patches: 3 (audiotest.au made it to upstream tarball)
    - Removed workaround for http://bugs.python.org/issue14774, discussed in
    http://bugs.python.org/issue15298 and fixed in revision 24d52d3060e8.

https://src.fedoraproject.org/rpms/python3/c/58f477b403222ea6c13d5d7358551b606cddc0f8?branch=master
msg332063 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-12-18 14:29
build.diff: Difference without/with the patch on Python build from source.

Example:

-	CPPFLAGS = "-I. -I./Include"
+	CPPFLAGS = "-I. -I/Include"

This change is wrong: /Include directory doesn't exist.

Another example:

-	LIBPL = "/usr/local/lib/python3.8/config-3.8dm-x86_64-linux-gnu"
+	LIBPL = "/usr/local/lib/python3.8/config-dm-x86_64-linux-gnu"

I don't understand why "3.8" is removed from the path.
msg342711 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-17 12:01
The patch is wrong. I'm not sure when/how C flags are duplicated. Anyway, it seems like the issue is somehow outdated or even gone, so I close the issue.
History
Date User Action Args
2019-05-17 12:01:04vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg342711

stage: patch review -> resolved
2018-12-18 14:29:32vstinnersetfiles: + build.diff

messages: + msg332063
2018-12-18 14:28:02vstinnersetfiles: + install.diff
nosy: + vstinner
messages: + msg332062

2014-12-11 22:02:08serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg232499
2014-12-11 12:12:23serhiy.storchakasetstage: needs patch -> patch review
versions: + Python 3.5, - Python 3.3
2014-05-25 14:36:15frougonsetnosy: + frougon
2014-04-26 20:04:19strogon14setnosy: + strogon14
messages: + msg217210
2013-04-10 08:59:07bkabrdasetfiles: + 00178-dont-duplicate-flags-in-sysconfig.patch
keywords: + patch
messages: + msg186483
2013-04-09 22:35:40ned.deilysetversions: + Python 2.7, Python 3.4
2013-04-09 22:35:03ned.deilysetversions: + Python 3.3, - Python 3.1
nosy: + ned.deily

messages: + msg186467

stage: needs patch
2013-04-09 13:51:48bkabrdacreate