classification
Title: distutils.sysconfig.customize_compiler() overrides CFLAGS var with OPT var if CFLAGS env var is set
Type: Stage: resolved
Components: Build Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: koobs, miss-islington, ned.deily, vstinner, yan12125
Priority: normal Keywords: patch

Created on 2019-03-08 12:23 by vstinner, last changed 2019-04-11 01:07 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12236 closed vstinner, 2019-03-08 12:34
PR 12348 merged vstinner, 2019-03-15 14:16
PR 12349 merged vstinner, 2019-03-15 14:19
PR 12380 closed yan12125, 2019-03-17 16:31
PR 12403 merged vstinner, 2019-03-18 11:40
PR 12415 merged vstinner, 2019-03-18 16:25
PR 12417 merged vstinner, 2019-03-18 16:32
PR 12751 merged vstinner, 2019-04-09 17:08
PR 12764 merged vstinner, 2019-04-10 12:56
PR 12768 merged miss-islington, 2019-04-10 23:39
Messages (19)
msg337468 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-08 12:23
When a C extension is built by distutils, distutils.sysconfig.customize_compiler() is used to configure compiler flags. Extract of the code:

    (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
        get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
                        'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')

    ...
    if 'CFLAGS' in os.environ:
        cflags = opt + ' ' + os.environ['CFLAGS']
        ldshared = ldshared + ' ' + os.environ['CFLAGS']
    if 'CPPFLAGS' in os.environ:
        cpp = cpp + ' ' + os.environ['CPPFLAGS']
        cflags = cflags + ' ' + os.environ['CPPFLAGS']
        ldshared = ldshared + ' ' + os.environ['CPPFLAGS']
    ...

If the CFLAGS environment variable is set, the 'CFLAGS' configuration variable is overriden with the 'OPT' configuration variable: cflags = opt + ...

This bug has been fixed since 2013 in Fedora and RHEL by this patch:
https://src.fedoraproject.org/rpms/python2/blob/master/f/00168-distutils-cflags.patch

RHEL bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=849994

I converted that patch to a pull request.
msg337505 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-03-08 16:16
This appears to be a duplicate of Issue969718.
msg337990 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-15 13:58
New changeset 86082c22d23285995a32aabb491527c9f5629556 by Victor Stinner in branch 'master':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236)
https://github.com/python/cpython/commit/86082c22d23285995a32aabb491527c9f5629556
msg337994 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-15 15:03
New changeset 37f6971777c05b5ca9c157606896b7ff458756a5 by Victor Stinner in branch '2.7':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) (GH-12349)
https://github.com/python/cpython/commit/37f6971777c05b5ca9c157606896b7ff458756a5
msg337995 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-15 15:03
New changeset 6c0e0d141a07cc3fd2441d9df8d762f56bf7edf2 by Victor Stinner in branch '3.7':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) (GH-12348)
https://github.com/python/cpython/commit/6c0e0d141a07cc3fd2441d9df8d762f56bf7edf2
msg338051 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2019-03-16 03:48
After this patch, test_distutils failed if $CPPFLAGS is not empty when building CPython. For example:

$ export CPPFLAGS=-DFOO=BAR
$ ./configure
$ make
$ ./python -m test test_distutils   
Run tests sequentially
0:00:00 load avg: 0.45 [1/1] test_distutils
test test_distutils failed -- Traceback (most recent call last):
  File "/home/yen/Projects/cpython/Lib/distutils/tests/test_sysconfig.py", line 99, in test_customize_compiler
    self.assertEqual(comp.exes['compiler'], 'my_cc --sysconfig-cflags --mycflags')
AssertionError: 'my_cc --sysconfig-cflags --mycflags -DFOO=BAR' != 'my_cc --sysconfig-cflags --mycflags'
- my_cc --sysconfig-cflags --mycflags -DFOO=BAR
?                                    ----------
+ my_cc --sysconfig-cflags --mycflags


test_distutils failed

== Tests result: FAILURE ==

1 test failed:
    test_distutils

Total duration: 2 sec 192 ms
Tests result: FAILURE

Tested with commit d2fdd1fedf6b9dc785cf5025b548a989faed089a.

This is an issue as Arch Linux uses CPPFLAGS="-D_FORTIFY_SOURCE=2" for creating packages [1].

[1] https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n39
msg338199 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 11:41
Chih-Hsuan Yen: After this patch, test_distutils failed if $CPPFLAGS is not empty when building CPython.

Aha, more sysconfig configuration variables and environment variables should be mocked. I wrote PR 12403 to mock all variables. Would you mind to test and review it?
msg338229 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2019-03-18 14:26
STINNER Victor: Thank you very much for that much better fix! I've tested it on Arch Linux. With that patch, test_distutils passes as usual. The pull request looks good, too. Just a question: I think it should be backported to 3.7 and 2.7 branches like https://github.com/python/cpython/pull/12236?
msg338252 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 16:19
New changeset 72c7b372cf145fded93a9a776acc742a60090f95 by Victor Stinner in branch 'master':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403)
https://github.com/python/cpython/commit/72c7b372cf145fded93a9a776acc742a60090f95
msg338254 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 16:37
> Just a question: I think it should be backported to 3.7 and 2.7 (...)

Right, I wrote PRs for that. I didn't use GitHub labels because the bot is currently broken.
msg338264 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 17:34
New changeset dd1cfefd67f254ce44f33995922e347c9d3f7b4e by Victor Stinner in branch '3.7':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) (GH-12415)
https://github.com/python/cpython/commit/dd1cfefd67f254ce44f33995922e347c9d3f7b4e
msg338265 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 17:34
New changeset 8c380e99e9b71387d5722373805fe3159e2d912c by Victor Stinner in branch '2.7':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) (GH-12417)
https://github.com/python/cpython/commit/8c380e99e9b71387d5722373805fe3159e2d912c
msg338267 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-18 17:35
Thanks for the review, I merged my PR and the test enhancements.
msg338268 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2019-03-18 17:41
Oh, I didn't know the bot is not working for now. Thank you for making the test more robust!
msg339785 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-09 17:02
Oh, the test failed on Python 2.7 on macOS:

x86-64 High Sierra 2.7
https://buildbot.python.org/all/#/builders/140/builds/211

======================================================================
FAIL: test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/distutils/tests/test_sysconfig.py", line 134, in test_customize_compiler
    'sc_cc -E')
AssertionError: '/usr/bin/clang -E' != 'sc_cc -E'
msg339795 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-09 17:54
New changeset 22de4ce498b656063e236350e8404981c13e1cd8 by Victor Stinner in branch '2.7':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12751)
https://github.com/python/cpython/commit/22de4ce498b656063e236350e8404981c13e1cd8
msg339859 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-10 12:56
Good, my change fixed x86-64 High Sierra 2.7 buildbot:
https://buildbot.python.org/all/#/builders/140/builds/216

I wrote PR 12764 to forward-port the fix to the master branch.
msg339899 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-10 23:38
New changeset a9bd8925c7fa50dd3cfab125b824ec192133ef49 by Victor Stinner in branch 'master':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12764)
https://github.com/python/cpython/commit/a9bd8925c7fa50dd3cfab125b824ec192133ef49
msg339901 - (view) Author: miss-islington (miss-islington) Date: 2019-04-10 23:58
New changeset d9b25a2627ff6f4e10d46b4de4fff941b63497c7 by Miss Islington (bot) in branch '3.7':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12764)
https://github.com/python/cpython/commit/d9b25a2627ff6f4e10d46b4de4fff941b63497c7
History
Date User Action Args
2019-04-11 01:07:51vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-04-10 23:58:58miss-islingtonsetnosy: + miss-islington
messages: + msg339901
2019-04-10 23:39:00miss-islingtonsetpull_requests: + pull_request12697
2019-04-10 23:38:51vstinnersetmessages: + msg339899
2019-04-10 12:56:46vstinnersetmessages: + msg339859
2019-04-10 12:56:02vstinnersetpull_requests: + pull_request12692
2019-04-09 17:54:28vstinnersetmessages: + msg339795
2019-04-09 17:08:17vstinnersetstage: resolved -> patch review
pull_requests: + pull_request12678
2019-04-09 17:02:29vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg339785
2019-03-18 17:41:35yan12125setmessages: + msg338268
2019-03-18 17:35:09vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg338267

stage: patch review -> resolved
2019-03-18 17:34:44vstinnerlinkissue969718 superseder
2019-03-18 17:34:35vstinnersetmessages: + msg338265
2019-03-18 17:34:34vstinnersetmessages: + msg338264
2019-03-18 16:37:35vstinnersetmessages: + msg338254
2019-03-18 16:32:16vstinnersetpull_requests: + pull_request12371
2019-03-18 16:25:01vstinnersetpull_requests: + pull_request12369
2019-03-18 16:19:04vstinnersetmessages: + msg338252
2019-03-18 14:26:39yan12125setmessages: + msg338229
2019-03-18 11:41:47vstinnersetmessages: + msg338199
2019-03-18 11:40:48vstinnersetpull_requests: + pull_request12358
2019-03-17 16:31:58yan12125setpull_requests: + pull_request12338
2019-03-16 03:48:40yan12125setnosy: + yan12125
messages: + msg338051
2019-03-15 15:03:53vstinnersetmessages: + msg337995
2019-03-15 15:03:46vstinnersetmessages: + msg337994
2019-03-15 14:19:36vstinnersetpull_requests: + pull_request12315
2019-03-15 14:16:44vstinnersetpull_requests: + pull_request12314
2019-03-15 13:58:03vstinnersetmessages: + msg337990
2019-03-12 03:03:42koobssetnosy: + koobs
2019-03-08 16:16:49ned.deilysetnosy: + ned.deily
messages: + msg337505
2019-03-08 12:34:54vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12223
2019-03-08 12:23:45vstinnercreate