C++ compilation support for distutils #42093

ahonkela · 2005-06-17T12:29:06Z

BPO	1222585
Nosy	@vstinner, @tiran, @benjaminp, @tarekziade, @merwok, @dhduvall, @skrah, @jdemeyer, @grimreaper, @ryandesign
Files	python_distutils_c++_support.patch: Patch against CVS HEAD to add C++ compilation support to distutils. py26_cxx_compiler.patch: Patch against release26-maint py25_cxx_compiler.patch: Patch against release25-maint distutils2-C++.patch: Patch for Distutils2 python-2.7-distutils-C++.patch python-3.3-distutils-C++.patch python-3.4-distutils-C++.patch python-3.x-distutils-C++.patch python-3.8-distutils-C++.patch python-3.x-distutils-C++.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-22.21:08:31.465>
created_at = <Date 2005-06-17.12:29:06.000>
labels = ['3.7', 'type-feature', 'library']
title = 'C++ compilation support for distutils'
updated_at = <Date 2020-10-22.21:08:31.465>
user = 'https://bugs.python.org/ahonkela'

bugs.python.org fields:

activity = <Date 2020-10-22.21:08:31.465>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-22.21:08:31.465>
closer = 'eric.araujo'
components = ['Distutils']
creation = <Date 2005-06-17.12:29:06.000>
creator = 'ahonkela'
dependencies = []
files = ['6696', '15389', '15390', '16742', '32218', '32219', '32220', '47046', '47669', '48016']
hgrepos = []
issue_num = 1222585
keywords = ['patch', 'needs review']
message_count = 35.0
messages = ['48477', '59307', '93211', '95629', '95631', '95633', '95640', '95641', '100804', '100805', '100806', '100807', '101588', '101602', '101626', '101627', '102216', '102231', '115762', '144117', '144191', '158200', '158223', '178660', '213211', '242706', '299396', '299403', '314859', '337380', '371688', '371689', '371757', '371840', '379348']
nosy_count = 21.0
nosy_names = ['mdboom', 'vstinner', 'christian.heimes', 'ahonkela', 'benjamin.peterson', 'tarek', 'eric.araujo', 'rpetrov', 'Arfrever', 'dhduvall', 'skrah', 'drkirkby', 'jdemeyer', 'eitan.adler', 'barcc', 'Michael.Crusoe', 'ehvatum', 'Christian H', 'Josef Kemetm\xc3\xbcller', 'bje', 'ryandesign']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1222585'
versions = ['Python 3.6', 'Python 3.7']

ahonkela · 2005-06-17T12:29:06Z

Add more proper C++ compilation support to distutils by
propagating information on used language to compiler as
well as linker. The information is used by
unixccompiler, cygwinccompiler and emxccompiler to call
a C++ compiler instead of a C compiler. Options for the
call are formed by as a union of options to both C and
C++ compilers.

The old version worked by accident on some platforms
where you can compile C++ programs by calling the C
compiler (e.g. gcc), but not on others (e.g. Tru64
Unix, probably also others not using gcc).

The more compilcated handling of options than with
linker is necessary to properly support manual
overrides such as "CXX=g++ -foo".

The attached patch is against CVS HEAD, but applies
with minimal changes at least to Python 2.3.

tiran · 2008-01-05T19:22:14Z

Isn't it already fixed? Need to check it at the upcoming bug day.

drkirkby · 2009-09-28T14:33:09Z

Just to add, I'd much appreciate if this bug could be squashed. I see
there is a patch, but it is against a 3-year old version, and is not a
small patch. Hence I am reluctant to try to apply it myself.

It is causing a problem in the Sage project, as I tried to compile Sage
with Sun's compiler on Solaris, rather than gcc.

mdboom · 2009-11-23T14:21:38Z

This is a rather serious bug for matplotlib -- since the C++ compiler is
not correctly selected, it can't be built with many Unix compilers, such
as Sun Studio.

tarekziade · 2009-11-23T14:29:37Z

Has someone tried to apply it with a recent Python ?

I can work on it but I'll need some help from someone matplotlib or Sage
I guess..

mdboom · 2009-11-23T15:43:34Z

I'm testing this against release25_maint and release26_maint as we
speak. I'll post new patches when that's done.

mdboom · 2009-11-23T18:05:26Z

Attached is a patch against the release26-maint branch. This works for
me on Sun Workshop 6.

Note that on Sun, the C compiler is 'cc' and the C++ compiler is 'CC',
but distutils uses 'cc' and 'c++'. However, the CXX environment
variable now works (after applying the patch), so this is less of an
issue, if it can be considered a bug at all.

mdboom · 2009-11-23T18:06:49Z

Here's another patch against release25_maint for the benefit of those
still using Python 2.5. It's different only in some whitespace changes.

Arfrever · 2010-03-10T19:26:47Z

I'm attaching my own patch, which has the following advantages:

Determination of target language is performed per source file instead of per list of source files passed to CCompiler.compile().
Proper determination of linker.
Proper support for CXXFLAGS and other variables.

tarekziade · 2010-03-10T19:59:26Z

Moved to distutils2 (distutils is feature frozen now)

mdboom · 2010-03-10T20:09:45Z

Does the reassignment only apply to the more recent more ambitious patch? The original patch (and my forward porting of it) is purely a bugfix, not a feature freeze. This is a showstopper bug for matplotlib on Solaris, for example, and I would hate to see its application further delayed.

tarekziade · 2010-03-10T20:40:27Z

If we consider that Distutils didn't provide c++ support, and that it partially worked by accident (through gcc), I would call it a new feature. Especially since it requires a new option.

We are freezing the API so we can't add options to methods/functions.

That's why it could be done on distutils2 instead, which will be released when 2.7 final is released (but not in the stdlib), and backward compatible with 2.4, 2.5, 2.6.

If we could do the proper job for C++ support in distutils2 that would be better.

Notice that distutils2 is a fork of how distutils was in trunk last month, so it should be fairly simple to depend on it in your project, to benefit from new features etc. The goal is to push it back in the stdlib in 3.3.

If you want to start a dev. branch of your project using distutils2, I am willing to help there.

Arfrever · 2010-03-23T17:28:57Z

To have proper support for C++ linking in distutils2 at least in when used with Python 2.7, Makefile should set a variable, which will specify default C++ linker. I think that LDCXXSHARED can be used as a name for this new variable. I'm attaching the patch for configure.in and Makefile.pre.in.

rpetrov · 2010-03-23T21:25:53Z

I could not found reasons python build to support C++ compiler as all source is C!
Another case is distutils where is possible external module to use C++ code.

Arfrever · 2010-03-24T08:08:49Z

Python can be compiled using C++ compiler.
See http://docs.python.org/extending/extending.html#writing-extensions-in-c

Arfrever · 2010-03-24T08:11:55Z

Also you would have to pass --with-cxx-main=<compiler> option to configure.

benjaminp · 2010-04-03T01:58:35Z

I think python-LDCXXSHARED.patch is fine.

tarekziade · 2010-04-03T08:47:27Z

LDCXXSHARED added in r79652 and r79657.

Now we can work on distutils2 side

merwok · 2010-09-07T13:48:06Z

Arfrever’s patch looks good. There is no tests for detect_language now; should the patch be blocked by that?

(Note: Remember to use “hg import --user 'Arfrever Frehtes Taifersar Arahesis <preferred email>'” to give proper credit)

djc · 2011-09-16T09:25:13Z

Could this get some attention, please?

merwok · 2011-09-17T15:16:46Z

Well, there’s no rush, Python 3.3b1 is not next week.

I’m willing to apply the patch to packaging and distutils2 (in a week or two when I’ve finished a massive cleanup and caught up with packaging), but I’d like tests for detect_language, and if possible tests for compiling C++ extensions/libs too.

djc · 2012-04-13T08:13:58Z

Ping, again. I'm sorry, I didn't write any of these patches and would not be a great fit for writing tests.

merwok · 2012-04-13T18:00:01Z

I should be able to port the patch and add tests for detect_language, but I know very little about C++ and may not be able to write a full test that really compiles and checks a C++ program. We’re having a sprint on the 21, I’ll see if I can work with another participant to do this. A sample short C++ source file would help (just use some Python/C function and print something).

Arfrever · 2012-12-31T10:10:33Z

I attach updated patches for distutils in case somebody wants to use them. (I privately update them once per week.)

merwok · 2014-03-12T07:54:25Z

Now that distutils2 development is stopped and that distutils is no longer under a feature freeze, this is a reasonable thing to add to 3.5. Thanks for your efforts Arfrever, and sorry you had to port and maintain this to so many different versions.

ehvatum · 2015-05-06T20:07:55Z

The 2012-12-31 of this patch (currently the latest) has the surprising effect of causing distutils CFLAGS to be dropped. See for reference: https://bugs.gentoo.org/show_bug.cgi?id=548776

The issue I am experiencing is resolved by changing the section of the patch that reads:

+- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++ (cc, cxx, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++ get_config_vars('CC', 'CXX', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++ cflags = ''
++ cxxflags = ''

to:

+- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++ (cc, cxx, cflags, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++ get_config_vars('CC', 'CXX', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++ cxxflags = cflags

This change causes the CFLAGS outputted by "python-config --cflags" to be used by distutils, as they are without the patch applied. To me, losing those CFLAGS, especially when building plain C python extensions, seems like a significant regression.

JosefKemetmller · 2017-07-28T15:29:27Z

python-3.4-distutils-C++.patch with changes of Erik Hvatum applied.

JosefKemetmller · 2017-07-28T16:04:55Z

python-3.4-distutils-C++.patch with Erik Hvatum's suggestions applied.

bje · 2018-04-03T04:01:38Z

Is there a simple workaround that one can put into setup.py in the meantime? I tried using compiler.compiler.remove('-Wstrict-prototypes') to no avail.

jdemeyer · 2019-03-07T10:03:32Z

I tried using compiler.compiler.remove('-Wstrict-prototypes') to no avail.

The -Wstrict-prototypes issue is a separate bug. It is fixed in Python >= 3.6 and there is an open backport PR for 2.7: #7476

ryandesign · 2020-06-16T18:22:28Z

What needs to happen to get this 15 year old bug fixed? It prevents C++ Python modules from being compiled in situations where the user needs to supply CXXFLAGS.

tiran · 2020-06-16T18:46:18Z

Please report the issue with setuptools. distutils is no longer under development. We recommend that all users use setuptools.

vstinner · 2020-06-17T16:21:34Z

What needs to happen to get this 15 year old bug fixed? It prevents C++ Python modules from being compiled in situations where the user needs to supply CXXFLAGS.

Someone has to propose a PR, someone else to review it, and a core developer has to take the responsibility to merge that PR. So far, there is no PR created.

ryandesign · 2020-06-18T22:12:40Z

Christian, thanks for the pointer. I think you're right, I probably am actually wanting this to be fixed in setuptools, not distutils, since setuptools is what people are using today. Since setuptools is an offshoot of distutils, I had assumed that the developers of setuptools would take ownership of any remaining distutils bugs that affected setuptools but I guess not. I looked through the setuptools issue tracker and this was the closest existing bug I could find: pypa/setuptools#1732

merwok · 2020-10-22T21:08:31Z

setuptools and other active build tools are the better target for this feature.

ahonkela mannequin added the stdlib Python modules in the Lib dir label Jun 17, 2005

drkirkby mannequin added the build The build process and cross-build label Sep 28, 2009

tarekziade mannequin self-assigned this Nov 23, 2009

tarekziade mannequin removed the stdlib Python modules in the Lib dir label Mar 10, 2010

merwok added type-feature A feature request or enhancement and removed build The build process and cross-build labels Jun 14, 2011

merwok assigned merwok and unassigned tarekziade Sep 17, 2011

merwok added the easy label Apr 13, 2012

merwok added stdlib Python modules in the Lib dir and removed easy labels Mar 12, 2014

marsam mannequin added the 3.7 (EOL) end of life label Dec 27, 2018

merwok closed this as completed Oct 22, 2020

merwok removed their assignment Oct 22, 2020

ezio-melotti transferred this issue from another repository Apr 9, 2022

vstinner mentioned this issue Jun 21, 2022

[WIP] gh-93744: Remove configure --with-cxx-main option #94063

Closed

haampie mentioned this issue Nov 3, 2022

python: 3.11.0 spack/spack#33507

Merged

C++ compilation support for distutils #42093

C++ compilation support for distutils #42093

Comments

ahonkela mannequin commented Jun 17, 2005

ahonkela mannequin commented Jun 17, 2005

tiran commented Jan 5, 2008

drkirkby mannequin commented Sep 28, 2009

mdboom mannequin commented Nov 23, 2009

tarekziade mannequin commented Nov 23, 2009

mdboom mannequin commented Nov 23, 2009

mdboom mannequin commented Nov 23, 2009

mdboom mannequin commented Nov 23, 2009

Arfrever mannequin commented Mar 10, 2010

tarekziade mannequin commented Mar 10, 2010

mdboom mannequin commented Mar 10, 2010

tarekziade mannequin commented Mar 10, 2010

Arfrever mannequin commented Mar 23, 2010

rpetrov mannequin commented Mar 23, 2010

Arfrever mannequin commented Mar 24, 2010

Arfrever mannequin commented Mar 24, 2010

benjaminp commented Apr 3, 2010

tarekziade mannequin commented Apr 3, 2010

merwok commented Sep 7, 2010

djc commented Sep 16, 2011

merwok commented Sep 17, 2011

djc commented Apr 13, 2012

merwok commented Apr 13, 2012

Arfrever mannequin commented Dec 31, 2012

merwok commented Mar 12, 2014

ehvatum mannequin commented May 6, 2015

JosefKemetmller mannequin commented Jul 28, 2017

JosefKemetmller mannequin commented Jul 28, 2017

bje mannequin commented Apr 3, 2018

jdemeyer commented Mar 7, 2019

ryandesign mannequin commented Jun 16, 2020

tiran commented Jun 16, 2020

vstinner commented Jun 17, 2020

ryandesign mannequin commented Jun 18, 2020

merwok commented Oct 22, 2020