currently, much of the configuration information is fetched by opening the config.h and the Makefile for the build. The locations of these files are derived from the prefix given at configure time.

If you want to have two distinct builds with the same prefix, but with different configuration options, this kind of configuration information is unsufficient.  The current use case is a normal build and a debug build, as used for the Fedora and Debian/Ubuntu python packages.

Fedora and Debian/Ubuntu carry a patch, adding a sys.pydebug attribute and patching "some" things to get the correct information for these builds using the same prefix.

This kind of information should not be derived by following some paths names, but be contained in the python binary itself, without needing to read the Makefile and the config.h files.

Attached is a patch currently applied in Debian/Ubuntu (and some variant for Fedora) which introduces a sys.pydebug attribute and derives the correct locations for the config.h and Makefile files. It's not meant as a patch ready to apply, but to get an idea what is needed to have two different python installations with the same prefix.

IMO there should not be any need to fetch information from config.h or the Makefile.
What about a sys.build_config dictionary containing all the necessary data?

+1

Amaury, you mention a sys.build_config dictionary, but I think it should actually be baked into the sysconfig module, possibly as a _sysconfig extension module.  sysconfig is the new goodness for getting at this, and I don't think it ought to have to go through a sys dictionary to get the information.

I'd be willing to work on this, if we can get some consensus.

For reference, the patch that I'm currently applying to Fedora's build of Python-3.2a1 can be seen at:
  http://pkgs.fedoraproject.org/gitweb/?p=python3.git;a=blob_plain;f=python-3.2a1-debug-build.patch;hb=HEAD

It appears to be very similar to Matthias' patch (it was originally based on an earlier version of Debian's python 2 patch, which I fixed up against Fedora's python 2, changed some aspects I wasn't happy with, then ported to python 3.1, then fixed up to 3.2a1 IIRC)

For further reference, Fedora's python3.spec has these comments that I wrote on the patch:
# Patch to support building both optimized vs debug stacks DSO ABIs, sharing
# the same .py and .pyc files, using "_d.so" to signify a debug build of an
# extension module.
#
# Based on Debian's patch for the same, 
#  http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/debug-build.dpatch
# 
# (which was itself based on the upstream Windows build), but with some
# changes:
#
#   * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so,
# but this can potentially find a module built against the wrong DSO ABI.  We
# instead search for just module_d.so in a debug build
#
#   * We remove this change from configure.in's build of the Makefile:
#   SO=$DEBUG_EXT.so
# so that sysconfig.py:customize_compiler stays with shared_lib_extension='.so'
# on debug builds, so that UnixCCompiler.find_library_file can find system
# libraries (otherwise "make sharedlibs" fails to find system libraries,
# erroneously looking e.g. for "libffi_d.so" rather than "libffi.so")
#
#   * We change Lib/distutils/command/build_ext.py:build_ext.get_ext_filename
# to add the _d there, when building an extension.  This way, "make sharedlibs"
# can build ctypes, by finding the sysmtem libffi.so (rather than failing to
# find "libffi_d.so"), and builds the module as _ctypes_d.so
#   
#   * Similarly, update build_ext:get_libraries handling of Py_ENABLE_SHARED by
# appending "_d" to the python library's name for the debug configuration
#
#   * We modify Modules/makesetup to add the "_d" to the generated Makefile
# rules for the various Modules/*.so targets
#
# This may introduce issues when building an extension that links directly
# against another extension (e.g. users of NumPy?), but seems more robust when
# searching for external libraries
#
#   * We don't change Lib/distutils/command/build.py: build.build_purelib to
# embed plat_specifier, leaving it as is, as pure python builds should be
# unaffected by these differences (we'll be sharing the .py and .pyc files)
#
#   * We introduce DEBUG_SUFFIX as well as DEBUG_EXT:
#     - DEBUG_EXT is used by ELF files (names and SONAMEs); it will be "_d" for
# a debug build
#     - DEBUG_SUFFIX is used by filesystem paths; it will be "-debug" for a
# debug build
#
#   Both will be empty in an optimized build.  "_d" contains characters that
# are valid ELF metadata, but this leads to various ugly filesystem paths (such
# as the include path), and DEBUG_SUFFIX allows these paths to have more natural
# names.  Changing this requires changes elsewhere in the distutils code.
#
#   * We add DEBUG_SUFFIX to PYTHON in the Makefile, so that the two
# configurations build parallel-installable binaries with different names
# ("python-debug" vs "python").
#
#   * Similarly, we add DEBUG_SUFFIX within python-config and
#  python$(VERSION)-config, so that the two configuration get different paths
#  for these.
#
#  * Patch runtests.sh to support supplying a value for PYTHON, so that we can
# run the tests against each of the builds

You are right; but the data could not be appended to sysconfig.py, because this file is shared by the different builds.
A new built-in C module is probably necessary.

@dmalcolm: I suspect you can reduce your diff for Python 3.2 now that PEP 3149 has landed.

On 09.09.2010 21:51, Dave Malcolm wrote:
> #   * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so,
> # but this can potentially find a module built against the wrong DSO ABI.  We
> # instead search for just module_d.so in a debug build

right, by design/complaint: people did complain that locally built debug 
extensions wouldn't be found in this case.  This is now obsolete for 3.2, 
because the debug extension has always added the "d" modifier.

@Amaury: yes, that makes perfect sense.  With PEP 3149, a debug (or any differently built) interpreter will pick up only the _sysconfig.blah.so that's appropriate for it, so baking it into there, with public API exposure through sysconfig seems like the right thing to do.

In reply to Barry A. Warsaw:
>@dmalcolm: I suspect you can reduce your diff for Python 3.2 now that PEP 3149 has landed.
Yeah, the patch I linked to is against the 3.2a1 tarball; I hoped to regenerate it for 3.2a2 but have been swamped this week :(

If the feature to be provided is "multiple Python installations with the same prefix", then I suggest to generalize this beyond the debug build. One approach would be:

1. add a --bin-suffix configure option (Debian: set this to _d when creating the debug build)
2. expose this as sys.bin_suffix
3. use this in many more places than this patch does, e.g. for python3-config (python3_d-config?), python3.pc, idle3, pydoc3, 2to3, and anything else that either is a binary or refers to one.

Agree with Barry: sysconfig is the new hotness for configuration info.

On #4359, I said that future improvements in sysconfig (http://bitbucket.org/tarek/distutils2/src/tip/docs/design/wiki.rst) will address part of the issue: It will use a configparser file to store installation directories.  There is nothing in the document about other build-time variables at the moment, but I’m +1 on any system that would remove the brittle Makefile parsing in sysconfig.  The sysconfig.cfg does not look like the best place to put this info, since it’s not editable configuration but more state.  +0 on generating a _sysconfig.c module.

I’ll have to read again the original patch and Martin’s proposal to understand the case of debug in the wider configure information picture.

A clarification: The sysconfig.cfg does not look like the best place to put this info, since [this info] is not editable configuration but more state.

My (I think fairly straightforward) idea is to just compile the useful values in Makefile and config.h into _sysconfig.c and arrange for sysconfig to import that and check it first (fallback to parsing mf and conf.h).

I'll see if I can whip up a patch for that for review and feedback.

Is there any point in creating another extension module?
We already have Modules/getpath.c which receives various configuration values at compile-time, themselves exposed as sys.prefix and friends. The simpler the better, IMO.

sysconfig can choose to expose those sys values in a funkier format if it wishes to do so.

> My (I think fairly straightforward) idea is to just compile the
> useful values in Makefile and config.h into _sysconfig.c and arrange
> for sysconfig to import that and check it first (fallback to parsing
> mf and conf.h).

You seem to suggest that by doing so, one can actually dispose of
pyconfig.h afterwards, as all information is in the module.

This is definitely not the case: pyconfig.h continues to be required,
as an include that is literally used when compiling extension modules.

@Martin: yep, I know we still need to install pyconfig.h and Makefile, but we shouldn't need to parse them to get programmatic access to the data they contain.

> @Martin: yep, I know we still need to install pyconfig.h and
> Makefile, but we shouldn't need to parse them to get programmatic
> access to the data they contain.

Hmm. What do you gain by not parsing them?

Le jeudi 16 septembre 2010 à 11:13 +0000, Martin v. Löwis a écrit :
> Martin v. Löwis <martin@v.loewis.de> added the comment:
> 
> > @Martin: yep, I know we still need to install pyconfig.h and
> > Makefile, but we shouldn't need to parse them to get programmatic
> > access to the data they contain.
> 
> Hmm. What do you gain by not parsing them?

Not having some complicate and brittle code to parse Makefiles would
certainly be a win, IMO.
(the canonical pyconfig.h is arguably much simpler)

> Not having some complicate and brittle code to parse Makefiles would
> certainly be a win, IMO.

Not sure how the patch would look like, but I would expect that any
patch to build a module to include Makefile settings makes it *less*
robust: every time you add or remove a Makefile variable, you have to
remember to adjust the module as well. Currently, you can access 
arbitrary Makefile variables.

I've only been on the periphery of the distutils/makefile discussion, but I thought the goal was to *autogenerate* a module containing the relevant information at python build time, instead of (as now) parsing the makefile at run time.  Whether or not this autogeneration involves or does not involve parsing the makefile at python build time is an orthogonal issue.

I was thinking along the lines that RDM outlined, IOW that _sysconfig.c or equivalent would be autogenerated at build time.  But I think there are really two issues here:

1) Avoiding parsing of pyconfig.h and Makefile to get variable values for the sysconfig module.  This is fairly easy, but also the less important one I think.  If we still want to do this, let's move it to a separate bug.

2) Allowing for pyconfig.h and Makefile for different build options to coexist.  You need this so that extensions will be built against the correct build parameters.  This is much more important and I think the issue that Doko really wants to solve.  The _d hack now used is only a partial solution because it doesn't take into account other build options.

One possibility would be to use $SO, $SOABI, or just the flags in the latter (if you don't want the 'cpython-32' redundancy in the name) in the filename to pyconfig.h and Makefile.  e.g.

>>> import sysconfig
>>> sysconfig.get_config_h_filename()
'/usr/local/include/python3.2/config-32m/pyconfig.h'
>>> sysconfig.get_makefile_filename() # see bug 9877
'/usr/local/lib/python3.2/config-32m/Makefile'

The related issue is the naming of the binaries to include the build flag.   Right now on Ubuntu we have python3-dbg for example.  Maybe instead we want python3-<SOABI-FLAGS> e.g. python3-m, python3-dm etc.  We can of course always use symlinks to get the old, or default names.

See issue 9878 for the "don't parse" bug.

Antoine:
> Is there any point in creating another extension module?
> We already have Modules/getpath.c which receives various configuration
> values at compile-time, themselves exposed as sys.prefix and friends.

I have no opinion on generating a new module vs. adding info to getpath.c, but I wouldn’t like to further bloat the sys module.

I have a some code available for review here:

http://codereview.appspot.com/2340041/

I've actually gone down a farther path than I originally intended, but I do kind of like where this is heading.  It will allow me to install different Python builds of the same version without collision, at least when you use 'make altinstall'.  There are different versions even of python-config (with an added --abiflag option) so you can tell exactly what you're getting.

Note that 'make install' still adds all the symlinks so that you get a 'python3' and 'python3-config' as defaults.

Still broken are importing extensions.  E.g. if I do the following:

./configure --prefix=/tmp/python && make altinstall
./configure --prefix=/tmp/python --with-wide-unicode --with-pydebug && make altinstall
(cd some simple module with an extension)
/tmp/python/bin/python3.2dmu setup.py install
/tmp/python/bin/python3.2m
>>> import myextension

it tries to import the 3.2dmu version and fails.  I'm still investigating that, but in the meantime, please go to codereview and let me know what you think so far.

dmalcom and doko remind me that we need to handle the .so when --enable-shared is included

<doko> so -Wl,-h <soname> is what counts  [17:48]
<doko> barry: and it would be good to have this soname available in this new module too  [17:49]

I see nothing obviously wrong in the patch.

Éric: which patch: file18807, or http://codereview.appspot.com/2340041/?

Sorry, I was replying to this:
“I'm still investigating that, but in the meantime, please go to codereview and let me know what you think so far.”

I meant that for the distutils part, I’ve seen nothing bad.  I have no idea about the import bug.

bzr branch lp:~barry/python/issue9807

https://code.edge.launchpad.net/~barry/python/issue9807

Let's try splitting this issue into separate patches.  This first one just exposes the ABI flags in the sys module and in the python-config script.  It doesn't change any installation behavior.

http://codereview.appspot.com/2478042/

Summarizing IRC discussion:

Tested on Fedora 13 x86_64 with:
  --enable-shared --with-wide-unicode
and with confdir != srcdir with:
  ../configure --enable-shared --with-wide-unicode --with-pydebug

Mostly working but, test_distutils fails:
  test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase) ... /usr/bin/ld: cannot find -lpython3.2
  collect2: ld returned 1 exit status

Each build makes a "libpython3.2.a"

Interestingly enough, the distutils failure that dmalcolm found was present in the trunk even before my patch.  If you build Python with --enable-shared, that distutils test fails because of the default path used for the -L option.  I fixed that in my patch, but it was unrelated to the changes I made to expose sys.abiflags.

I wonder if we should try to get a buildbot up that uses --enable-shared?

> I fixed that in my patch, but it was unrelated to the changes I made to expose sys.abiflags.

Would you mind committing that part independently of the rest?

> I wonder if we should try to get a buildbot up that uses --enable-shared?

If the option has significant use, why not.

(barry)
> I wonder if we should try to get a buildbot up that uses --enable-
> shared?
(éric)
> If the option has significant use, why not.

Well, it's all the more significant that most Linux distros use shared libraries, so they will use that option.
I'll look into changing a buildbot to use --enable-shared, then.

I've done so on one of the stable buildbots. Let's see how it behaves:

http://www.python.org/dev/buildbot/builders/x86%20Ubuntu%20Shared%203.x/

Barry: I had misunderstood your message, so disregard my request for commit (since the fix you mention *is* committed).

two fixes, the configure.in differentiates the name for the static library, as mentioned in msg118832.

the python-config.in fix prints the library name with the abiflags.

Index: configure.in
===================================================================
--- configure.in	(Revision 85644)
+++ configure.in	(Arbeitskopie)
@@ -585,7 +585,7 @@
 AC_MSG_CHECKING(LIBRARY)
 if test -z "$LIBRARY"
 then
-	LIBRARY='libpython$(VERSION).a'
+	LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
 fi
 AC_MSG_RESULT($LIBRARY)
 
Index: Misc/python-config.in
===================================================================
--- Misc/python-config.in	(Revision 85644)
+++ Misc/python-config.in	(Arbeitskopie)
@@ -45,7 +45,7 @@
 
     elif opt in ('--libs', '--ldflags'):
         libs = getvar('LIBS').split() + getvar('SYSLIBS').split()
-        libs.append('-lpython'+pyver)
+        libs.append('-lpython'+pyver+sys.abiflags)
         # add the prefix/lib/pythonX.Y/config dir, but only if there is no
         # shared library in prefix/lib/.
         if opt == '--ldflags':

Index: Misc/python.pc.in
===================================================================
--- Misc/python.pc.in	(Revision 85644)
+++ Misc/python.pc.in	(Arbeitskopie)
@@ -8,6 +8,6 @@
 Requires: 
 Version: @VERSION@
 Libs.private: @LIBS@
-Libs: -L${libdir} -lpython@VERSION@
+Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
 Cflags: -I${includedir}/python@VERSION@

the name of the library should not differ for the static and the shared library.

Here now is the second half of the patch, which installs the binary, library, and includes into names with abiflags.  I understand this is more controversial, but it will help continue the discussion.

- As configure script add new "substitute variable" LDVERSION" why do not use LDVERSION in Makefile instead $(VERSION)$(ABIFLAGS). Note first to replace in configure script LDVERSION="$(VERSION)" to LDVERSION="$VERSION" !

- May be $(LN) -s is not portable. What about to add macro AC_PROG_LN_S in configure script and to use $(LN_S) in makefile ?

the python.pc installation name should be changed too, and a symlink added.

Lib/distutils/command/install.py () needs updates in INSTALL_SCHEMES/headers.

FTR, the ld bug with --enable-shared is tracked in #10126.

@rpetrov: reusing $LDVERSION does make sense, but then I think the variable is misnamed.  I don't like $ABIVERSION either but maybe $BUILDVERSION?

Also I think we need to let the substitution for $LDVERSION happen in the Makefile rather than in the configure script.  I tried it the way you suggested and IIRC, it had some problems, but I don't remember the details.

@doko: Agreed.  I'll have an updated patch that fixes python.pc.  I'll look at INSTALL_SCHEME/headers too.

Attached is the remaining patch against py3k.  This installs the symlinks, fixes the distutils 'install_headers' location, and uses abiflags in the python3.pc path.

I think this branch is done, pending approval and commit.

this adds the modifier to the <prefix>/python3.2<abi>/config directory.

Now you end up with both
  
  <prefix>/python3.2<abi>/config

and

    <prefix>/python3.2/

for anything else. This is not what Debian, Fedora and Ubuntu are currently doing. Is this really wanted? I'd rather like to see this as

    <prefix>/python3.2/config-<abi>

so that everything is again in one prefix.


$ ls -l <prefix>/lib/pkgconfig/
total 4
-rw-r--r-- 1 doko doko 282 Nov 14 02:44 python-3.2.pc
lrwxrwxrwx 1 doko doko  13 Nov 14 02:44 python-3.2mu.pc -> python-3.2.pc

python-3.2mu.pc should be the file, python-3.2.pc the symlink.

the change to python.pc should make the abi change to includedir, not Cflags.

Could you also fix issue #10262, which is related to this issue?

Matthias,

you mean $prefix/lib/python3.2 and $prefix/lib/python3.2$abiflags, right?  The latter has just a config directory, and you'd rather see that become $prefix/lib/python3.2/config-$abiflags, right?

Matthias,

When you say the python.pc change should change includedir instead of Cflags, that seems weird.  includedir does not currently include the 'pythonX.Y' subdirectory so there's no place to put the abiflags.  Or are you suggesting that we move the pythonX.Y<abiflags> subdir to includedir and remove it from Cflags?

On 15.11.2010 23:57, Barry A. Warsaw wrote:
> you mean $prefix/lib/python3.2 and $prefix/lib/python3.2$abiflags, right?  The latter has just a config directory, and you'd rather see that become $prefix/lib/python3.2/config-$abiflags, right?

yes.

On 16.11.2010 00:01, Barry A. Warsaw wrote:
> When you say the python.pc change should change includedir instead of Cflags, that seems weird.  includedir does not currently include the 'pythonX.Y' subdirectory so there's no place to put the abiflags.  Or are you suggesting that we move the pythonX.Y<abiflags>  subdir to includedir and remove it from Cflags?

sorry, I'm wrong.

Here's an updated patch which address's Matthias's last concerns.

looks good. checked with a plain and a debug build and installation.

r86731