New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
deriving configuration information for different builds with the same prefix #54016
Comments
currently, much of the configuration information is fetched by opening the config.h and the Makefile for the build. The locations of these files are derived from the prefix given at configure time. If you want to have two distinct builds with the same prefix, but with different configuration options, this kind of configuration information is unsufficient. The current use case is a normal build and a debug build, as used for the Fedora and Debian/Ubuntu python packages. Fedora and Debian/Ubuntu carry a patch, adding a sys.pydebug attribute and patching "some" things to get the correct information for these builds using the same prefix. This kind of information should not be derived by following some paths names, but be contained in the python binary itself, without needing to read the Makefile and the config.h files. Attached is a patch currently applied in Debian/Ubuntu (and some variant for Fedora) which introduces a sys.pydebug attribute and derives the correct locations for the config.h and Makefile files. It's not meant as a patch ready to apply, but to get an idea what is needed to have two different python installations with the same prefix. |
IMO there should not be any need to fetch information from config.h or the Makefile. |
+1 |
Amaury, you mention a sys.build_config dictionary, but I think it should actually be baked into the sysconfig module, possibly as a _sysconfig extension module. sysconfig is the new goodness for getting at this, and I don't think it ought to have to go through a sys dictionary to get the information. I'd be willing to work on this, if we can get some consensus. |
For reference, the patch that I'm currently applying to Fedora's build of Python-3.2a1 can be seen at: It appears to be very similar to Matthias' patch (it was originally based on an earlier version of Debian's python 2 patch, which I fixed up against Fedora's python 2, changed some aspects I wasn't happy with, then ported to python 3.1, then fixed up to 3.2a1 IIRC) For further reference, Fedora's python3.spec has these comments that I wrote on the patch: # Based on Debian's patch for the same, # * Debian's patch to dynload_shlib.c looks for module_d.so, then module.so, # * We remove this change from configure.in's build of the Makefile: # * We change Lib/distutils/command/build_ext.py:build_ext.get_ext_filename # * We modify Modules/makesetup to add the "_d" to the generated Makefile # This may introduce issues when building an extension that links directly # * We don't change Lib/distutils/command/build.py: build.build_purelib to # * We introduce DEBUG_SUFFIX as well as DEBUG_EXT: # Both will be empty in an optimized build. "_d" contains characters that # * We add DEBUG_SUFFIX to PYTHON in the Makefile, so that the two # * Similarly, we add DEBUG_SUFFIX within python-config and # * Patch runtests.sh to support supplying a value for PYTHON, so that we can |
You are right; but the data could not be appended to sysconfig.py, because this file is shared by the different builds. |
On 09.09.2010 21:51, Dave Malcolm wrote:
right, by design/complaint: people did complain that locally built debug |
If the feature to be provided is "multiple Python installations with the same prefix", then I suggest to generalize this beyond the debug build. One approach would be:
|
Agree with Barry: sysconfig is the new hotness for configuration info. On bpo-4359, I said that future improvements in sysconfig (http://bitbucket.org/tarek/distutils2/src/tip/docs/design/wiki.rst) will address part of the issue: It will use a configparser file to store installation directories. There is nothing in the document about other build-time variables at the moment, but I’m +1 on any system that would remove the brittle Makefile parsing in sysconfig. The sysconfig.cfg does not look like the best place to put this info, since it’s not editable configuration but more state. +0 on generating a _sysconfig.c module. I’ll have to read again the original patch and Martin’s proposal to understand the case of debug in the wider configure information picture. |
A clarification: The sysconfig.cfg does not look like the best place to put this info, since [this info] is not editable configuration but more state. |
My (I think fairly straightforward) idea is to just compile the useful values in Makefile and config.h into _sysconfig.c and arrange for sysconfig to import that and check it first (fallback to parsing mf and conf.h). I'll see if I can whip up a patch for that for review and feedback. |
Is there any point in creating another extension module? sysconfig can choose to expose those sys values in a funkier format if it wishes to do so. |
You seem to suggest that by doing so, one can actually dispose of This is definitely not the case: pyconfig.h continues to be required, |
@martin: yep, I know we still need to install pyconfig.h and Makefile, but we shouldn't need to parse them to get programmatic access to the data they contain. |
Hmm. What do you gain by not parsing them? |
Le jeudi 16 septembre 2010 à 11:13 +0000, Martin v. Löwis a écrit :
Not having some complicate and brittle code to parse Makefiles would |
Not sure how the patch would look like, but I would expect that any |
I've only been on the periphery of the distutils/makefile discussion, but I thought the goal was to *autogenerate* a module containing the relevant information at python build time, instead of (as now) parsing the makefile at run time. Whether or not this autogeneration involves or does not involve parsing the makefile at python build time is an orthogonal issue. |
I was thinking along the lines that RDM outlined, IOW that _sysconfig.c or equivalent would be autogenerated at build time. But I think there are really two issues here:
One possibility would be to use $SO, $SOABI, or just the flags in the latter (if you don't want the 'cpython-32' redundancy in the name) in the filename to pyconfig.h and Makefile. e.g. >>> import sysconfig
>>> sysconfig.get_config_h_filename()
'/usr/local/include/python3.2/config-32m/pyconfig.h'
>>> sysconfig.get_makefile_filename() # see bug 9877
'/usr/local/lib/python3.2/config-32m/Makefile' The related issue is the naming of the binaries to include the build flag. Right now on Ubuntu we have python3-dbg for example. Maybe instead we want python3-<SOABI-FLAGS> e.g. python3-m, python3-dm etc. We can of course always use symlinks to get the old, or default names. |
See bpo-9878 for the "don't parse" bug. |
Antoine:
I have no opinion on generating a new module vs. adding info to getpath.c, but I wouldn’t like to further bloat the sys module. |
I have a some code available for review here: http://codereview.appspot.com/2340041/ I've actually gone down a farther path than I originally intended, but I do kind of like where this is heading. It will allow me to install different Python builds of the same version without collision, at least when you use 'make altinstall'. There are different versions even of python-config (with an added --abiflag option) so you can tell exactly what you're getting. Note that 'make install' still adds all the symlinks so that you get a 'python3' and 'python3-config' as defaults. Still broken are importing extensions. E.g. if I do the following: ./configure --prefix=/tmp/python && make altinstall
./configure --prefix=/tmp/python --with-wide-unicode --with-pydebug && make altinstall
(cd some simple module with an extension)
/tmp/python/bin/python3.2dmu setup.py install
/tmp/python/bin/python3.2m
>>> import myextension it tries to import the 3.2dmu version and fails. I'm still investigating that, but in the meantime, please go to codereview and let me know what you think so far. |
dmalcom and doko remind me that we need to handle the .so when --enable-shared is included |
Summarizing IRC discussion: Tested on Fedora 13 x86_64 with: Mostly working but, test_distutils fails: Each build makes a "libpython3.2.a" |
Interestingly enough, the distutils failure that dmalcolm found was present in the trunk even before my patch. If you build Python with --enable-shared, that distutils test fails because of the default path used for the -L option. I fixed that in my patch, but it was unrelated to the changes I made to expose sys.abiflags. I wonder if we should try to get a buildbot up that uses --enable-shared? |
Would you mind committing that part independently of the rest?
If the option has significant use, why not. |
(barry)
Well, it's all the more significant that most Linux distros use shared libraries, so they will use that option. |
I've done so on one of the stable buildbots. Let's see how it behaves: http://www.python.org/dev/buildbot/builders/x86%20Ubuntu%20Shared%203.x/ |
Barry: I had misunderstood your message, so disregard my request for commit (since the fix you mention *is* committed). |
two fixes, the configure.in differentiates the name for the static library, as mentioned in msg118832. the python-config.in fix prints the library name with the abiflags. Index: configure.in --- configure.in (Revision 85644)
+++ configure.in (Arbeitskopie)
@@ -585,7 +585,7 @@
AC_MSG_CHECKING(LIBRARY)
if test -z "$LIBRARY"
then
- LIBRARY='libpython$(VERSION).a'
+ LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
fi
AC_MSG_RESULT($LIBRARY)
Index: Misc/python-config.in
===================================================================
--- Misc/python-config.in (Revision 85644)
+++ Misc/python-config.in (Arbeitskopie)
@@ -45,7 +45,7 @@
elif opt in ('--libs', '--ldflags'):
libs = getvar('LIBS').split() + getvar('SYSLIBS').split()
- libs.append('-lpython'+pyver)
+ libs.append('-lpython'+pyver+sys.abiflags)
# add the prefix/lib/pythonX.Y/config dir, but only if there is no
# shared library in prefix/lib/.
if opt == '--ldflags': |
Index: Misc/python.pc.in --- Misc/python.pc.in (Revision 85644)
+++ Misc/python.pc.in (Arbeitskopie)
@@ -8,6 +8,6 @@
Requires:
Version: @VERSION@
Libs.private: @LIBS@
-Libs: -L${libdir} -lpython@VERSION@
+Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
Cflags: -I${includedir}/python@VERSION@ |
the name of the library should not differ for the static and the shared library. |
Here now is the second half of the patch, which installs the binary, library, and includes into names with abiflags. I understand this is more controversial, but it will help continue the discussion. |
|
the python.pc installation name should be changed too, and a symlink added. |
Lib/distutils/command/install.py () needs updates in INSTALL_SCHEMES/headers. |
FTR, the ld bug with --enable-shared is tracked in bpo-10126. |
@RPetrov: reusing $LDVERSION does make sense, but then I think the variable is misnamed. I don't like $ABIVERSION either but maybe $BUILDVERSION? Also I think we need to let the substitution for $LDVERSION happen in the Makefile rather than in the configure script. I tried it the way you suggested and IIRC, it had some problems, but I don't remember the details. @Doko: Agreed. I'll have an updated patch that fixes python.pc. I'll look at INSTALL_SCHEME/headers too. |
Attached is the remaining patch against py3k. This installs the symlinks, fixes the distutils 'install_headers' location, and uses abiflags in the python3.pc path. I think this branch is done, pending approval and commit. |
this adds the modifier to the <prefix>/python3.2<abi>/config directory. Now you end up with both <prefix>/python3.2<abi>/config and
for anything else. This is not what Debian, Fedora and Ubuntu are currently doing. Is this really wanted? I'd rather like to see this as
so that everything is again in one prefix. $ ls -l <prefix>/lib/pkgconfig/
total 4
-rw-r--r-- 1 doko doko 282 Nov 14 02:44 python-3.2.pc
lrwxrwxrwx 1 doko doko 13 Nov 14 02:44 python-3.2mu.pc -> python-3.2.pc python-3.2mu.pc should be the file, python-3.2.pc the symlink. |
the change to python.pc should make the abi change to includedir, not Cflags. |
Could you also fix issue bpo-10262, which is related to this issue? |
Matthias, you mean $prefix/lib/python3.2 and $prefix/lib/python3.2$abiflags, right? The latter has just a config directory, and you'd rather see that become $prefix/lib/python3.2/config-$abiflags, right? |
Matthias, When you say the python.pc change should change includedir instead of Cflags, that seems weird. includedir does not currently include the 'pythonX.Y' subdirectory so there's no place to put the abiflags. Or are you suggesting that we move the pythonX.Y<abiflags> subdir to includedir and remove it from Cflags? |
On 15.11.2010 23:57, Barry A. Warsaw wrote:
yes. |
On 16.11.2010 00:01, Barry A. Warsaw wrote:
sorry, I'm wrong. |
Here's an updated patch which address's Matthias's last concerns. |
looks good. checked with a plain and a debug build and installation. |
r86731 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: