classification
Title: Add python-config.sh for use during cross compilation.
Type: enhancement Stage: resolved
Components: Cross-Build Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Ray.Donnelly, doko, eric.araujo, georg.brandl, loewis, python-dev, trent
Priority: normal Keywords: patch

Created on 2012-10-14 21:55 by Ray.Donnelly, last changed 2013-01-26 10:46 by Ray.Donnelly. This issue is now closed.

Files
File name Uploaded Description Edit
0000-add-python-config-sh.patch Ray.Donnelly, 2012-10-14 21:55 First version of patch to add python-config.sh review
0000-add-python-config-sh.patch Ray.Donnelly, 2012-11-04 11:56
0000-add-python-config-sh.patch Ray.Donnelly, 2012-11-08 13:35
0000-add-python-config-sh.patch Ray.Donnelly, 2012-11-30 14:44 review
0001-add-python-config-sh.patch doko, 2013-01-25 18:03 review
Messages (17)
msg172925 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-10-14 21:55
Creates python-config.sh which aims to behave exactly the same as python-config except it doesn't depend on a Python interpreter on the build machine (only depends on a posixy shell, tested with bash, dash, MSYS-bash).

I use this for cross compiling gdb with Python pretty printer support.
msg173710 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-10-24 21:38
Earlier, we had the shell script, now we have the python implementation, which doesn't work for cross builds. On the other hand, I don't like to have two implementations which need to be kept in sync.

So my proposal would be to discard the python implementation in favour of the shell implementation, provided that it works on all platforms (don't know about windows).

just did see that the patch doesn't support --configdir for 3.3.
msg174788 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-04 11:56
I agree in principle about not having to maintain two implementations but I would worry about breaking external packages that depend on the python implementation.

In the meantime, please find an updated version of this patch that adds support for --configdir
msg174790 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-04 12:21
I also checked the Windows releases and python-config isn't included. However, I don't think there's a good reason for that. python-config would in theory work fine if people wanted to link their programs to Python on Windows, as would python-config.sh given a posix-y shell that e.g. Cygwin or MSYS provide.
msg175092 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-11-07 12:32
see issue #1161914 for the original script.

> 2) Since we are Pythoneers, why write this script as a
> shell-script instead of a Python script? (sh may not even be
> available on Windows).

python-config is usally not used by python module builds, but third party build systems. When cross-building these packages you usually are unable to run the python interpreter for the target.

Georg, Martin, do you have any opinion if we should replace the python code with shell code (if it works on all targets), or if it should be added as a separate script?

Ray, some issues:

 - is `local' available in all shells? just avoid it.

 - is there a need for the built vs. installed prefix?
   this is logic not found in the python implementation.
   what is this supposed to do?

 - LDLIBRARY and LDSHARED are expressed in terms of Makefile
   macros, leading to syntax errors.

 - --includes doesn't include the plat specific include dir

 - abi flags are missing from the include and lib dirs.

Please recheck, and compare the output of the python implementation with the one of the shell implementation.
msg175095 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-07 12:52
> is there a need for the built vs. installed prefix?
>    this is logic not found in the python implementation.
>    what is this supposed to do?

You are right, it is not found in the original python implementation, but I feel that it's useful. Without it, you couldn't give someone a -devel style precompiled Python archive that they can extract where-ever they want and embed in other software without requiring them to install it to the same prefix used during the build process.

> is `local' available in all shells? just avoid it.

I can't be sure about all shells, but local is *very* well supported (bash, dash, csh); I'm happy to remove it though as it's not necessary.

> LDLIBRARY and LDSHARED are expressed in terms of Makefile
>   macros, leading to syntax errors.

This shouldn't be the case, see:

# Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+	sed -e "s,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g" < Misc/python-config.sh >python-config.sh

> - --includes doesn't include the plat specific include dir

You are right, I will correct this oversight.

> - abi flags are missing from the include and lib dirs.

ABIFLAGS propagates through other variables, namely LDLIBRARY and LDVERSION, but again, there are some issues for me to fix with this.

I will write a new part on my build script that compares the output from the shell script with that from the python script and post a new patch when they agree 100% on all my test platforms.
msg175101 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-11-07 13:20
Am 07.11.2012 13:52, schrieb Ray Donnelly:
>
> Ray Donnelly added the comment:
>
>> is there a need for the built vs. installed prefix?
>>     this is logic not found in the python implementation.
>>     what is this supposed to do?
>
> You are right, it is not found in the original python implementation, but I feel that it's useful. Without it, you couldn't give someone a -devel style precompiled Python archive that they can extract where-ever they want and embed in other software without requiring them to install it to the same prefix used during the build process.

hmm, but python.pc, _sysconfigdata.py and Makefile still have this information. 
How would you test the output and compare it with the python implementation?

>> is `local' available in all shells? just avoid it.
>
> I can't be sure about all shells, but local is *very* well supported (bash, dash, csh); I'm happy to remove it though as it's not necessary.
>
>> LDLIBRARY and LDSHARED are expressed in terms of Makefile
>>    macros, leading to syntax errors.
>
> This shouldn't be the case, see:
>
> # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> +	sed -e "s,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g" < Misc/python-config.sh >python-config.sh

ahh, ok, only looked at the file in the build location.

>> - --includes doesn't include the plat specific include dir
>
> You are right, I will correct this oversight.

maybe should only be included, if it's different.
msg175110 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-07 17:16
> hmm, but python.pc, _sysconfigdata.py and Makefile still have this information.

In my experience, as long as PYTHONHOME is set correctly before the Python .so, .dll or .dylib is loaded then everything works correctly.

From what I've seen, .pc files are also useless for the purposes of a relocatable -devel package as there's no capability for relative directories in pkg-config.

> maybe should only be included, if it's different.

Ok.
msg175154 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-08 13:35
I've addressed everything (local, ABIFLAGS) except the platform includes.

AFAICT, platform includes only differs from includes during the Python build phase. This script can only be run post-install.

I also added --extension-suffix, tidied it up and allowed it to return multiple values in a single invocation, as per the Python script.
msg176676 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-11-29 23:51
looks fine, just one more minor issue: please use @LIBPL@ directly.

there's no feedback from others, so I think this should go in as the extra script first, and not replace the original one immediately.
msg176677 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-11-30 00:03
oh, the equivalent for

        flags = ['-I' + sysconfig.get_path('include'),
                 '-I' + sysconfig.get_path('platinclude')]

is still missing. I know that both scripts versions are only run post-install, but let's have the same output.

background: I'm (mis)using this for a Debian/Ubuntu multiarch installation.
msg176688 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2012-11-30 14:44
New patch attached:

LIBPL changed to a more direct AC_SUBST and PLATINCDIR used for --includes and --cflags.
msg180602 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-01-25 18:03
so here is what I intend to commit.

 - --help now does exit with 0 (same as the python script)
 - removed the abi safety check (always compares the same
   two strings)
 - build the script for the build target, so that it can be
   checked before installing it.
 - install the old script in $(LIBPL). Maybe we can remove
   this later, or add this functionality into sysconfig.
msg180622 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2013-01-25 20:59
Great. My only query is whether the changes to sed will work on non-GNU sed.

It used to be like:
sed < in >out

you've now got:
sed in >out

I'm not saying it won't work, I'm not in a position to check, but I think it's safer to keep it as it was.
msg180655 - (view) Author: Roundup Robot (python-dev) Date: 2013-01-26 10:39
New changeset c0370730b364 by doko in branch 'default':
- Issue #16235: Implement python-config as a shell script.
http://hg.python.org/cpython/rev/c0370730b364
msg180656 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-01-26 10:45
now committed, using stdin for sed.
msg180657 - (view) Author: Ray Donnelly (Ray.Donnelly) * Date: 2013-01-26 10:46
Thank you Matthias!
History
Date User Action Args
2013-01-26 10:46:34Ray.Donnellysetmessages: + msg180657
2013-01-26 10:45:07dokosetstatus: open -> closed
resolution: fixed
messages: + msg180656

stage: patch review -> resolved
2013-01-26 10:39:48python-devsetnosy: + python-dev
messages: + msg180655
2013-01-25 20:59:09Ray.Donnellysetmessages: + msg180622
2013-01-25 18:03:31dokosetfiles: + 0001-add-python-config-sh.patch

messages: + msg180602
2012-11-30 14:44:32Ray.Donnellysetfiles: + 0000-add-python-config-sh.patch

messages: + msg176688
2012-11-30 00:03:20dokosetmessages: + msg176677
2012-11-29 23:51:07dokosetmessages: + msg176676
2012-11-08 13:35:27Ray.Donnellysetfiles: + 0000-add-python-config-sh.patch

messages: + msg175154
2012-11-07 17:16:25Ray.Donnellysetmessages: + msg175110
2012-11-07 13:20:16dokosetmessages: + msg175101
2012-11-07 12:52:10Ray.Donnellysetmessages: + msg175095
2012-11-07 12:32:47dokosetnosy: + loewis, georg.brandl
2012-11-07 12:32:07dokosetmessages: + msg175092
2012-11-04 12:21:17Ray.Donnellysetmessages: + msg174790
2012-11-04 11:56:47Ray.Donnellysetfiles: + 0000-add-python-config-sh.patch

messages: + msg174788
2012-10-24 21:38:24dokosetmessages: + msg173710
2012-10-19 16:43:21eric.araujosetnosy: + eric.araujo
2012-10-18 11:53:36ezio.melottisetnosy: + doko

stage: patch review
2012-10-16 03:40:14trentsetnosy: + trent
2012-10-14 21:55:14Ray.Donnellycreate