classification
Title: Two copies of python-config
Type: behavior Stage: resolved
Components: Demos and Tools Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, doko, larry, lukasz.langa, python-dev, r.david.murray, ronaldoussoren
Priority: Keywords:

Created on 2013-06-18 15:04 by ronaldoussoren, last changed 2017-09-26 01:56 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
issue-18257.txt ronaldoussoren, 2013-07-07 11:35 review
Messages (23)
msg191416 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-18 15:04
Changeset c0370730b364 introduced a shell-script implementation of python-config (see issue #16235). The older python implementation is still present in $(srcdir)/Misc and generated by Makefile.pre.in.
msg191417 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-18 15:21
Also, the shell-script version of python-config uses the -f option of readlink and that option is not supported on OSX 10.8 (or older versions, I haven't looked at the 10.9 beta yet)

Furthermore the entire readlink command is not present in HP-UX (and possibly other "enterpricy" unix flavors)
msg192524 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-07 08:52
This is IMHO a release blocker: the the shell-script version of python-config doesn't work on a major platform (OSX), and (older) commercial unix systems.

The easiest workaround is to remove the shell script and keep using the python script.
msg192533 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-07-07 10:41
How about if the shell script detected that it was running on OS X and exited with an error instructing the user to use the Python script instead?

I don't agree that this is a release blocker.  Most people on OS X use the prebuilt binaries.
msg192534 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 10:46
Proposing to remove the shell script as the first step looks wrong.  Do you know about a substitute for readlink on an "enterpricy" unix flavor like MacOSX?  Using ls -l and interpreting the last argument of the output comes to mind.
msg192538 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-07 11:13
Why use a shell script in the first place? The shell script doesn't do everything the python script does (an example of this is that on OSX distutils and sysconfig can tweak the CFLAGS and LDFLAGS as needed based on the curent OSX version and the installed compiler, which is needed to be able to build extensions out of the box on all supported OSX releases) and doesn't work everywhere because it uses non-standard shell commands.

I'll provide a patch that ensures that the python version gets used on OSX, that will reduce the number of issues we get about this. 

Larry: python-config is shipped in the binary installers, and as it currently is doesn't work.  That's a problem because some software uses python-config to get the compiler options needed to use this python (for embedding, building extensions using a makefile, ...).  

I don't use python-config myself, but do try to keep the OSX version as closely as possible a normal unix install (another example of this: in a framework install, such as the binary installers, we ship a "libpython.a" at the expected location that is a symlink to the real *shared* library. That symlink isn't need for Python itself, but was added because some popular Linux software (can't remember which package) assumed that the library would be there).

Anyway, I'll push a patch for this when I'm home.  I'm slowly closing down my involvement with the EP sprints and will then head to the airport.
msg192544 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 11:30
> Why use a shell script in the first place?

you can't run python for a cross build, unless you "tweak" the interpreter.  you can do that with a shell script only.  As suggested in the original issue/patch, I did propose to remove the python implementation, so a fix for the shell script seems to be the better approach.  Tweaking the CFLAGS in the shell script should be possible too.  Maybe you could describe what exactly needs tweaking, as I cannot find this in the python implementation either.
msg192545 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 11:32
> Furthermore the entire readlink command is not present in HP-UX

The use of readlink is guarded, the use of readlink -f apparently not.
msg192546 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-07 11:35
This should do the trick, the shell script version is replaced by the python version when building on OSX.

And yes, the use of readlink is guarded but that doesn't help because you then use a command-line flag that isn't supported on OSX:

$ readlink -h .
readlink: illegal option -- h
usage: readlink [-n] [file ...]
msg192549 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 11:40
the proposed patch won't work once the python implementation is removed.  Is -f supported on MacOSX? There seems to be a typo on your side trying -h, which isn't supported on Linux either.
msg192550 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-07 11:47
Sorry about the confusion, "-f" isn't supported either that's why I noticed there is a problem.

$ readlink -f .
readlink: illegal option -- f
usage: readlink [-n] [file ...]

Again, why is does does have to be a shell script anyway? 

I really don't like the propect of having to reimplement the logic in _osx_support in shell code...  I've written loads of large, portable shell scripts in the past (when portable meant supporting a dozen or so different unix flavors), and that's not really an experience worth repeating :-/
msg192551 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 13:09
> Again, why is does does have to be a shell script anyway? 

please see above. I explained it.
msg192553 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-07-07 13:12
and see issue16235 (as mentioned in NEWS) for the complete discussion.
msg192560 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-07 14:29
Matthias: sorry, I completely missed the message where you explained why the script is now a shell script. 

The customization is performed by _osx_xsupport.customize_config_vars.

What this is used for:

* We ship binaries build with a specific compiler configuration
  (IIRC Xcode 3 on OSX 10.5 for the 32-bit installers)

* That configuration does not work on all supported platforms:

  - OSX 10.3 (which is more less supported on a "we'll try to fix
    if you scream basis") does not support "-arch" flags or
    "-isysroot" at all.
 
  - On OSX 10.4 you must use an SDK to build fat binaries (such as 
    for the 32-bit binary installers).

  - Xcode 4, which must be used on more recent systems (at least 10.7
    and 10.7, possibly 10.6 as well), is different. A particular worry
    is that the "gcc" command for Xcode 4 is a gcc frontend for the 
    LLVM system and that miscompiles parts of Python 3.3. Therefore we
    use clang instead of gcc when detecting that compiler. In future
    versions of Xcode there will likely not be a gcc compiler at all.

  - We cannot use the same SDK on all platforms, because Xcode ships
    with a limited number of SDK (basicly just the SDK for the current
    and previous releases). We therefore automaticly select the most
    appropriate SDK at runtime.

    (Different SDKs isn't as problematic as it sounds, you can build
    with the OSX 10.8 SDK and deploy to OSX 10.4; except for PPC you
    must use Xcode 3 for PPC support because Xcode 4 does not contain
    a compiler that does that).

  - OSX 10.9 (currently in beta) also contains changes that affect 
    compiler support. I won't comment on those changes right now because
    I haven't fully investigated them yet, and because 10.9 is 
    in a closed SDK.

I may have missed some details because I haven't looked at the code while
writing this, but this should at least explain why I was upset with a change to a shell script.  

The customization can be written as shell scripts, but it is more code to maintain in an inelegant scripting system.
msg193980 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-07-31 07:08
Is there any resolution for this likely to happen soon?  I'm hoping to cut Python 3.4a1 in about two days, so one of three things is gonna happen:

1) This gets fixed.
2) This gets lowered from "release blocker".
3) The release slips.
msg193990 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-31 09:10
I won't be able to work on this before 3.4a1 is released, and that shouldn't be a problem.

I do think this should be fixed before the first beta though.

(That said, I'd rather not work on this at all, I didn't introduce this problem)
msg194034 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-08-01 08:22
Okay, downgrading to "deferred blocker".
msg194073 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-01 13:32
New changeset 7b165c7ab7ef by doko in branch 'default':
- Issue #18257: Fix readlink usage in python-config.  Install the python
http://hg.python.org/cpython/rev/7b165c7ab7ef
msg194074 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-08-01 13:35
checked in a fix for the readlink issue, and Ronald's proposed solution for Darwin.  And documented for now, that we do have two versions of this script.  This works as long as you don't cross-build for Darwin or on Darwin.
msg194097 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-08-01 18:08
Thanks, Matthias!  Now I can cut 3.4a1 on time.  :)
msg213713 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-03-16 05:15
Can I mark this closed?  I'm tagging 3.4.0 final soon.
msg302983 - (view) Author: Ɓukasz Langa (lukasz.langa) * (Python committer) Date: 2017-09-25 22:55
Downgrading the priority, it's not a deferred blocker for sure. I think we could probably just close it?
msg302994 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-09-26 01:56
Since it hasn't been an issue for a few releases, I say we close it.  If there is some problem remaining, it probably deserves its own tracker issue anyway.
History
Date User Action Args
2017-09-26 01:56:43r.david.murraysetstatus: open -> closed

nosy: + r.david.murray
messages: + msg302994

resolution: fixed
stage: needs patch -> resolved
2017-09-25 22:55:15lukasz.langasetpriority: deferred blocker ->
nosy: + lukasz.langa
messages: + msg302983

2014-03-16 05:15:05larrysetmessages: + msg213713
2013-08-01 18:08:55larrysetmessages: + msg194097
2013-08-01 13:35:25dokosetmessages: + msg194074
2013-08-01 13:32:59python-devsetnosy: + python-dev
messages: + msg194073
2013-08-01 08:22:13larrysetpriority: release blocker -> deferred blocker

messages: + msg194034
2013-07-31 09:10:35ronaldoussorensetmessages: + msg193990
2013-07-31 07:08:07larrysetmessages: + msg193980
2013-07-07 14:29:33ronaldoussorensetmessages: + msg192560
2013-07-07 13:12:59dokosetmessages: + msg192553
2013-07-07 13:09:15dokosetmessages: + msg192551
2013-07-07 11:47:11ronaldoussorensetmessages: + msg192550
2013-07-07 11:40:22dokosetmessages: + msg192549
2013-07-07 11:35:51ronaldoussorensetfiles: + issue-18257.txt

messages: + msg192546
2013-07-07 11:32:56dokosetmessages: + msg192545
2013-07-07 11:30:15dokosetmessages: + msg192544
2013-07-07 11:24:49Arfreversetnosy: + Arfrever
2013-07-07 11:13:32ronaldoussorensetmessages: + msg192538
2013-07-07 10:46:13dokosetmessages: + msg192534
2013-07-07 10:41:33larrysetmessages: + msg192533
2013-07-07 08:52:41ronaldoussorensetpriority: normal -> release blocker
nosy: + larry
messages: + msg192524

2013-06-18 15:21:53ronaldoussorensetmessages: + msg191417
components: + Demos and Tools
2013-06-18 15:04:32ronaldoussorencreate