Title: use different mechanism for pythonw on osx
Type: behavior Stage: committed/rejected
Components: Macintosh Versions: Python 3.2, Python 3.1, Python 2.7, Python 2.6
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: loewis, ned.deily, robince, ronaldoussoren
Priority: normal Keywords: easy

Created on 2009-09-03 14:24 by ronaldoussoren, last changed 2010-01-08 09:08 by ned.deily. This issue is now closed.

File name Uploaded Description Edit
pythonw.c ronaldoussoren, 2009-09-22 10:48
Messages (11)
msg92210 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-09-03 14:24
Note: this is mostly a reminder for myself to clean up the pythonw stub 

The current implementation of pythonw on OSX uses exec to start an 
executable inside the framework, this is needed to be able to use GUI 
functionality from the command-line without resorting to undocumented 
and unsupported system APIs. To deal with selection between 32-bit and 
64-bit the framework contains a number of python executables.

Using "posix_spawnattr_setbinpref_np", "posix_spawnattr_setflags" and 
"posix_spawn" it is possible to do away with the additional executables, 
leaving a simpler situation. 

Nice to have features:
* python(1) on SnowLeopard has a system preference to select between 32-
bit and 64-bit:

   $ defaults read  
    "Prefer-32-Bit" = 1;

(The "versioner" appears to be a private Apple library/tool, 
reimplementing the functionality would be fairly trivial)

* It would be nice to have a command-line switch as well

* It would be nice if the stub executable could be reused by tools like 
virtualenv without recompilation
msg92215 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-09-03 18:43
Can you kindly report how architecture selection works? Is there a
separate binary which execs? Some other magic?

Asking primarily out of curiosity, but if it's a launcher, then
(sym)linking it into a virtualenv might be sufficient.
msg92232 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-09-04 06:10
Let me first explain in more detail why pythonw is needed in the first 
place: MacOSX only allows access to a large number of GUI-related APIs 
from application bundles (the symbols are of course always available, 
but the APIs fail when you are calling them from an executable that is 
not in an application bundle)

This is why we introduced pythonw in framework builds of Python. Pythonw 
is a small executable that does nothing else but execv the real python 
interpreter that is in and .app bundle inside the Python framework 
n). The full path to the execv-ed interpreter is hardcoded into the 
pythonw executable.

To be able to switch between 32-bit and 64-bit versions of Python a 4-
way universal build of Python creates 3 variations of the Python 
executable inside the .app bundle: Python-32, Python-64 and Python (the 
latter being 4-way universal).  There are also 3 variations of the 
pythonw executable (such as pythonw-32) that execv the proper version of 
the interpreter.

Using posix_spawn and the other API's I mention it should be fairly easy 
to create a simpler situation where we need only one copy of Python in 
the .app bundle, with the pythonw wrapper selecting the required 
architecture(s) before executing the real interpreter.

Making pythonw fully reusable by virtualenv requires some more work, and 
requires more support code in virtualenv itself as well. One fairly easy 
way to accomplish easy reusability without requiring a C compiler is to 
use a large buffer for storing the path to the real interpreter, 
virtualenv could then patch the executable. 

Another option is to link the pythonw executable to Python.framework and 
then use dyld APIs to find the path to the Python framework and from 
that the executable.

As an aside, virtualenv copies the shared library in Python.framework 
into virtual environments, this is needed because the framework build 
uses the location of that shared library to determine sys.prefix.
msg92981 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-09-22 10:48
The attached file 'pythonw.c' is a first version of a better pythonw 

This version uses posix_spawn rather than execv to start the real 
interpreter. The main advantage of the new implementation is that 'arch 
-ppc pythonw' works as expected, with the current version of pythonw the 
'arch' command only affects the pythonw executable and not the real 

* I'm not use if the '-X32bit' option is a good idea or not.

  The basic idea of this is to provide an easy way to force python to 
  start in 32bit mode (for use in #! lines) for scripts that need it
  (basicly anything that needs to access Carbon GUI APIs, including most
  current Python GUI libraries)

* The implementation of '-X32bit' is sucky, it contains a copy of the
  getopt format string from Py_Main in Modules/main.c

* The path to the real executable is hardcoded for a standard framework
  install of 2.7 (easily changed to the same mechanism as used by
  the current edition of pythonw)

  What I'd like to do is link pythonw to the framework and use dyld
  introspection to deduce the path to the real executable. That's 
  slightly more complicated, but would provide a clean way to reuse
  the executable in tools like virtualenv without recompiling.
msg93057 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-09-24 07:30
> This version uses posix_spawn rather than execv to start the real 
> interpreter. 

In what way is that better? It creates a new process (IIUC); therefore,
I think that using it is worse than using execv. Anybody killing the
pythonw process would only kill the wrapper.

> The main advantage of the new implementation is that 'arch 
> -ppc pythonw' works as expected, with the current version of pythonw the 
> 'arch' command only affects the pythonw executable and not the real 
> interpreter.

I suppose this would also be possible through execv?
msg94634 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-10-28 17:58
For people searching the bug tracker, I've modified the title of the 
issue to make it clearer that there is a problem here on OS X 10.6 Snow 
Leopard with multiple architecture builds.

As Ronald mentions above, the effect of using the pythonw "launcher" on 
10.6 as it currently stands is to always prefer x86_64 (64-bit) over 
i386 (32-bit) when both are available.  arch -i386 only forces the 
launcher to run as 32-bit; the execv runs the interpreter in 64-bit mode 
(if available).  To get a multi-arch (32/64) interpreter to run in 32-
bit, one workaround is to arch -i386 directly to the interpreter binary 
in the framework app bundle, typically:
But then, presumably, the GUI functionality no longer works.

A more robust workaround would be to build a 32-bit-only Python (but 
ensure that -arch is forced on the build - see Issue7184).
msg94638 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-10-28 20:16
Ned: I'm planning to use the attached version of pythonw, or a slightly 
updated one, for 2.7 and 3.2. This version will not be used for 2.6.5 or 
3.1.2 due to backward compatibility constraints.

I will look into the OSX launching issues though, it was my intention that 
universal builds had a way to select a 32-bit or 64-bit python but that 
code may be broken.
msg95947 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-12-03 21:07
Another reminder: when implementing, make sure platform.architecture() 
always returns the correct results for bits.  It seems the Apple patches 
in 10.6 don't handle that (it seems to always report 64-bit), probably 
because the code in platform.architecture() doesn't look quite robust 
enough.  (I'll open a bug with Apple.)
msg96184 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-12-09 21:46
FWIW: I now have a 2.7 tree with the new pythonw on my machine, open 
issues are:

* Ensure gets build in such a way that the GUI works for
  all supported universal binaries (including a 4-way build on 10.5, 
  where the system Tk doesn't work in 64-bit mode)

* Update NEWS file

* Port to 3.2
msg96857 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-12-24 14:04
I've committed the completed patch as r77031 (trunk) and r77032 (py3k)

I will not backport to 2.6 and 3.1 because this is not a bugfix.
msg97398 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-01-08 09:08
Note, r77031 and r77032 cause compile errors for OS X deployment targets of 10.4 or earlier.  See Issue7658 for details and patch.
Date User Action Args
2010-01-08 09:08:34ned.deilysetmessages: + msg97398
2009-12-24 14:04:10ronaldoussorensetstatus: open -> closed
resolution: accepted
messages: + msg96857

stage: needs patch -> committed/rejected
2009-12-09 21:46:52ronaldoussorensetmessages: + msg96184
2009-12-03 21:07:54ned.deilysetmessages: + msg95947
2009-10-28 20:16:55ronaldoussorensetmessages: + msg94638
title: use different mechanism for pythonw on osx: universal builds always run 64-bit on OS X 10.6 -> use different mechanism for pythonw on osx
2009-10-28 18:25:04robincesetnosy: + robince
2009-10-28 18:09:04ned.deilysetversions: + Python 2.6, Python 3.1
2009-10-28 17:58:45ned.deilysettitle: use different mechanism for pythonw on osx -> use different mechanism for pythonw on osx: universal builds always run 64-bit on OS X 10.6
nosy: + ned.deily

messages: + msg94634

type: enhancement -> behavior
2009-09-24 07:30:16loewissetmessages: + msg93057
2009-09-22 10:48:54ronaldoussorensetfiles: + pythonw.c

messages: + msg92981
2009-09-04 06:10:53ronaldoussorensetmessages: + msg92232
2009-09-03 18:43:20loewissetnosy: + loewis
messages: + msg92215
2009-09-03 14:24:43ronaldoussorencreate