classification
Title: urllib ctypes error on Mac OS X Server 10.5
Type: crash Stage:
Components: ctypes, macOS Versions: Python 3.0, Python 3.1, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: asoki, ned.deily, ronaldoussoren, theller
Priority: normal Keywords: patch

Created on 2009-03-03 23:55 by asoki, last changed 2009-09-20 11:00 by ronaldoussoren. This issue is now closed.

Files
File name Uploaded Description Edit
urllib5413.patch ronaldoussoren, 2009-03-30 21:30
Messages (9)
msg83111 - (view) Author: Attila Soki (asoki) Date: 2009-03-03 23:55
i trying to compile Python 2.6.1 from scratch on Mac OS X Server (ppc).
After configure/make/make install i test the compiled python with
the followig file (t.py) (example from
http://docs.python.org/library/urllib.html):


Contents of /tmp/t.py:
-------------
import urllib
params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
print f.read()
-------------

and i get this error:

python /tmp/t.py 
Traceback (most recent call last):
  File "/tmp/t.py", line 3, in <module>
    f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 82, in urlopen
    opener = FancyURLopener()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 611, in
__init__
    URLopener.__init__(self, *args, **kwargs)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 129, in
__init__
    proxies = getproxies()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1555, in
getproxies
    return getproxies_environment() or getproxies_macosx_sysconf()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1449, in
getproxies_macosx_sysconf
    _CFSetup(sc)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1330, in
_CFSetup
    sc.CFStringCreateWithCString.argtypes = [ c_void_p, c_char_p, c_int32 ]
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
366, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not found
Exception AttributeError: "FancyURLopener instance has no attribute
'tempcache'" in <bound method FancyURLopener.__del__ of
<urllib.FancyURLopener instance at 0x3743a0>> ignored


than tested it whit this code (Thanks to Ned Deily)
See the mailinglist thread at:
http://mail.python.org/pipermail/python-list/2009-March/703067.html

Contents of /tmp/tt.py:
-----------------------
from ctypes import cdll
from ctypes.util import find_library
sc = cdll.LoadLibrary(find_library("SystemConfiguration"))
x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
-----------------------
/usr/local/test/python/bin/python2.6 /tmp/tt.py
Traceback (most recent call last):
  File "/tmp/tt.py", line 4, in <module>
    x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
366,
in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
371,
in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not
found

Both test fails on my PPC Mac and runs without error on my Intel MacBook

Some detailed info about the macs:

ppc:
Model Name: Xserve G5
Model Identifier: RackMac3,1
Processor Name: PowerPC G5  (3.1)
Processor Speed: 2.3 GHz
Number Of CPUs: 2
L2 Cache (per CPU): 512 KB
Boot ROM Version: 5.1.7f2
System Version: Mac OS X Server 10.5.6 (9G55)
Kernel Version: Darwin 9.6.0


intel:
Model Name: MacBook Pro
Model Identifier: MacBookPro3,1
Processor Name: Intel Core 2 Duo
Processor Speed: 2.4 GHz
Number Of Processors: 1
Total Number Of Cores: 2
L2 Cache: 4 MB
Bus Speed: 800 MHz
Boot ROM Version: MBP31.0070.B07
SSC Version: 1.16f8
System Version: Mac OS X 10.5.6 (9G55)
Kernel Version: Darwin 9.6.0


I recompiled Python-2.6.1 with MACOSX_DEPLOYMENT_TARGET=10.3, but no
change..
I updated developertools to xcode312_2621, no change..

Btw.. Python-2.5.4 compiled from scratch works fine.

I dig a bit deeper with otool and discover some differences:

Intel MacOS.so: Carbon, libSystem.B.dylib, CoreServices,
ApplicationServices

PPC MacOS.so: Carbon, libmx.A.dylib, libSystem.B.dylib

Intel _ctypes.so: libSystem.B.dylib
PPC _ctypes.so: libmx.A.dylib, libSystem.B.dylib

-----------
Intel with deploymant target 10.3, xcode312:
-----------
sh-3.2# otool
-L /usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so
/usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so:
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
(compatibility version 2.0.0, current version 136.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)
    
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
(compatibility version 1.0.0, current version 32.0.0)
    
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Applicat
ionServices (compatibility version 1.0.0, current version 34.0.0)

-------

sh-3.2# otool
-L /usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so
/usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

----------
PPC with deployment_target 10.3, xcode312_2621
----------

otool -L /usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so
/usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so:
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
(compatibility version 2.0.0, current version 136.0.0)
    /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version
47.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

-------


otool -L /usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so
/usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so:
    /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version
47.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)


How i compiled it:

export MACOSX_DEPLOYMENT_TARGET=10.5
configure --prefix=/usr/local/test/python
make
make install

How i test:

export
PYTHONPATH=/usr/local/test/python:/usr/local/test/python/lib/python2.6/site-packages
/usr/local/test/python/bin/ is in the PATH
python /tmp/t.py
msg83112 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2009-03-04 00:32
For more info, see discussion starting here:
<http://mail.python.org/pipermail/python-list/2009-March/703067.html>

I suspect a difference in the proxy configuration available between OS X 
and OS X Server, although I don't have a OS X Server configuration at hand 
to test it.  (For 2.6, the OS X proxy configuration code was rewritten to 
use ctypes instead of the deprecated mac Carbon wrappers.)
msg83117 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-04 07:08
I don't have time to investigate this right now, I do have access to a 
10.5 server box though. With some luck I'll be able to create a patch 
later today.

Looking at the code in urllib I found the most likely reason for the 
problem: the Python script tries to resolve "CFStringCreateWithCString" 
through the SystemConfiguration framework. It may well be that this 
doesn't work on PPC systems and that this symbol should explicitly be 
loaded from the CoreFoundation framework.

BTW. This probably also affects 3.x.
msg84654 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-03-30 21:30
I don't have a PPC machine handy at the moment, but I hope the attached 
copy of urllib fixes the issue. Could you please test if that's indeed the 
case?
msg84990 - (view) Author: Attila Soki (asoki) Date: 2009-04-01 08:58
no luck.
output for t.py:

Traceback (most recent call last):
  File "t.py", line 3, in <module>
    f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 82, in
urlopen
    opener = FancyURLopener()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 611, in
__init__
    URLopener.__init__(self, *args, **kwargs)
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 129, in
__init__
    proxies = getproxies()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 1558, in
getproxies
    return getproxies_environment() or getproxies_macosx_sysconf()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 1452, in
getproxies_macosx_sysconf
    _CFSetup(sc)
TypeError: _CFSetup() takes exactly 2 arguments (1 given)
Exception AttributeError: "FancyURLopener instance has no attribute
'tempcache'" in <bound method FancyURLopener.__del__ of
<urllib.FancyURLopener instance at 0x372288>> ignored


output for tt.py:

Traceback (most recent call last):
  File "tt.py", line 4, in <module>
    x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
  File "/var/root/pytest/python/lib/python2.6/ctypes/__init__.py", line
366, in __getattr__
    func = self.__getitem__(name)
  File "/var/root/pytest/python/lib/python2.6/ctypes/__init__.py", line
371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not found

notes:
t.py fails on both with the same error
tt.py runs on Intel/Mac OS X and fails on PPC/Mac OS X Server
msg85004 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-04-01 13:32
I'll install 10.5 server on my PPC test machine when I'm back from PyCon, 
this seem to need some serious debugging.
msg86165 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-04-19 11:05
The attached patch is not entirely correct, I'm not yet uploading an 
updated patch because the fix is obvious.

I cannot reproduce this with a the 2.6.2 installer (build on PPC) nor 
with a recent build from subversion (python-trunk, build on X86). My 
tests were done on a PPC G4 iMac running the following version of OSX:

ProductName:	Mac OS X
ProductVersion:	10.4.11
BuildVersion:	8S165

Xcode is:

$ xcodebuild -version
Component versions: DevToolsCore-660.0; DevToolsSupport-651.0

I'm currently installing 10.5 server in a VM, and will try to test that 
as well (using Rosetta). I don't have access to a PPC machine running 
10.5 server, the 10.4 system I mention above cannot be easily 
reinstalled.

BTW. I intend to commit the fixed version of my patch regardless of 
whether this fixes the issue because it is slightly more correct to 
fetch CoreFoundation symbols directly from the CoreFoundation framework 
instead of relying on SystemConfiguration's "connection" to that 
framework.
msg86763 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-04-28 20:11
Seems ronald takes care of this issue.
msg92888 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-09-20 11:00
I've just committed a fix for another issue and that completely removes 
the ctypes-based code that causes this issue (replacing it by a C 
extension)

Therefore this issue should no longer occur and hence can be closed.
History
Date User Action Args
2009-09-20 11:00:47ronaldoussorensetstatus: open -> closed
resolution: fixed
messages: + msg92888
2009-04-28 20:11:12thellersetassignee: theller -> ronaldoussoren
messages: + msg86763
2009-04-19 11:05:35ronaldoussorensetmessages: + msg86165
2009-04-01 13:32:47ronaldoussorensetmessages: + msg85004
2009-04-01 08:59:00asokisetmessages: + msg84990
2009-03-30 21:30:14ronaldoussorensetfiles: + urllib5413.patch
keywords: + patch
messages: + msg84654
2009-03-04 07:08:09ronaldoussorensetmessages: + msg83117
versions: + Python 3.0, Python 3.1
2009-03-04 00:32:19ned.deilysetnosy: + ronaldoussoren, ned.deily
messages: + msg83112
2009-03-03 23:55:43asokicreate