Title: test_urllib2 crashes on OS X 10.3 attempting to retrieve network proxy configuration
Type: crash Stage:
Components: macOS Versions: Python 2.7, Python 2.6
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: ned.deily, orsenthil, ronaldoussoren
Priority: normal Keywords:

Created on 2010-03-09 09:53 by ned.deily, last changed 2010-08-11 08:27 by ned.deily. This issue is now closed.

File name Uploaded Description Edit
Python.crash.log ned.deily, 2010-03-09 09:58
issue-scconfig-10_3-trunk-26.txt ned.deily, 2010-03-09 22:01
issue8095.txt ronaldoussoren, 2010-04-18 19:31
Messages (7)
msg100708 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-03-09 09:53
The current mechanism for urllib and urllib2 on OS X to retrieve network proxy information is to call _scproxy.c to query the OS X SystemConfiguration Framework.  _scproxy.c uses a schema key, kSCPropNetProxiesExcludeSimpleHostnames, that was introduced in OS X 10.4.  Building on 10.3 results in a compile error.  Current OS X installers are built using the 10.4u SDK with a deployment target of 10.3 to allow running on OS X 10.3.9.  When such pythons are installed on 10.3, attempts to use proxy support, such as is tested in test_urllib2, result in a crash:

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000000

Thread 0 Crashed:
0       	0x000a77ac get_proxy_settings + 0x6c

Whether this is worth fixing is debatable, considering that 10.3 has not been supported by Apple for many years and that the use of network proxy  servers has likely been declining.  At a minimum, it would be better for it to fail without a crash but perhaps a README item would suffice.
msg100710 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-03-09 10:09
I don't have a 10.3 system to test on, and definitely don't want to spent effort on enabling compiles on 10.3

The patch below would probably fix the crash when running a binary created on 10.4 or later on osx 10.3:

Index: ../Mac/Modules/_scproxy.c
--- ../Mac/Modules/_scproxy.c	(revision 78807)
+++ ../Mac/Modules/_scproxy.c	(working copy)
@@ -64,13 +64,18 @@
 	result = PyDict_New();
 	if (result == NULL) goto error;
-	aNum = CFDictionaryGetValue(proxyDict, 
+	if (kSCPropNetProxiesExcludeSimpleHostnames != NULL) {
+		aNum = CFDictionaryGetValue(proxyDict, 
-	if (aNum == NULL) {
-		v = PyBool_FromLong(0);
-	} else {
-		v = PyBool_FromLong(cfnum_to_int32(aNum));
+		if (aNum == NULL) {
+			v = PyBool_FromLong(1);
+		} else {
+			v = PyBool_FromLong(cfnum_to_int32(aNum));
+		}
+	}  else {
+		v = PyBool_FromLong(1);
 	if (v == NULL) goto error;
 	r = PyDict_SetItemString(result, "exclude_simple", v);

The patch hasn't been compiled yet, but the idea should be clear: test if kSCPropNetProxiesExcludeSimpleHostnames has a valid value before using it, default to 'True'.

(This also changes the default on 10.4/10.5, IMHO defaulting to true would be better: I haven't seen an enviroment yet where local systems should be accessed through a proxy).

BTW. Removing 3.1 and 3.2 because _scproxy isn't in those releases yet (mostly because porting requires signifant changes to the C code and I haven't had time to do that yet)
msg100763 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-03-09 22:01
OK, I tried the patch.  Reversing the default sense causes the proxy tests in test_urllib2 to fail on 10.6 et al.  So I changed the sense of the tests in the patch to match previous behavior; the modified patch is attached.  Unfortunately, it didn't help on 10.3; test_urllib2 still gets a bus error:

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000000

Thread 0 Crashed:
0       	0x003b57ac get_proxy_settings + 0x6c
1   org.python.python 	0x00498dfc PyEval_EvalFrameEx + 0x535c
msg103521 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-04-18 19:31
Ned: I've attached a new patch, could you please test this one on an 10.3 box?

It turns out that testing if a weaklinked variable is defined should be done by testing the address of the variable, not its value.

I've tested this new approach with a symbol that is defined on 10.6 but not 10.5, but cannot test on 10.3.
msg103683 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-04-20 08:55
I've committed my proposed fix in r80243 (trunk)
msg111471 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-24 12:49
Ned: could you please check if the issue is present in Python 2.7?

I don't have access to machines that are capable of running OSX 10.3.
msg113595 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-08-11 08:27
The fix in r80243 for 27 and similar fixes for 26, 31, and py3k do prevent crashes on OS X 10.3 when using network proxies so this issue can be closed.  However, the problem I mentioned above with changing the default sense which caused tests to fail was documented in Issue8455 and subsequently fixed for 10.4+.  The fixes did not correct the 10.3 code paths; that is now documented in Issue9568 with tested patches.
Date User Action Args
2010-08-11 08:27:25ned.deilysetstatus: open -> closed

messages: + msg113595
2010-07-24 12:49:11ronaldoussorensetmessages: + msg111471
2010-04-20 08:55:14ronaldoussorensetmessages: + msg103683
2010-04-18 19:31:25ronaldoussorensetfiles: + issue8095.txt

messages: + msg103521
2010-03-09 22:01:51ned.deilysetfiles: + issue-scconfig-10_3-trunk-26.txt

messages: + msg100763
2010-03-09 10:09:03ronaldoussorensetmessages: + msg100710
versions: - Python 3.1, Python 3.2
2010-03-09 09:58:34ned.deilysetfiles: + Python.crash.log
2010-03-09 09:53:38ned.deilycreate