classification
Title: urllib2 unrelease KQUEUE on Mac OSX 10.9+
Type: crash Stage: resolved
Components: Library (Lib), macOS Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: Andrew.Gross, Lita.Cho, ned.deily, orsenthil, rcarz, ronaldoussoren
Priority: high Keywords:

Created on 2014-02-10 18:46 by Andrew.Gross, last changed 2020-10-19 18:09 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
urllib2.crash Andrew.Gross, 2014-02-10 18:45 Kernel Crash Report and lsof output
python_2014-07-18-171109.crash rcarz, 2014-07-19 01:51 Crash report with _scproxy in the stack trace on OS X 10.9
Messages (9)
msg210860 - (view) Author: Andrew Gross (Andrew.Gross) Date: 2014-02-10 18:45
In the latest OSX, 10.9, it looks like there have been some security changes related to inheriting file descriptors from parent to child processes.  While in the past it would allow you to inherit the parents open FDs, now it will kill the process. 

It looks like urllib2 does not properly unsubscribe from the KQUEUE after opening a remote connection, causing the process to be killed if you try use "evecv" style commands, or fork.

Simple Reproduction:
import urllib2
request = urllib2.urlopen('http://www.python.org')
response = request.read()
request.close()
os.execvp('ssh', ['ssh', 'user@1.2.3.4'])
# Killed: 9

I have attached the diagnostic crash report from OSX, appended to the end is a snippet from `lsof` showing the open KQUEUE file descriptor.
msg210862 - (view) Author: Andrew Gross (Andrew.Gross) Date: 2014-02-10 19:43
For some additional context on how I came across this bug, check out https://github.com/coderanger/pychef/issues/29
msg210900 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-02-11 06:57
It looks like the culprit is _scproxy, an internal C extension, that urllib2 (py2) and urllib.request (py3) use on OS X to access the system configuration for network proxies.  If you aren't using any proxies, a workaround is to define an environment variable:

    export no_proxy='*'

which causes urllib2/urllib to bypass checking the system configuration.

http://docs.python.org/2/library/urllib2.html#urllib2.ProxyHandler
msg210964 - (view) Author: Andrew Gross (Andrew.Gross) Date: 2014-02-11 16:33
Thanks, the workaround fixes my issue.
msg216914 - (view) Author: Lita Cho (Lita.Cho) * Date: 2014-04-20 18:47
Going to try working on this.
msg223457 - (view) Author: Bob Carroll (rcarz) Date: 2014-07-19 01:51
I'm not forking, but this is possibly related my issue. I have a twisted application that periodically crashes when attempting to make an HTTP request. Based on the stack trace showing _scproxy, I tried the work around and it seems to have helped. The crash log is attached.
msg247125 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-22 14:58
I don't think it is possible to fix this crash other than by removing the use of _scproxy (proxy autodection on OSX) completely.

Problem is that Apple's higher level APIs (such as those used in _scproxy) don't take the use-case of calling fork(2), but not exec(2), to create child processes into account. 

BTW. I'm against removing _scproxy because that makes platform integration worse.

I guess this should be documented somewhere, and possibly just in the documentation for os.fork() with a warning about not using just fork on OSX when there's any chance that some C extension you use calls into a higher level API.

BTW. I could reproduce the crash on 10.9, but failed to do so on 10.10 earlier today (but that might just mean I have to try harder...)
msg378954 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-10-19 15:29
I cannot reproduce this issue on macOS 10.15 (with python 3.9).

I intent to close this issue because (a) this is a platform problem and (b) seems no longer to be a problem with the latest macOS and Python versions.
msg378974 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-10-19 18:09
I can't reproduce with 3.9 on 10.9, either. If someone can reproduce with current Pythons and/or macOS without involving forking, feel free to reopen with details.
History
Date User Action Args
2020-10-19 18:09:10ned.deilysetstatus: pending -> closed

messages: + msg378974
2020-10-19 15:29:26ronaldoussorensetstatus: open -> pending
resolution: out of date
messages: + msg378954

stage: needs patch -> resolved
2015-07-22 14:58:45ronaldoussorensetmessages: + msg247125
2015-02-27 08:48:06ned.deilylinkissue23533 superseder
2014-07-19 01:51:12rcarzsetfiles: + python_2014-07-18-171109.crash
nosy: + rcarz
messages: + msg223457

2014-04-20 18:47:54Lita.Chosetnosy: + Lita.Cho
messages: + msg216914
2014-02-11 16:33:09Andrew.Grosssetmessages: + msg210964
2014-02-11 06:57:04ned.deilysetnosy: + ned.deily

messages: + msg210900
stage: needs patch
2014-02-10 23:21:49ned.deilysetpriority: normal -> high
nosy: + orsenthil

versions: + Python 3.3, Python 3.4
2014-02-10 19:43:54Andrew.Grosssetmessages: + msg210862
2014-02-10 18:46:03Andrew.Grosscreate