Title: OSX 10.12 crash in urllib.request getproxies_macosx_sysconf and proxy_bypass_macosx_sysconf
Type: Stage: resolved
Components: Library (Lib), macOS Versions: Python 3.5
Status: closed Resolution: duplicate
Dependencies: Superseder: exception error in when called after fork
View: 13829
Assigned To: Nosy List: ned.deily, ronaldoussoren, thehesiod
Priority: normal Keywords:

Created on 2016-10-02 23:48 by thehesiod, last changed 2016-10-04 19:06 by ned.deily. This issue is now closed.

File name Uploaded Description Edit
python_urllib_crash.txt thehesiod, 2016-10-02 23:48
Messages (7)
msg277917 - (view) Author: Alexander Mohr (thehesiod) * Date: 2016-10-02 23:48
I have a unittest which spawns several processes repeatedly.  One of these subprocesses uses botocore, which itself uses the above two methods through the calls proxy_bypass and getproxies.  It seems after re-spawning the methods a few times the titled calls eventually repeatedly cause python to crash on 10.12.  I have a core file if that would help, zip'd it's ~242MB.

I've attached a file that shows the lldb callstack and python callstack.
msg277921 - (view) Author: Alexander Mohr (thehesiod) * Date: 2016-10-03 02:22
interestingly I haven't been able to get this to crash in a separate test app.  There must be either timing related to some interaction with another module.  let me know how you guys would like to proceed.  I can definitely reproduce it consistently in our application.
msg277922 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-03 02:33
This is likely a duplicate of issues like #27126 and #13829.  In #24273 there is a suggested workaround: set environment variable "no_proxy" to "*".
msg277968 - (view) Author: Alexander Mohr (thehesiod) * Date: 2016-10-03 17:34
ya I did a monkey patch which resolved it.

if sys.platform == 'darwin':
    import botocore.vendored.requests.utils, urllib.request
    botocore.vendored.requests.utils.proxy_bypass = urllib.request.proxy_bypass_environment
    botocore.vendored.requests.utils.getproxies = urllib.request.getproxies_environment

    urllib.request.proxy_bypass = urllib.request.proxy_bypass_environment
    urllib.request.getproxies = urllib.request.getproxies_environment
msg277970 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-03 17:56
Glad you have it working.  Did you try the suggested workaround of setting environment variable "no_proxy" to "*"?  I believe that should have had the same net effect as the monkey patching.
msg278001 - (view) Author: Alexander Mohr (thehesiod) * Date: 2016-10-03 21:37
I'm sure it would work, I just wanted a solution that didn't changes to our build infrastructure. btw if we're marking this as a duplicate of the other bug, can we update the other bug to say it affects python3.x as well?  Thanks!
msg278080 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-04 19:06
OK, I'm closing this a duplicate and I've updated #13829 to include currently 3.x versions.
Date User Action Args
2016-10-04 19:06:27ned.deilysetstatus: open -> closed

nosy: + ronaldoussoren
messages: + msg278080

components: + macOS
stage: resolved
2016-10-03 21:37:33thehesiodsetstatus: pending -> open

messages: + msg278001
2016-10-03 17:56:32ned.deilysetstatus: open -> pending

messages: + msg277970
2016-10-03 17:34:39thehesiodsetstatus: pending -> open

messages: + msg277968
2016-10-03 02:33:19ned.deilysetstatus: open -> pending
2016-10-03 02:33:05ned.deilysetnosy: + ned.deily
messages: + msg277922
resolution: duplicate

superseder: exception error in when called after fork
2016-10-03 02:22:59thehesiodsetmessages: + msg277921
2016-10-02 23:48:54thehesiodcreate