classification
Title: Seg Fault on OSX when multiprocessing
Type: crash Stage:
Components: macOS Versions: Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: alvis, davin, ned.deily, pitrou, ronaldoussoren, timstonis, xtreak
Priority: normal Keywords:

Created on 2019-07-25 08:07 by alvis, last changed 2019-10-10 07:49 by ronaldoussoren.

Messages (5)
msg348429 - (view) Author: alvis (alvis) Date: 2019-07-25 08:07
Encountered a crash with the following logs.
I have already added no_proxy=*, and it fixed the issue which i can replicate consistent. However, this crash does not happen every time, maybe once in a few week.

Process:               Python [66435]
Path:                  /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.6.8 (3.6.8)
Code Type:             X86-64 (Native)
Parent Process:        Python [65077]
Responsible:           Python [66435]
User ID:               501

Date/Time:             2019-07-24 21:45:29.722 +0800
OS Version:            Mac OS X 10.14.2 (18C54)
Report Version:        12
Bridge OS Version:     3.5 (16P5125)
Anonymous UUID:        3F729BF3-2BB1-3830-B6FE-D22CA431DE5A


Time Awake Since Boot: 1600000 seconds

System Integrity Protection: enabled

Crashed Thread:        2

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000109b23aa2
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [66435]

VM Regions Near 0x109b23aa2:
    MALLOC_LARGE           0000000109aa3000-0000000109b23000 [  512K] rw-/rwx SM=COW  
--> 
    __TEXT                 0000000109b27000-0000000109b3d000 [   88K] r-x/rwx SM=COW  /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/.dylibs/libgcc_s.1.dylib

Application Specific Information:
*** multi-threaded process forked ***
crashed on child side of fork pre-exec 

Thread 2 Crashed:
0   libsystem_trace.dylib         	0x00007fff7d6fdab3 _os_log_cmp_key + 4
1   libsystem_c.dylib             	0x00007fff7d5aaa5e rb_tree_find_node + 53
2   libsystem_trace.dylib         	0x00007fff7d6fbc81 os_log_create + 370
3   libnetwork.dylib              	0x00007fff7c04058f nwlog_legacy_init_once + 47
4   libsystem_pthread.dylib       	0x00007fff7d6e1ce4 __pthread_once_handler + 66
5   libsystem_platform.dylib      	0x00007fff7d6d7acb _os_once_callout + 18
6   libsystem_pthread.dylib       	0x00007fff7d6e1c7f pthread_once + 56
7   libnetwork.dylib              	0x00007fff7c024ccd networkd_settings_init + 669
8   libnetwork.dylib              	0x00007fff7c024982 networkd_settings_get_bool + 18
9   libnetwork.dylib              	0x00007fff7c0244f0 -[NWConcrete_nw_parameters initWithStack:] + 160
10  libnetwork.dylib              	0x00007fff7c023d97 nw_path_create_evaluator_for_endpoint + 487
11  libnetwork.dylib              	0x00007fff7c2451bc nw_nat64_v4_address_requires_synthesis + 220
12  libsystem_info.dylib          	0x00007fff7d609e83 _gai_nat64_v4_address_requires_synthesis + 67
13  libsystem_info.dylib          	0x00007fff7d609210 _gai_nat64_second_pass + 512
14  libsystem_info.dylib          	0x00007fff7d5e7b4a si_addrinfo + 1978
15  libsystem_info.dylib          	0x00007fff7d5e7262 _getaddrinfo_internal + 242
16  libsystem_info.dylib          	0x00007fff7d5e715d getaddrinfo + 61
17  _socket.cpython-36m-darwin.so 	0x00000001092c24ed socket_getaddrinfo + 621
18  org.python.python             	0x0000000106d63697 _PyCFunction_FastCallDict + 183
19  org.python.python             	0x0000000106de5747 call_function + 439
20  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
21  org.python.python             	0x0000000106de61af _PyEval_EvalCodeWithName + 2447
22  org.python.python             	0x0000000106de6ae1 fast_function + 545
23  org.python.python             	0x0000000106de5721 call_function + 401
24  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
25  org.python.python             	0x0000000106de61af _PyEval_EvalCodeWithName + 2447
26  org.python.python             	0x0000000106de6ae1 fast_function + 545
27  org.python.python             	0x0000000106de5721 call_function + 401
28  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
29  org.python.python             	0x0000000106de6a3d fast_function + 381
30  org.python.python             	0x0000000106de5721 call_function + 401
31  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
32  org.python.python             	0x0000000106de61af _PyEval_EvalCodeWithName + 2447
33  org.python.python             	0x0000000106de6ae1 fast_function + 545
34  org.python.python             	0x0000000106de5721 call_function + 401
35  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
36  org.python.python             	0x0000000106de61af _PyEval_EvalCodeWithName + 2447
37  org.python.python             	0x0000000106de6dfb _PyFunction_FastCallDict + 763
38  org.python.python             	0x0000000106d1a6b7 _PyObject_FastCallDict + 247
39  org.python.python             	0x0000000106d1a7d5 _PyObject_Call_Prepend + 149
40  org.python.python             	0x0000000106d1a4f0 PyObject_Call + 96
41  org.python.python             	0x0000000106d7badd slot_tp_init + 125
42  org.python.python             	0x0000000106d77d09 type_call + 313
43  org.python.python             	0x0000000106d1a685 _PyObject_FastCallDict + 197
44  org.python.python             	0x0000000106de5648 call_function + 184
45  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
46  org.python.python             	0x0000000106de6a3d fast_function + 381
47  org.python.python             	0x0000000106de5721 call_function + 401
48  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
49  org.python.python             	0x0000000106de6a3d fast_function + 381
50  org.python.python             	0x0000000106de5721 call_function + 401
51  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
52  org.python.python             	0x0000000106de6a3d fast_function + 381
53  org.python.python             	0x0000000106de5721 call_function + 401
54  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
55  org.python.python             	0x0000000106de6a3d fast_function + 381
56  org.python.python             	0x0000000106de5721 call_function + 401
57  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
58  org.python.python             	0x0000000106de6a3d fast_function + 381
59  org.python.python             	0x0000000106de5721 call_function + 401
60  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
61  org.python.python             	0x0000000106de6a3d fast_function + 381
62  org.python.python             	0x0000000106de5721 call_function + 401
63  org.python.python             	0x0000000106de1eb7 _PyEval_EvalFrameDefault + 27559
64  org.python.python             	0x0000000106de6c5c _PyFunction_FastCallDict + 348
65  org.python.python             	0x0000000106d1a6b7 _PyObject_FastCallDict + 247
66  org.python.python             	0x0000000106d1a7d5 _PyObject_Call_Prepend + 149
67  org.python.python             	0x0000000106d1a4f0 PyObject_Call + 96
68  org.python.python             	0x0000000106e2d396 t_bootstrap + 70
69  org.python.python             	0x0000000106e26be9 pythread_wrapper + 25
70  libsystem_pthread.dylib       	0x00007fff7d6e3305 _pthread_body + 126
71  libsystem_pthread.dylib       	0x00007fff7d6e626f _pthread_start + 70
72  libsystem_pthread.dylib       	0x00007fff7d6e2415 thread_start + 13
msg348430 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-07-25 08:14
Thanks for the report. It would be helpful to have a standalone reproducer for this issue to try reproducing it in 3.6 and other versions.
msg349514 - (view) Author: alvis (alvis) Date: 2019-08-13 02:21
Hi, sorry i am unable to provide a standalone script that cause this issue, as it happen once every few weeks. Is it similar to the no_proxy=* issue?
msg354243 - (view) Author: Timothy Stonis (timstonis) Date: 2019-10-09 02:09
Hi, I have the exact same issue and same stack trace from getaddrinfo (except Python 2.7). Same as original poster, it only happens every couple of weeks and I don't know how to reproduce. For me it's in an Odoo instance that spawns multiple workers with connections to a postgres database. I think this is a MacOS bug, since the issue does not happen on High Sierra (10.13).
msg354325 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2019-10-10 07:49
Looking at the stack trace this is a similar issue as the "no_proxy" one, but in some sense worse because it happens when just calling a regular unix API (not an Apple framework API).

The first few lines of the stack trace are for code in the "Network" framework which is new in macOS 10.14 and is based on lib dispatch and  hence uses threads. That results in the same problem as the "no_proxy" one: code that causes problems when called after os.fork().

That's a bug in macOS.

In Python 3.8 the multiprocessing module switched to "spawn" mode instead of "fork" mode because of this.  You can do this manually for earlier versions (https://docs.python.org/3.7/library/multiprocessing.html#contexts-and-start-methods)

P.S. A reproducer would still be nice as that would make it easier to raise and issue with Apple about this.
History
Date User Action Args
2019-10-10 07:49:23ronaldoussorensetversions: + Python 3.6
nosy: + ned.deily, ronaldoussoren

messages: + msg354325

components: + macOS
2019-10-09 02:09:42timstonissetnosy: + timstonis

messages: + msg354243
versions: + Python 2.7, - Python 3.6
2019-08-13 02:21:29alvissetmessages: + msg349514
2019-07-25 08:14:15xtreaksetnosy: + xtreak, pitrou, davin
messages: + msg348430
2019-07-25 08:07:01alviscreate