classification
Title: 3.7.0b2 Interpreter crash in dev mode (or with PYTHONMALLOC=debug) with 'python -X dev -c 'import os; os.fork()'
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, jmadden, miss-islington, vstinner, xiang.zhang
Priority: normal Keywords: patch

Created on 2018-03-05 22:40 by jmadden, last changed 2018-03-06 15:01 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6001 merged vstinner, 2018-03-06 11:18
PR 6003 merged miss-islington, 2018-03-06 13:32
Messages (8)
msg313296 - (view) Author: Jason Madden (jmadden) * Date: 2018-03-05 22:40
At the request of Victor Stinner on twitter, I ran the gevent test suite with Python 3.7.0b2 with the new '-X dev' argument and discovered an interpreter crash. With a bit of work, it boiled down to a very simple command:

$ env -i .runtimes/snakepit/python3.7.0b2 -X dev -c 'import os; os.fork()'
*** Error in `.runtimes/snakepit/python3.7.0b2': munmap_chunk(): invalid pointer: 0x0000000001c43a80 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f5a971607e5]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7f5a9716d698]
.runtimes/snakepit/python3.7.0b2(_PyRuntimeState_Fini+0x30)[0x515d90]
.runtimes/snakepit/python3.7.0b2[0x51445f]
.runtimes/snakepit/python3.7.0b2[0x42ce40]
.runtimes/snakepit/python3.7.0b2(_Py_UnixMain+0x7b)[0x42eaab]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f5a97109830]
.runtimes/snakepit/python3.7.0b2(_start+0x29)[0x42a0d9]
======= Memory map: ========
00400000-00689000 r-xp 00000000 08:01 177409                             //.runtimes/versions/python3.7.0b2/bin/python3.7
00888000-00889000 r--p 00288000 08:01 177409                             //.runtimes/versions/python3.7.0b2/bin/python3.7
00889000-008f3000 rw-p 00289000 08:01 177409                             //.runtimes/versions/python3.7.0b2/bin/python3.7
008f3000-00914000 rw-p 00000000 00:00 0
01b84000-01c64000 rw-p 00000000 00:00 0                                  [heap]
7f5a96052000-7f5a96068000 r-xp 00000000 08:01 265946                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96068000-7f5a96267000 ---p 00016000 08:01 265946                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96267000-7f5a96268000 rw-p 00015000 08:01 265946                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5a96268000-7f5a96273000 r-xp 00000000 08:01 268943                     /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96273000-7f5a96472000 ---p 0000b000 08:01 268943                     /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96472000-7f5a96473000 r--p 0000a000 08:01 268943                     /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96473000-7f5a96474000 rw-p 0000b000 08:01 268943                     /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f5a96474000-7f5a9647a000 rw-p 00000000 00:00 0
7f5a9647a000-7f5a96485000 r-xp 00000000 08:01 268947                     /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96485000-7f5a96684000 ---p 0000b000 08:01 268947                     /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96684000-7f5a96685000 r--p 0000a000 08:01 268947                     /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96685000-7f5a96686000 rw-p 0000b000 08:01 268947                     /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f5a96686000-7f5a9669c000 r-xp 00000000 08:01 268927                     /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9669c000-7f5a9689b000 ---p 00016000 08:01 268927                     /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689b000-7f5a9689c000 r--p 00015000 08:01 268927                     /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689c000-7f5a9689d000 rw-p 00016000 08:01 268927                     /lib/x86_64-linux-gnu/libnsl-2.23.so
7f5a9689d000-7f5a9689f000 rw-p 00000000 00:00 0
7f5a9689f000-7f5a968a7000 r-xp 00000000 08:01 268938                     /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a968a7000-7f5a96aa6000 ---p 00008000 08:01 268938                     /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96aa6000-7f5a96aa7000 r--p 00007000 08:01 268938                     /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96aa7000-7f5a96aa8000 rw-p 00008000 08:01 268938                     /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f5a96acc000-7f5a96b4c000 rw-p 00000000 00:00 0
7f5a96b4c000-7f5a96b4e000 r-xp 00000000 08:01 184551                     //.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96b4e000-7f5a96d4e000 ---p 00002000 08:01 184551                     //.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d4e000-7f5a96d4f000 r--p 00002000 08:01 184551                     //.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d4f000-7f5a96d51000 rw-p 00003000 08:01 184551                     //.runtimes/versions/python3.7.0b2/lib/python3.7/lib-dynload/_heapq.cpython-37m-x86_64-linux-gnu.so
7f5a96d51000-7f5a96e11000 rw-p 00000000 00:00 0
7f5a96e11000-7f5a970e9000 r--p 00000000 08:01 133586                     /usr/lib/locale/locale-archive
7f5a970e9000-7f5a972a9000 r-xp 00000000 08:01 268930                     /lib/x86_64-linux-gnu/libc-2.23.so
7f5a972a9000-7f5a974a9000 ---p 001c0000 08:01 268930                     /lib/x86_64-linux-gnu/libc-2.23.so
7f5a974a9000-7f5a974ad000 r--p 001c0000 08:01 268930                     /lib/x86_64-linux-gnu/libc-2.23.so
7f5a974ad000-7f5a974af000 rw-p 001c4000 08:01 268930                     /lib/x86_64-linux-gnu/libc-2.23.so
7f5a974af000-7f5a974b3000 rw-p 00000000 00:00 0
7f5a974b3000-7f5a975bb000 r-xp 00000000 08:01 268926                     /lib/x86_64-linux-gnu/libm-2.23.so
7f5a975bb000-7f5a977ba000 ---p 00108000 08:01 268926                     /lib/x86_64-linux-gnu/libm-2.23.so
7f5a977ba000-7f5a977bb000 r--p 00107000 08:01 268926                     /lib/x86_64-linux-gnu/libm-2.23.so
7f5a977bb000-7f5a977bc000 rw-p 00108000 08:01 268926                     /lib/x86_64-linux-gnu/libm-2.23.so
7f5a977bc000-7f5a977be000 r-xp 00000000 08:01 268937                     /lib/x86_64-linux-gnu/libutil-2.23.so
7f5a977be000-7f5a979bd000 ---p 00002000 08:01 268937                     /lib/x86_64-linux-gnu/libutil-2.23.so
7f5a979bd000-7f5a979be000 r--p 00001000 08:01 268937                     /lib/x86_64-linux-gnu/libutil-2.23.so
7f5a979be000-7f5a979bf000 rw-p 00002000 08:01 268937                     /lib/x86_64-linux-gnu/libutil-2.23.so
7f5a979bf000-7f5a979c2000 r-xp 00000000 08:01 268932                     /lib/x86_64-linux-gnu/libdl-2.23.so
7f5a979c2000-7f5a97bc1000 ---p 00003000 08:01 268932                     /lib/x86_64-linux-gnu/libdl-2.23.so
7f5a97bc1000-7f5a97bc2000 r--p 00002000 08:01 268932                     /lib/x86_64-linux-gnu/libdl-2.23.so
7f5a97bc2000-7f5a97bc3000 rw-p 00003000 08:01 268932                     /lib/x86_64-linux-gnu/libdl-2.23.so
7f5a97bc3000-7f5a97bdb000 r-xp 00000000 08:01 268929                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5a97bdb000-7f5a97dda000 ---p 00018000 08:01 268929                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5a97dda000-7f5a97ddb000 r--p 00017000 08:01 268929                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5a97ddb000-7f5a97ddc000 rw-p 00018000 08:01 268929                     /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5a97ddc000-7f5a97de0000 rw-p 00000000 00:00 0
7f5a97de0000-7f5a97e06000 r-xp 00000000 08:01 268928                     /lib/x86_64-linux-gnu/ld-2.23.so
7f5a97e10000-7f5a97fb5000 rw-p 00000000 00:00 0
7f5a97fb5000-7f5a97fdc000 r--p 00000000 08:01 135047                     /usr/lib/locale/C.UTF-8/LC_CTYPE
7f5a97fdc000-7f5a97fe1000 rw-p 00000000 00:00 0
7f5a97ffd000-7f5a97ffe000 rw-p 00000000 00:00 0
7f5a97ffe000-7f5a98005000 r--s 00000000 08:01 529048                     /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f5a98005000-7f5a98006000 r--p 00025000 08:01 268928                     /lib/x86_64-linux-gnu/ld-2.23.so
7f5a98006000-7f5a98007000 rw-p 00026000 08:01 268928                     /lib/x86_64-linux-gnu/ld-2.23.so
7f5a98007000-7f5a98008000 rw-p 00000000 00:00 0
7fff79aeb000-7fff79b0c000 rw-p 00000000 00:00 0                          [stack]
7fff79b1d000-7fff79b20000 r--p 00000000 00:00 0                          [vvar]
7fff79b20000-7fff79b22000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

The crash is reproducible on Ubuntu 16.04 with a pyenv-built 3.7.0b2 and on macOS 10.13 with the python.org build.

Individually setting PYTHONMALLOC=debug also triggers the crash:

$ PYTHONMALLOC=debug /usr/local/bin/python3.7 -c 'import os; os.fork()'
Python(16996,0x7fffb1879340) malloc: *** error for object 0x7f90e6d01ff0: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
msg313297 - (view) Author: Jason Madden (jmadden) * Date: 2018-03-05 23:01
I built a local version of master (6821e73) and was able to get some line numbers (they're off by one for some reason, it appears):

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fff78972e3e __pthread_kill + 10
1   libsystem_pthread.dylib       	0x00007fff78ab1150 pthread_kill + 333
2   libsystem_c.dylib             	0x00007fff788cf312 abort + 127
3   libsystem_malloc.dylib        	0x00007fff789cc866 free + 521
4   python.exe                    	0x0000000100fba715 _PyRuntimeState_Fini + 37 (pystate.c:90)
5   python.exe                    	0x0000000100fb9d73 Py_FinalizeEx + 547 (pylifecycle.c:1231)
6   python.exe                    	0x0000000100fddd80 pymain_main + 5808 (main.c:2664)
7   python.exe                    	0x0000000100fdec82 _Py_UnixMain + 178 (main.c:2697)
8   libdyld.dylib                 	0x00007fff78823115 start + 1
msg313318 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-03-06 11:25
This bug is likely a regression on my work on Py_Main(): see bpo-32030. I proposed a fix: see attached PR 6001.
msg313324 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-03-06 13:31
New changeset 5d92647102fac9e116b98ab8bbc632eeed501c34 by Victor Stinner in branch 'master':
bpo-33005: Fix _PyGILState_Reinit() (#6001)
https://github.com/python/cpython/commit/5d92647102fac9e116b98ab8bbc632eeed501c34
msg313325 - (view) Author: miss-islington (miss-islington) Date: 2018-03-06 13:52
New changeset 31e2b76f7bbcb8278748565252767a8b7790ff27 by Miss Islington (bot) in branch '3.7':
bpo-33005: Fix _PyGILState_Reinit() (GH-6001)
https://github.com/python/cpython/commit/31e2b76f7bbcb8278748565252767a8b7790ff27
msg313326 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-03-06 14:30
Thanks Jason Madden for your bug report! I should now be fixed.

You might want to try the 3.7 or master branch until the next 3.7 release: PEP 537 ("3.7.0 beta 3: 2018-03-26").
msg313328 - (view) Author: Jason Madden (jmadden) * Date: 2018-03-06 14:59
Thank you! I can confirm that git commit 31e2b76f7bbcb8278748565252767a8b7790ff27 on the 3.7 branch fixes the issue for me.
msg313329 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-03-06 15:01
> Thank you! I can confirm that git commit 31e2b76f7bbcb8278748565252767a8b7790ff27 on the 3.7 branch fixes the issue for me.

Cool. You can now continue your gevent tests using -X dev ;-)
History
Date User Action Args
2018-03-06 15:01:11vstinnersetmessages: + msg313329
2018-03-06 14:59:31jmaddensetmessages: + msg313328
2018-03-06 14:30:30vstinnersetstatus: open -> closed
versions: + Python 3.8
messages: + msg313326

resolution: fixed
stage: patch review -> resolved
2018-03-06 13:52:29miss-islingtonsetnosy: + miss-islington
messages: + msg313325
2018-03-06 13:32:10miss-islingtonsetpull_requests: + pull_request5768
2018-03-06 13:31:40vstinnersetmessages: + msg313324
2018-03-06 11:25:18vstinnersetmessages: + msg313318
2018-03-06 11:18:42vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5766
2018-03-06 03:30:22xiang.zhangsetnosy: + vstinner, eric.snow, xiang.zhang
2018-03-05 23:01:20jmaddensetmessages: + msg313297
2018-03-05 22:40:09jmaddencreate