classification
Title: SIGSEGV (Address boundary error)
Type: crash Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: int(s), float(s) and others may cause segmentation fault
View: 34087
Assigned To: Nosy List: inada.naoki, matrixise, mdk, xiang.zhang, ybon
Priority: normal Keywords:

Created on 2018-08-08 06:47 by ybon, last changed 2018-08-27 11:42 by inada.naoki. This issue is now closed.

Messages (8)
msg323263 - (view) Author: Yohan Boniface (ybon) Date: 2018-08-08 06:47
Hi!

Just installed 3.7 (ArchLinux) and I've a SIGSEGV on one of my projects.
I've a hard time reducing to a minimal testcase, because it seems whatever random piece of code I remove the crash disappears at some point.

Here is the repository:

https://framagit.org/ybon/trefle

To reproduce, install the project in a 3.7 venv with `python setup.py develop` then run `python trefle/bin.py` (or even `python -c 'from trefle import routine'`).

Here is the output I have:

```
Initializing config
Done initializing config
fish: “python trefle/bin.py” terminated by signal SIGSEGV (Address boundary error)
```

Here are some elements:

- if I run the code with PYTHONMALLOC=debug, I have no crash
- the project is using quite a lot of unicode (French content written in config files), even in some file names
- the project is using asyncio (but it does not seem directly related at first look)
- it is running without issue as is on python 3.6

Here is a gdb backtrace:

```
$ gdb python                                                                                                                                GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run trefle/bin.py 
Starting program: /home/ybon/.virtualenvs/trefle/bin/python trefle/bin.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Initializing config
Done initializing config

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff791a9ce in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
(gdb) backtrace 
#0  0x00007ffff791a9ce in PyObject_Malloc () from /usr/lib/libpython3.7m.so.1.0
#1  0x00007ffff79fec6e in ?? () from /usr/lib/libpython3.7m.so.1.0
#2  0x00007ffff7a05874 in PyParser_ASTFromStringObject () from /usr/lib/libpython3.7m.so.1.0
#3  0x00007ffff7a693f2 in Py_CompileStringObject () from /usr/lib/libpython3.7m.so.1.0
#4  0x00007ffff7a695c3 in ?? () from /usr/lib/libpython3.7m.so.1.0
#5  0x00007ffff795963f in _PyMethodDef_RawFastCallDict () from /usr/lib/libpython3.7m.so.1.0
#6  0x00007ffff79597d1 in _PyCFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#7  0x00007ffff79f7e16 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#8  0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#9  0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#10 0x00007ffff79f3142 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#11 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#12 0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#13 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#14 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#15 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#16 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#17 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#18 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#19 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#20 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#21 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#22 0x00007ffff793a08b in _PyFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#23 0x00007ffff7949888 in ?? () from /usr/lib/libpython3.7m.so.1.0
#24 0x00007ffff79b71b9 in _PyObject_CallMethodIdObjArgs () from /usr/lib/libpython3.7m.so.1.0
#25 0x00007ffff792e285 in PyImport_ImportModuleLevelObject () from /usr/lib/libpython3.7m.so.1.0
#26 0x00007ffff79f4434 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#27 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#28 0x00007ffff7939f34 in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#29 0x00007ffff7939f5c in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#30 0x00007ffff7a05a64 in ?? () from /usr/lib/libpython3.7m.so.1.0
#31 0x00007ffff7959709 in _PyMethodDef_RawFastCallDict () from /usr/lib/libpython3.7m.so.1.0
#32 0x00007ffff79597d1 in _PyCFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#33 0x00007ffff79f7e16 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#34 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#35 0x00007ffff7980982 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#36 0x00007ffff79f6933 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#37 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#38 0x00007ffff79f2225 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#39 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#40 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#41 0x00007ffff79807db in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#42 0x00007ffff79f23cd in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#43 0x00007ffff793a08b in _PyFunction_FastCallDict () from /usr/lib/libpython3.7m.so.1.0
#44 0x00007ffff7949888 in ?? () from /usr/lib/libpython3.7m.so.1.0
#45 0x00007ffff79b71b9 in _PyObject_CallMethodIdObjArgs () from /usr/lib/libpython3.7m.so.1.0
#46 0x00007ffff792e285 in PyImport_ImportModuleLevelObject () from /usr/lib/libpython3.7m.so.1.0
#47 0x00007ffff79f4434 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#48 0x00007ffff7939069 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#49 0x00007ffff7939f34 in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#50 0x00007ffff7939f5c in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#51 0x00007ffff7a68770 in ?? () from /usr/lib/libpython3.7m.so.1.0
#52 0x00007ffff7a6a54a in PyRun_FileExFlags () from /usr/lib/libpython3.7m.so.1.0
#53 0x00007ffff7a6bac5 in PyRun_SimpleFileExFlags () from /usr/lib/libpython3.7m.so.1.0
#54 0x00007ffff7a6da8f in ?? () from /usr/lib/libpython3.7m.so.1.0
#55 0x00007ffff7a6e420 in _Py_UnixMain () from /usr/lib/libpython3.7m.so.1.0
#56 0x00007ffff7dc9003 in __libc_start_main () from /usr/lib/libc.so.6
#57 0x000055555555477a in _start ()
```

Thanks for your help on tracking this! :)

Yohan
msg323267 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-08-08 07:30
Can reproduce with python3.7 from Debian packages, but can't reproduce with a python3.7 built with --with-pydebug.
msg323270 - (view) Author: Stéphane Wirtel (matrixise) * (Python triager) Date: 2018-08-08 09:50
With the last revision of 3.7 (w/o --with-debug), I don't get this issue on Fedora 28 :/
msg323271 - (view) Author: Stéphane Wirtel (matrixise) * (Python triager) Date: 2018-08-08 09:51
and what's the issue with asyncio ?
msg323272 - (view) Author: Yohan Boniface (ybon) Date: 2018-08-08 10:46
Thanks all :)

As noted by Julien, to reproduce the test cases, one also needs to install the dev requirements (or just `pip instal minicli hupper`):

pip install -r requirements-dev.txt

> and what's the issue with asyncio ?

Nothing specific as far as I can tell.
I mentioned it because not all projects use asyncio so I thought it was an significative point to have in mind, just in case.
Also I blindly checked "asyncio" in the "Components" because the project does use asyncio, without foreseeing that this would point asyncio as a guilty and that asyncio maintainers would have been specifically CCed.
Thanks for fixing this.

One other thing to notice just in case: the code base do have unicode chars (in comments, strings and raw strings).
msg323304 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-08-08 21:52
After noticing that without pydebug I can reproduce in v3.7.0 but not in master I ran a git bisect, the following commit looks like it fixes the issue:

```
commit 16dfca4d829e45f36e71bf43f83226659ce49315
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Sat Jul 14 12:06:43 2018 +0900

    bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274)
    
    `_PyUnicode_TransformDecimalAndSpaceToASCII()` missed trailing NUL char.
    It caused buffer overflow in `_Py_string_to_number_with_underscores()`.
    
    This bug is introduced in 9b6c60cb.
```
msg324048 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2018-08-25 07:44
So this is a dupliate of #34087 and we can close it?
msg324169 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2018-08-27 11:42
I think so.
History
Date User Action Args
2018-08-27 11:42:54inada.naokisetstatus: open -> closed
superseder: int(s), float(s) and others may cause segmentation fault
messages: + msg324169

resolution: duplicate
stage: resolved
2018-08-25 07:44:32xiang.zhangsetnosy: + inada.naoki, xiang.zhang
messages: + msg324048
2018-08-08 21:52:35mdksetmessages: + msg323304
2018-08-08 10:46:23ybonsetmessages: + msg323272
2018-08-08 09:52:58matrixisesetnosy: - asvetlov, yselivanov
components: - asyncio
2018-08-08 09:51:31matrixisesetmessages: + msg323271
2018-08-08 09:50:09matrixisesetnosy: + matrixise
messages: + msg323270
2018-08-08 07:30:27mdksetnosy: + mdk
messages: + msg323267
2018-08-08 06:47:22yboncreate