classification
Title: Crash on REPL mode with long text copy and paste
Type: crash Stage: resolved
Components: Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, corona10, methane, miss-islington, pablogsal, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-01-04 14:29 by corona10, last changed 2020-01-06 16:27 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17860 merged pablogsal, 2020-01-06 10:28
PR 17867 merged miss-islington, 2020-01-06 15:59
Messages (5)
msg359290 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-01-04 14:29
When I copy and paste the pretty long text into REPL shell.
REPL shell is crash down with segment fault.

This issue is only reproducible on macOS,
but Linux REPL doesn't look like normal behavior.

[origin text]
<?xml version="1.0" encoding="iso-8859-1"?>
<test>
    <Users>
        <fun25>
            <limits>
                <total>0KiB</total>
                <kbps>0</kbps>
                <rps>1.3</rps>
                <connections>0</connections>
            </limits>
            <usages>
                <total>16738211KiB</total>
                <kbps>237.15</kbps>
                <rps>1.3</rps>
                <connections>0</connections>
            </usages>
            <time_to_refresh>never</time_to_refresh>
            <limit_exceeded_URL>none</limit_exceeded_URL>
        </fun25>
    </Users>
</test>

[macOS]

Python 3.9.0a2+ (heads/master:7dc72b8d4f, Jan  4 2020, 23:22:45)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
... <test>
...     <Users>
...         <fun25>
...             <limits>
...                 <total>0KiB</total>
...                 <kbps>0</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </limits>
...             <usages>
...                 <total>16738211KiB</total>
...                 <kbps>237.15</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </usages>
...             <time_to_refresh>never</time_to_refresh>
...             <limit_exceeded_URL>none</limit_exceeded_URL>
...         </fun25>
...     </Users>
... </test>
... """
Assertion failed: ((intptr_t)(int)(a - line_start) == (a - line_start)), function parsetok, file Parser/parsetok.c, line 324.
[1]    13389 abort      ./python.exe


[linux]

Python 3.9.0a2+ (heads/master-dirty:7dc72b8, Jan  4 2020, 23:22:11)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
<test>
    <Users>
        <fun25>
            <limits>
                <total>0KiB</total>
                <kbps>0</kbps>
                <rps>true</rps>
                <connections>0</connections>
            </limits>
            <usages>
                <total>16738211KiB</total>
                <kbps>237.15</kbps>
                <rps>true</rps>
                <connections>0</connections>
            </usages>
            <time_to_refresh>never</time_to_refresh>
            <limit_exceeded_URL>none</limit_exceeded_URL>
        </fun25>
    </Users>
</test>
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... """
>>> a
'<?xml version="1.0" encoding="iso-8859-1"?>\n<test>\n    <Users>\n        <fun25>\n            <limits>\n                <total>0KiB</total>\n                <kbps>0</kbps>\n                <rps>true</rps>\n                <connections>0</connections>\n            </limits>\n            <usages>\n                <total>16738211KiB</total>\n                <kbps>237.15</kbps>\n                <rps>true</rps>\n                <connections>0</connections>\n            </usages>\n            <time_to_refresh>never</time_to_refresh>\n            <limit_exceeded_URL>none</limit_exceeded_URL>\n        </fun25>\n    </Users>\n</test>\n'
>>>
msg359414 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-06 10:13
I bisected this to this commit:

995d9b92979768125ced4da3a56f755bcdf80f6e is the first bad commit
commit 995d9b92979768125ced4da3a56f755bcdf80f6e
Author: Anthony Sottile <asottile@umich.edu>
Date:   Sat Jan 12 20:05:13 2019 -0800

    bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

 Lib/test/test_ast.py                               | 19 +++++++++++++
 Lib/test/test_fstring.py                           | 32 +++++++++-------------
 Lib/test/test_opcodes.py                           |  2 +-
 Lib/test/test_string_literals.py                   |  8 +++---
 Misc/ACKS                                          |  1 +
 .../2018-10-20-18-05-58.bpo-16806.zr3A9N.rst       |  1 +
 Parser/parsetok.c                                  | 15 ++++++++--
 Parser/tokenizer.c                                 |  7 +++++
 Parser/tokenizer.h                                 |  5 ++++
 Python/ast.c                                       | 10 +++++--
 Python/importlib.h                                 | 16 +++++------
 Python/importlib_external.h                        | 18 ++++++------
 Python/importlib_zipimport.h                       |  8 +++---
 13 files changed, 91 insertions(+), 51 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2018-10-20-18-05-58.bpo-16806.zr3A9N.rst
msg359416 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-01-06 11:47
@pablogsal

Works correct with PR 17860 :)


Python 3.9.0a2+ (heads/pr/17860:958541d67c, Jan  6 2020, 20:45:49)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
... <test>
...     <Users>
...         <fun25>
...             <limits>
...                 <total>0KiB</total>
...                 <kbps>0</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </limits>
...             <usages>
...                 <total>16738211KiB</total>
...                 <kbps>237.15</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </usages>
...             <time_to_refresh>never</time_to_refresh>
...             <limit_exceeded_URL>none</limit_exceeded_URL>
...         </fun25>
...     </Users>
... </test>
... """
>>> a
'<?xml version="1.0" encoding="iso-8859-1"?>\n<test>\n    <Users>\n        <fun25>\n            <limits>\n                <total>0KiB</total>\n                <kbps>0</kbps>\n                <rps>1.3</rps>\n                <connections>0</connections>\n            </limits>\n            <usages>\n                <total>16738211KiB</total>\n                <kbps>237.15</kbps>\n                <rps>1.3</rps>\n                <connections>0</connections>\n            </usages>\n            <time_to_refresh>never</time_to_refresh>\n            <limit_exceeded_URL>none</limit_exceeded_URL>\n        </fun25>\n    </Users>\n</test>\n'
>>>
msg359432 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-06 15:59
New changeset 5ec91f78d59d9c39b984f284e00cd04b96ddb5db by Pablo Galindo in branch 'master':
bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)
https://github.com/python/cpython/commit/5ec91f78d59d9c39b984f284e00cd04b96ddb5db
msg359435 - (view) Author: miss-islington (miss-islington) Date: 2020-01-06 16:26
New changeset b2e281aaa2e4a1f24671d293dfd06b23bb052e47 by Miss Islington (bot) in branch '3.8':
bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)
https://github.com/python/cpython/commit/b2e281aaa2e4a1f24671d293dfd06b23bb052e47
History
Date User Action Args
2020-01-06 16:27:13pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-01-06 16:26:17miss-islingtonsetnosy: + miss-islington
messages: + msg359435
2020-01-06 15:59:37miss-islingtonsetpull_requests: + pull_request17284
2020-01-06 15:59:13pablogsalsetmessages: + msg359432
2020-01-06 11:47:03corona10setmessages: + msg359416
2020-01-06 10:28:11pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request17279
2020-01-06 10:14:41pablogsalsetnosy: + methane
2020-01-06 10:13:32pablogsalsetnosy: + Anthony Sottile
messages: + msg359414
2020-01-04 14:31:31corona10setnosy: + vstinner, serhiy.storchaka
2020-01-04 14:29:32corona10create