classification
Title: parsetok(): Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed, when running get-pip.py
Type: Stage: patch review
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, lukasz.langa, pablogsal, serhiy.storchaka, vstinner
Priority: release blocker Keywords: 3.8regression, patch

Created on 2019-07-16 09:52 by vstinner, last changed 2019-07-29 14:05 by pablogsal.

Files
File name Uploaded Description Edit
get-pip2.py vstinner, 2019-07-16 09:52
Pull Requests
URL Status Linked Edit
PR 14789 closed vstinner, 2019-07-16 10:04
Messages (5)
msg348008 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-16 09:52
bpo-16806 introduced a regression with the following change:

commit 995d9b92979768125ced4da3a56f755bcdf80f6e (refs/bisect/bad)
Author: Anthony Sottile <asottile@umich.edu>
Date:   Sat Jan 12 20:05:13 2019 -0800

    bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

get-pip.py does now crash:

$ ./python get-pip.py --help
python: Parser/parsetok.c:266: parsetok: Assertion `(intptr_t)(int)(a - line_start) == (a - line_start)' failed.
Aborted (core dumped)

Python 3.8 is also affected.

get-pip.py comes from: https://bootstrap.pypa.io/get-pip.py

Reproducing depends if REALLOC() returns a new pointer, see below.

I attached get-pip2.py: simplified version which crash when using "./python get-pip2.py --help". If it doesn't crash, duplicate lines to make the file larger.

--

The root issue seems to be that tok->multi_line_start isn't updated on REALLOC() in tok_nextc():

                newbuf = (char *)PyMem_REALLOC(newbuf,
                                               newsize);
                if (newbuf == NULL) {
                    tok->done = E_NOMEM;
                    tok->cur = tok->inp;
                    return EOF;
                }
                tok->buf = newbuf;
                tok->cur = tok->buf + cur;
                tok->line_start = tok->cur;

I guess that multi_line_start should also be updated there?

                tok->multi_line_start = tok->cur;
msg348027 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-16 16:05
PR 14433 seems to fix this issue and it contains a test.

My PR 14789 fixed get-pip.py, but it introduces another bug: I closed it.
msg348655 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-07-29 13:08
This is marked as release blocker but since BPO-37433 is still in review, I'm releasing 3.0b3 as is. Please resolve this by b4, I will block the last beta on this issue.
msg348660 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-29 13:23
> This is marked as release blocker but since BPO-37433 is still in review, I'm releasing 3.0b3 as is. Please resolve this by b4, I will block the last beta on this issue.

That sounds reasonable to me.

FYI I just approved (but not merged) PR 14433.
msg348663 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-07-29 14:05
I just merged PR 14433
History
Date User Action Args
2019-07-29 14:05:50pablogsalsetmessages: + msg348663
2019-07-29 13:23:11vstinnersetmessages: + msg348660
2019-07-29 13:08:21lukasz.langasetmessages: + msg348655
2019-07-16 16:05:54vstinnersetmessages: + msg348027
2019-07-16 10:04:01vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request14586
2019-07-16 09:52:56vstinnersetfiles: + get-pip2.py
2019-07-16 09:52:31vstinnersetnosy: + inada.naoki, serhiy.storchaka, pablogsal
2019-07-16 09:52:21vstinnercreate