classification
Title: itertools.groupby() can fail a C assert()
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: arigo, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2017-05-11 21:10 by arigo, last changed 2017-09-26 20:16 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1557 merged serhiy.storchaka, 2017-05-12 13:41
PR 3770 merged python-dev, 2017-09-26 18:48
PR 3772 merged serhiy.storchaka, 2017-09-26 19:57
Messages (8)
msg293517 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2017-05-11 21:10
This triggers an assert() failure on debug-mode Python (or a leak in release Python):

from itertools import groupby

def f(n):
    print("enter:", n)
    if n == 5:
        list(b)
    print("leave:", n)
    return n != 6

for (k, b) in groupby(range(10), f):
    print(list(b))

With current trunk we get: python: ./Modules/itertoolsmodule.c:303: _grouper_next: Assertion `gbo->currkey == NULL' failed.
msg293519 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-11 22:19
Do you have a suggested fix?
msg293549 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-12 13:46
PR 1557 implements the simplest fix -- remove the assert and use Py_XSETREF() instead of simple assignment. Since the resulting code in _grouper_next() is identical to the code in groupby_next(), it was shared between these two functions.
msg293579 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-12 21:56
At first glance, the suggested PR looks fine.  Unless there is a rush, I would like to hold-off spend more time thinking about Issue 30346 before moving forward with this one.
msg302948 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-25 11:55
Hmm, the sole issue30346 is not enough for fixing this crash. I don't know if there is a simpler fix based on issue30346, but since PR 1557 removes the code duplication I'm going to merge it in any case.
msg303062 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-26 18:47
New changeset c740e4fe8a9bc5815dc18c38d7f7600b128c3c51 by Serhiy Storchaka in branch 'master':
bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (#1557)
https://github.com/python/cpython/commit/c740e4fe8a9bc5815dc18c38d7f7600b128c3c51
msg303065 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-26 19:20
New changeset 69b2dc8637ba924d78f9869be592c5a545510f10 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (GH-1557) (#3770)
https://github.com/python/cpython/commit/69b2dc8637ba924d78f9869be592c5a545510f10
msg303071 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-26 20:15
New changeset d0b9dc33676bdad217d5074954c1b37d4ca54a10 by Serhiy Storchaka in branch '2.7':
[2.7] bpo-30347: Stop crashes when concurrently iterate over itertools.groupby() iterators. (GH-1557). (#3772)
https://github.com/python/cpython/commit/d0b9dc33676bdad217d5074954c1b37d4ca54a10
History
Date User Action Args
2017-09-26 20:16:27serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-09-26 20:15:38serhiy.storchakasetmessages: + msg303071
2017-09-26 19:57:28serhiy.storchakasetpull_requests: + pull_request3759
2017-09-26 19:20:26serhiy.storchakasetmessages: + msg303065
2017-09-26 18:48:15python-devsetkeywords: + patch
pull_requests: + pull_request3757
2017-09-26 18:47:58serhiy.storchakasetmessages: + msg303062
2017-09-25 11:56:06serhiy.storchakasetversions: - Python 3.5
2017-09-25 11:55:57serhiy.storchakasetmessages: + msg302948
2017-09-25 09:21:32rhettingersetassignee: rhettinger -> serhiy.storchaka
2017-05-12 21:56:04rhettingersetmessages: + msg293579
2017-05-12 13:46:47serhiy.storchakasetmessages: + msg293549
stage: needs patch -> patch review
2017-05-12 13:41:26serhiy.storchakasetpull_requests: + pull_request1653
2017-05-11 22:19:56rhettingersetmessages: + msg293519
2017-05-11 21:33:56serhiy.storchakasetversions: + Python 3.5, Python 3.6
nosy: + rhettinger, serhiy.storchaka

assignee: rhettinger
components: + Extension Modules, - Interpreter Core
stage: needs patch
2017-05-11 21:10:21arigocreate