classification
Title: Keeping an exception in cache can segfault the interpreter
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: 45757, mdk, nitishch, serhiy.storchaka, terry.reedy, zunger
Priority: normal Keywords:

Created on 2017-12-24 02:27 by zunger, last changed 2018-03-17 09:28 by zunger. This issue is now closed.

Messages (10)
msg308978 - (view) Author: Yonatan Zunger (zunger) Date: 2017-12-24 02:27
Using the following decorator to memoize a function causes the interpreter to segfault on exit if any exceptions were stored in the cache. Changing the bad line to instead store copy.copy(e) makes the problem go away.

Almost certainly some kind of problem with the GC of the traceback not happening at the moment the interpreter expects it, but I haven't had time to investigate any more deeply yet.

def memoize(keyFunc=lambda x: x, cacheExceptions: bool=True):
    def wrapper(func):
        return _Memo(func, keyFunc, cacheExceptions)
    return wrapper

class _Memo(object):
    def __init__(self, func, keyFunc, cacheExceptions: bool) -> None:
        self.func = func
        self.keyFunc = keyFunc
        self.cacheExceptions = cacheExceptions
        self.lock = threading.Lock()
        self.cache = {}

    def __call__(self, *args, **kwargs) -> Any:
        key = self.keyFunc(*args, **kwargs)
        assert isinstance(key, collections.Hashable)
        with self.lock:
            if key in self.cache:
                value = self.cache[key]
                if isinstance(value, BaseException):
                    raise value
                return value

        try:
            result = self.func(*args, **kwargs)
        except BaseException as e:
            if self.cacheExceptions:
                with self.lock:
                    self.cache[key] = e   # BAD LINE
            six.reraise(*sys.exc_info())
        else:
            with self.lock:
                self.cache[key] = result
            return result
msg309027 - (view) Author: Nitish (nitishch) * Date: 2017-12-25 04:37
What is six.reraise in except block?
msg309028 - (view) Author: Yonatan Zunger (zunger) Date: 2017-12-25 05:26
It's just a clean version of reraise. Part of the six (Py2/Py3
compatibility) library, but pretty useful on its own anyway.

On Sun, Dec 24, 2017 at 8:37 PM, Nitish <report@bugs.python.org> wrote:

>
> Nitish <nitishch@protonmail.com> added the comment:
>
> What is six.reraise in except block?
>
> ----------
> nosy: +nitishch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32421>
> _______________________________________
>
msg309061 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2017-12-26 11:26
Tested with:

    @memoize()
    def foo(x):
        raise Exception("From foo")


    for _ ∈ range(5):
        try:
            foo(42)
        except Exception as err:
            print(err)

using a python 3.5.4, 3.6.4, and master(13a6c09), and was unable to reproduce it, which version of Python are you using? How are you reproducing it?
msg309062 - (view) Author: Yonatan Zunger (zunger) Date: 2017-12-26 11:48
3.6.2. But Nick Coghlan pointed out elsewhere that the underlying issue is
more likely to be one of the other objects being pinned in-memory because
of the traceback, so the exception may actually be just an incidental
thing. I'm going to have to work on a better minimal repro case -- this
particular example had both a subprocess.Popen open at the time and a
message-channel object that holds some os.pipes, so there are all sorts of
fun things that may be causing the trouble.

I'll update the bug when I have a chance to do that and narrow it down a
bit more.

On Tue, Dec 26, 2017 at 3:26 AM, Julien Palard <report@bugs.python.org>
wrote:

>
> Julien Palard <julien+python@palard.fr> added the comment:
>
> Tested with:
>
>     @memoize()
>     def foo(x):
>         raise Exception("From foo")
>
>
>     for _ ∈ range(5):
>         try:
>             foo(42)
>         except Exception as err:
>             print(err)
>
> using a python 3.5.4, 3.6.4, and master(13a6c09), and was unable to
> reproduce it, which version of Python are you using? How are you
> reproducing it?
>
> ----------
> nosy: +mdk
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32421>
> _______________________________________
>
msg309209 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-12-30 00:04
Yonathon, when replying by email, please delete the quoted message, except possibly for a specific line,  as it is redundant noise when posted on the web page.
msg313862 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-03-15 09:41
>     for _ ∈ range(5):

What version of Python are you using?
msg313863 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2018-03-15 09:50
Serhiy: Sorry the ∈ comes from emacs' pretty-mode, I should copy from outside emacs. Version tested are listed in my previous message.
msg313888 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-15 15:45
I repeated Julian's test with installed 3.7.0b2 on Windows 10 and there is no crash when running either from a command line or from an IDLE editor.  So either the issue has been fixed since 3.6.2, or it is specific to some undisclosed system or class of systems, or the specific claim in the title is a misinterpretation of behavior.  In any case, there is currently no possibility of action on our part.

Yonatan, if you have the promised minimal reproducible test case, for current python, please post it, with version and OS.  If not, we should close this ('not a bug' fits best) until there is one.
msg313990 - (view) Author: Yonatan Zunger (zunger) Date: 2018-03-17 09:28
Apologies -- I've been completely swamped and haven't had time to build a repro case. I don't think I have enough state on the way I found this to reconstruct it easily right now, so I'll close it and re-open if I find it again.
History
Date User Action Args
2018-03-17 09:28:50zungersetstatus: open -> closed
resolution: not a bug
messages: + msg313990

stage: test needed -> resolved
2018-03-15 15:45:29terry.reedysetmessages: + msg313888
stage: test needed
2018-03-15 09:50:28mdksetmessages: + msg313863
2018-03-15 09:41:01serhiy.storchakasetmessages: + msg313862
2017-12-30 00:04:03terry.reedysetnosy: + terry.reedy
messages: + msg309209
2017-12-26 11:48:18zungersetmessages: + msg309062
2017-12-26 11:26:31mdksetnosy: + mdk
messages: + msg309061
2017-12-25 05:26:16zungersetmessages: + msg309028
2017-12-25 04:37:30nitishchsetnosy: + nitishch
messages: + msg309027
2017-12-24 22:22:1545757setnosy: + 45757
2017-12-24 08:20:57serhiy.storchakasetnosy: + serhiy.storchaka
2017-12-24 02:27:03zungercreate