classification
Title: defaultdict raises SystemError, __missing__ returned NULL in thread
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: iter(classmethod, sentinel) broken for Argument Clinic class methods?
View: 30524
Assigned To: vstinner Nosy List: Peter Parente, louielu, serhiy.storchaka, vstinner, xiang.zhang
Priority: critical Keywords:

Created on 2017-05-25 12:21 by Peter Parente, last changed 2017-07-01 22:25 by vstinner. This issue is now closed.

Messages (10)
msg294473 - (view) Author: Peter Parente (Peter Parente) Date: 2017-05-25 12:21
Working on this PR (https://github.com/maxpoint/spylon/pull/49), I encountered an unexpected SystemError in Python 3.6 on my Mac and on Travis Linux. 

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/Users/parente/miniconda3/envs/spylon-dev/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/parente/miniconda3/envs/spylon-dev/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/parente/projects/spylon/spylon/spark/progress.py", line 99, in _spark_progress_thread_worker
    td = datetime.datetime.now() - start_times[stage_id]
SystemError: <built-in method __missing__ of collections.defaultdict object at 0x106013098> returned NULL without setting an error

The exception only occurs in Python 3.6, not in Python 3.5 or 3.4: https://travis-ci.org/maxpoint/spylon/builds/235992988

The defaultdict is both created and used as a local variable in a threading.Thread run() function. It's never accessed outside that context (https://github.com/maxpoint/spylon/pull/49/commits/2bd47dc32f6129f5f6a4824be1eaed568351df11#diff-1ba10d1cd92a35a380442cb586e310e2R179)

I didn't see a related issue on the tracker here nor did I see anything in the 3.6 changelog about a behavior change.
msg294477 - (view) Author: Peter Parente (Peter Parente) Date: 2017-05-25 13:30
Much simpler example: https://gist.github.com/parente/a4772297459f05e43e12a5820051431b

Wrapping the datetime.datetime.now in a lambda avoids the issue.
msg294482 - (view) Author: Louie Lu (louielu) * Date: 2017-05-25 14:37
Another way to reproduce this problem:

>>> import datetime
>>> from collections import defaultdict as dd
>>> d = dd(datetime.datetime.now)
>>> d['foo']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: <built-in method __missing__ of collections.defaultdict object at 0x7f072e55b3b8> returned NULL without setting an error

This problem doesn't appear on Python 2.7, too.
msg294484 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-05-25 14:53
Interestingly I could only reproduce the failure on 3.6 but not master.
msg294497 - (view) Author: Louie Lu (louielu) * Date: 2017-05-25 16:59
This bug is introduce at commit: 37e4ef7b17ce6e98ca725c0a53ae14c313c0e48c, then fixed at commit: 998c20962ca3e2e693c1635efe76c0144dde76fc
msg294504 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-25 17:57
In 3.6 _PyStack_UnpackDict() returns just args if there are no keyword arguments. But if args is NULL, the result of _PyStack_UnpackDict() is treated as error.

Thank you for the reproducer Louie!
msg296485 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 20:34
This bug is a duplicate of bpo-30524.

Louie Lu: "This bug is introduce at commit: 37e4ef7b17ce6e98ca725c0a53ae14c313c0e48c, then fixed at commit: 998c20962ca3e2e693c1635efe76c0144dde76fc"

This is the fix for master. For Python 3.6, the fix is the commit f0ff849adc6b4a01f9d1f08d9ad0f1511ff84541. I added wrote unit tests for this bug, but also for many ways to calls functions: commit b7577456c430283f8b7ec4e914b701cb943cc69b.

> Much simpler example: https://gist.github.com/parente/a4772297459f05e43e12a5820051431b

I'm able to reproduce the bug in Python 3.6 before f0ff849adc6b4a01f9d1f08d9ad0f1511ff84541, and also confirm that this commit fixes the bug.

The good news is that Python 3.6.2rc1 is already available and contains my fix, and 3.6.2 final should be released soon.
msg296487 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-20 20:35
Oh, and thank you Peter Parente for the bug report, and for the script to reproduce the bug. It was usual to validate that the bug is now fixed.
msg297466 - (view) Author: Peter Parente (Peter Parente) Date: 2017-07-01 01:59
Glad to hear it. Cheers, Victor!
msg297498 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-01 22:25
> It was usual to validate that the bug is now fixed.

Crap, I wanted to write: useful, not usual!?
History
Date User Action Args
2017-07-01 22:25:16vstinnersetmessages: + msg297498
2017-07-01 01:59:29Peter Parentesetmessages: + msg297466
2017-06-20 20:35:17vstinnersetmessages: + msg296487
2017-06-20 20:34:13vstinnersetstatus: open -> closed
superseder: iter(classmethod, sentinel) broken for Argument Clinic class methods?
messages: + msg296485

resolution: duplicate
stage: needs patch -> resolved
2017-05-25 17:57:08serhiy.storchakasetpriority: normal -> critical

assignee: vstinner
components: + Interpreter Core

nosy: + serhiy.storchaka
messages: + msg294504
stage: needs patch
2017-05-25 16:59:40louielusetmessages: + msg294497
2017-05-25 14:53:20xiang.zhangsetnosy: + xiang.zhang
messages: + msg294484
2017-05-25 14:37:55louielusetnosy: + louielu
messages: + msg294482
2017-05-25 13:30:34Peter Parentesetmessages: + msg294477
2017-05-25 12:30:57serhiy.storchakasetnosy: + vstinner
2017-05-25 12:21:49Peter Parentecreate