I got a weird crash in an interpreter session. Here is what I did:

$ ./python 
Python 3.5.0a0 (default:fd658692db3a+, Oct 15 2014, 23:13:43) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('toto', 'ab')
>>> f.write(b'bb')
2
>>> f.flush()
>>> 
>>> f = open('toto', 'ab')
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='toto'>
python: Objects/dictobject.c:855: insertdict: Assertion `ep->me_key != ((void *)0) && ep->me_key != (&_dummy_struct)' failed.
Abandon (core dumped)

Here are the inner frames of the traceback:

(gdb) bt
#0 0x00007f27e691df77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f27e69215e8 in __GI_abort () at abort.c:90
#2 0x00007f27e6916d43 in __assert_fail_base (fmt=0x7f27e6a6df58 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0x6878d0 "ep->me_key != ((void *)0) && ep->me_key != (&dummy_struct)", file=file@entry=0x6874f2 "Objects/dictobject.c",
line=line@entry=855, function=function@entry=0x6880a0 <PRETTY_FUNCTION.10152> "insertdict") at assert.c:92
#3 0x00007f27e6916df2 in __GI___assert_fail (assertion=0x6878d0 "ep->me_key != ((void *)0) && ep->me_key != (&dummy_struct)",
file=0x6874f2 "Objects/dictobject.c", line=855, function=0x6880a0 <PRETTY_FUNCTION.10152> "insertdict") at assert.c:101
#4 0x00000000004b65d0 in insertdict (mp=0x7f27e76f9838, key='f', hash=-9123380860235530973, value=<io.BufferedWriter at remote 0x7f27e766e758>)
at Objects/dictobject.c:855
#5 0x00000000004b752a in PyDict_SetItem (
op={'f': <io.BufferedWriter at remote 0x7f27e766e758>, '__builtins': <module at remote 0x7f27e7750358>, '__spec': None, '__warningregistry__': {'version': 0}, '__doc__': None, 'rlcompleter': <module at remote 0x7f27e5f0b358>, '__name__': '__main__', '__cached__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/antoine/.pythonrc.py') at remote 0x7f27e765d468>, 'readline': <module at remote 0x7f27e5f0b058>},
key='f', value=<_io.BufferedWriter at remote 0x7f27e766e758>) at Objects/dictobject.c:1245
#6 0x00000000005a9f7c in PyEval_EvalFrameEx (f=Frame 0x7f27e7704d78, for file <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:2065

Here are the hash initialization values:

(gdb) p _Py_HashSecret
$1 = {uc = "\260\306\vA\a\342\274\337\341\026\003\bbq\366\f\"\032E\232jb%\023", fnv = {prefix = -2324734786846079312, suffix = 934058638581110497},
siphash = {k0 = 16122009286863472304, k1 = 934058638581110497}, djbx33a = {padding = "\260\306\vA\a\342\274\337\341\026\003\bbq\366\f",
suffix = 1379617070853200418}, expat = {padding = "\260\306\vA\a\342\274\337\341\026\003\bbq\366\f", hashsalt = 1379617070853200418}}
(gdb) p PyHash_Func
$2 = {hash = 0x5ee557 <siphash24>, name = 0x6b2020 "siphash24", hash_bits = 64, seed_bits = 128}

The crash seems difficult to reproduce simply by typing the lines above. Perhaps by forcing the hash initialization values as above.

Ok, here is how to reproduce deterministically:

1. apply randomsecret.patch to hardcode the siphash initialization values
2. run "cat dictcrash.py | ./python -i":

Python 3.5.0a0 (default:030fda7b1de8+, Oct 16 2014, 14:27:32) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> >>> >>> >>> __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='x'>
python: Objects/dictobject.c:855: insertdict: Assertion `ep->me_key != ((void *)0) && ep->me_key != (&_dummy_struct)' failed.
Abandon

Looking down into the insertdict frame:

(gdb) p key
$14 = 'f'
(gdb) p old_value
$15 = <unknown at remote 0x7f27e766e678>
(gdb) p *ep
$16 = {me_hash = -3761688987579986997, me_key = 0x0, me_value = 0x0}

So this seems to have hit the following line in insertdict():

    if (old_value != NULL) {
        assert(ep->me_key != NULL && ep->me_key != dummy);
        *value_addr = value;
--->    Py_DECREF(old_value); /* which **CAN** re-enter */
    }

And it *has* reentered because the ResourceWarning inserted the __warningregistry__ attribute into the __main__ dict.

Note the reentrancy looks safe: the dict should be in a stable state at this point (?). But the assert at the end of insertdict assumes the dict hasn't mutated... Why?

So after moving away that specific assert, there doesn't seem to be any problem (I ran the whole test suite in the same interpreter).

If it's really a case of the assert being too strict, then the issue isn't very severe, as most people would you a non-debug build.

The assertion on line 855
is a duplicate of the assertion on line 821, which is just before the reentrant DECREF.
The assertion on line 821 appears to be correct.

I don't know why the assertion is at the end of the function rather than earlier, but hg blame says its my code :(

I'll put a patch together this weekend, once I've had time to reproduce the failure.

Here is a patch, with a test that fails deterministically without the fix.

Le 16/10/2014 15:43, Mark Shannon a écrit :

I'll put a patch together this weekend, once I've had time to reproduce the failure.

Sorry, I was already writing mine when you said that :-)

Crash when running test_reentrant_insertion is reproducible also in 3.3 branch, but not in 3.2 branch.
insertdict.patch applies and works in 3.3 branch.

But 3.3 only receives security fixes, and this is not a security issue. It's up to Georg whether we really wants to backport the fix.

This is not a crash, but an abort, so it's not a security issue.

New changeset 9ec84f9b61c6 by Antoine Pitrou in branch '3.4':
Issue bpo-22653: Fix an assertion failure in debug mode when doing a reentrant dict insertion in debug mode.
https://hg.python.org/cpython/rev/9ec84f9b61c6

New changeset 4ff865976bb9 by Antoine Pitrou in branch 'default':
Issue bpo-22653: Fix an assertion failure in debug mode when doing a reentrant dict insertion in debug mode.
https://hg.python.org/cpython/rev/4ff865976bb9

Ok, then I've committed the patch to 3.4 and 3.5.