Unsure how to re-produce. Have a CoreDump file with the below backtrace. Our module is socc, we were running a Twisted application and Python core dumped giving us the following backtrace.

Looks like we failed to spawn a process wsing os.fork() due to a double free in PyThread_ReInitTLS.

System Details:
CentOS 6.5
Kernel - 2.6.32-696.el6.x86_64
Python 2.7.13 (default, Feb  6 2017, 15:27:35) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-10)] on linux2
GDB - GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el6


File "/usr/local/lib/python2.7/site-packages/socc/cloudTransport/transport.py", line 128, in executeProcessInThread

line +128
ret = reactor.spawnProcess(execProtocol, args[0], args, env=os.environ, childFDs={0: "w", 1: "r", 2: "r"})


Python backtrace:

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/process.py", line 382, in _fork
    self.pid = os.fork()
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/process.py", line 677, in __init__
    self._fork(path, uid, gid, executable, args, environment, fdmap=fdmap)
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 345, in spawnProcess
    processProtocol, uid, gid, childFDs)
  File "/usr/local/lib/python2.7/site-packages/socc/cloudTransport/transport.py", line 128, in executeProcessInThread
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 196, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/local/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/threading.py", line 774, in __bootstrap
    self.__bootstrap_inner()

/* Forget everything not associated with the current thread id.
 * This function is called from PyOS_AfterFork().  It is necessary
 * because other thread ids which were in use at the time of the fork
 * may be reused for new threads created in the forked process.
 */
void
PyThread_ReInitTLS(void)
{
    long id = PyThread_get_thread_ident();
    struct key *p, **q;

    if (!keymutex)
        return;

    /* As with interpreter_lock in PyEval_ReInitThreads()
       we just create a new lock without freeing the old one */
    keymutex = PyThread_allocate_lock();

    /* Delete all keys which do not match the current thread id */
    q = &keyhead;
    while ((p = *q) != NULL) {
        if (p->id != id) {
            *q = p->next;
            free((void *)p); <-------------------------FAILED HERE
            /* NB This does *not* free p->value! */
        }
	else
            q = &p->next;
    }
}

Installed
python27-Logbook-debuginfo-0.9.1-1.el6_7.x86_64.rpm
python27-PyYAML-debuginfo-3.12-1.x86_64.rpm
python27-Twisted-debuginfo-14.0.0-1.x86_64.rpm
python27-cffi-debuginfo-1.9.1-1.el6_7.x86_64.rpm
python27-cryptography-debuginfo-1.7.2-1.el6_7.x86_64.rpm
python27-debuginfo-2.7.13-2.el6_7.x86_64.rpm
python27-psutil-debuginfo-5.1.3-1.x86_64.rpm
python27-simplejson-debuginfo-3.10.0-1.el6_7.el6_7.x86_64.rpm
python27-zope-interface-debuginfo-4.3.3-1.x86_64.rpm

Backtrace is:

#0  0x00007ffff6db0625 in raise () from /lib64/libc.so.6
#1  0x00007ffff6db1e05 in abort () from /lib64/libc.so.6
#2  0x00007ffff6dee537 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff6df3f4e in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff6df6cad in _int_free () from /lib64/libc.so.6
#5  0x00007ffff7b082a9 in PyThread_ReInitTLS () at Python/thread.c:413
#6  0x00007ffff7b0fbb4 in PyOS_AfterFork () at Modules/signalmodule.c:1004
#7  0x00007ffff7b12cf6 in posix_fork (self=<optimized out>, noargs=<optimized out>) at Modules/posixmodule.c:3875
#8  0x00007ffff7ac8460 in call_function (oparg=<optimized out>, pp_stack=0x7fffdc043fb0) at Python/ceval.c:4336
#9  PyEval_EvalFrameEx (f=f@entry=0x36bb930, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#10 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=7, kws=kws@entry=0x36b54d0, kwcount=1, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3584
#11 0x00007ffff7ac8800 in fast_function (nk=<optimized out>, na=7, n=<optimized out>, pp_stack=0x7fffdc0441f0, func=<optimized out>) at Python/ceval.c:4447
#12 call_function (oparg=<optimized out>, pp_stack=0x7fffdc0441f0) at Python/ceval.c:4372
#13 PyEval_EvalFrameEx (f=f@entry=0x36b5270, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#14 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffdeaa5e38, argcount=10, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x7fffeb543068, 
    defcount=defcount@entry=3, closure=0x0) at Python/ceval.c:3584
#15 0x00007ffff7a40760 in function_call (func=0x7fffeb429398, arg=0x7fffdeaa5e20, kw=0x0) at Objects/funcobject.c:523
#16 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffeb429398, arg=arg@entry=0x7fffdeaa5e20, kw=kw@entry=0x0) at Objects/abstract.c:2547
#17 0x00007ffff7a1cec5 in instancemethod_call (func=0x7fffeb429398, arg=0x7fffdeaa5e20, kw=0x0) at Objects/classobject.c:2602
#18 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffdc068820, arg=arg@entry=0x7fffea19c2d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#19 0x00007ffff7a7c5bf in slot_tp_init (self=0x7fffdc059710, args=0x7fffea19c2d0, kwds=0x0) at Objects/typeobject.c:5806
#20 0x00007ffff7a78ecf in type_call (type=<optimized out>, args=0x7fffea19c2d0, kwds=0x0) at Objects/typeobject.c:765
#21 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0xb89410, arg=arg@entry=0x7fffea19c2d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#22 0x00007ffff7ac452e in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffdc044820, func=<optimized out>) at Python/ceval.c:4569
#23 call_function (oparg=<optimized out>, pp_stack=0x7fffdc044820) at Python/ceval.c:4374
#24 PyEval_EvalFrameEx (f=f@entry=0x36b0c00, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#25 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=4, kws=kws@entry=0x36acac0, kwcount=2, defs=0x7fffedb795a8, defcount=7, 
    closure=0x0) at Python/ceval.c:3584
#26 0x00007ffff7ac8800 in fast_function (nk=<optimized out>, na=4, n=<optimized out>, pp_stack=0x7fffdc044a60, func=<optimized out>) at Python/ceval.c:4447
#27 call_function (oparg=<optimized out>, pp_stack=0x7fffdc044a60) at Python/ceval.c:4372
#28 PyEval_EvalFrameEx (f=f@entry=0x36ac900, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#29 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffdc068928, argcount=3, kws=kws@entry=0x7ffff7fa4068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
---Type <return> to continue, or q <return> to quit---
    defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3584
#30 0x00007ffff7a40835 in function_call (func=0x7fffdc0af140, arg=0x7fffdc068910, kw=0x7fffdc060c58) at Objects/funcobject.c:523
#31 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffdc0af140, arg=arg@entry=0x7fffdc068910, kw=kw@entry=0x7fffdc060c58) at Objects/abstract.c:2547
#32 0x00007ffff7ac38d2 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffdc044d70, func=0x7fffdc0af140) at Python/ceval.c:4666
#33 PyEval_EvalFrameEx (f=f@entry=0x7fffdc078238, throwflag=throwflag@entry=0) at Python/ceval.c:3028
#34 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffdeac0ea0, argcount=6, kws=kws@entry=0x7ffff7fa4068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3584
#35 0x00007ffff7a40835 in function_call (func=0x7fffecf8f050, arg=0x7fffdeac0e88, kw=0x7fffdc07ad70) at Objects/funcobject.c:523
#36 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffecf8f050, arg=arg@entry=0x7fffdeac0e88, kw=kw@entry=0x7fffdc07ad70) at Objects/abstract.c:2547
#37 0x00007ffff7ac38d2 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffdc045070, func=0x7fffecf8f050) at Python/ceval.c:4666
#38 PyEval_EvalFrameEx (f=f@entry=0x7fffdc07b620, throwflag=throwflag@entry=0) at Python/ceval.c:3028
#39 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffdeac0f08, argcount=6, kws=kws@entry=0x7ffff7fa4068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3584
#40 0x00007ffff7a40835 in function_call (func=0x7fffecf8f230, arg=0x7fffdeac0ef0, kw=0x7fffdc054d70) at Objects/funcobject.c:523
#41 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffecf8f230, arg=arg@entry=0x7fffdeac0ef0, kw=kw@entry=0x7fffdc054d70) at Objects/abstract.c:2547
#42 0x00007ffff7ac38d2 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffdc045370, func=0x7fffecf8f230) at Python/ceval.c:4666
#43 PyEval_EvalFrameEx (f=f@entry=0x36ab6a0, throwflag=throwflag@entry=0) at Python/ceval.c:3028
#44 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffeb4344e8, argcount=1, kws=kws@entry=0x7ffff7fa4068, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3584
#45 0x00007ffff7a40835 in function_call (func=0x7fffdc07f2a8, arg=0x7fffeb4344d0, kw=0x7fffdc061910) at Objects/funcobject.c:523
#46 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffdc07f2a8, arg=arg@entry=0x7fffeb4344d0, kw=kw@entry=0x7fffdc061910) at Objects/abstract.c:2547
#47 0x00007ffff7ac38d2 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffdc045670, func=0x7fffdc07f2a8) at Python/ceval.c:4666
#48 PyEval_EvalFrameEx (f=f@entry=0x7fffdc075cc8, throwflag=throwflag@entry=0) at Python/ceval.c:3028
#49 0x00007ffff7ac891b in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fffdc045800, func=<optimized out>) at Python/ceval.c:4437
#50 call_function (oparg=<optimized out>, pp_stack=0x7fffdc045800) at Python/ceval.c:4372
#51 PyEval_EvalFrameEx (f=f@entry=0x36aaa60, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#52 0x00007ffff7ac891b in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fffdc0459a0, func=<optimized out>) at Python/ceval.c:4437
#53 call_function (oparg=<optimized out>, pp_stack=0x7fffdc0459a0) at Python/ceval.c:4372
#54 PyEval_EvalFrameEx (f=f@entry=0x7fffdc076e50, throwflag=throwflag@entry=0) at Python/ceval.c:2989
#55 0x00007ffff7ac9d0d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fffedb4e8e8, argcount=1, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3584
#56 0x00007ffff7a40760 in function_call (func=0x7fffed216c08, arg=0x7fffedb4e8d0, kw=0x0) at Objects/funcobject.c:523
#57 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffed216c08, arg=arg@entry=0x7fffedb4e8d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
---Type <return> to continue, or q <return> to quit---
#58 0x00007ffff7a1cec5 in instancemethod_call (func=0x7fffed216c08, arg=0x7fffedb4e8d0, kw=0x0) at Objects/classobject.c:2602
#59 0x00007ffff7a0e203 in PyObject_Call (func=func@entry=0x7fffdc068370, arg=arg@entry=0x7ffff7fa4050, kw=<optimized out>) at Objects/abstract.c:2547
#60 0x00007ffff7ac01c7 in PyEval_CallObjectWithKeywords (func=0x7fffdc068370, arg=0x7ffff7fa4050, kw=<optimized out>) at Python/ceval.c:4221
#61 0x00007ffff7b0d9c2 in t_bootstrap (boot_raw=0xdef990) at Modules/threadmodule.c:620
#62 0x00007ffff77a4aa1 in start_thread () from /lib64/libpthread.so.0
#63 0x00007ffff6e6693d in clone () from /lib64/libc.so.6

Core dump available upon request

Unsure if related to:
https://bugs.python.org/issue10517
or
https://bugs.python.org/issue13156

Python 2.7 uses its own implementation of Thread Local Storage: see find_key() in Python/thread.c. This implementation uses a lock and a chained list. A fork only clones the current thread in the child process, all other threads are "removed", so Python has to manually remove all TLS variables of the other threads using PyThread_ReInitTLS().

I don't think that Python 3 is affected by such bug, since Python 3 uses native TLS APIs like pthread pthread_{get,set}specific() on UNIX/BSD.

This bug seems similar to issue #29640. Good news: it contains a fix (not merged yet) :-)

Anoter previous change related to PyThread_ReInitTLS(), issue #13817, unlikely related to your bug.

commit e0e88b0483d73c925e8f1809d24031acd6bfdf01
Author: Charles-François Natali <neologix@free.fr>
Date:   Thu Feb 2 19:57:19 2012 +0100

    Issue #13817: After fork(), reinit the ad-hoc TLS implementation earlier to fix
    a random deadlock when fork() is called in a multithreaded process in debug
    mode, and make PyOS_AfterFork() more robust.

This commit is part of Python 2.7 since Python 2.7.3.

Indeed, #29640 seems remarkably similar. Will attempt to run attached POC code. Thank you very much for your help.

Sorry we didn't really reach a conclusion here, Thomas, but as 2.7 is now EOL I'm going ahead and closing the issue.