Last August I started running a bot for AIX using xlc_r as the compiler, rather than gcc that the other AIX bot uses.

Initially, I had no issues with the test_multiprocess* tests, but of late (last two+ months I am guessing) I have been having regular issues when the bot builds, but not when I would run the tests (all 418) or individually - when run manually.

The last two weeks I have invested time - and have been repaid - in that I can now get a regular failure when running the tests.

Your assistance is appreciated. I'll continue to work on this when I have time.


Short version:

This looks like there is a statement "crafted" to cause a crash:
(dbx) where
PyDict_GetItem(op = 0x2002ddc8, key = 0x30061e68), line 1320 in "dictobject.c"
_PyDict_GetItemId(dp = 0xdbdbdbdb, key = 0xdbdbdbdb), line 3276 in "dictobject.c"
...

This is based on bot run that failed (Python 3.8.0a0 (heads/master:0785889468)).

The test message that comes back is:

test_multiprocessing_fork failed
Process Process-94:
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/_test_multiprocessing.py", line 2847, in _putter
    manager.connect()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/managers.py", line 512, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 796, in XmlClient
    return ConnectionWrapper(Client(*args, **kwds), _xml_dumps, _xml_loads)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 629, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 79] Connection refused
test test_multiprocessing_fork failed -- Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/_test_multiprocessing.py", line 2865, in test_rapid_restart
    queue = manager.get_queue()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/managers.py", line 701, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/managers.py", line 584, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 796, in XmlClient
    return ConnectionWrapper(Client(*args, **kwds), _xml_dumps, _xml_loads)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/multiprocessing/connection.py", line 629, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 79] Connection refused

I had a hard time finding anything - because I was looking for a permission issue in the Socket "domain", but what seems more likely is that the "server" thread/process is crashing with a segmentation fault and a "client" thread is getting "refused" because the server no longer exists and/or never got successfully started.

I finally managed to capture a core dump and I hope that this will help you help me with getting deeper and closer to a resolution/understanding on what is going on.

buildbot@x064:[/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue]area/coredumps/core.12582914.25123239                        <
Type 'help' for help.
warning: The core file is not a fullcore. Some info may
not be available.
[using memory image in /home/buildbot/buildarea/coredumps/core.12582914.25123239]
reading symbolic information ...

Segmentation fault in PyDict_GetItem at line 1320 in file "Objects/dictobject.c"
 1320       if (!PyDict_Check(op))
(dbx) where
PyDict_GetItem(op = 0x2002ddc8, key = 0x30061e68), line 1320 in "dictobject.c"

From other data about the program I expect the segmentation error is caused by the key value. Unless the program has done a mmap/shmap request for memory allocation (something not done by default) the address 0x30000000-0x3fffffff is not a valid address.

Summary:

This looks like there is a statement "crafted" to cause a crash:
(dbx) where
PyDict_GetItem(op = 0x2002ddc8, key = 0x30061e68), line 1320 in "dictobject.c"
_PyDict_GetItemId(dp = 0xdbdbdbdb, key = 0xdbdbdbdb), line 3276 in "dictobject.c"
...

Gory details:
(dbx) where
PyDict_GetItem(op = 0x2002ddc8, key = 0x30061e68), line 1320 in "dictobject.c"
_PyDict_GetItemId(dp = 0xdbdbdbdb, key = 0xdbdbdbdb), line 3276 in "dictobject.c"
unnamed block in _PyEval_EvalFrameDefault(f = 0x1000eb04, throwflag = 807503860), line 957 in "ceval.c"
unnamed block in _PyEval_EvalFrameDefault(f = 0x1000eb04, throwflag = 807503860), line 957 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1000eb04, throwflag = 807503860), line 957 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100111d0, throwflag = 537350752), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x1009f4b0, globals = 0x202f476c, locals = (nil), args = (nil), argcount = 24, kwnames = 0x30211d88, kwargs = 0x202f4700, kwcount = -2013255516, kwstep = 2, defs = 0x302187f4, defcount = 1, kwdefs = (nil), closure = (nil), name = 0x300022e8, qualname = 0x30214928), line 3969 in "ceval.c"
_PyFunction_FastCallDict(func = 0x1010ac50, args = 0x20021c28, nargs = 539970704, kwargs = 0x820028af), line 380 in "call.c"
_PyObject_FastCallDict(callable = 0x10209a1c, args = 0x30a17028, nargs = 539969536, kwargs = 0x2820424f), line 100 in "call.c"
_PyObject_Call_Prepend(callable = 0x100206ac, obj = 0x20077e84, args = 0x202f4880, kwargs = 0x20088734), line 906 in "call.c"
slot_tp_init(self = 0x100de870, args = 0x00000074, kwds = 0x20126740), line 6638 in "typeobject.c"
unnamed block in type_call(type = 0x100d9ed8, args = 0x20021c28, kwds = 0x202f4940), line 954 in "typeobject.c"
type_call(type = 0x100d9ed8, args = 0x20021c28, kwds = 0x202f4940), line 954 in "typeobject.c"
_PyObject_FastCallKeywords(callable = 0x30665450, stack = 0x3008c1b0, nargs = 805573048, kwnames = 0x103521ac), line 201 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 807520936), line 4658 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de870, throwflag = 813135108), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3021dbf8, globals = 0x2008d708, locals = 0xcfb98979, args = 0x30037e30, argcount = 807520936, kwnames = 0x3021a4f0, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = 0x30218a04, defcount = 1, kwdefs = (nil), closure = (nil), name = 0x3021caa8, qualname = 0x3021caa8), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x202f50b8, stack = 0x300d3ef8, nargs = 539971696, kwnames = 0x222022cf), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x20089a38, throwflag = 807373136), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = (nil), throwflag = 807476128), line 581 in "ceval.c"
function_code_fastcall(co = 0x3021df78, args = 0x2008d708, nargs = 538317872, globals = 0x00000034), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x3091f5a0, stack = 0x30041b50, nargs = 815894632, kwnames = 0x30142e64), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 807366808), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100cc074, throwflag = 806932108), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x202f61bc, globals = 0x3021a570, locals = 0x202f5d10, args = 0x30037e30, argcount = 269083832, kwnames = 0x301f49f0, kwargs = 0x202f5d10, kwcount = 537429812, kwstep = 1, defs = 0x3022123c, defcount = 2, kwdefs = (nil), closure = (nil), name = 0x301f7098, qualname = 0x301f7098), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x1018c538, stack = 0x30a158e8, nargs = 537037592, kwnames = 0x30142e64), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805546776), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100cc074, throwflag = 807787624), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x202f686c, globals = 0x301f4c70, locals = 0x202f63c0, args = 0x30037e30, argcount = 269083832, kwnames = 0x301edbf0, kwargs = 0x202f63c0, kwcount = 537429812, kwstep = 1, defs = 0x30218464, defcount = 1, kwdefs = (nil), closure = (nil), name = 0x3003ab18, qualname = 0x3003ab18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x1003470c, stack = 0x200058f8, nargs = 539976896, kwnames = 0x20088734), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100c3c64, throwflag = 807867476), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x301f19e0, throwflag = 807867476), line 581 in "ceval.c"
function_code_fastcall(co = 0x301f5418, args = 0x2008d708, nargs = -796895527, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x1018c538, stack = 0x3096a9b8, nargs = 814795680, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x3091e488, throwflag = 805546776), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100cc074, throwflag = 810945964), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x202f758c, globals = 0x301edc70, locals = 0x202f70e0, args = 0x15e8c00d, argcount = 269083832, kwnames = 0x309ec970, kwargs = 0x202f70e0, kwcount = 537429812, kwstep = 1, defs = 0x301f3884, defcount = 1, kwdefs = (nil), closure = (nil), name = 0x3003ab18, qualname = 0x3003ab18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x3091f540, stack = 0x30005840, nargs = 815871240, kwnames = 0x42204842), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100c1d90, throwflag = 537410828), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 814835560), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x309ec970, locals = 0x202f7770, args = 0x20088734, argcount = 269346004, kwnames = 0x20021c28, kwargs = 0x202f7770, kwcount = 815881976, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x2007d04c, locals = 0x202f77e0, args = 0x200058f8, argcount = 269237348, kws = 0x300ba1a8, kwcount = 539981808, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x30a15f00, globals = 0x300ba1ac, locals = 0x202f7820), line 558 in "ceval.c"
builtin_exec(module = 0x1000eb04, args = (nil), nargs = 539981952, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x100c6724, args = 0x300ba1ac, nargs = 0), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x100a0d3c, self = 0x103461d0, args = (nil), nargs = 815035112, kwargs = 0x309435f0), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x3007f8a8, args = 0x309ec9b0, nargs = 539982304, kwargs = 0x220048cf), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = (nil), kwargs = 0x300927d0), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 815019488), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x100c1d90, stack = 0x200058f8, nargs = 539984160, kwnames = 0x4420228f), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 814745524), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 815712688), line 581 in "ceval.c"
function_code_fastcall(co = 0x202f8bdc, args = 0x3096ecf0, nargs = 539985712, globals = 0x309007ac), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x3096ed20, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x00000049), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 815253484), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 815253488), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x10010cf8, stack = 0x300cf030, nargs = 805768508, kwnames = 0x4420228f), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 814745132), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 814745132), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x00000030, stack = 0x202dc120, nargs = -260629964, kwnames = 0x202d8840), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000008, throwflag = 815253044), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 537009192), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = 0x200058f8, nargs = 539990624, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x3002dfa8, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 539990896, kwargs = 0x302dc830), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = "0-\307\3600-\307\260"), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x302beee8, 0x3003a7a0, 0x0, 0x4ea54, 0xdb, 0x30828de0), line 1250 in "call.c"
import_find_and_load(abs_name = 0x309044f8), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x302dc428, globals = 0x309ecaf0, locals = 0x202f9d20, fromlist = 0x300bc190, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x302dc428, globals = 0x309ecaf0, locals = 0x202f9d20, fromlist = 0x300bc190, level = 269346004), line 1748 in "import.c"
import_name(f = 0x100c5e54, name = 0x10338aa8, fromlist = 0x202f9d80, level = 0x20088734), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x10010cf8, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815072624), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x309ecaf0, locals = 0x202fa3b0, args = 0x20088734, argcount = 269346004, kwnames = 0x10338aa8, kwargs = 0x309e8e60, kwcount = 537434348, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x200058f8, locals = 0x202fa450, args = 0x20088734, argcount = 269246788, kws = (nil), kwcount = 539993120, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x00000003, globals = 0x300ba1ac, locals = (nil)), line 558 in "ceval.c"
builtin_exec(module = 0x10337030, args = (nil), nargs = 539993344, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x00000003, args = 0x307bbe28, nargs = 805795944), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x3007df24, self = 0x103461d0, args = (nil), nargs = 815035480, kwargs = 0x30943478), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x00000004, args = 0x0000008d, nargs = 806068652, kwargs = 0x30089ad8), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = 0x300ba1a8, kwargs = 0x30031290), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 815019112), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x100c1d90, stack = 0x200058f8, nargs = 539995488, kwnames = 0x4420228f), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 814744724), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 815713072), line 581 in "ceval.c"
function_code_fastcall(co = 0x202fb81c, args = 0x3096e990, nargs = 539997040, globals = 0x3090048c), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x3096e900, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x0000004f), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 815252260), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 815252264), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x10010cf8, stack = 0x300cf030, nargs = 805768508, kwnames = 0x4420228f), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 814744332), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 814744332), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x00000002, stack = (nil), nargs = 538317872, kwnames = 0x304e9d30), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000006, throwflag = 815035836), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 537028216), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = 0x20161430, nargs = 540001952, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x30828de0, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540002224, kwargs = 0x20088734), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = "0N\235\2600N\235p"), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x304a86e8, 0x3003a7a0, 0x0, 0x4e71c, 0xdb, 0x30828de0), line 1250 in "call.c"
import_find_and_load(abs_name = 0x3097dbf8), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x30286ee8, globals = 0x309ecc70, locals = 0x202fc960, fromlist = 0x300bc190, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x30286ee8, globals = 0x309ecc70, locals = 0x202fc960, fromlist = 0x300bc190, level = 269346004), line 1748 in "import.c"
import_name(f = 0x100c5e54, name = 0x10338aa8, fromlist = 0x202fc9c0, level = 0x20088734), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x10010cf8, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815071504), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x309ecc70, locals = 0x202fcff0, args = 0x20088734, argcount = 269346004, kwnames = 0x10338aa8, kwargs = 0x309e8f78, kwcount = 537434348, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x200058f8, locals = 0x202fd090, args = 0x20088734, argcount = 269246788, kws = (nil), kwcount = 540004448, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x00000003, globals = 0x300ba1ac, locals = (nil)), line 558 in "ceval.c"
builtin_exec(module = 0x10337030, args = (nil), nargs = 540004672, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x00000003, args = 0x307bbca8, nargs = 805795944), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x3007df24, self = 0x103461d0, args = (nil), nargs = 815036216, kwargs = 0x306a9bd0), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x00000004, args = 0x0000008d, nargs = 806068652, kwargs = 0x30089ad8), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = 0x300ba1a8, kwargs = 0x30031290), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 812293056), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x100c1d90, stack = 0x200058f8, nargs = 540006816, kwnames = 0x4420228f), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 812375540), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 815713456), line 581 in "ceval.c"
function_code_fastcall(co = 0x202fe45c, args = 0x3096e3c0, nargs = 540008368, globals = 0x306bddec), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x3096e270, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x0000004e), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 815251852), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 815251856), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x10010cf8, stack = 0x300cf030, nargs = 805768508, kwnames = 0x4420228f), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812375148), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 812375148), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x00000002, stack = (nil), nargs = 538317872, kwnames = 0x304e9d30), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000005, throwflag = 812342084), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 537028216), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = 0x20161430, nargs = 540013280, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x3082e5a0, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540013552, kwargs = 0x20088734), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = "0N\235\2600N\235p"), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x3091cfa0, 0x3003a7a0, 0x0, 0x4e40f, 0xdb, 0x3082e5a0), line 1250 in "call.c"
import_find_and_load(abs_name = 0x30977e28), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x30097df0, globals = 0x309ecdf0, locals = 0x202ff5a0, fromlist = 0x442042c4, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x30097df0, globals = 0x309ecdf0, locals = 0x202ff5a0, fromlist = 0x442042c4, level = 269346004), line 1748 in "import.c"
import_name(f = 0x1000eb04, name = 0x10338aa8, fromlist = 0x202ff600, level = 0x200898ec), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x10010cf8, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815011856), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x309ecdf0, locals = 0x202ffc30, args = 0x20088734, argcount = 269346004, kwnames = 0x10338aa8, kwargs = 0x309e8e60, kwcount = 537434348, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x200058f8, locals = 0x202ffcd0, args = 0x20088734, argcount = 269246788, kws = (nil), kwcount = 540015776, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x00000003, globals = 0x300ba1ac, locals = (nil)), line 558 in "ceval.c"
builtin_exec(module = 0x10337030, args = (nil), nargs = 540016000, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x00000003, args = 0x307bbb28, nargs = 805795944), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x3007df24, self = 0x103461d0, args = (nil), nargs = 812148536, kwargs = 0x306a9a58), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x00000004, args = 0x0000008d, nargs = 806068652, kwargs = 0x30089ad8), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = 0x300ba1a8, kwargs = 0x30031290), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 812292680), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x100c1d90, stack = 0x200058f8, nargs = 540018144, kwnames = 0x4420228f), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 812374340), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 815713840), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030109c, args = 0x30973f30, nargs = 540019696, globals = 0x306bd93c), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x30973f60, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x0000004a), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812358836), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 812358840), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x10010cf8, stack = 0x300cf030, nargs = 805768508, kwnames = 0x4420228f), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812373948), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 812373948), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x100c6724, stack = 0x00000311, nargs = 540023088, kwnames = 0x6f636e65), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000005, throwflag = 812195788), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 806149408), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = (nil), nargs = 540024608, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x3082e5a0, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540024880, kwargs = 0x20088734), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = "0^N^Z00^Mop"), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x3091cf58, 0x3003a7a0, 0x0, 0x4e09d, 0xdb, 0x3082e5a0), line 1250 in "call.c"
import_find_and_load(abs_name = 0x30977338), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x3009d568, globals = 0x309ecf70, locals = 0x203021e0, fromlist = 0x482042c4, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x3009d568, globals = 0x309ecf70, locals = 0x203021e0, fromlist = 0x482042c4, level = 269346004), line 1748 in "import.c"
import_name(f = 0x203022f0, name = 0x300652f8, fromlist = 0x20302250, level = 0x4220422f), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x10010cf8, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815201728), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x309ecf70, locals = 0x20302870, args = 0x20088734, argcount = 269346004, kwnames = 0x10338aa8, kwargs = 0x309e87a8, kwcount = 537434348, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x200058f8, locals = 0x20302910, args = 0x20088734, argcount = 269246788, kws = (nil), kwcount = 540027104, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x00000003, globals = 0x300ba1ac, locals = (nil)), line 558 in "ceval.c"
builtin_exec(module = 0x10337030, args = (nil), nargs = 540027328, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x00000003, args = 0x307bba28, nargs = 805795944), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x3007df24, self = 0x103461d0, args = (nil), nargs = 809333848, kwargs = 0x306bd188), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x00000004, args = 0x0000008d, nargs = 806068652, kwargs = 0x30089ad8), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = 0x300ba1a8, kwargs = 0x30031290), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 812372344), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x102097b0, stack = 0x300b9770, nargs = 537028768, kwnames = 0x20088734), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 812356772), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 815714224), line 581 in "ceval.c"
function_code_fastcall(co = 0x20303cdc, args = 0x30946720, nargs = 540031024, globals = 0x306b949c), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x309469c0, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812355980), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 812355984), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x100c6084, stack = 0x00000001, nargs = 540032768, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812325596), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 812325596), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x000084b1, stack = 0x0000405d, nargs = 540164848, kwnames = 0x000084a1), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000001, throwflag = 812314220), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 537009192), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = 0x309b7d50, nargs = 814817192, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x100c30bc, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540036208, kwargs = 0x0000014d), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = warning: Unable to access address 0xdbdbdbdb from core
(invalid char ptr (0xdbdbdbdb))), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x304e9fa8, 0x3003a7a0, 0x0, 0x4deb7, 0xcb, 0x306aee88), line 1250 in "call.c"
import_find_and_load(abs_name = 0x3097abf8), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x30005840, globals = 0x306bc9f0, locals = 0x20304e20, fromlist = 0x20088734, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x30005840, globals = 0x306bc9f0, locals = 0x20304e20, fromlist = 0x20088734, level = 269346004), line 1748 in "import.c"
import_name(f = 0x000083df, name = 0x20304f6c, fromlist = 0x20304e80, level = 0x42204842), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x100c1d90, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815247744), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x306bc9f0, locals = 0x203054b0, args = 0x20088734, argcount = 269346004, kwnames = 0x20021c28, kwargs = 0x203054b0, kwcount = 815697696, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x2007d04c, locals = 0x20305520, args = 0x200058f8, argcount = 269237348, kws = 0x300ba1a8, kwcount = 540038448, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x309e8f28, globals = 0x300ba1ac, locals = 0x20305560), line 558 in "ceval.c"
builtin_exec(module = 0x1000eb04, args = (nil), nargs = 540038592, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x0000001a, args = 0x20077e84, nargs = 815031568), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x100a0d3c, self = 0x103461d0, args = (nil), nargs = 812326392, kwargs = 0x306a9d48), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x3007f8a8, args = 0x306bca70, nargs = 540038944, kwargs = 0x220048cc), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = (nil), kwargs = 0x300927d0), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 812293432), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x1018c538, stack = 0x300b9770, nargs = 540040960, kwnames = 0x20088734), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 812373140), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 812370544), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030691c, args = 0x30946510, nargs = 540042352, globals = 0x306bd48c), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x309466c0, stack = 0x30055ea8, nargs = 537028192, kwnames = 0x300553e8), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812357204), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 812357208), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x100a0d3c, stack = 0x3002c958, nargs = 540044096, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 812372748), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 812372748), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x202d2e20, stack = 0x202d6130, nargs = 805955712, kwnames = 0x20161430), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x00000007, throwflag = 812345196), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 814615216), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = (nil), nargs = 540047264, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x30055b68, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540047536, kwargs = 0x22de5185), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = "0k\312\2600k\312\360"), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x304a8ba8, 0x3003a7a0, 0x0, 0x4d0b7, 0xdb, 0x30653a40), line 1250 in "call.c"
import_find_and_load(abs_name = 0x30960178), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x309a5418, globals = 0x306acf30, locals = 0x20307a60, fromlist = 0x20088734, level = 269346004), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x309a5418, globals = 0x306acf30, locals = 0x20307a60, fromlist = 0x20088734, level = 269346004), line 1748 in "import.c"
import_name(f = 0x00008505, name = 0x20307bac, fromlist = 0x20307ac0, level = 0x42204242), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x100c1d90, throwflag = 537410828), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x00000001, throwflag = 815384032), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x30061e28, globals = 0x306acf30, locals = 0x203080f0, args = 0x20088734, argcount = 269346004, kwnames = 0x20021c28, kwargs = 0x203080f0, kwcount = 814950664, kwstep = 2, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = (nil), qualname = (nil)), line 3969 in "ceval.c"
PyEval_EvalCodeEx(_co = 0x10109cb0, globals = 0x2007d04c, locals = 0x20308160, args = 0x200058f8, argcount = 269237348, kws = 0x300ba1a8, kwcount = 540049776, defs = 0x20088734, defcount = 0, kwdefs = (nil), closure = (nil)), line 4000 in "ceval.c"
PyEval_EvalCode(co = 0x30932910, globals = 0x300ba1ac, locals = 0x203081a0), line 558 in "ceval.c"
builtin_exec(module = 0x1000eb04, args = (nil), nargs = 540049920, ??), line 1040 in "bltinmodule.c"
builtin_exec(module = 0x100c6724, args = 0x300ba1ac, nargs = 0), line 317 in "bltinmodule.c.h"
_PyMethodDef_RawFastCallDict(method = 0x100a0d3c, self = 0x103461d0, args = (nil), nargs = 812195432, kwargs = 0x30798a58), line 532 in "call.c"
_PyCFunction_FastCallDict(func = 0x3007f8a8, args = 0x306ac730, nargs = 540050272, kwargs = 0x220042cf), line 585 in "call.c"
PyCFunction_Call(func = 0x100da54c, args = (nil), kwargs = 0x300927d0), line 791 in "call.c"
do_call_core(func = 0x200af330, callargs = 0x300ba1a8, kwdict = 0x30031290), line 4681 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x1034b668, throwflag = 805419800), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1009f9b0, throwflag = 813271624), line 581 in "ceval.c"
_PyEval_EvalCodeWithName(_co = 0x3007bbf8, globals = 0x2008d708, locals = 0x055abdc8, args = 0x30037e30, argcount = 805547336, kwnames = 0x2008d708, kwargs = 0x20161430, kwcount = 0, kwstep = 1, defs = (nil), defcount = 0, kwdefs = (nil), closure = (nil), name = 0x3001bb18, qualname = 0x3001bb18), line 3969 in "ceval.c"
_PyFunction_FastCallKeywords(func = 0x100c3dc4, stack = (nil), nargs = 540052288, kwnames = 0x30600d78), line 437 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 814590276), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30055ce8, throwflag = 812304176), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030955c, args = 0x308e0b70, nargs = 540053680, globals = 0x308da93c), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x308e0f30, stack = 0x30055ea8, nargs = 537028192, kwnames = 0xcbcbcbcb), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 815270276), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x3006e348, throwflag = 815270280), line 581 in "ceval.c"
function_code_fastcall(co = 0x30062568, args = 0x2008d708, nargs = 135563570, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x100c6724, stack = 0x30831cf0, nargs = 540055440, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x1009f88c, throwflag = 814588684), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x1011900c, throwflag = 814588684), line 581 in "ceval.c"
function_code_fastcall(co = 0x300625d8, args = 0x2008d708, nargs = 1019748152, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x30061e28, stack = 0x30998070, nargs = 540057104, kwnames = 0x306b2480), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = (nil), throwflag = 812195060), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100de860, throwflag = 812316976), line 581 in "ceval.c"
function_code_fastcall(co = 0x1009f88c, args = 0xcbcbcbcb, nargs = 540058576, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x100a0d3c, args = (nil), nargs = 537556816, kwargs = (nil)), line 324 in "call.c"
_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540058864, kwargs = 0x306ac8a0), line 100 in "call.c"
object_vacall(callable = 0x100de658, vargs = ""), line 1200 in "call.c"
_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x307a6a28, 0x3003a7a0, 0x0, 0x4b651, 0xcb, 0x30691ba8), line 1250 in "call.c"
import_find_and_load(abs_name = 0x3061ee50), line 1652 in "import.c"
unnamed block in PyImport_ImportModuleLevelObject(name = 0x30691bb0, globals = 0x20021c28, locals = 0x2030a6a0, fromlist = 0x880042c4, level = 269329740), line 1748 in "import.c"
PyImport_ImportModuleLevelObject(name = 0x30691bb0, globals = 0x20021c28, locals = 0x2030a6a0, fromlist = 0x880042c4, level = 269329740), line 1748 in "import.c"
import_name(f = 0x100c1d90, name = 0xffffffff, fromlist = 0x2030a710, level = 0x42002224), line 4836 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x100de860, throwflag = 812215736), line 2722 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30344060, throwflag = 812151856), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030b1bc, args = 0x30621c00, nargs = 540060944, globals = 0x30696dac), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x100fe3d4, stack = 0x305608a4, nargs = 540061056, kwnames = 0x20088734), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x10109334, throwflag = 812316528), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x100fc8d0, throwflag = -889192245), line 581 in "ceval.c"
function_code_fastcall(co = 0x30022868, args = 0x20077e84, nargs = 807677136, globals = (nil)), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x200009f0, args = 0x20077e84, nargs = 537028216, kwargs = 0x3024c82e), line 324 in "call.c"
_PyObject_FastCallDict(callable = 0x00000002, args = 0x20077e84, nargs = 16, kwargs = (nil)), line 100 in "call.c"
_PyObject_Call_Prepend(callable = 0x30245e00, obj = 0x306ac5b0, args = (nil), kwargs = 0x306acfb0), line 906 in "call.c"
method_call(method = 0x1021f5ac, args = (nil), kwargs = 0x2030b570), line 304 in "classobject.c"
PyObject_Call(callable = 0x100a2254, args = 0x20021c28, kwargs = 0x2030b570), line 247 in "call.c"
do_call_core(func = 0x308ddf90, callargs = 0x30250ca0, kwdict = 0x20026660), line 4709 in "ceval.c"
_PyEval_EvalFrameDefault(f = 0x300bae00, throwflag = 812198764), line 3285 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30048aa8, throwflag = 812306352), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030c08c, args = 0x308dd090, nargs = 540064736, globals = 0x30037e30), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = (nil), stack = (nil), nargs = 0, kwnames = (nil)), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = 0x100c6144, throwflag = 812196632), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x30254658, throwflag = 812306352), line 581 in "ceval.c"
function_code_fastcall(co = 0x2030c6fc, args = 0x308dd090, nargs = 540066384, globals = 0x20088734), line 285 in "call.c"
_PyFunction_FastCallKeywords(func = 0x30825b80, stack = 0x200058f8, nargs = 540066496, kwnames = 0x308ddb08), line 410 in "call.c"
_PyEval_EvalFrameDefault(f = (nil), throwflag = -875836469), line 4655 in "ceval.c"
PyEval_EvalFrameEx(f = 0x10144cb8, throwflag = 812131072), line 581 in "ceval.c"
function_code_fastcall(co = 0x202d2110, args = 0x202d2118, nargs = 1548420902, globals = 0x0005e716), line 285 in "call.c"
_PyFunction_FastCallDict(func = 0x10010cf8, args = 0x2030c9d0, nargs = 1548420902, kwargs = 0x0005e715), line 324 in "call.c"
_PyObject_FastCallDict(callable = 0xd0127034, args = 0xcbcbcbcb, nargs = 540068256, kwargs = 0xf0766408), line 100 in "call.c"
_PyObject_Call_Prepend(callable = 0x1000f0f0, obj = 0x175a9b70, args = 0x2030ca20, kwargs = 0x20088734), line 906 in "call.c"
method_call(method = 0x100fd6d4, args = 0x200058c8, kwargs = 0x2030ca70), line 304 in "classobject.c"
PyObject_Call(callable = 0x20081d14, args = 0x3082ab90, kwargs = 0x2030cab0), line 247 in "call.c"
t_bootstrap(boot_raw = 0xcbcbcbcb), line 994 in "_threadmodule.c"
pythread_wrapper(arg = (nil)), line 174 in "thread_pthread.h"
(dbx)

OK, I have gone as far back as "where" in dbx can bring me.

Could this, somehow, be related with changes made in issue-33015 ?

In any case, does it seem correct that "pthread_wrapper(void *arg) can be correct if arg is nil?

  +155  /* bpo-33015: pythread_callback struct and pythread_wrapper() cast
  +156     "void func(void *)" to "void* func(void *)": always return NULL.
  +157
  +158     PyThread_start_new_thread() uses "void func(void *)" type, whereas
  +159     pthread_create() requires a void* return value. */
  +160  typedef struct {
  +161      void (*func) (void *);
  +162      void *arg;
  +163  } pythread_callback;
  +164
  +165  static void *
  +166  pythread_wrapper(void *arg)
  +167  {
  +168      /* copy func and func_arg and free the temporary structure */
  +169      pythread_callback *callback = arg;
  +170      void (*func)(void *) = callback->func;
  +171      void *func_arg = callback->arg;
  +172      PyMem_RawFree(arg);
  +173
  +174      func(func_arg);
  +175      return NULL;
  +176  }

(dbx) print boot
0x2030cab0
(dbx) which boot
_threadmodule.t_bootstrap.boot
(dbx) print boot
0x2030cab0
(dbx) print *boot
(interp = 0x2030cb00, func = 0xcbcbcbcb, args = 0x1018c460, keyw = 0xcbcbcbcb, tstate = 0xcbcbcbcb)
(dbx) which arg
thread.pythread_wrapper.arg
(dbx) print arg
(nil)
(dbx) print *arg
reference through nil pointer
(dbx) which callback
thread.pythread_wrapper.callback
(dbx) print callback
0x2030cb00
(dbx) print *callback
(func = 0x2030cb50, arg = 0x35000000)

I am "nervous" when it comes to accepting a value such as 0xcbcbcbcb or 0xdbdbdbdb as values for pointers to objects - or even "int" or unsigned values - look so "specific pattern" like.

Suggestions on how to look at this "better" would be appreciated.

Michael

OK. being more specific about the test situation.

When I run ./python -m test test_multiprocessing_fork all is fine. However, when I run it as: ./python -m test -j2 test_multiprocessing_main_handling test_multiprocessing_fork

test_multiprocessing_main_handling PASSes and test_multiprocessing_fork crashes (and has a segmentation core dump in the process).

After enabling PYTHONTHREADDEBUG=1 I got the dprintf output.

I added line info (as fixed text) asin:
Python/thread_pthread.h:
  +511  PyLockStatus
  +512  PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds,
  +513                              int intr_flag)
  +514  {
  +515      PyLockStatus success = PY_LOCK_FAILURE;
  +516      pthread_lock *thelock = (pthread_lock *)lock;
  +517      int status, error = 0;
  +518
  +519      dprintf(("519: PyThread_acquire_lock_timed(%p, %lld, %d) called\n",
  +520               lock, microseconds, intr_flag));
  +521
  +522      if (microseconds == 0) {
  +523          status = pthread_mutex_trylock( &thelock->mut );
  +524          if (status != EBUSY)
  +525              CHECK_STATUS_PTHREAD("pthread_mutex_trylock[1]");
  +526      }
  +527      else {
  +528          status = pthread_mutex_lock( &thelock->mut );
  +529          CHECK_STATUS_PTHREAD("pthread_mutex_lock[1]");
  +530      }

and can establish that USE_SEMAPHORES is not being used.

There are many reasons why - I expect - something re: Python3.5 (issue23428) talks about this routine and also something about CLOCk_MONOTONIC versus CLOCK_REALTIME (hope I spelled those right).

Further, back in Python 2.3 days - issue525532 added POSIX support for semaphores.

I would love to proceed - but particularly, issue23428 makes me think I should not think that the logic that keeps USE_SEMAPHORE is incorrect.

Help appreciated!

p.s. - deeper details

with PYTHONTHREADDEBUG=1 I no longer get a segmentation fault. Instead I get:

Total duration: 1 min 53 sec
Tests result: NO TEST RUN
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/libregrtest/runtest_mp.py", line 145, in run
    stop = self._runtest()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/libregrtest/runtest_mp.py", line 135, in _runtest
    result = json.loads(result)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/libregrtest/runtest_mp.py", line 145, in run
    stop = self._runtest()
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/test/libregrtest/runtest_mp.py", line 135, in _runtest
    result = json.loads(result)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/issue/Lib/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
?

This is prefixed by:
PyThread_allocate_lock called
PyThread_allocate_lock() -> 200a9b40
519: PyThread_acquire_lock_timed(200a9b40, 0, 0) called
577: PyThread_acquire_lock_timed(200a9b40, 0, 0) -> 1
PyThread_release_lock(200a5fe0) called
519: PyThread_acquire_lock_timed(200a9b40, 0, 0) called
577: PyThread_acquire_lock_timed(200a9b40, 0, 0) -> 0
519: PyThread_acquire_lock_timed(200a9b40, 29999996, 1) called
519: PyThread_acquire_lock_timed(200fe220, 0, 0) called
577: PyThread_acquire_lock_timed(200fe220, 0, 0) -> 1
PyThread_release_lock(200fe220) called
519: PyThread_acquire_lock_timed(200fe220, 0, 0) called
577: PyThread_acquire_lock_timed(200fe220, 0, 0) -> 1
PyThread_release_lock(200fe220) called
519: PyThread_acquire_lock_timed(20156c00, 0, 0) called
577: PyThread_acquire_lock_timed(20156c00, 0, 0) -> 1
PyThread_release_lock(20156c00) called
519: PyThread_acquire_lock_timed(20156c00, 0, 0) called
577: PyThread_acquire_lock_timed(20156c00, 0, 0) -> 1
PyThread_release_lock(20156c00) called
519: PyThread_acquire_lock_timed(200fe1a0, 0, 0) called
577: PyThread_acquire_lock_timed(200fe1a0, 0, 0) -> 1
PyThread_release_lock(200fe1a0) called
PyThread_free_lock(200fe1a0) called
PyThread_free_lock(200fe220) called
PyThread_free_lock(20156c00) called
519: PyThread_acquire_lock_timed(200a5fe0, 0, 0) called
577: PyThread_acquire_lock_timed(200a5fe0, 0, 0) -> 1
519: PyThread_acquire_lock_timed(200a5fe0, 0, 0) called
577: PyThread_acquire_lock_timed(200a5fe0, 0, 0) -> 0
PyThread_release_lock(200a9b40) called
PyThread_release_lock(200a5fe0) called
577: PyThread_acquire_lock_timed(200a9b40, 29999996, 1) -> 1
519: PyThread_acquire_lock_timed(200a5fe0, 0, 0) called
577: PyThread_acquire_lock_timed(200a5fe0, 0, 0) -> 1
PyThread_release_lock(200a9b40) called
PyThread_free_lock(200a9b40) called
519: PyThread_acquire_lock_timed(200a5fe0, 0, 0) called
577: PyThread_acquire_lock_timed(200a5fe0, 0, 0) -> 0
PyThread_release_lock(200a5fe0) called
PyThread_free_lock(200c44c0) called
PyThread_free_lock(200a81c0) called
519: PyThread_acquire_lock_timed(200b7e30, 0, 0) called
577: PyThread_acquire_lock_timed(200b7e30, 0, 0) -> 1
PyThread_release_lock(200b7e30) called
PyThread_release_lock(200fd1f0) called
PyThread_free_lock(200fd1f0) called
PyThread_free_lock(200a9ac0) called
PyThread_free_lock(200a6d30) called
PyThread_free_lock(200a5fe0) called
519: PyThread_acquire_lock_timed(200a3ac0, -1, 0) called
577: PyThread_acquire_lock_timed(200a3ac0, -1, 0) -> 1
PyThread_release_lock(200a3ac0) called
PyThread_exit_thread called

== Tests result: NO TEST RUN ==


Hope this helps!

I am still trying to get further with this, but I won't get far enough without some help on how to best dig deeper.

For one, it should be leaving a core dump, but it never seems to leave the core dump in the working directory. I know it is doing core dump because the "errpt" system tells me it is.

Further, anyone who can help me better understand messages such as:
Warning -- Dangling processes: {<SpawnProcess name='SyncManager-1168' pid=13041826 parent=6815746 stopped exitcode=-SIGSEGV>, <SpawnProcess name='SyncManager-1202' pid=8454174 parent=6815746 stopped exitcode=-SIGSEGV>}

All I can figure out is that the "parent" process has children that "die". Is there anything useful in the SpawnProcess name info?

Another message that surprises me is:
Warning -- multiprocessing.process._dangling was modified by test_multiprocessing_spawn
  Before: <_weakrefset.WeakSet object at 0x3076e810>
  After:  <_weakrefset.WeakSet object at 0x3076e390>

Normally speaking the address 0x30000000 and higher should be "out of bounds" aka a SEGV - unless special actions are taken to open a memory region above 0x2fffffff - for default AIX memory model.

Just calling malloc(), afaik, does not do automatically create a new memory section (0x0000000-0x0fffffff is reserved for kernel CODE, 0x10000000-0x1fffffff is reserved for application CODE, and 0x20000000-0x2fffffff is .data,.bss, "empty aka not-active", .stack).
The area between .end_bss and .stack is the area that sbrk() provides memory from (for calls to malloc(), e.g.)

In short, is there something I do not know understand about Python object pointers that 0x30000000+ values are actually living in 0x20000000-0x2fffffff space?

Also - this looks like a core dump was 'seen', but later removed.

Warning -- files was modified by test_multiprocessing_forkserver
  Before: []
  After:  ['core']

What can I change so that ot does not cleanup the core file?

I see I already asked howto better utilize this info:

ConnectionRefusedError: [Errno 79] Connection refused
Warning -- files was modified by test_multiprocessing_fork
  Before: []
  After:  ['core']

-- so, more specific -- which module, or file, is doing this check - as I would like to do postmortem analysis of the core dump.

Thx.

See also bpo-36752.

"""
...

_PyObject_FastCallDict(callable = (nil), args = 0x2008bf70, nargs = 540036208, kwargs = 0x0000014d), line 100 in "call.c"

object_vacall(callable = 0x100de658, vargs = warning: Unable to access address 0xdbdbdbdb from core
(invalid char ptr (0xdbdbdbdb))), line 1200 in "call.c"

_PyObject_CallMethodIdObjArgs(obj = 0x30061db0, name = 0x20039710, ... = 0x304e9fa8, 0x3003a7a0, 0x0, 0x4deb7, 0xcb, 0x306aee88), line 1250 in "call.c"

import_find_and_load(abs_name = 0x3097abf8), line 1652 in "import.c"

...
"""

Extract of _PyObject_CallMethodIdObjArgs:

PyObject *
_PyObject_CallMethodIdObjArgs(PyObject *obj,
                              struct _Py_Identifier *name, ...)
{
    va_list vargs;
    PyObject *callable, *result;

    if (obj == NULL || name == NULL) {
        return null_error();
    }

    callable = _PyObject_GetAttrId(obj, name);
    if (callable == NULL) {
        return NULL;
    }

    va_start(vargs, name);
    result = object_vacall(callable, vargs);
    va_end(vargs);

    Py_DECREF(callable);
    return result;
}

I don't know how va_list and va_start are implemented in XLC, but 0xDBDBDBDB looks like the byte pattern for released memory in Python: you can now use _PyMem_IsPtrFreed() function from pycore_pymem.h to check if a pointer looks like "freed":

static inline int _PyMem_IsPtrFreed(void *ptr)
{
    uintptr_t value = (uintptr_t)ptr;
#if SIZEOF_VOID_P == 8
    return (value == (uintptr_t)0xCDCDCDCDCDCDCDCD
            || value == (uintptr_t)0xDDDDDDDDDDDDDDDD
            || value == (uintptr_t)0xFDFDFDFDFDFDFDFD);
#elif SIZEOF_VOID_P == 4
    return (value == (uintptr_t)0xCDCDCDCD
            || value == (uintptr_t)0xDDDDDDDD
            || value == (uintptr_t)0xFDFDFDFD);
#else
#  error "unknown pointer size"
#endif
}

Maybe modify object_vacall() to add "assert(!_PyMem_IsPtrFreed(vargs));". You may need #include "pycore_pymem.h".

Good luck with debug :-(

3.8 is no longer being maintained. Is this an issue on 3.9+?

I stopped running my bot. I have no idea if this is also an issue on 3.9+

And since noone will look at this in it's present state. I'd recommend closing.