classification
Title: dictobject infinite loop in module set-up
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7, Python 2.6, 3rd party
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Daniel.Farina, Mark.Shannon, Max Khon, alex, benjamin.peterson, gregory.p.smith, iritkatriel, rhettinger, terry.reedy
Priority: normal Keywords:

Created on 2012-05-24 20:19 by Daniel.Farina, last changed 2020-11-07 19:57 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
offender.py Daniel.Farina, 2013-09-04 06:12
Messages (14)
msg161527 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2012-05-24 20:19
I seem to be encountering somewhat rare an infinite loop
in hash table probing while importing _socket, as triggered by
init_socket.c in Python 2.6, as seen/patched shipped with Ubuntu 10.04
LTS.  The problem only reproduces on 32 bit machines, on both -O2 and
-O0 builds (which is how I have managed to retrieve the detailed stack
traces below).  To cut to the chase, the bottom of the stack trace
invariably looks like this, in particular the "key" (and therefore
"hash") value is always the same:

#0  0x08088637 in lookdict_string (mp=0xa042714, key='SO_RCVTIMEO',
   hash=612808203) at ../Objects/dictobject.c:421
#1  0x080886cd in insertdict (mp=0xa042714, key='SO_RCVTIMEO', hash=612808203,
   value=20) at ../Objects/dictobject.c:450
#2  0x08088cac in PyDict_SetItem (op=<unknown at remote 0x37>, key=
   'SO_RCVTIMEO', value=20) at ../Objects/dictobject.c:701
#3  0x0808b8d4 in PyDict_SetItemString (v=
   {'AF_INET6': 10, 'SocketType': <type at remote 0x8275e00>,
'getaddrinfo': <built-in function getaddrinfo>,
'TIPC_MEDIUM_IMPORTANCE': 1, 'htonl': <built-in function htonl>,
'AF_UNSPEC': 0, 'TIPC_DEST_DROPPABLE': 129, 'TIPC_ADDR_ID': 3,
'PF_PACKET': 17, 'AF_WANPIPE': 25, 'PACKET_OTHERHOST': 3, 'AF_AX25':
3, 'PACKET_BROADCAST': 1, 'PACKET_FASTROUTE': 6, 'TIPC_NODE_SCOPE': 3,
'inet_pton': <built-in function inet_pton>, 'AF_ATMPVC': 8,
'NETLINK_IP6_FW': 13, 'NETLINK_ROUTE': 0, 'TIPC_PUBLISHED': 1,
'TIPC_WITHDRAWN': 2, 'AF_ECONET': 19, 'AF_LLC': 26, '__name__':
'_socket', 'AF_NETROM': 6, 'SOCK_RDM': 4, 'AF_IRDA': 23, 'htons':
<built-in function htons>, 'SOCK_RAW': 3, 'inet_ntoa': <built-in
function inet_ntoa>, 'AF_NETBEUI': 13, 'AF_NETLINK': 16,
'TIPC_WAIT_FOREVER': -1, 'AF_UNIX': 1, 'TIPC_SUB_PORTS': 1,
'HCI_TIME_STAMP': 3, 'gethostbyname_ex': <built-in function
gethostbyname_ex>, 'SO_RCVBUF': 8, 'AF_APPLETALK': 5,
'SOCK_SEQPACKET': 5, 'AF_DECnet': 12, 'PACKET_OUTGOING': 4,
'SO_SNDLOWAT': 19, 'TIPC_SRC_DROPPABLE':...(truncated), key=0x81ac5fb
"SO_RCVTIMEO", item=20) at ../Objects/dictobject.c:2301
#4  0x080f6c98 in PyModule_AddObject (m=<module at remote 0xb73cac8c>, name=
   0x81ac5fb "SO_RCVTIMEO", o=20) at ../Python/modsupport.c:615
#5  0x080f6d0b in PyModule_AddIntConstant (m=<module at remote 0xb73cac8c>,
   name=0x81ac5fb "SO_RCVTIMEO", value=20) at ../Python/modsupport.c:627
#6  0x081321fd in init_socket () at ../Modules/socketmodule.c:4708

Here, we never escape from lookdict_string.  The key is not in the
dictionary, but at this stage Python is trying to figure out that is
the case, and cannot seem to exit because of the lack of a dummy
entry.  Furthermore, every single reproduced case has a dictionary
with a suspicious looking violation of an invariant that I believe is
communicated by the source of dictobject.c, with emphasis on the
values of ma_fill, ma_used, and ma_mask, which never deviate in any
reproduced case.  It seems like no hash table should ever get this
full, per the comments in the source:

$3 = {ob_refcnt = 1, ob_type = 0x81c3aa0, ma_fill = 128, ma_used = 128,
 ma_mask = 127, ma_table = 0xa06b4a8, ma_lookup =
   0x8088564 <lookdict_string>, ma_smalltable = {{me_hash = 0, me_key = 0x0,
     me_value = 0x0}, {me_hash = 1023053529, me_key = '__name__', me_value =
   '_socket'}, {me_hash = 1679430097, me_key = 'gethostbyname', me_value =
   <built-in function gethostbyname>}, {me_hash = 0, me_key = 0x0, me_value =
   0x0}, {me_hash = 779452068, me_key = 'gethostbyname_ex', me_value =
   <built-in function gethostbyname_ex>}, {me_hash = -322108099, me_key =
   '__doc__', me_value = None}, {me_hash = -1649837379, me_key =
   'gethostbyaddr', me_value = <built-in function gethostbyaddr>}, {
     me_hash = 1811348911, me_key = '__package__', me_value = None}}}

The Python program that is running afoul this bug is using gevent, but
the stack traces suggest that all gevent is doing at the time this
crashes is importing "socket", and this is done at the very, very
beginning of program execution.

Finally, what's especially strange is that I had gone a very long time
running this exact version of Python, libraries, and application quite
frequently: it suddenly started cropping up a little while ago (maybe
a few weeks).  It could have been just coincidence, but if there are
code paths in init_socket.c that may somehow be sensitive to the
network somehow, this could have been related.  I also have a limited
suspicion that particularly unlucky OOM (these systems are configured
in a way where malloc and friends will return NULL, i.e. no overcommit
on Linux) could be related.
msg161529 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-05-24 21:26
Can you reproduce on current 2.7.3 or 3.2.3 or even 3.3.0 (which has changed dict implementation)? Or can you upload a short program that exhibits the problem, so someone else can try? 2.6 gets security fixes only and I do not believe this qualifies. If not a current problem, this should be closed as out of date, even if disconcerting.
msg161683 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2012-05-26 21:29
Unfortunately it's not so easy to upgrade the system's Python, however, it is something we might try.  The reproducing test case would appear to be akin to:

"import _socket"

And, within the gevent stack trace program, https://github.com/schmir/gevent/blob/master/gevent/monkey.py#L109

But to be precise, it is stuck at: https://github.com/heroku/WAL-E/blob/master/wal_e/cmd.py#L17.  None of this is very helpful, I realize, but this program (in this case, when archiving PostgreSQL write-ahead-log archives) is started up millions of times in a day, putting the defect rate at not far from the nominally figurative phrase "one in a million".
msg161684 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-05-26 22:08
I was suggesting alternate installations, not that you touch your system Python (a bad idea I have read). Such should be easy on Ubuntu. Whether you can run this particular program with alt installs is a different matter. If you have questions about doing so, also try python-ideas list or the news.gmane mirror, where there are several *nix users, including Ubuntu.
msg190952 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2013-06-11 08:55
Hello folks,

After long delay, I can confirm this issue reproduces in a similar way inside of init_socket in 2.7.
msg191254 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2013-06-16 02:39
In addition to being re-verified on 2.7, this occurred on a 64 bit system, so I've changed the title accordingly.
msg196889 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2013-09-04 05:45
I've confirmed this in a non-gevent program (actually a very trivial one, I can include the source if anyone asks), in 'initposix'.  This is on a quiet system, so OOM is not a very likely explanation.  Perhaps signal handling?


#0  0x000000000054662d in ?? ()
#1  0x000000000054d12e in ?? ()
#2  0x00000000005209e9 in PyDict_SetItem ()
#3  0x00000000004430cd in ?? ()
#4  0x0000000000456fcc in initposix ()
#5  0x0000000000456d2b in ?? ()
#6  0x000000000055fd03 in ?? ()
#7  0x00000000005600cb in ?? ()
#8  0x00000000004a0893 in ?? ()
#9  0x0000000000560964 in ?? ()
#10 0x0000000000553eab in ?? ()
#11 0x00000000004bf2a6 in PyObject_Call ()
#12 0x00000000004bf5a6 in PyEval_CallObjectWithKeywords ()
#13 0x000000000046870b in PyEval_EvalFrameEx ()
#14 0x000000000057bd02 in PyEval_EvalCodeEx ()
#15 0x000000000057cda8 in PyImport_ExecCodeModuleEx ()
#16 0x000000000055f6d1 in ?? ()
#17 0x00000000005600cb in ?? ()
#18 0x00000000004a0893 in ?? ()
#19 0x0000000000560964 in ?? ()
#20 0x0000000000553eab in ?? ()
#21 0x00000000004bf2a6 in PyObject_Call ()
#22 0x00000000004bf5a6 in PyEval_CallObjectWithKeywords ()
#23 0x000000000046870b in PyEval_EvalFrameEx ()
#24 0x000000000057bd02 in PyEval_EvalCodeEx ()
#25 0x000000000057cda8 in PyImport_ExecCodeModuleEx ()
#26 0x000000000055f6d1 in ?? ()
#27 0x00000000005600cb in ?? ()
#28 0x00000000004a0893 in ?? ()
#29 0x0000000000560964 in ?? ()
#30 0x0000000000553eab in ?? ()
#31 0x00000000004bf2a6 in PyObject_Call ()
#32 0x00000000004943c5 in PyObject_CallFunction ()
#33 0x000000000054dfbe in PyImport_Import ()
#34 0x000000000050df4d in ?? ()
#35 0x0000000000511ae2 in ?? ()
#36 0x0000000000512764 in Py_Main ()
#37 0x00007f88142bc76d in __libc_start_main () from
/lib/x86_64-linux-gnu/libc.so.6
#38 0x000000000041ba51 in _start ()
msg196891 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2013-09-04 05:49
If you could supply the source that'd be great.
msg196892 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2013-09-04 06:12
Attached.  The program's function is to take a base64 encoded string and arguments as input and then to materialize this program on disk and run it with its arguments.

Notably, this one contains no socket interaction at all, unlike the other examples, which narrows the cause quite a bit.  It is run via 'ssh'.  Like the other case, it's during module dictionary set-up.
msg196893 - (view) Author: Daniel Farina (Daniel.Farina) Date: 2013-09-04 06:17
altered title now that it's been seen in init_posix.
msg221011 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-06-19 19:04
Can you provide specific details of exactly which python package from which distro is installed on the machines?

Are the machines hardware or VMs?  if they are VMs, what version of what VM system and what hardware are the VMs running on?

I'm asking because someone at work is seeing a potentially similar problem from an ubuntu python2.7-minimal package - i don't know which version - they can pipe in here and provide that and any other details that may be relevant.
msg270868 - (view) Author: Max Khon (Max Khon) Date: 2016-07-20 12:21
I reproduced the problem with Python 2.7.5 as shipped with CentOS 7:

root@192.168.0.86 /home/padmin # python -V
Python 2.7.5
root@192.168.0.86 /home/padmin # rpm -q python
python-2.7.5-34.el7.x86_64
root@192.168.0.86 /home/padmin # 

(gdb) bt
#0  lookdict_string (mp=<optimized out>, key='RPMTAG_OPTFLAGS', hash=411442822543039667)
    at /usr/src/debug/Python-2.7.5/Objects/dictobject.c:461
#1  0x00007f92d6d9f2c9 in insertdict (mp=0x2502600, key='RPMTAG_OPTFLAGS', hash=411442822543039667, 
    value=1122) at /usr/src/debug/Python-2.7.5/Objects/dictobject.c:559
#2  0x00007f92d6d9f3b0 in dict_set_item_by_hash_or_entry (
    op={'RPMTAG_HEADERREGIONS': 64, 'RPMTAG_EXCLUSIVEOS': 1062, 'fi': <type at remote 0x7f92c87ef6c0>, 'RPMTAG_CHANGELOGNAME': 1081, 'RPMTAG_CONFLICTNEVRS': 5044, 'RPMTAG_FILECAPS': 5010, 'RPMTAG_FILERDEVS': 1033, 'RPMTAG_COLLECTIONS': 5029, 'RPMTAG_BUGURL': 5012, 'setStats': <built-in function setStats>, 'RPMTAG_FILEDIGESTALGO': 5011, 'RPMTAG_DEPENDSDICT': 1145, 'RPMTAG_CLASSDICT': 1142, 'RPMTAG_FILEMODES': 1030, 'RPMTAG_FILEDEPENDSN': 1144, 'RPMTAG_BUILDTIME': 1006, 'ii': <type at remote 0x7f92c87f0280>, 'RPMTAG_INSTALLCOLOR': 1127, 'RPMTAG_CHANGELOGTEXT': 1082, 'RPMTAG_HEADERCOLOR': 5017, 'RPMTAG_CONFLICTNAME': 1054, 'RPMTAG_CONFLICTS': 1054, 'setLogFile': <built-in function setLogFile>, 'versionCompare': <built-in function versionCompare>, 'RPMTAG_CONFLICTVERSION': 1055, 'RPMTAG_NVRA': 1196, 'RPMTAG_NOPATCH': 1052, 'RPMTAG_HEADERI18NTABLE': 100, 'RPMTAG_LONGARCHIVESIZE': 271, 'RPMTAG_FILEREQUIRE': 5002, 'RPMTAG_FILEDEPENDSX': 1143, 'RPMTAG_EVR': 5013, 'RPMTAG_INSTALLTIME': 1008, 'RPMTAG_NAME': 1000, 'RPMTAG_LONG...(truncated), key=<optimized out>, hash=<optimized out>, 
    ep=<optimized out>, value=<optimized out>) at /usr/src/debug/Python-2.7.5/Objects/dictobject.c:774
#3  0x00007f92d6da18a8 in PyDict_SetItemString (
    v={'RPMTAG_HEADERREGIONS': 64, 'RPMTAG_EXCLUSIVEOS': 1062, 'fi': <type at remote 0x7f92c87ef6c0>, 'RPMTAG_CHANGELOGNAME': 1081, 'RPMTAG_CONFLICTNEVRS': 5044, 'RPMTAG_FILECAPS': 5010, 'RPMTAG_FILERDEVS': 1033, 'RPMTAG_COLLECTIONS': 5029, 'RPMTAG_BUGURL': 5012, 'setStats': <built-in function setStats>, 'RPMTAG_FILEDIGESTALGO': 5011, 'RPMTAG_DEPENDSDICT': 1145, 'RPMTAG_CLASSDICT': 1142, 'RPMTAG_FILEMODES': 1030, 'RPMTAG_FILEDEPENDSN': 1144, 'RPMTAG_BUILDTIME': 1006, 'ii': <type at remote 0x7f92c87f0280>, 'RPMTAG_INSTALLCOLOR': 1127, 'RPMTAG_CHANGELOGTEXT': 1082, 'RPMTAG_HEADERCOLOR': 5017, 'RPMTAG_CONFLICTNAME': 1054, 'RPMTAG_CONFLICTS': 1054, 'setLogFile': <built-in function setLogFile>, 'versionCompare': <built-in function versionCompare>, 'RPMTAG_CONFLICTVERSION': 1055, 'RPMTAG_NVRA': 1196, 'RPMTAG_NOPATCH': 1052, 'RPMTAG_HEADERI18NTABLE': 100, 'RPMTAG_LONGARCHIVESIZE': 271, 'RPMTAG_FILEREQUIRE': 5002, 'RPMTAG_FILEDEPENDSX': 1143, 'RPMTAG_EVR': 5013, 'RPMTAG_INSTALLTIME': 1008, 'RPMTAG_NAME': 1000, 'RPMTAG_LONG...(truncated), key=key@entry=0x7f92c83bf537 "RPMTAG_OPTFLAGS", 
    item=item@entry=1122) at /usr/src/debug/Python-2.7.5/Objects/dictobject.c:2448
#4  0x00007f92d6e181f2 in PyModule_AddObject (m=m@entry=<module at remote 0x24d35c8>, 
    name=name@entry=0x7f92c83bf537 "RPMTAG_OPTFLAGS", o=o@entry=1122)
    at /usr/src/debug/Python-2.7.5/Python/modsupport.c:616
#5  0x00007f92d6e182d8 in PyModule_AddIntConstant (m=m@entry=<module at remote 0x24d35c8>, 
    name=name@entry=0x7f92c83bf537 "RPMTAG_OPTFLAGS", value=value@entry=1122)
    at /usr/src/debug/Python-2.7.5/Python/modsupport.c:628
#6  0x00007f92c85e2b20 in addRpmTags (module=<module at remote 0x24d35c8>) at rpmmodule.c:200
#7  initModule (m=<module at remote 0x24d35c8>) at rpmmodule.c:343
#8  init_rpm () at rpmmodule.c:281
#9  0x00007f92d6e13ed9 in _PyImport_LoadDynamicModule (name=name@entry=0x24f69d0 "rpm._rpm", 
    pathname=pathname@entry=0x24f79e0 "/usr/lib64/python2.7/site-packages/rpm/_rpmmodule.so", 
    fp=<optimized out>) at /usr/src/debug/Python-2.7.5/Python/importdl.c:53
...

The infinite loop happens when "import rpm" is called in low-memory conditions (e.g. when handling ENOMEM or exceptions.MemoryError - RedHat/CentOS abrt-addon-python package which installs sys.excepthook handler).
msg380515 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-11-07 19:10
Is this a python 2-only issue?
msg380521 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-11-07 19:57
I suspect so.  If any modern supported python 3.x version runs into an issue like this I think opening a fresh bugreport is good.

Closing as not reproducable / obsolete.
History
Date User Action Args
2020-11-07 19:57:28gregory.p.smithsetstatus: open -> closed
resolution: out of date
messages: + msg380521

stage: test needed -> resolved
2020-11-07 19:10:20iritkatrielsetnosy: + iritkatriel
messages: + msg380515
2016-07-20 12:21:40Max Khonsetnosy: + Max Khon
messages: + msg270868
2014-06-19 19:04:07gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg221011
2013-09-04 06:17:06Daniel.Farinasetmessages: + msg196893
title: dictobject infinite loop while importing socket -> dictobject infinite loop in module set-up
2013-09-04 06:12:05Daniel.Farinasetfiles: + offender.py

messages: + msg196892
2013-09-04 05:49:54alexsetnosy: + alex
messages: + msg196891
2013-09-04 05:45:41Daniel.Farinasetmessages: + msg196889
2013-06-21 17:56:15Daniel.Farinasettitle: dictobject infinite loop -> dictobject infinite loop while importing socket
2013-06-16 02:39:19Daniel.Farinasetmessages: + msg191254
title: dictobject infinite loop on 2.6.5 on 32-bit x86 -> dictobject infinite loop
2013-06-11 09:15:13serhiy.storchakasetnosy: + rhettinger, benjamin.peterson
2013-06-11 08:55:57Daniel.Farinasetversions: + Python 2.7
2013-06-11 08:55:42Daniel.Farinasetmessages: + msg190952
2012-05-26 22:08:29terry.reedysetmessages: + msg161684
2012-05-26 21:29:19Daniel.Farinasetmessages: + msg161683
2012-05-24 21:26:48terry.reedysetnosy: + terry.reedy

messages: + msg161529
stage: test needed
2012-05-24 21:15:56terry.reedysetnosy: + Mark.Shannon
2012-05-24 20:19:44Daniel.Farinacreate